MAJOR PUBLIC ENERGY UTILITY ENSURES 24/7 OPERATIONS FOR 1 MILLION+ CUSTOMERS WITH MULTIPLE BACKUP TAPE COPIES

Ameren – formerly Union Electric – is one of the country’s largest public energy utilities, serving more than 1 million customers in Missouri, Illinois and Iowa. As with any utility company, Ameren is critically concerned with keeping operations running on a 24/7 basis, with an extreme high priority on systems and data availability. To meet that objective, Ameren operates with a corporate policy that requires duplicate copies of all backup tapes so that one copy can be kept safely off-site in case of a catastrophic data loss.

Ameren’s computer installation at the company’s St. Louis headquarters encompasses 1500 nodes with 120 heterogeneous file servers running NetWare, Windows NT, Sun Solaris and HP-UX. All departmental data, along with custom applications developed for time reporting, customer service systems and outage management, is backed up nightly to two centralized tape servers.

To handle the data backup requirements Ameren uses two HP LXe Pro servers operating as backup servers, with each server handling backups for 60 file servers. The expansive installation generates a nightly incremental backup load of up to 100 gigabytes, and 400 to 500 gigabyte backups for the weekly full system backups. Each centralized HP tape server is connected to two Storage Technology 9714 tape libraries containing 100 tapes and equipped with four or six DLT7000 tape drives with Legato NetWorker as the backup automation software.

Using this backup architecture, Ameren’s IT staff sought to satisfy the corporate dictum of duplicate backup tapes by using the tape cloning feature within NetWorker, which provides off-line tape-to-tape copy functions after the backup process is completed. While NetWorker’s tape cloning feature performed as expected in creating a set of off-site backup tapes, it also presented a problem, according to Vance Bufalo, Senior Engineer in Networking Engineering at Ameren.

"My servers are doing so much work every night and the off-site tape has to be ready for pickup by 11 a.m. I did not have time to do the backup at night and do the tape-to-tape copy in the morning before the vendor showed up to pick up the tapes," said Bufalo.

"The other problem I ran into is that the two tape systems support approximately 1500 users in our main office complex and we do recovers on a daily basis. If the tape drives are tied up doing tape-to-tape copies during the morning, then those drives would not be available for tape recovers. So I needed a solution where I was getting true tape mirroring – real time copy as the first tape was being written, the second tape was being created at the same time."

The need to do real time tape copying led Ameren to Orange County, Calif.-based Ultera Systems for a solution built around Ultera’s ShadowMaster™ tape controller technology that provides real-time tape mirroring. A mirrored tape architecture essentially creates a RAID-1 environment, simultaneously writing data to a mirrored set of tape drives, automatically creating a duplicate set of tapes for off-site storage in the time it takes to do the original backup and solving Ameren’s backup window problem.

Ultera’s mirroring tape controller technology also appealed to Ameren because it is 100 percent software transparent, allowing the company to continue to use Legato NetWorker as its backup software platform with no changes required. Ultera controllers also work seamlessly with all other leading backup software applications for UNIX, NetWare and Windows NT.

For the Ameren installation, Ultera’s solution went beyond simple tape drive mirroring and into a new concept pioneered by Ultera, RAIL (Redundant Array of Independent Libraries). Ultera created a solution that actually mirrors a pair of STK 9714 libraries with one library dedicated to storing on-site tapes, and the other used exclusively for tapes created for off-site storage. As shown in Figure 1, each pair of mirrored drives is driven by a dedicated ShadowMaster controller. Another ShadowMaster mirroring controller manages the robotic media changers in each library.

This innovative solution delivered the real-time tape mirroring that Ameren required, but also made media management a painless process, said Bufalo. "We have a 30-day tape rotation because that’s our data retention policy, and tapes never leave the on-site library. They’re in there all the time for tape recovers and also for the writes at night, and the off-site tapes are rotated on a daily basis."

Ameren’s RAIL solution is designed so that it is impossible to perform any backup operations without creating the required off-site set of tapes. Any read operations will read data from tapes in the primary library drive, but if a write command is issued, the Ultera ShadowMaster will automatically write data to the two mirrored drives. If the second drive is not ready, an error message will be generated and the backup aborted. "We never want to have a broken pair on a mirrored tape," said Bufalo.

"What that gives us is first off, real-time mirroring which met our requirements; off-site copies which are ready by 11 a.m., and we can still do all of our data recovers from the on-site tape library for the last 30 days without any tape handling by the operators. We defined our requirements and Ultera worked with us to design a system that met our specific needs."

The Ultera library mirroring solution worked so well that Ameren recently implemented an Ultera-based library solution on a larger scale, deploying an identical mirrored-library configuration using Ultera controllers with an HP9000 K-Class Enterprise Server running HP-UX with two STK 9710 libraries. Each library accommodates 512 tape slots and 10 DLT7000 drives. "We’re doing the same thing only on a much larger scale, only we’re doing it with UNIX and the bigger libraries," said Bufalo.

Still, Ameren has even larger plans for Ultera-based tape library solutions. Ameren is working on the larger solution, but it is for the ever-expanding data requirements and not for the backups of the corporation's 5000 nodes. The current development plan is to incorporate Ultera mirroring controllers with large-scale Storage Technology Powderhorn libraries. "We’ve used Ultera for our past tape requirements, our present and now we’re looking to Ultera for the future."

For further information, call (949) 367-8800, email inform@ultera.com or visit Ultera's web site at http://www.ultera.com.