MAJOR PUBLIC ENERGY
UTILITY ENSURES 24/7 OPERATIONS FOR 1 MILLION+
CUSTOMERS WITH MULTIPLE BACKUP TAPE
Ameren formerly Union Electric is one of the
countrys largest public energy utilities, serving more than 1 million customers in
Missouri, Illinois and Iowa. As with any utility company, Ameren is critically concerned
with keeping operations running on a 24/7 basis, with an extreme high priority on systems
and data availability. To meet that objective, Ameren operates with a corporate policy
that requires duplicate copies of all backup tapes so that one copy can be kept safely
off-site in case of a catastrophic data loss.
Amerens computer installation at the
companys St. Louis headquarters encompasses 1500 nodes with 120 heterogeneous file
servers running NetWare, Windows NT, Sun Solaris and HP-UX. All departmental data, along
with custom applications developed for time reporting, customer service systems and outage
management, is backed up nightly to two centralized tape servers.
To handle the data backup requirements Ameren uses two HP LXe Pro servers operating as
backup servers, with each server handling backups for 60 file servers. The expansive
installation generates a nightly incremental backup load of up to 100 gigabytes, and 400
to 500 gigabyte backups for the weekly full system backups. Each centralized HP tape
server is connected to two Storage Technology 9714 tape libraries containing 100 tapes and
equipped with four or six DLT7000 tape drives with Legato NetWorker as the backup
Using this backup architecture, Amerens IT staff sought to satisfy the corporate
dictum of duplicate backup tapes by using the tape cloning feature within
provides off-line tape-to-tape copy functions after the backup process is completed. While
NetWorkers tape cloning feature performed as expected in creating a set of off-site
backup tapes, it also presented a problem, according to Vance
Bufalo, Senior Engineer in
Networking Engineering at Ameren.
"My servers are doing so much work every night and the off-site tape has to be
ready for pickup by 11 a.m. I did not have time to do the backup at night and do the
tape-to-tape copy in the morning before the vendor showed up to pick up the tapes,"
"The other problem I ran into is that the two tape systems support approximately
1500 users in our main office complex and we do recovers on a daily basis. If the tape
drives are tied up doing tape-to-tape copies during the morning, then those drives would
not be available for tape recovers. So I needed a solution where I was getting true tape
mirroring real time copy as the first tape was being written, the second tape was
being created at the same time."
The need to do real time tape copying led Ameren to Orange County, Calif.-based Ultera
Systems for a solution built around Ulteras ShadowMaster tape controller
technology that provides real-time tape mirroring. A mirrored tape architecture
essentially creates a RAID-1 environment, simultaneously writing data to a mirrored set of
tape drives, automatically creating a duplicate set of tapes for off-site storage in the
time it takes to do the original backup and solving Amerens backup window problem.
Ulteras mirroring tape controller technology also appealed to Ameren because it
is 100 percent software transparent, allowing the company to continue to use Legato
NetWorker as its backup software platform with no changes required. Ultera controllers
also work seamlessly with all other leading backup software applications for UNIX, NetWare
and Windows NT.
For the Ameren installation, Ulteras solution went beyond simple tape drive
mirroring and into a new concept pioneered by Ultera, RAIL (Redundant Array of Independent
Libraries). Ultera created a solution that actually mirrors a pair of STK 9714 libraries
with one library dedicated to storing on-site tapes, and the other used exclusively for
tapes created for off-site storage. As shown in Figure 1, each pair of mirrored drives is
driven by a dedicated ShadowMaster controller. Another ShadowMaster mirroring controller
manages the robotic media changers in each library.
This innovative solution delivered the real-time tape mirroring that Ameren required,
but also made media management a painless process, said Bufalo. "We have a 30-day
tape rotation because thats our data retention policy, and tapes never leave the
on-site library. Theyre in there all the time for tape recovers and also for the
writes at night, and the off-site tapes are rotated on a daily basis."
Amerens RAIL solution is designed so that it is impossible to perform any backup
operations without creating the required off-site set of tapes. Any read operations will
read data from tapes in the primary library drive, but if a write command is issued, the
Ultera ShadowMaster will automatically write data to the two mirrored drives. If the
second drive is not ready, an error message will be generated and the backup aborted.
"We never want to have a broken pair on a mirrored tape," said
"What that gives us is first off, real-time mirroring which met our requirements;
off-site copies which are ready by 11 a.m., and we can still do all of our data recovers
from the on-site tape library for the last 30 days without any tape handling by the
operators. We defined our requirements and Ultera worked with us to design a system that
met our specific needs."
The Ultera library mirroring solution worked so well that Ameren recently implemented
an Ultera-based library solution on a larger scale, deploying an identical
mirrored-library configuration using Ultera controllers with an HP9000 K-Class Enterprise
Server running HP-UX with two STK 9710 libraries. Each library accommodates 512 tape slots
and 10 DLT7000 drives. "Were doing the same thing only on a much larger scale,
only were doing it with UNIX and the bigger libraries," said
Still, Ameren has even larger plans for Ultera-based tape library solutions. Ameren is
working on the larger solution, but it is for the ever-expanding data requirements and not
for the backups of the corporation's 5000 nodes. The current development plan is to
incorporate Ultera mirroring controllers with large-scale Storage Technology Powderhorn
libraries. "Weve used Ultera for our past tape requirements, our present and
now were looking to Ultera for the future."
For further information, call (949) 367-8800, email firstname.lastname@example.org or visit Ultera's web site at http://www.ultera.com.
# # #
Back to Articles of Interest