Business Continuity For Better Business Health
Magellan Health Cashes In With Reality Checks.
As the country's leader in behavioral health, employee assistance programs and humanservices, Magellan Behavioral Health focuses on helping people in crisis. To ensure thatcritical operations in the event of a disruption to its data center operations wouldcontinue, the company built a comprehensive business continuity program.Terry Shaughnessy
agellan Behavioral Health, a division of the $2 billion Magellan Health Services, Inc.,(Columbia, Md.), em-ploys 6,000 people and serves more than 64 million members through anationwide provider network and regional service centers operating in the U.S., Canada andPuerto Rico. A disaster affecting their IT environment would have a disastrous dominoeffect. "Claims wouldn't be processed. And that means checks wouldn't be cut fordoctors, therapists, clinicians and other providers," says Tina Yeargain, manager oftechnical operations for Magellan Behavioral Health.
"Additionally, we've developed several client-server applications that rely oncustomer-specific information extracted from our midrange systems. If the midrangeenvironment goes down, the client-server platforms can't get the data they require andthis has an impact across the organization and on our ability to effectively interfacewith our clients." Magellan's AMISYS (version 9.52) healthcare claim processingapplication uses an HP 3000 Model 997/800 running MPE/iX 5.5. Also available are two IBMAS/400s (a 530 running OS/400 V4R3 and a 620 running V4R1) running its home-grownapplications like the Claims Account Payable System (CAPS) application Employee AssistanceProgram (EAP), Managed Care Program (MCP) and Integrated Product applications.
Yeargain, along with George Conway, operations support specialist, were charged withmanaging Magellan Behavioral Health's business continuity program for the midrangeplatforms. "Because we are focused on managed mental healthcare, a lot of the impactsaren't quantifiable," says Yeargain. "If someone calls a hotline to talk to atherapist because they're suicidal and that therapist doesn't have access to the client'srecords or history because the systems are down, the impact is hard to quantify. There's ahuman factor that absolutely cannot be ignored."
Previously, the pair had worked together at a large electrical manufacturing companywhere they were able to identify an hourly dollar amount of how much revenue themanufacturer lost if the systems were down in terms of sales orders, inventory and theoverall manufacturing process. With assistance from Comdisco's (Rosemont Ill.)professional services organization, Magellan Behavioral Health conducted a Business ImpactAnalysis (BIA) to identify quantifiable as well as unquantifiable factors.
Also in use is an HP 9000 (a Model 170 running HP-UX 10.20 for Costware TechnologyCorp.'s Datagate STC, an EDI/transaction-based application that pre-processes membershipand claim files from Magellan Behavioral Health's customers into a standard format. Thestandardized files are then batch-loaded into the claim processing applications running onthe HP 3000 and the AS/400 systems. Lastly, the company has built a data warehouse (usingIBM's DB2 running on an RS/6000 Model R40) that captures data on claims from the othersystems using Bridgeware from Quest/Taurus. All the systems are connected via a TCP/IPnetwork using a Cisco 5500 switch.
Once the BIA was complete and the needs were identified, Magellan Behavioral Healthbegan developing and implementing its program. Using ComPAS, Comdisco's businesscontinuity planning software tool, Conway led the development of the business continuityplan. "The Comdisco solution," says Conway, "supported a multitude ofplatforms throughout a network of recovery facilities." Magellan Behavioral Healthfinalized its recovery strategy, selecting to relocate to a Comdisco recovery facilitylocated in Carlstadt, N.J., where it has access to an HP 3000, HP 9000, RS/6000 and twoAS/400s, which they need to successfully recover their critical applications.
Although Magellan Behavioral Health has been able to successfully restore its systems in half the time of its stated recovery time, the company continues to seek ways to improve. However, data and applications -- particularly the AMISYS claim processing application -- are growing at a rapid rate. That, in turn, takes up more disk space and slows recovery time.
To help alleviate the problem, both in a recovery scenario as well as for day-to-day back-up operations, Magellan Behavioral Health is in the process of implementing an enterprise back-up solution that uses a StorageTek 9710 DLT tape library to back up all its platforms. Currently the company's midrange systems are being backed up by utilities indigenous to each system and then to tape drives specific to each system.
"We're always looking for ways to try to get the systems to restore faster and will be working closely with Comdisco to integrate the tape library into our business continuity program as it will impact how we recover," says Conway. "The DLT media is supposed to be more reliable and have a higher tolerance for faults and through an enterprise back-up approach, we're hoping the new tape library will help further quicken recovery."
Another recommendation is not to underestimate the amount of disk needed to recover. "Allow for as much disk as needed to completely rebuild your systems," said Conway. "It will take less time to recover and you'll be less prone to mistakes than if you're trying to pull pieces together from different disks at time of disaster."
The company also is implementing a hot-node network solution that will provide it withimmediate network connectivity from the recovery facility to its headquarters in Columbia,Md. and its other business locations throughout North America. "It may cost less tohave network equipment stored away that's just pulled out once a year for testing, but youalso have to factor in the time to set it up and you also run the risk that it may notwork," said Yeargain. "Having a solution like a hot-node network that provides anear real-time switch over may cost a little more, but the trade-off is that it's readilyavailable. You know it's going to work and it's one less issue to contend with duringrecovery."
Yeargain and Conway are also strong advocates of thorough testing to help ensure thatMagellan's environment remains recoverable. "A lot of companies have a plan, but itdoesn't get tested enough. Or they just go through the motions of testing but don't reallydo all the verification and validation they need," said Yeargain. Currently,Magellan's recovery time objective is 72 hours. However, successfully tested recoverieshave occurred in half that time. "What you learn about the recovery process throughtesting can make the difference between a successful recovery and chaos."
Magellan Behavioral Health essentially starts its tests from scratch -- completelyrestoring the systems and ensuring they're running as expected. "We try to replicatea normal production day as close as we can for each of the platforms," said Yeargain."Once the systems are restored, we have a pre-determined list of the production batchjobs we want to test. We have users at a few remote locations set up to log on and test tomake sure they can get into the systems, process claims, conduct searches and anythingelse they normally would do on any given day."
Having conducted several on-site tests at the Carlstadt recovery facility, MagellanBehavioral Health is trying a new approach: testing remotely. "The ability to testremotely, as well as to recover remotely if needed, appealed to us," said Yeargain."We didn't want to be in a situation where if we couldn't get to the Carlstadtfacility, we wouldn't be able to recover. [Now], we can recover remotely from [Comdisco's]St. Louis location or in their Wood Dale, Ill. location outside of Chicago."
According to Conway, the remote testing worked quite similar to on-site testing."Our hardware remained in New Jersey, but our employees participating in the testwere located in the Wood Dale facility and interfacing with Comdisco technicians and ourrecovery environment set up in Carlstadt. "We had our business users log into therecovery systems and conduct production activities as we would with any other test. And itall ran smoothly."
-- Terry Shaughnessy is director of distributed systems for Comdisco,Inc. (Rosemont, IL.; www.comdisco.com).