Disaster Recovery Plan, 1. Hurricane, 0.
The solution businesses need most but never want to use, and rarely think about, is disaster recovery. But disasters can happen--even in what seem to be the safest places.
Allison Manufacturing Co., headquartered in New York, produces screen printed apparel products under contract for the Walt Disney Company. The $70 million firm has plants in three states, with all of its administrative functions—including operations, finance, and MIS--centered in Albemarle, NC.
A thoroughly integrated JIT operation, Allison Manufacturing relies on in-house developed systems for product development, costing and production scheduling. All ordering and order fulfillment with retail customers is done via EDI. Bar code labels and price tags are printed by Allison to customers' specifications. The company uses an imaging system with optical disk storage to archive all of its business documents.
The heart of Allison Manufacturing's IT operations is a Model 620 RISC-based AS/400, an upgrade from a Model 510 completed in April, 1997. "We depend on one machine," explains Glenn Wood, vice president of MIS, "but the AS/400 is the most 'up-time' machine there is. Availability and accuracy is what we were aiming for to begin with. But if it goes it could put a big hurt on us."
Albemarle is a quiet place. Thirty miles from Charlotte, it is far from geological fault lines, political terrorists and the normal track of dangerous storms. Yet Wood received what he calls a "wake-up call" in 1989 from remnants of Hurricane Hugo, the storm that pushed inland from Charleston, spawning tornadoes as far as western North Carolina.
Fortunately, Allison Manufacturing was spared by Hugo, but there was enough local damage, and utility companies were thrown into such turmoil, that Wood decided it was wise to develop a disaster recovery plan for the company's MIS facilities.
A prudent person, Wood's approach was to plan on the worst case. "Nobody can predict a disaster," he observes. "Risk analysis may help you decide where to put your dollars, but it's not an absolute--it's like buying life insurance that only covers a heart attack. I thought: Let's think about losing this place completely. Anything less we can manage our way through, but if we plan for the worst we'll be covered."
By 1991, Wood had a plan in place, backed up by IBM's Business Recovery Services (BRS). He could have chosen a "hot site" option, in which IBM BRS would provide not only a replacement AS/400 but office space as well. However, since Allison Manufacturing had a building that could be used as an alternate site, a warehouse about 2,000 yards from their main building, Wood chose an alternate site plan, in which, in case of disaster, IBM would immediately ship a replacement system to them there.
The company did the basic cabling work in the warehouse for a replacement AS/400, if it was ever needed. In addition, they developed a floor diagram indicating where the different departments and people would be situated and LANs wired, if they ever had to be.
"From the day we finished the plan we started cultivating the local vendors we knew we could count on to deliver PCs and peripherals as fast as they could overnight them to us," Wood says. "We experimented. We got the plan where we wanted it--so when we did lose it all, we lost it according to plan."
On Friday, July 18, 1997, Hurricane Danny came ashore over the Alabama Gulf coast and headed slowly northeast. By Wednesday, it had crossed three states and arrived over Albemarle. A lightweight storm compared to Hugo, Danny nevertheless brought prodigious rainfall, dumping 15 inches on the town in one three-hour period.
By early Thursday, Allison Manufacturing's single story building was under five feet of water from a nearby creek, never known to have flooded before, and the data center was a total loss. At a higher elevation than the main building, the warehouse alternate site was untouched by the flood.
By 10 am Thursday, the MIS staff, and any assistance they could find, were at work wiring the warehouse for the installation of computers, networks and phones. By 10:30, Wood had called IBM BRS and 30 hours later a brand new AS/400 620 was delivered to the alternate site. In two days, he had 70 PCs from his vendors, configured exactly as specified.
By the following Monday, Allison's systems were again in operation, including a damaged goods inventory application to evaluate the business impact of the lost production facilities in Albemarle. Some data that had been backed up on PCs was lost. But critical data, including receivables and the legacy systems, had been backed up on the AS/400 the day of the flood and was safe.
How would the company have fared if they had had to pick up the pieces by themselves? For one thing, they had no guarantee that an AS/400 would have been available for them on a day's notice. It took two weeks to purchase the 620 the previous April, Wood says, and another two weeks to configure the system to their needs--experience that was to prove invaluable, it turned out.
But replacing damaged hardware was not the point. The real question is, could Allison Manufacturing have pulled through at all, given its dependence on IT? "It would have devastated the business," Wood says. "Our customers are big. They don't take orders that are shipped late. We felt from the beginning that we couldn't afford to be down for a month or even a couple of weeks. I don't think we would have survived."
"We hoped for the best but planned on the worst, and the worst happened," he continues. "The problem is you don't know the extent of a potential disaster. That's what we learned in 1989. Hurricane Danny wasn't supposed to be as bad as Hugo. It was a shock to everybody. But it's a mistake to think you are immune from anything, a real mistake."