In-Depth
The Human Dimension in Disaster Recovery
Your disaster recovery plan must consider more than just protecting and restoring your data. Your staff has to be recovered as well.
One thing that may surprise you about the state of disaster recovery preparedness today is that not much has changed since September 11, 2001.
What this means, experts say, is that the industries which were most affected by the events of September 11—financial services firms, airlines, healthcare organizations, and government agencies—already had business continuity plans and disaster recovery procedures in place beforehand. Indeed, analysts say, most Global 2000 firms have conducted business continuity or disaster recovery planning for years.
What’s surprising, however, is that the events of September 11 haven’t precipitated a mad rush of small and medium enterprises anxious to follow suit. There’s plenty of interest, to be sure, but when it comes down to signing on the dotted line, CFOs are balking at the prospect of forking over precious IT dollars to underwrite expensive business continuity planning (BCP) or disaster recovery efforts. The culprit, of course, is a prolonged economic downturn that has forced all companies to make tough decisions about IT spending. “They’re certainly evaluating [BCP and disaster recovery solutions], but part of the problem has been that … the economy has been slow and funding has been slow,” acknowledges John Sensenich, director of product management with BCP and disaster recovery services vendor Sungard.
Adds Thom Carroll, global director of business continuity with IT outsourcing and services provider Computer Sciences Corp. (CSC): “In spite of the financial slowdown, you’re still seeing people paying attention to disaster recovery and business continuity planning, coming to us, asking questions about it all. Not that they’ve implemented anything, they’re just asking the questions.”
While investment in BCP and disaster recovery solutions might seem like a no-brainer decision in the post-9/11 business environment, Tony Adams, a principal support analyst with research firm Gartner Inc.’s worldwide infrastructure support service, says that it’s anything but an automatic slam dunk for struggling small- and mid-size companies. Many don’t have the money, and those which have spent on BCP or disaster recovery have done so as a result of regulatory requirements, such as Sarbanes-Oxley, which specify rules for the retention of documents and written communications. “If they were regulated already, they’re already doing it, and if they weren’t regulated, they’re not doing it, and they’re not going to change their behaviors,” he observes
Then there’s the avoidance factor. BCP can be a ghoulish practice, Adams points out, and investment in disaster recovery services and technologies is hardly an inexpensive proposition. “This is not a sexy discipline. It is very difficult and requires close examination of a lot of things that are going to be uncomfortable,” he argues, noting that information systems aren’t the only resources that need to be recovered in the event of a catastrophe. “[CFOs] have finite resources and they can either boost their prospects for more revenue, or they can buy this thing [BCP or disaster recovery services], well, that’s really like a casket. That’s the decision that they have to make—should I hire another salesperson, or should I spend an enormous amount of money for a consultant to come in for a month and look at my underside?”
In many small- and mid-size companies, Adams notes, concerns about immediate survival are trumping horizontal issues such as BCP and disaster recovery preparedness. “If [small and mid-size companies are] spending money [on IT], they’re focusing on areas where they feel that they can get immediate return on investment,” he says. “For them, it’s not a simple proposition that if you don’t spend on [BCP or disaster recovery], you could lose your business. For them, there’s a real possibility that if they spend now [on BCP or disaster recovery], they will lose their business.”
Many Companies Still Unprepared
Just because a company has invested in business continuity or disaster recovery planning and services doesn’t mean that it should rest on its laurels.
For a variety of reasons, says Sungard’s Sensenich, companies that have developed business continuity or disaster recovery plans are often unprepared to resume or restore their mission-critical operations in the event of a disaster. He cites a Sungard survey of 200 IT organizations in which 80 percent of respondents received failing grades for overall disaster preparedness. Many had already invested in BCP or disaster recovery, he says, but nevertheless failed to anticipate all that could go wrong. “They hadn’t planned for how the people who needed the information would reconnect to it,” he explains. “They had plans in place for recovering, protecting, and restoring their data, but they hadn’t any plans for recovering their people.”
People. As the tragic events of September 11 demonstrated, companies must also factor the loss of human life into a business continuity or disaster recovery equation. What’s more, experts say, they’ve also got to allow for human nature, especially in the context of a catastrophic disaster-- such as 9/11-- when employees are understandably concerned about their families. “When a disaster strikes, people are going to take care of their families. It’s human nature. They’re going to look to their families first—then comes the technology,” observes Steve Higgins, director of business continuity marketing for EMC Corp.
Adds Gartner’s Adams: “The issue is one of interdependency and human fallibility. Let’s say that a nuclear bomb goes off and everything turns to glass— how long are [employees] going to sit in that center? And what about their families? Are they really going to want to stay in there, not knowing what happened to their families?”
In post-9/11 business continuity and disaster recovery planning paradigms, industry watchers say, companies are making allowances for the unavailability of their human resources and are trying to preserve the business-critical knowledge that they possess. This is especially true of corporate executives. “I see some companies that were directly involved in 9/11… their organizations have built in practices now that identify key roles, that duplicate those roles,” says Gartner’s Adams. “Some companies even have specific days where executives work at home, and now there are rules about flying, so you don’t have the whole management team on a plane.”
Organizations aren’t just minding their executives, however. After all, notes CSC’s Carroll, the folks in the trenches need to be recovered, too. And if for one reason or another they can’t be recovered, their skills and knowledge must be preserved for those who remain. “With the catastrophic loss of personnel that we experienced on 9/11, we’re finding that business continuity planning works in concert with succession planning, and that’s one spot where you painstakingly have to take these guys and gals that have all of the knowledge embedded in their heads and write it all down, put a process around it, and allow a transfer of knowledge,” he explains. “Then you need to make sure that you have a contingent grouping of people who can pick that function up in a situation if the unthinkable happens.”
According to Sungard’s Sensenich, almost all of his company’s customers have revisited their business continuity or disaster recovery plans in the wake of 9/11. One thing that these customers are focusing on, he notes, is the importance of recovering their human resources when disaster strikes. “Sungard Planning Associates [a risk assessment services group] has worked with many of their clients to review the plans that they have in place, and a lot of them are now asking about expanding their strategies to include the people part of the business,” he comments.
Of course, one problem with many existing disaster recovery plans is that organizations have, in fact, written it all down—so much so that when disaster strikes, people on the ground often aren’t able to make all that much sense of it. “At a variety of levels, a lot of these disaster recovery plans were useless [on September 11], because there was just too much information to work with in the face of a catastrophic event,” Carroll continues. “So when we work with clients, we’re eliminating a lot of the minutiae and paring everything back to the bare essential information.”
What’s considered essential when disaster strikes? “There’s a real focus on how can the reader quickly interpret this and figure out his role in the plan, how are they going to effectively function in that role, are they going to have to assume the role of someone else, and how will they be able to quickly understand that role [if they do have to replace someone], and that has dramatically changed what and how we do things,” Carroll concludes.
Kevin Coyne, director of business operations with Sun Microsystems Inc.’s services unit, says that his company has developed a novel solution to this problem. “What we did inside Sun is we put together a wallet-sized card, a communications card, that is provided to every employee, and that basically tells them what they need to do to follow the [disaster recovery] process immediately and know exactly what to do.”
Even when an organization does attempt to control for the human factor in its business continuity and disaster recovery planning, there’s probably another variable skulking around somewhere in the works. That’s because so few organizations test—that is, really test—their disaster recovery preparedness, says Gartner’s Adams.
He tells a story about an Italian client that had a very thorough business continuity and disaster recovery plan. It brought in a consulting company to vet its disaster preparedness and was shocked when the company’s consultants asked if it had ever “thrown the switch,” so to speak, to test its ability to recover in the event of a complete outage. “[The Italian company said,] ‘We’ve never really thrown the switch because we don’t know what would happen!’” Adams relates. “The lesson is that even if you really are confident in your [business continuity and disaster recovery] planning, there’s always this lingering fear that you’ll have a major disruption. And let’s face it, how many organizations can really afford to ‘throw the switch’ to test this out?”