E-Business and Disaster Recovery Planning: More Art than Science
Business continuity planning strategies must constantly adapt to changes in information technology and in the business processes they support if they are to help ensure the survival of the company that deploys them.
With the arrival of e-business -- whether manifested in the form of e-commerce dotcoms, application service providers, or business-to-business portals -- many organizations are seeking to define exactly what business continuity strategies are relevant and worthwhile. Todd Gordon, General Manager of IBM Business Continuity and Recovery Services, says that the problems of business continuity planning in a Web-based world are complex and that they are made more so by the constant infusion of new technology to support a shifting business model.
"How many BCRS customers are doing e-business?" asks Gordon, "That is almost a trick question. My best guess is that 30 percent of our client base is actively doing e-business today. Another 30 percent are putting their toes in the e-business water. The remainder are not there yet."
Gordon says that the inquiries received by BCRS related to specific business continuity consulting services is growing "exponentially," "There is a great demand of continuity planning competencies in the e-business world. We have been building out a cadre of continuity experts in specific areas, including customer relationship management (CRM), enterprise resource planning (ERP), e-commerce and business-to-business (B2B) exchange to address the growth of interest."
Gordon concedes that, while demand for e-business contingency planning expertise is increasing, "an exponential increase of a small number is still a small number." In other words, he does not believe that many companies adequately understand what risks and exposures they are incurring as they extend their business onto the Web.
"People are still working in a hurricane mindset. Hurricanes don’t happen very often, but having a contingency plan worked out in advance is a good policy [for coping] if and when a hurricane occurs. Most businesses are not aware of how frequently disasters occur on the Web. They aren’t aware of security risks or network unreliability until an event occurs. They rarely appreciate how frequently interruptions are actually occurring."
Added to the potential for interruption, Gordon says that disasters can and do result from successes in e-business initiatives. When a company has not worked out the impact from a transaction standpoint that a successful e-business venture will have on existing Web and non-Web infrastructure, technology failures and business process interruptions are the likely results.
Jon Derome, Senior Analyst within the Business to Business Commerce and Applications practice of The Yankee Group (Boston, MA), believes that many companies have been adopting Web-based ERP with an eye toward realizing big gains from new technology. There is a widespread tendency in these adoptions to accept the defaults of the application and to redefine business processes to match the presumed "best practices" believed to be embodied in the software.
"Without being too terribly cynical, it should be pointed out that there is no new or improved best practice capability in the software. Vendors have spent the majority of their money recently not to add smarts to their packages, but to capitalize on the interest in e-business -- to Web enable packages."
Such an approach to application adoption, Gordon offers, may shift a company to a "supply chain model" before they have had the opportunity to assess the potential points of failure -- both in terms of business processes and technology infrastructure -- that they new model represents.
Consulting Assistance at the Ready
Luis Hernandez, National Principal within IBM’s BCRS consulting practice, says that his organization is able to draw upon extensive resources within IBM to facilitate customer business continuity planning. "When a prospective client calls us who is interested in contingency planning, the first thing we ask is where they are in their plan journey. We find that most companies have been doing planning in IT areas only. Y2K changed that somewhat, as companies did need to look at risk management from a more enterprise-wide focus."
Hernandez adds that current and prospective customers are "a little smarter" than they once were, "they can see that continuity planning makes business sense." The problem is that they may not have the internal resources or skills to act. The range of issues may be too broad for a single planner within an organization.
"Some issues may be resolved by a change in platform architecture, and customers may be willing to consider this if we can show a business benefit. If they are willing to restructure an application to achieve better recoverability, we can bring in IBM system engineers to support recovery at the desired speed," Hernandez offers.
He adds that large companies with a long disaster recovery heritage tend to be more open to discussions of application rearchitecture than are dotcoms. "Dotcoms are generally not as focused on recovery and, as a result, their back end systems are not as protected."
Part of the problem may have to do with lack of experience-based knowledge in the dotcom arena, he notes. Performance, capacity planning and tuning are a science in the world of the brick and mortar data center, but in the world of the Web, performance monitoring, anticipating resource demands, and predicting problems so they can be resolved proactively is still as much art as science. Says Hernandez, a consulting service -- with experience gleaned from numerous e-business implementations -- can be an enormous asset.
Application Service Providers Offer Value
To David Goldschlag, Chief Technology Officer for US Internetworking (USi) (Annapolis, MD), continuity guarantees are an important component in the evaluation process for companies considering the use of Application Service Providers (ASPs). An ASP delivers applications via the Internet or private connection to a client company on a subscription basis. According to most analysts, the future of the ASP mode of application delivery is bright. USi, an early pioneer in the field, boasts one of the largest customer bases in the industry.
Goldschlag says that the company has dedicated resources "at the highest levels" to determine what baseline capabilities should be offered to customers and how to "productize" additional levels of continuity services and configurations.
"Currently, USi includes an obligation in its standard contract to provide a certain level of business continuity to the customer. The bare level provision is for a recovery at an alternative site within two weeks if our current facility becomes unavailable. If our data center in Annapolis became unavailable, we would seek to recover the customer at our Milpitas, CA data center within two weeks, which is in line with our contracts with vendors who would replace our inventory," says Goldschlag.
For customers requiring a shorter recovery timeframe, Goldshlag offers that there are only two alternatives that an ASP (or a Web hosting company) can offer, "They can offer to improve the client’s placement in the recovery queue. For example, if one of our customers running PeopleSoft Financials has an outage near month’s end, an especially important time in their processing cycle, we might be able to assign their recovery priority over other clients."
"Another approach is to provide a more expensive contract that provides for the replacement of resources in a shorter timeframe. We may provide an additional service of data replication across our two data centers, providing an equivalent site that may be held in reserve or used on on-going basis as additional capacity and resources."
Goldschlag notes that one recent customer contract obtained by USi does not engage the firm to provide ASP services at all, but only engages them as a hot backup to a competitor’s hosting arrangement. "The customer involved is hosting their applications elsewhere, but they liked our service and wanted us to provide a disaster recovery backup site for them. We like this concept a lot. It shows we can provide a service that provides value to customers and also to competing ASPs."
He adds that USi’s continuity planning service offerings are evolving, "Business continuity is a very thoughtful, very expensive enterprise. It doesn’t command as much attention as other [priorities] and it is a hard competency to build." At the same time, he notes that resiliency and fault tolerance is part of the design process for USi offerings.
"Our networks are a good example. We can connect customers via the Internet or private Frame Relay access. In both cases, we emphasize vendor and carrier diversity [to ensure fault tolerance]. We actually handle the line from the cloud to the customer. We recommend redundancy to ensure that a single carrier’s outage will not cause a service outage for the customer. We try to get the customer to pay for the redundant link by turning it into an operational advantage: in operation, we can load balance between multiple T-1s. We can do the integration of the two circuits for redundancy, adding value to the service."
E-Business Protection More Than Compontent Fault Tolerance
IBM BCRS General Manager Gordon says that he is familiar with the work USi is doing in high availability. "High availability offers protection of components. However, redundant networks and platforms do not define business continuity. Companies need to look at how the new Web-based processes are cross-connected with legacy processes and to develop a plan that addresses both."
Great unknowns exacerbate the complexity of e-business continuity planning. According to Gordon and others, these impair the effective application of traditional business continuity planning methodologies to the world of the Web. Over time and with experience, techniques and technology will improve so that on-line brokerages need not hold 70 percent of their resources in "hot standby" on the off chance that a market change will result in a spike in transactions, and so that full redundancy is not the only solution for continuity at Web speed.
In the meantime, companies doing business on the Web are urged to examine their Web hosting, application service provisioning, and b-to-b service provisioning agreements closely. Asking the following questions may save enormous headaches downstream, if and when interruptions occur:
Does the service contract explicitly assign the responsibility for an outage to the service provider, or is the responsibility "shared" -- among the customer, service provider, and/or the provider’s providers (i.e., third party hardware, software or network vendors)? Shared responsibility clauses may be an open door for time delays and finger pointing if an outage occurs.
Is a timeframe for service restoration in the event of an outage explicitly stated in the contract? Avoid "best effort" clauses or other open-ended statements that do not bind the service provider to a guaranteed result.
Are "enhanced recovery options" such as redundancy or mirroring available? Can they be cost justified by the potential financial impact of an outage? Can "enhanced recovery options" be leveraged to sweeten the deal that is being made with the vendor?
What is the current load on service provider facilities? Can the Web hosting service or ASP or portal provider honestly guarantee recovery at an alternate facility? Is the facility or network "overbooked?"
How is the guaranteed service level to be monitored? What notifications are provided for disaster potentials that are not immediately obvious (say, half of the country cannot visit the e-commerce site due to a networks service interruption)?
What redress is provided by the service provider for outages? Is a pro-rated discount on the monthly bill adequate to offset the loss of business that accrued to an outage period?
Does the provider have an internal business continuity plan? Has it been audited for routine testing and currency?
If the service provider offers backup services to other service providers, how does this potentially affect the availability of resources for your business?
Will the service provider assist you if an internal interruption occurs that does not directly involve the services they provide to you under contract?
For B2B portals, have the portal providers taken any measures to ascertain the continuity planning readiness of member organizations? Like any supply chain relationship, B2B partners establish interdependencies that can have a ripple effect if disaster impacts a partner.
The above list of questions is by no means comprehensive, but it should give e-business enthusiasts a starting point for considering the business continuity impact of new ventures. The business potential of the Web is only now being explored. Like any adventure into uncharted territory, the risks are as abundant as the potential returns.