Enterprise Rent-A-Car Driven to Disaster Recovery Planning
Angela Swartz isn’t the type who comes immediately to mind when one envisions a disaster recovery planner. She lacks most of the archetypical features of the so-called “Deans of DR” – the spreading midsection, the silver-gray receding hairline, the face deeply etched by the burden of thinking about the unthinkable for too many years. She’s even missing the signature indentation of the lip, tailor-made for holding a cigar.
In stark contrast to the stereotype, Angela Swartz is an extraordinarily bright and charming woman with an easy sense of humor, friendly nature and disarming smile that belies a deep and comprehensive understanding of both the business and technical requirements of her employer for business continuity. She is the manager and chief architect of Enterprise Rent-A-Car’s business continuity capability and may well be the archetype of the next generation of disaster recovery planners.
Enterprise Rent-A-Car was an early pioneer in the rental car business. Since 1962, the organization has grown into the largest player in the market, with over 4,000 offices situated within 15 minutes of 90 percent of the US population. Enterprise has grown its fleet of vehicles from 17 in 1963 to more than 427,000 in 2000. The company enjoyed revenues of $4.73 billion worldwide in 1999 -- up from $3.68 billion in 1997 -- which Swartz attributes largely to customer service excellence and rates that are nearly 20 percent lower than its closest competitors.
Enterprise enjoys a special relationship with its customers, says Swartz, often picking them up at their homes, offices, or repair shops and racing them to Enterprise rental offices. The company also capitalizes on its strong ties to insurance companies: substantial portions of Enterprise rentals are made in connection with insurance adjustments following automobile accidents, mechanical breakdowns or thefts.
Coordinating this business are administrative offices covering various regions of the US -- as well as Canada and Europe -- and Enterprise’s corporate headquarters in St. Louis, MO. The administrative offices run client/server applications to support human resources activities, vehicle ordering, insurance issue resolution, accounting, and vehicle administration. Invoices from the United States are handled entirely by headquarters, but they are receipted at the administrative office level, according to Swartz.
*Satellite Infrastructure Simplifies Local Office Planning
A unique feature of Enterprise Rent-A-Car’s technology infrastructure is the deliberate choice made by the company to equip its rental offices with satellite terminals, rather than systems. The company’s computing power is largely centralized in the St. Louis home office. There, an IT staff of more than 780 persons oversees 35 AS/400s -- 19 of which are currently dedicated to handling an in-house vehicle rental application that handles an average of 1.3 million transactions each hour for Enterprise's rental offices. (This application is currently being ported to an Oracle-based application running on IBM RS/6000 platforms.) The multiple AS/400s ensure redundancy and availability and also serve different time zones. Access to these applications from rental offices is made via Enterprise’s own private satellite and hub network. Users enjoy a response time of less than two seconds, while AS/400s treat the communications like a multidrop leased line with multiple control units – the product of some “protocol spoofing” by the hub.
Within the local office, the dish transfers data to an SNA control unit, which then passes data to one or more connected terminals. Using the dumb terminal architecture for its rental offices, says Swartz, enables the company to field new offices quickly. It also simplifies her job as continuity planner to a degree, she concedes.
“Operationally speaking, we manage our risk in the field through our decisions regarding where to put equipment and how to manage equipment. Every branch is equipped with a very small aperture terminal (VSAT) – a small dish on the roof. The use of dumb terminals limits the potential for hardware problems. If an office has an incident, the manager can go to another office [and reestablish his terminal identity]. We are also looking at developing a trailer that can be used as a mobile office in an emergency,” Swartz says. She adds that the only additional requirement for office level risk management is backing up the controller software image, which is downloaded to the SNA controller at each site.
*Home Office And Administrative Office Protection is Paramount
Much of Swartz’s time and effort is spent on planning for the protection of business processes supported by the company’s centralized data center, which houses IBM AS/400 and RS/6000 systems, as well as servers from Sun Microsystems. She is quick to share the credit for the corporate continuity capability with the engineers and applications staff who “have dedicated many hours to what we have accomplished. Without them, it would have been impossible.”
Having said this, Swartz acknowledges that it is a management practice at Enterprise to demonstrate its commitment to a project by assigning a visible individual to assume responsibility for its completion. In the area of AS/400 recovery planning, few candidates have better credentials for the job than Swartz.
In 1997, she sat down with IBM Business Recovery Services (now Business Continuity and Recovery Services) to discuss the special recovery needs of large AS/400 users. With 20 AS/400s on the floor at Enterprise, the company was part of an “elite group” of firms – including Viacom and Home Depot – with a shared interest in large systems recovery.
“Up to that time,” she recalls, “large AS/400 shops needed to turn to multiple [systems recovery] vendors in order to accomplish a recovery in the event of a disaster. IBM had a benchmark testing facility in Rochester, MN. We wrote a paper proposing the use of the facility as a large AS/400 recovery facility. IBM took the proposal and opened the recovery facility in April 1998. It was just moved into BCRS’ primary recovery facility in Sterling Forest, NY. Large users are now more and more in the mainstream.”
She observes that the headquarters data center is a focal point of business enablement. In addition to the resources of 19 AS/400s that “strictly serve the rental side of the business,” Enterprise also fields an assortment of dedicated and shared systems, including Sun Microsystems and Intel servers, for such functions as data warehousing and data mining, and to support the operations of the company’s Internet presence.
“We use the Internet to deliver some applications to our customers today. This is growing. We want to offer direct access to applications to some of our business partners, whether via direct connection or the public Internet. We are working toward making it as easy as possible to do business with Enterprise,” she notes.
To protect this effort from natural and man-made disaster potentials, the five year old Enterprise Rent-A-Car headquarters data center is hardened against earthquakes and tornadoes, and equipped with a state-of-the-art fire detection and suppression capability. Swartz notes that the current facility lies within the little known New Madrid fault system or seismic zone, which includes large sections of Illinois, Missouri, Kentucky, Tennessee and Arkansas. “New Madrid is an earthquake wildcard,” she acknowledges, pointing to the fact to explain why the company embraced the BOCA National Building Code, developed in response to earthquake damage studies, when building the data center.
The facility is protected from tornadoes by the use of “armor-plated, bullet proof glass,” Swartz reports, adding that the walls of the center are fire rated for three hours, and that additional fire protection is provided by a combination of preaction water sprinklers and clean agent (FM-200) suppression systems “in three zones.”
Power protection was also a priority for Enterprise in constructing its current data center. Swartz describes the details of the solution, “We have a substation on our corporate campus with two sources of power, a transfer switch, multiple uninterruptible power supplies, and a generator to provide five nines of uptime. We had a power outage last Monday morning and switched over to internally generated power within 15 seconds.”
Swartz says that this data center is about to be augmented with a new high availability data center the company is building in Weldon Springs, MO. She says that the new facility is “huge on the priority list.”
“With the advent of the World Wide Web and business-to-business [processes], management is increasingly IT savvy. They recognize that there is a high availability requirement to get to where they want to be in business. We believe in satisfying the customer 100 percent, and part of that is being available. Continuity planning is no longer an IT subject, it is a real business issue,” she observes.
When complete, the new data center will be linked via satellite and dark fiber to the current center, providing opportunities for increased redundancy and availability strategies, Swartz says.
*Business Sense Prevails
Working with her staff of two, maintaining the “buy in from senior management,” and coordinating the work of point persons from company departments (as well as technical input from the corporate IT department), are part of Swartz’s daily work. Focusing narrowly on contingencies and their mitigation, she says it is easy to see how a planner might lose sight of the “big picture” of business operations.
“The challenge is always that you want to provide safeguards against every risk. [Thinking them through,] you can discover that the safeguard you are proposing costs three times what the potential loss might be. You need to focus on the business, maximize your disaster recovery strategy within the constraints imposed by business operations.”
Swartz says that she has been fortunate to have the support of enlightened senior management, which, as a member of IBM BCRS’ Customer Advisory Council, she realizes is not always the case for her peers. She notes that management interest has increased as the company has come to depend increasingly on technology to accomplish business goals. “In the past, technology managers were responsible for continuity planning. Today, I report to the Assistant Vice President of Operations. Our plan is audited periodically by an outside firm.”
In addition to nurturing management support and maintaining a business focus, Swartz says that the other challenge in her work is keeping pace with technology. She notes that her background in technical purchasing and project management has provided essential tools for doing her job.
She also emphasizes the importance of testing to ensure that the continuity strategy keeps pace with changes within the company’s business systems and networks. “We test at BCRS twice per year for computer operations, and print and mail. These are live tests using backup systems. We switch to backup routers and bring up the systems at BCRS via backup T1s.”
In addition, she says, “We periodically test a strategy for recovering our national reservation center at our training center.”
Testing helps to provide feedback on plan adequacy. Maintenance of the plan itself is accomplished electronically, using a combination of Microsoft Word documents, Excel Spreadsheets and Visio diagrams. “Department representatives are key to keeping plan components up-to-date by maintaining documentation on-line. We put the nitty-gritty procedures into an on-line help utility that can be queried simply when needed.”
*Homegrown Is Key
Swartz may not look like an old guard DR planner, but she sounds like one. Like the rugged individualists of the early days of Big Iron recovery planning, she prefers a homegrown planning effort to an outsourced solution. Ultimately, she and her co-workers at Enterprise will need to implement whatever strategy they deploy, so it makes little sense to her to have contingency plans developed by an organization outside of the company. She says that management philosophy at Enterprise is entirely consistent with her homegrown approach.
“The philosophy of Enterprise management,” she notes, “is that if you want to make something a priority, make it someone’s main focus.” She says that her job is made considerably easier by management support, on the one hand, and the enthusiastic support of both technical and business personnel at Enterprise.
Swartz typifies a new breed of planners: her work straddles two worlds of business operations and information technology. She recognizes that the appropriate focus of her work is not the recovery of a specific platform or network, but the business processes that the technology infrastructure supports.
Her proactive planning approach and her attention to high availability further distinguishes her as a next-generation “master of disaster.” Given the increasingly tight relationship between business and technology, and the increasing complexity of technology itself, planners can no longer be content to develop recovery strategies after the fact. Like Swartz, next generation planners need to be directly involved at the conception of new business applications, to contribute their insights to the design of software and hosting platforms, so that their recoverability is assured.
To Swartz, as to Enterprise Rent-A-Car senior management, business interruptions are anathema—not events to be feared, but risks to be engineered out of the system. Like the true next generation contingency planner, Swartz is ultimately an optimist.
BIO: Jon William Toigo is an independent consultant and can be reached via email at jtoigo@intnet.net.