In-Depth
System Availability is a Must for AUL Pension Division
by George Ferguson
Customer service is the number one priority for the Pension Division of Indianapolis-based American United Life (AUL) Insurance Company. Delivering the level of service required to keep customers satisfied -- in the competitive pensions market and in the Internet era -- makes system availability and business recovery critical areas of concern.
Complicating matters is the Pension Division’s ongoing migration from an HP e3000 to a UNIX environment. For the past 15 years, the division has used the HP e3000 platform to run its core software, a pension record-keeping application. That application is nearing the end of its service life, and the division has decided to replace it with software that provides additional functionality, including the ability to link record keeping with workflow and document management. The problem is that the new record-keeping application does not run on the HP e3000’s MPE/iX operating system but on Windows NT and HP-UX.
To ensure high availability and recoverability both during and after the migration, the division relies on hardware redundancy, multiple levels of data back-up, mission critical support and business recovery services from HP and other vendors.
Soup-to-Nuts Service
AUL's Pension Division administers 401(k), Profit-Sharing, Money Purchase, Tax Deferred Annuity, Deferred Compensation Plan and Individual Retirement Account plans. Currently, more than 400,000 participants invest in an AUL pension product. The division has revenue of more than $1 billion per year. One of four AUL divisions, the Pension Division employs about half of AUL’s 1,500 employees.
With 26 sales locations throughout the United States, the Pension Division’s target market is the 25- to 250-employee company. To serve these customers and differentiate itself in the pensions marketplace, the division provides a variety of options and services to employers and the employees participating in its pension plans.
Unlike large corporate customers, which typically have an in-house department to administer retirement plans, many of the Pension Division’s customers rely on AUL for administrative/financial services that range far beyond transferring funds and issuing checks. For example, the division administers IRS 1099 forms for its business customers and supports their 401(k) IRS/DOL compliance needs.
In addition, for employees participating in the retirement plan, AUL provides interactive voice response and Internet access to account transactions, fund performance and investment education. Future plans incorporate more services for both employees and employers.
"Pension plans are an unbelievably competitive part of the financial services market," says Jack Snyder, manager of planning and technical support with AUL’s Pension Division. "Many of the big financial services firms cherry-pick the large employers to be their pension customers, but AUL is concentrating on the smaller employers. We provide a total package of services that meets all the retirement administration needs of the employer and its employees, and we provide those services extremely efficiently."
Availability is Non-Negotiable
By providing this level of service, AUL has become a leading pension plan provider for small to medium-sized employers. While successful, this marketing strategy has serious ramifications for IT: High systems availability is essential to deliver the level of service required by customers.
As a result, virtually every piece of hardware and software in the division’s IT environment is mission critical. System availability is essential, because neither internal nor external customers will tolerate gaps in service. Plan participants expect to be able to access information about their pension accounts around the clock by phone or via the Internet.
"Our customers use their 401(k) plans for a huge amount of their retirement and savings goals. It doesn’t matter if it’s 2:00 pm on a Saturday, 3:00 am on a Tuesday or 9:00 pm on a Sunday, they want to be able to access their balance via the Internet or Interactive Voice Response," Snyder says.
In addition to round-the-clock availability, the division faces stringent day-in, day-out performance demands on the HP e3000 system by both internal customers (customer service and operations personnel) and external customers (plan administrators and participants).
Challenges include:
- A meteoric growth in number of pension transactions flowing through AUL’s Web site. The number of transactions rose by 800 percent between 1999 and mid-2000. In April 2000, the site experienced 4,000 transactions per day.
- A high volume of phone calls from plan participants with inquiries about their accounts. The division receives 15,000 to 20,000 phone calls per month through an IVR (Interactive Voice Response) system that communicates with the HP e3000.
- Heavy periods of database-transaction processing. The system must at times perform 100 million database transactions within four hours. Transactions include deposits of the plan participant’s contributions and the employer’s matching amounts into multiple mutual or stock funds—all multiplied by the hundreds of thousands of participants in the plans.
- A high number of concurrent user sessions. More than 2,000 user sessions may be active on the division’s system at any one time.
- The need to keep IT systems available during data backups and nightly batch processing. Currently, the division’s average batch cycle runs more than 1,000 jobs per night.
The goal of quickly providing to customer service reps and salespeople—and to plan participants and sponsors via the Internet—answers that may require thousands of calculations. Depending on the plan and the transaction or inquiry, "We may need to do up to 26,000 calculations to get a plan participant’s account balance. And we do it in seconds," Snyder says.
Assuring Business Continuity
Meeting these challenges requires both high system availability and superior business recovery capabilities. In fact, the division maintains greater than 99 percent IT uptime for the HP e3000 environment. Redundant hardware and HP Critical Systems Support (HP CSS) are pillars of the system’s high availability.
The division’s current environment includes several HP e3000 Business Servers. The two that have historically been used for production are HP e3000 Model 997 servers. These two HP e3000 databases shadow each other using SharePlex software from Quest Software, Inc.
In addition, the Pension Division has one HP e3000 Model 979 Business Server. Until recently, this was designated a development server. With Internet transactions and inquiries escalating so rapidly—and development on the HP e3000 being reduced—the division has upgraded the development server and redeployed it as a dedicated Internet/IVR server. "In this way, we can assure that our customers that use the Internet and IVR will not be adversely affected when we have high volumes of daily processing," according to Snyder.
The HP e3000 Model 997 servers and the Model 979 server are all protected by HP Critical Systems Support (CSS), which includes preventive and reactive service to assure system availability. HP CSS provides an assigned mission-critical support team that performs systems monitoring, preventive assistance, monthly patch analysis, change management planning and technical consulting.
The support package also provides, in the event of a failure, automated configuration tracking, remote diagnostics, immediate access to a system recovery specialist, immediate dispatch of a hardware specialist and a six-hour hardware call-to-repair commitment.
Backup Plan
To ensure rapid, transparent recovery of systems and data in case of a natural disaster or some less dramatic business interruption, the division has crafted a business recovery plan for the HP e3000 environment that includes daily and weekly data back-ups and twice-annual business recovery rehearsals.
In preparation for data loss caused by unplanned downtime, the division has multiple levels of data backup in place. The division conducts daily and weekly back-ups onsite. "As our division moves more and more into client/server computing, with the servers all talking to each other, backups are becoming extraordinarily important," Snyder says.
For now, the division uses a disk array product from EMC Corp. to create copies of back-up data: a primary copy, and a secondary/mirror copy. In the future, there will be a tertiary business copy. In the event of a business interruption and for business recovery rehearsals, the division will back up from the business copy. The division currently uses HP DDS3 DAT drives for back-ups.
A business interruption of only three or four days would cost hundreds of thousands of dollars and create hardship for the division’s customers. Therefore, its goal in case of an actual disaster is to recover, with no loss of data, within 24 hours. "We are looking to reduce this potential outage to hours in the future," says Snyder
To make sure it can meet this goal, the division works with HP Business Recovery Services to conduct recovery rehearsals. It conducted the first rehearsal in 1992. Every six to 12 months, the Pension Division works jointly with Comdisco, and the team at the HP Recovery Center in Wayne, Pa., to perform a recovery rehearsal.
Results of the rehearsals prove the division could be back in operation within 24 hours even if a disaster knocked out all systems and data. "We’ve never had less than 100 percent success with any rehearsal we’ve conducted," Snyder reports.
The rehearsals have yielded knowledge useful in not only recovering the system but in shaping the division's computing environment. For example, the division has used the HP recovery center to test hardware, software and operating system upgrades. Such testing is performed as a preventive measure to avoid a business interruption.
The Big Picture
Although HP products and services play a large role in the division’s IT operations, particularly for the pension record-keeping application and business recovery, the Pension Division also uses solutions from other vendors and relies heavily on support from AUL’s corporate IT department.
"HP doesn’t make solutions to meet all of our needs, and sometimes we need to choose other vendors," Snyder says. For example, the company standardized on Compaq laptops because the desired feature set was not available in the HP product line at the time of purchase.
AUL’s corporate IT department and the Pension Division run help desks to aid internal employees and the company’s sales reps and customer service reps. Corporate IT also manages Internet security for all AUL divisions. The company’s firewall security is based on Raptor. Internet transactions are protected with 128 bit encryption and Microsoft’s Secure Sockets Layer (SSL).
The corporate IT department supports many areas from the desktop to voice, Compaq Windows NT servers, Lotus Notes, Domino as well as many platforms for multiple AUL Divisions. Lotus Domino is an integrated messaging and Web application.
AUL’s corporate IBM mainframe is on a subnet with the Lotus Domino servers, to provide users company-wide with mail and browser capabilities. The subnet was built using Cisco switches and routers. The Pension Division uses the IBM mainframe for many applications and print management, currently.
For the Future
The environment definitely is in flux, however, as the division executes its migration from the HP e3000-based system to a client/server environment featuring HP 9000 servers and third-party HP-UX applications for pension record-keeping, document management and workflow. The division is gradually adding hardware, software and support for the new environment and expects the migration to be complete in two years, with some applications starting on-line in the fourth quarter of 2000.
The client/server environment will include five to eight HP 9000 K-Class and N-Class servers; service and support from HP, Comdisco and the AUL corporate IT department; and software from more than 20 independent software vendors. Key HP-UX applications will be products of SunGard, CSC, Quest Software, Inc.
Data storage in the new environment will move the division toward a Storage Area Network (SAN) solution. In addition to an HP SureStore 1200e Optical Juke Box, components of the storage solution will include the HP SureStore DLT 20/700 tape library and expansion of its EMC storage capabilities.
Both the DLT library and the EMC disk support multiple operating systems and server platforms, which is essential when building a SAN solution. The HP SureStore DLT 20/700 will run software with disaster recovery in mind for multiple servers and platforms.
"The combination of the EMC and DLT library will enable us to tie all our HP-UX production servers into one set of disks. We will be able to back up all the HP 9000s at the same time. Without that capability, we would end up with a time-sequencing problem during disaster recovery. We must be able to back up all the systems at the same time, because in our environment data is constantly flowing from multiple systems to multiple systems," Snyder says.
For its client/server pension record-keeping application, the division selected SunGard’s OmniPlus enterprise solution. OmniPlus is a scalable retirement-plan administration system. For document management and workflow, the division chose SunGard’s PowerImage application. PowerImage digitally captures key customer documents as they enter the workplace and automatically integrates them into the workflow to enhance customer service.
Output management will be handled by the Vista Plus enterprise report management application from Quest. Vista Plus, PowerImage and OmniPlus all will be protected by HP MC/ServiceGuard when they become operational in late 2000 and 2001.
HP Multi-Computer/ServiceGuard (MC/ServiceGuard) is a specialized facility for protecting mission-critical applications from various hardware and software failures. This service monitors the health of each system and quickly responds to failures in a way that minimizes or eliminates application downtime. The product can automatically detect and respond to failures in system processors, system memory, LAN media and adapters, system processes and application processes.
Service and support also will continue to play an important role in assuring system availability and business continuity as the division migrates to client/server. HP and Comdisco will continue to provide business recovery services, and HP will continue to provide Critical Systems Support for production servers in the new environment.
George Ferguson is Worldwide Marketing Manager for HP's Business Recovery Services.