Philadelphia Stock Exchange Invests in Technology

The nation's oldest securities exchange puts stock in SunGard Recovery Services to help develop and implement a reduced recovery window.

• 08/26/1999

In 1990, the Philadelphia Stock Exchange celebrated its 200th birthday. That makes it the nation’s oldest securities exchange, predating the New York Stock Exchange by two years. But even though it is nearly as old as the republic itself, the Philadelphia Stock Exchange hasn’t run out of new ideas. To compete against its rivals, the Exchange relies on creative and continuous innovation, including such product introductions as currency options, sectors index options and listed stock options.

When the Philadelphia Stock Exchange’s IS team faced the need to reduce its recovery window to an internal target of just two hours, it turned to SunGard Recovery Services to help develop and implement that solution.

"We already had a recovery contract with SunGard," says Frank Reidy, First Vice President of Information Services for the Philadelphia Stock Exchange. "But under the previous recovery scenario, it was a lengthy process to load our IBM system environment onto the SunGard systems. A two-hour recovery just wasn’t achievable." That was in 1996.

Reidy noted that because of the business resumption requirements of the Exchange’s back office processing, he needed to narrow the recovery window in the event a disaster was experienced during the business day.

The Exchange’s mainframe with EMC Symmetrix DASD supports the clearing and settlement processes, which are critical to the public. A disaster could wipe out the only records of a day’s trades, leaving buyers and sellers holding the bag, so to speak. Securities worth hundreds of millions of dollars change hands each day on the Exchange, so losing trading records for a day or even a few hours could certainly have a dramatic impact on the Exchange, its members and its customers.

Unlike many other disaster recovery customers, the Philadelphia Stock Exchange wouldn’t be able to hit its new two-hour target using the standard backup tape routine.

In such cases – that is, those cases in which the recovery window is too narrow even to allow loading tapes, let alone time to secure backup tapes and travel to a recovery facility – the only solution is data mirroring.

Data mirroring replicates information as it is created, simultaneously transmits that information via high-speed fiber optic circuits and records it at a remote site, effectively storing transactions at two locations. Once information is stored and protected at a second site, it becomes immediately available in the event of a processing interruption at the originating site. Mirroring closes the recovery window for mission-critical applications and can eliminate the need for a duplicate data center. However, it’s not easy to shop for mirroring services.

You can see or read about most recovery options. They have a track record. Mirroring is different.

"Mirroring wasn’t a new technology," says Reidy, "but in the past, it was used primarily between data centers. Files were duplicated at a second data center, which required duplicate environmentals, CPU, disk, tape, printing and communications, if it was also to be utilized as a disaster recovery site."

The cost of creating a duplicate data center made the use of mirroring an expensive technology that appeared only in those rare cases when the need to recover almost instantaneously superseded financial considerations.

Obviously, a mirroring solution was not something the Philadelphia Stock Exchange could buy "off-the-shelf," and mirroring’s prohibitive cost left the Exchange with no real track record to base its judgement on. But the idea of mirroring from an operational data center to a disaster recovery company was new, and the cost structure of such a solution was shifting. Thus, at the same time the Philadelphia Stock Exchange’s recovery window was closing, the window for a cost-effective mirroring solution was opening.

The Philadelphia Stock Exchange was looking for a way to narrow its back office recovery window. Meanwhile, SunGard was expanding its relationship with EMC and had just completed the disaster recovery industry’s first mirroring test. SunGard had seen the technology’s tremendous potential, and it had begun recruiting candidates to exploit that potential.

In January 1996, Reidy sent Howard Lambert, Director of Production Support, to SunGard to attend a presentation on SunVault services, which include SunGard’s data mirroring service, and to evaluate the feasibility, costs and benefits of mirroring.

Lambert returned with this assessment: Compared to building a duplicate data center, mirroring could be a cost-effective solution; mirroring would enable a more efficient recovery because it obviated the need to retrieve backup tapes from their off-site storage and the need to reload all their data. Instead of the eight to 24 hours the Exchange had been budgeting for a recovery operation, mirroring could provide a relatively instantaneous recovery – all the Exchange would have to do is run a system IPL.

Reidy saw that SunGard’s new mirroring solution was the Exchange’s best chance to meet its two-hour recovery goal. "Mirroring was a big relief," says Reidy, "because we weren’t so sure how to recover in the necessary time frame without something like mirroring."

"We were definitely interested, but there were no reference sites to emulate as examples because this was a relatively new technology. We needed to find out whether we should commit, whether the technology worked and whether we could afford it."

Shortly after the "leading edge" entered the high-tech dictionary, it was followed by the "bleeding edge," a term used to describe technology so new that its implementation often resulted in productivity, budget or career casualties (i.e., high-risk stuff). With mirroring, the Exchange was dangerously close to – if not on – the bleeding edge.

"The Exchange had some skepticism because the proposed system couldn’t be demonstrated," says Reidy. "No one else had this system."

The Exchange was also concerned about the cost of mirroring, which is significantly lower than it was just a few years ago, but still significantly higher than the more traditional disaster recovery solutions. However, what eventually overcame these obstacles was the solution’s potential to meet the Exchange’s needs and SunGard’s cooperation in providing incentives for the Exchange to participate in the testing.

"The overwhelming incentive," says Reidy, "was an internal incentive – we needed to recover in a more rapid time frame. If mirroring couldn’t narrow our recovery window, no incentive would have persuaded us. An external incentive was that because we were the first company in the world to do this, SunGard and EMC were willing to offer what amounted to a no-risk clause. SunGard and EMC installed the entire infrastructure – communications, Sonet rings and disk storage at SunGard’s Philadelphia MegaCenter – without any written obligation from us. By doing that, SunGard raised the level of confidence we had in the product." The next step for both was to implement the idea.

In April 1996, after all the necessary approvals were in place, the Exchange and SunGard began the mirroring project. It was a win-win situation: The Philadelphia Stock Exchange got a risk-free test, and SunGard got to provide the industry’s first real-time disaster recovery mirroring solution. To tackle a project in which the technology was virtually new to everyone, the Exchange set up a team with members from the Exchange, SunGard and EMC.

SunGard MegaCenters are interconnected to telecommunications carriers with high-speed fiber optic circuits, so SunGard set up a SONET ring, using T3 lines in a closed loop with dual paths to SunGard.

Using network hardware and telecommunications circuits, SunGard connected the Exchange’s existing EMC DASD to its own EMC Symmetrix using the EMC Symmetrix Remote Data Facility.

Having developed the solution and introduced it to the Exchange management, all Reidy and Lambert had left to do was prove it would work.

First, they installed the infrastructure and tested various configuration and communication scenarios between the Exchange and SunGard. Then, Reidy scheduled a final acceptance test of the mirroring solution for August 1996 – to prove mirroring would work.

"We stopped mirroring on Friday at 7:30 p.m.," Reidy recalls, "and sent our people to SunGard’s MegaCenter to see if we could parallel our production processing using the data we’d been mirroring. When we arrived at SunGard, the situation and our objectives were the same as they were for previous tests.’’

"We used a matrix to test all sorts of alternate scenarios," Reidy says. "Between the two sites, we tested every possible system initiation combination, and every test was successful. We ran our computer systems utilizing disks at SunGard, and we ran disks at the Exchange from computer systems at SunGard. We booted both systems using SYSRES packs at each facility. We processed the overnight production stream in parallel. We ran all batch and communication processing, and we did a bit-by-bit comparison to our production system. Everything matched up perfectly. Doing the regular production workload at the Exchange was the final proof that the disks did mirror and remain in sync for a full day’s processing."

The only difference in the recovery process was the time. It took the team just 58 minutes to bring the Exchange’s back office systems online and ready to process, thanks to the hours saved by not having to recover its DASD.

Reidy and his team, in partnership with SunGard and EMC, hit all of their milestones and accomplished all of their goals. "We proved the concept," Reidy says, "and we exceeded expectations on this final acceptance test. We ran 1,035 jobs, all successfully, and it was by far our best disaster recovery test ever."

The overwhelming success of the August 1996 test was the green light for the Philadelphia Stock Exchange to add SunGard’s Electronic Vaulting Services to its services contract.

Predictably, the new recovery scenario has prompted some changes.

"We currently mirror about half our total DASD under our contract with SunGard," Reidy says. "We maintained traditional recovery methodologies for development and other non-critical data."

Since completing its inaugural test in just under one hour, the Exchange has continued to pare its recovery times, getting its system online and ready to process in as little as 24 minutes.

About the Author: John Lindeman is Director of Enterprise Solutions at SunGard Recovery Services, Inc. He can be reached at (215) 351-1300, or at jlindeman@sungardrs.com