Kicking Up High-Availability At The World Cup -- Enterprise Systems

Kicking Up High-Availability At The World Cup

HP Prepared for the Worst, While Expecting the Best

12/31/1998

"The World Cup and all the organizations that it entails is a microcosm of theElectronic World," said Alex Sozonoff, HP vice president and general manager for HP'sMarketing Operations Group. So, it was within that e-world paradigm and as the 1998 WorldCup's official technology supplier that HP employees around the globe provided anelaborate and sophisticated e-business and e-commerce IT platform.

With all the international attention focused on the last World Cup in this century, HPhad to see to it that everything involved with the Games -- ticket sales, athlete andmedia accreditation, scoring, providing game statistics and play-by-play information andmany other details -- happened without snafus of the kind (the content server went downfor three days) that garnered IBM so much negative publicity during its handling of the1996 Summer Olympic Games. But it was just more than a HP marketing opportunity.

"The World Cup was a great learning experience in building a collection ofmission-critical systems that had to be up and running all the time. And really make themperform," said HP's CEO Lew Platt. "[And] what we learned from doing [at theWorld Cup], we think we can apply to a lot of customer situations."

Like any large IT project, HP didn't go it alone. Long before the first game of theWorld Cup was played, HP and its partners -- EDS, France Telecom and Sybase --demonstrated their own kind of teamwork. Employees found the devils in the details of anelaborate and sophisticated computer and communications IP-based network -- and exorcisedthem during the weeks from June 10 to July 12.

Although there were many lessons to be learned, the most important, by far, was theneed to meet the business continuity and high-availability requirements necessary "tobe up and running all the time." Delivering World Cup tickets was one of the firstmajor challenges for Philippe Verveer, the IT Director for the Comite Francaisd'Organisation. (CFO; the World Cup organizing committee). The 16th World Cup also camewith new challenges -- of the Internet kind. Two Web sites -- www.france98.com (availableworldwide over the Internet) and www.infofrance98.com (an intranet available only to themedia) -- were known collectively as the World Cup Online. World Cup Online, stored allthe information and data about the games, feeding statistics and tournament highlightsthat included video clips and updated news to the world's public and international media.

Using Sybase's PowerDynamo Web server, this central database, (running on a cluster ofHP 9000 D370/2), generated Web content then transferred information to the france98 andinfofrance98 Web sites. Data from the content engine was replicated (using Sybase'sReplication Server) in real-time to each of the 10 stadium sites, the CFO headquarters andthe International Media Center (IMC) to Web Servers. According to HP World Cup 1998Support Coordinator, Joel Dubedat, the Web server architecture included a HP NetServer LCII running Netscape Enterprise Server on top of Microsoft Windows NT 4.0 at each location.

These Web servers received updates, like scores, news and weather forecasts inreal-time, from the content engine. "In case of failure," explains Dubedat,"of ANY one of these servers, any other server (at another stadium) could bedesignated as a backup server. Even if the switch process was manual, we designed it to beeasy to implement. There was no need to modify the desktop environments," saysDubedat.

The Internet backbone was connected to four mirrored sites: one in Paris, France(hosted by France Telecom); one in Plano, Texas (hosted by EDS); one in Herndon, Virginia(hosted by PSINet); and one in Santa Clara, California (hosted by PSINet). Global loadbalancing between mirror sites was done using a Cisco Distributed Director and likedevices. "This is a two-level balancing mechanism (distributed then local),"recalls Dubedat.

As the official public Web site, france98.com set four Guinness Book world records:

1.137 billion hits -- most visited Web site (June 10 to July 12)
10.29 million hits -- most hits in one hour (June 29)
235,356 hits -- most hits in one minute (June 29)
73 million hits -- most hits on an Internet site in one day, June 30. The previous one-day hit record was 59 million set by IBM's official Nagano, Japan Web site during the 1998 Winter Olympic Games.

The average response time was 4.78 seconds, says Dubedat, as measured by Keynotesystems. (www.keynote.com/measures/worldcup). Also online for the first time wasstore.france98.com, an e-store selling World Cup memorabilia. Store.france98.com wasdesigned, configured and managed by HP's Operation Systems Division (HP OSD is located inthe U.K.). The e-commerce site maintained an average rate of 760,000 hits per day. HP alsoused its Web Quality of Service (Web QoS) technology for the first time to ensure aroundthe clock Web availability.

In creating the World Cup infrastructure, HP supplied a panoply of computing hardware,software and services (see Giving it Up For the Cup sidebar). While some IT managers arequick to seek out the latest and greatest products, the savvier ones know that"leading edge" is perilously close to the "bleeding edge."

The only glitch occurred when one venue was forced to run independently of the centralhost, but the users were never aware of the problem, says Dick Wiles, director ofinformation services for EDS Global Sports.

But Dubedat says, "We decided to use proven technology. We never tried to installthe latest product out of the Lab. We decided to use HP technology that had alreadydemonstrated its abilities in mission-critical applications."

--S.R. Rodefer and Lane Cooper of the Washington News Bureau (Washington,D.C.) also contributed to this article. HP Professional would also like to thank LinaHoriuchi of Cunningham Communications and Joel Dubedat and Christian Hostelet ofHewlett-Packard France for their helpful assistance.

THE HIGH-AVAILABILITY CONFIGURATION OF CHAMPIONS

High-availability was not only required, it was essential. So, two independent MC/ServiceGuard clusters were configured as follows: Cluster 1 was dedicated to the ticketing system. It was composed of a HP 9000 two-way K410 and a uniprocessor K210. The K410 (1GB RAM, 2x FWD SCSI-2 interface, 4GB internal storage for dumps, 5x2GB mirrored external storage using HP MirrorDisk/UX and UPS) acted as the "main" node and the K210 (512MB RAM, 2xFWD SCSI-2 interface, 4GB internal storage and UPS) served as the "backup node" of this two-node cluster. The K210 also supported some non-critical portions of the application; queries for statistics, for instance. The ticketing system supported the selling of 2.5 million tickets. There was no connection with the Internet. These two machines were running HP-UX 10.10 and Sybase System 11.

The second cluster, comprised of three two-way D370s (512MB RAM, 2xFWD SCSI-2 interface, 2GB internal storage and UPS) and 2 external disk bays (4x9GB hot swap disk modules and 2xFWD SCSI-2 connectors and redundant power supplies) was configured for multi-purposes. Each D370 node had physical access to the two disk bays to allow applications failover even in case of a dual-fault. That is, if node 1 failed, then application switched to node 2. If node 2 failed, application switched to node 3.

The HP 9000 D370/2 was configured with 1.5GB of RAM running HP-UX 10.20 with the latest patches, OnLine JFS (for faster recovery), Netscape Enterprise Server 2.x, 2 x HSC 100BaseT network card and High Availability disk bay (2 x 4.3GB High storage with MirrorDisk/UX).

In standard mode, EDS's SCORE!, which included data on athlete and media accreditation, HR information about the volunteers as well as protocol and transportation data was running alone on node 1; the intranet content engine and e-mail (12,000 mailboxes handled by HP OpenMail) was running on node 2 while the Internet Content Engines was running on node 3.

The two "Content Engines" located physically on node 2 and node 3 contained nearly identical content. This was done to isolate the Internet activity (potentially large number of users, unpredictable behaviour) from the intranet activity. Both of these content engines were feeding their associated Web servers (intranet or Internet).

The WAN connectivity or bandwidth available at each of these sites to the Internet were: 155Mbps for Plano, 45Mbps for Paris (plus an additional 45Mbps link for backup), 100Mbps for Santa Clara and 100Mbps for Herndon, according to Dubedat.

GIVING IT UP FOR THE CUP

EDS

As the systems integrator, EDS (Plano, Texas) personnel started working with the CFO and HP Consulting in January 1995 making several early risk assessments, including one by management consulting firm and wholly-owned EDS subsidiary, A.T. Kearney (Chicago, Ill.). The results included a crisis center to handle external problems and two central hubs of information -- a joint partner's solution and a central command center in Paris that acted as a clearinghouse for problem resolution.

EDS, also adapted its ticketing application (based on Sybase's Sybase 11 database) to run on HP-UX (see Sybase below). The ticketing system interfaced with a network of HP 9000 CAD workstations that allowed visualization of seat assignments. Other capabilities included management of all seat assignments, phone, mail and Minitel (See France Telecom below) reservations, waiting lists, cash management, bookkeeping, taxation and ticket production for accredited officials, VIPs, media and sponsors.

France Telecom

France Telecom provided the intranet backbone via a fully redundant Frame Relay WAN. That allowed each stadium (and the IMC) to be linked at 2Mbits/s with a hot backup link with automatic switchover (the open shortest path first, OSPF, protocol was used). At the application level, it was possible to reach any node from anywhere (inside the intranet boundaries) for database replication, file distribution, system and application administration. According to Dubedat, "This allowed us to devise recovery procedures on this highly redundant infrastructure."

As part of the ticketing and online services, France Telecom's Minitel, one of the first widespread online services in the world, fulfilled 60 percent of all domestic ticket sales through its 8 million text-based terminals installed in homes throughout France.

Hewlett-Packard

In creating the World Cup infrastructure, HP supplied a panoply of computing hardware, software and services consisting of over 50 HP 9000 systems, 65 NetServers, 2,000 PCs and laptops, 290 AdvanceStack hubs, switches and routers and 600 networked printers.

All that hardware was interconnected throughout Paris and the 10 World Cup stadium sites via HP AdvanceStack switches over 150 HP 100VG-AnyLAN LANs and connected by approximately 250 10BaseT and 100Mbits LAN hubs and 50 HP AdvanceStack routers. At the CFO's datacenter located in Le Blanc-Mensil, network and systems management consisted of several HP OpenView components: Network Node Manager, OmniBack II, NetMetrix, PerfView and IT/Operations.

Each site featured local area load balancing, which spread the connections over the number of available servers. If a server was busy or failed, no more requests were sent to that server until the problem was corrected. Overlaid onto that was a WAN that sent users to the desired site within an optimal response time, says John Watson, director of business development for PSINet's application Web services group. PSINet prepared for the expected traffic volume by using HP's OpenView suite of tools to do proactive maintenance.

Sybase

SCORE!, an EDS application (used in the 1994 World Cup on Sun Microsystem platforms) was migrated to run on HP-UX 10.2 and Sybase System 11. Aided by EDS Resources, the company's strategic business unit in France, a global team developed the core applications in the U.S. Five Sybase products were used to build the SCORE! applications: Sybase System 11, Sybase's flagship database; PowerDesigner, an application design tool; PowerBuilder, a fourth-generation application development tool; Adaptive Server Anywhere, a mobile database software product; Adaptive Server Enterprise, a large database product; and PowerDynamo, a tool required to do dynamic HTML publishing of the information available on the Web. All logic was written in PowerDynamo's DynaScript and displayed as pure HTML. And the record-breaking Web site traffic gave the company's professional services arm a workout.