In-Depth

Welcome to Wendy's, How May I Protect Your Server Data? Centralized Backup Management at Wendy's International

• 07/01/1999

Customers around the world have learned to expect fast, prompt and courteous service when visiting any one of the 7,000 restaurants of Wendy’s International, Inc. (Dublin, Ohio). Similar expectations exist within the company – especially when ensuring the reliability, availability and recoverability of a growing amount of data residing on corporate network servers.

While the vast majority of all critical application processing at Wendy’s occurs in the MVS mainframe environment, Wendy’s IT professionals have witnessed an exponential leap in the number of network servers required to support a multitude of new, distributed systems. Connected to such growth is the challenge of implementing effective file backup, recovery and archival procedures.

"With the implementation of new distributed systems during the past five years, we’ve seen our server data grow from approximately 250,000 files in 8.9 gigabytes to 1.9 million files in nearly 1/2-terabyte," says Ed Ohanian, Director of Enterprise Technology Services. "Early on, we began to struggle with a method for including server data in our enterprise recovery plans."

However, as a mainframe-centric environment with established procedures for data center automation, Wendy’s required a solution that would integrate server storage management with the tasks controlled by its job scheduling system and the vaulting capabilities provided by its tape management system.

"Centralization and automation was crucial," says Ohanian. "It was imperative that we implement a toolset that could handle both incremental and full server backups and restores. It was also imperative that the toolset allowed us to manage our servers alongside of everything else."

After evaluating a number of methods and tools, Wendy’s implemented FDR/UPSTREAM from Innovation Data Processing (IDP; Little Falls, N.J.). FDR/UPSTREAM is a storage management tool for centralized, automated and unattended backup/restore and archival from PC/LAN/UNIX systems to OS/390 MVS mainframe server tape and DASD. The product provides full, incremental and differential backup and supports PC servers running NT, Novell, OS/2, as well as UNIX servers and client workstations.

The implementation enabled the company to further exploit the automation tasks already in existence in the mainframe environment through integration with established backup/recovery procedures. Additionally, it provided a centralized location for administering server-based storage management tasks with the added benefit of managing by exception.

"Before we installed FDR/UPSTREAM, we were required to monitor logs and keep an eye on a separate PC console to determine if the backups were successful," says Ohanian. "Now, we receive automatic notification only in the event of a failure."

Arguably, one of the most popular ad campaigns ever launched by Wendy’s was, the "Where’s the Beef?" slogan uttered by Clara Peller. However, this clever slogan could possibly have originated several years ago from within the halls of the company in the form of "Where’s the File?"

The drive toward client/server applications back then caused a great deal of critical data to be created on a growing number of OS/2 servers in a token ring network. Like many companies, changing business requirements eventually saw the technology evolve to include more than 40 Windows NT servers.

"The servers were used to support both critical and non-critical applications," says Ohanian. "We needed to ensure that this distributed data was protected and recoverable in a fashion similar to what we’ve performed in the mainframe environment."

Although continually evolving disaster recovery plans were in place for the mainframe environment, Wendy’s realized it needed to expand those plans to include its servers. In some cases, PC users implemented their own procedures for backups, using a combination of PKZIP (PKWARE Inc., Brown Deer, Wis.) and CA-XCOM (Computer Associates, Islandia, N.Y.) to compact and transmit files to mainframe storage.

While this type of activity provided a small level of comfort for users who infrequently remembered to perform backups, it provided no assurance that data could be recovered in a coordinated fashion in the event of a major contingency recovery effort. Also lacking was the ability to accurately monitor backups or report on file availability.

"It almost got to the point where everyone wanted their own little tape drive on their desk," says Ohanian. "Obviously, it was no way to manage recovery."

Seeking to centralize the backup/ recovery activities, Wendy’s installed a standalone file server backup tool. According to Ohanian, while the product accurately performed file backups, the process was exceedingly slow. "It was a single-thread process that operated on one server at a time," says Ohanian. "With the growth we were experiencing, it became readily apparent that the backup time would soon cut into the time the servers were required to be available for the business."

Another drawback, Ohanian notes, was a distinct lack of automation that required constant manual intervention by the operations staff to review the success and failure of the backups – notification was through a log that was created only after the backup task had completed. Most important though, technology constraints prohibited the integration of the product with the automated operations technology and the backup/ recovery and archival procedures already in place for the mainframe.

"We wanted to take advantage of the procedures and tools already in place on the mainframe," says Ohanian. "That means using our scheduling system, using our tape management system and its vaulting capabilities, and integrating server storage management with mainframe storage management."

The implementation of IDP’s FDR/ UPSTREAM enabled Wendy’s to exploit the automation of its mainframe-based PLATINUM AutoSys/Zeke – from PLATINUM technology Inter-national, Inc. (Oakbrook Terrace, Ill.) – job scheduling and its CA-1 from Computer Associates tape management products. Further integration with IDP’s mainframe storage management tools FDR/ABR helped the company transform its disaster recovery plan into an enterprisewide solution encompassing both mainframe and server environments. Equally important, the product provided a solution that placed no limits on a constantly-evolving, ever-changing technology infrastructure.

One of the first steps performed by Wendy’s was the creation of a "baseline" set of backup jobs. These full-volume backups capture all of the files on each of the distributed servers. The process is initiated when the ZEKE mainframe job scheduling submits a job that, in turn, links to the server environment through a TCP/IP or LU6.2 connection where it submits a request to perform the backup. Under the covers, FDR/UPSTREAM monitors each of the file names and records additional information, such as date and time of the backup, the name of the server, creation date and last modification date. Additionally, controls within the product provide immediate notification in the event of task failure, or in the event that the task was unable to perform a backup of specific files.

"Events and exceptions from the server environment are immediately displayed on the operator console alongside of the events and exceptions that may occur in the mainframe environment," says Ohanian. "From an operations perspective, we need only to go to one place to check the status of our backups."

After baseline creation, Wendy’s then employs the FDR/UPSTREAM Forward Merge Backup procedure that helps guarantee speed and provides the most protection for files while minimizing tape utilization and network traffic. Employing workload balancing techniques based on the amount of data stored on the various servers throughout the organization, full volume backups used for disaster recovery are performed during different days of the week. For example, full backups for servers containing the databases for Lotus NOTES and Voice Response Units (VRUs) may be performed each Monday, while the full backups for IBM’s MQSeries and remote servers may be performed each Tuesday.

To ensure the availability of the servers for production business users, a couple of techniques were exploited to increase the speed of the backup.

One technique used was through an FDR/UPSTREAM process named MVS Migration. MVS Migration performs disk pooling and places backup data directly from the server onto a mainframe 3380 DASD device. From there, an MVS Migration job consolidates all backups from multiple server data onto a single tape, thereby reducing tape utilization. Regardless of the technique employed, all tasks are initiated by the AutoSys/Zeke job scheduling system and all created tapes are placed under the control of the CA-1 tape management system.

Additionally, the company implemented standards that requires FDR/UPSTREAM to be installed on all production servers as well as a corporate directive to maintain critical data on these LAN servers. This enables IT to provide business users with file recovery capabilities for up to three weeks.

"Although we have the technical ability to backup data from each of the desktops," says Ohanian, "it is an administrative nightmare to keep track of each of those units – especially in a mobile environment with laptops." He notes that most desktop computers are like thin clients – they contain a small amount of storage and exploit the applications and storage running on the network.

While backup speed is crucial to ensure the availability of files on production servers, equally important is the capability for complete recovery and efficient tape utilization. "Even in light of full disaster recovery capabilities, most times a restore is required only for a single file," says Ohanian.

And for the people that require a restore, it is vital that the file be as current as possible, or available from several most recently changed or worked on. That is why Wendy’s implemented the Forward Merge backup while also maintaining Upstream’s incremental file level granularity. Ohanian has decided to allow users to keep up to 21 days of incremental backups for file selection.

Incremental backups in the server environment operate in similar fashion to those employed on the mainframe: Backups only occur for those files that have been modified since the previous backup. According to Ohanian though, the difference provided with FDR/UPSTREAM is the ability to reduce tape utilization and network data traffic with a process named Forward Merge Backup. With Forward Merge Backup, FDR/UPSTREAM creates a new, synthesized image of the server or workstation without sending or transmitting any unchanged files. The Forward Merge backup facility allows FDR/UPSTREAM to construct a complete full backup without the client server or workstation having to read or send all the files across the network resulting in extraordinary performance.

Using a sophisticated technique, the workstation sends a "picture" of the drives to be backed up to FDR/ UPSTREAM MVS which uses prior backups plus changed files to construct the "full merge," which properly reflects deleted files, changed files and renamed files. The completion of the forward merge results in a single mainframe tape or DASD dataset that is representative of the server at that time of the backup and provides quick access for restores. "The task completes in a fraction of the time, compared to a full volume backup," says Ohanian.

Recovery is the opposite side of the same storage management coin. Proof of accuracy for backups can only be realized through a successful recovery process. At Wendy’s, FDR/UPSTREAM recovery falls into three primary categories: Individual restores for lost or corrupted files; full volume restores for disaster recovery purposes; and a combination of file/directory restores using new names, typically used for creation of test beds or retention of a duplicate copy of a file or directory without overwriting the original.

"The ability to restore files to a new directory or with a new name is especially valuable for user documentation files or for Lotus Notes NSF files," says Ohanian. "The original file can remain in place and users can perform comparisons between the two."

Equally important, he notes, is the ability to perform physical disk backups and restores that enable the recreation of an entire OS/2 or MS-Windows NT server from a single diskette. "We want our servers to look exactly as they did before the disaster struck, including the registry," says Ohanian.

Additionally, the company has taken advantage of the SAF-compliant interfaces within the product to extend mainframe security rules to the server environment and further protect its files. The same centralized management capabilities provided for backups extends to restores and is managed by the System Administration group. FDR/ UPSTREAM maintains a database of files backed up and through a GUI interface using a Windows Explorer-like screen. Members of the systems administration staff can easily view, select files, directories and/or subdirectories, or volumes with wild card support to easily perform restores. Similar capabilities are available in the MVS TSO/ISPF environment.

"We can employ two layers of security to protect files that have been backed up with the product," explains Ohanian. "The first layer is our traditional mainframe security system, the second layer is within the server component of FDR/UPSTREAM itself."

While Wendy’s considers the implementation of FDR/UPSTREAM to be a success, a continually evolving information infrastructure has presented the company with some challenges. One challenge, noted Ohanian, was an evident reduction in throughput time during backups after a network conversion from SNA to TCP/IP. The problem was rectified after research revealed some configuration modifications were required.

Other challenges currently faced by Wendy’s is an ongoing evolution in the WAN and LAN environments and the implementation of EMC’s (Hopkinton, Mass.) new Symmetrix DASD with Enterprise Storage Platform. "We’re putting in frame relay support for our remote sites on the WAN and moving to Ethernet from token-ring on our LANs," says Ohanian. "On top of that, we’re installing EMC DASD and will need to provide recovery support for those volumes." Wendy’s is considering using FDR/UPSTREAM companion products, FDRSOS and UPSTREAM/SOS to transmit data from EMC DASD through the mainframe channel and avoid the network.

Continued growth translates into the need for continuous fine-tuning of the backup processing. For example, backup processing for a 20GB corporate Lotus NOTES server may need to be changed from a daily full backup to using the NOTES incremental backup support of FDR/UPSTREAM to save time and tapes. And, as more and more users place increasing amounts of data in the server environment, Wendy’s is also exploring implementation of automatic archive and recall, or disk grooming functions that will reduce the workload of the system administration group.

Even without the changing networks or addition of new hardware, Wendy’s continues to experience growth in server storage utilization as DBMSs demand greater amounts of space contained within new in-house developed applications and commercial-off-the-shelf products. While these DBMSs are currently included within the company’s backup/restore processing, greater efficiency can be realized through the implementation of specific DBMS support within FDR/UPSTREAM.

For now, the company will continue to expand coverage as the environment continues to grow and change. The methodologies provided by FDR/ UPSTREAM have enabled the company to fully integrate the storage management activities of its server environment with the automated procedures in place on the mainframe, helping to build comprehensive, enterprisewide recovery capabilities.

About the Author: Philip E. Courtney is a marketing consultant and technical journalist with nearly 20 years experience in the IT industry. He can be reached at (602) 684-2854, or at www.philcourtney.com.