In-Depth

Evolving Effective Enterprise-Wide Distributed Processing Solutions Using CICS, MQSeries and DB2

• 12/01/1998

In its Northern California region, one of the nation’s oldest and largest health maintenance organizations has been in distributed computing since the mid-’80s with its use of mainframes networked to geographically dispersed 8100s. In the late ’80s this distributed strategy was abandoned when the 8100 product line came to an end and the applications were brought back to host machines. Distributed computing took a different form when the challenge was to provide real-time data exchange between MVS, Tandem and DEC host machines connected to more than 20,000 terminals. Distributed computing advanced again with the need to provide peer-to-peer connectivity with departmental systems on Intel and UNIX systems located throughout the region.

The region has 31 medical centers and offices interconnected by IP and SNA networks. Over a five-year period, starting in 1994, the organization created a highly successful Clinical Data Repository (CDR) for its 2.5 million members and caregivers on an MVS mainframe using DB2 and an asynchronous message processing paradigm. The key business drivers for the CDR were to:

1. Create a centralized repository for a longitudinal electronic patient record that could be accessed by terminals or computers throughout the region.
1. Provide more timely and accurate information for patient care.
2. Reduce reliance on paper charts and manual filing.
3. Use the data for new functionality like preventive health prompting, patient alerts and basic decision support.

The CDR is a patient centered DB2 database that is currently 1/2 of a terabyte with 20 data types including: patient demographics, lab results, pharmacy prescriptions, admissions, outpatient visits, appointments, radiology and transcription reports. The strategy was simple even if the execution was extremely challenging: gather together all clinically relevant data for patient care from operational systems into one database and develop a carefully controlled layer of access to that data. The first user interface to exploit that layer was called the Clinical Information Presentation System (CIPS), developed as a 3270 CICS application. It represents a high water mark of what can be done with a 3270 presentation. Menus are created dynamically. Navigation is expedited with menu bars and short cut keys. There is extensive on-line help. Response time is optimized by caching screens after the initial retrieval of data. CIPS is used daily by almost 10,000 regional caregivers. By following a multi-tier strategy separating presentation, application and retrieval (data server) logic, the foundation was laid to broaden the accessibility of the data through distribution and the refinement, extension and reuse of the data retrieval layer. In addition, it provided new opportunities to take advantage of rapidly improving and lower cost distributed technologies to capture new data and provide greater functionality to the caregivers via a workstation interface.

The source operational systems for the CDR are centrally located MVS, Tandem and DEC hosts, as well as distributed Intel-OS/2 and Sun servers. As previously mentioned, the strategy was simple, but the execution was challenging. The CDR is an integration project and it required close collaboration between the CDR project and operational systems teams. The CDR project team had to go to the project manager of each operational system and negotiate an interface with that application. Each interface required that an application insert critical additional code to capture into a recoverable queue the data being applied to the operational database. To achieve the goal that the CDR would eventually become an electronic record that would support patient care, it was crucial that the insertion into a queue be part of the application’s unit of work. There was natural resistance to this from the operational system project managers because:

• The additional work could slow down the existing application.
• If there were problems with the queue, the application would not be able to continue until the problem was fixed.

The operational system project managers were doing their duty in protecting their users by requiring that every reasonable measure be taken to prevent the above from happening. The technical obstacles were easier to solve than the human conflicts. The achievements of the CDR are a testament to the management vision and close cooperation that eventually resolved these conflicts.

The distributed message delivery system is known as the Event Processor (EVE). EVE consists of:

• network programs
• recoverable message queues
• distributors (processes that make copies of messages)
• backup processes
• message reformatting processes
• background error alerting processes
• process initiators

EVE relies upon IBM MQSeries, as well as home grown queuing mechanisms and network programs. Over 1.6 million messages are delivered each day to the CDR. A highly successful by-product of the capture of messages by EVE is the ability to feed non-CDR applications as well. Those applicatons are deployed on distributed OS/2, HP/UX, AIX and NT servers running MQSeries.

A follow-on phase to the CDR called for the creation of multiple partitions of the database hosted on lower-cost mid-range UNIX machines. These machines would be located on LANs close to a user community provided with PC workstations. Like the Central CDR, the Distributed CDR would be fed through EVE and a routing subsystem called Intelligent EVE. Intelligent EVE uses medical events occurring at specific locations to determine which Distributed CDR to route patient data to. The technology that supports the logical architecture includes: RS/6000s, Distributed CICS for AIX, MQSeries and DB2 for AIX. Client workstations running Windows NT or 95 can access data from either the Central or a Distributed CDR using the CICS Client. Navigation is determined by a process on the Distributed CDR machine called the Data Access Manager, which runs under Distributed CICS and uses a patient data directory stored in DB2. A typical request for data consists of a patient medical record number, a date range and the data server desired. A data server provides a row and column result set of a specific data type.

The goals of the Distributed CDR were:

• Improved availability by having data spread on multiple machines to eliminate single points of failure
• Improved accessibility from distributed workstation and servers
• Improved performance resulting from smaller, regularly pruned, local data stores
• Increased scalability from being able to divide and conquer with more machines

The complete Distributed CDR strategy provides the following options:

• Workstation access to CDR data (mainframe or distributed)
• Ability to locate distributed CDR platforms close to users for maximum performance capabilities or at a central location for best manageability. The performance trade-off is between a LAN versus a WAN remote procedure call.
• Ability to refine schemes for distribution of data to provide a strong likelihood that local data will be used.
• Ability to move beyond data retrieval in the user interface to capturing new data via a workstation. That data would update both the Distributed and Central CDR asynchronously.
• As the Distributed CDR usage increases, the Central CDR would take on the role of a warehouse where data is indefinitely stored and become a candidate for research.
• Ability to scale using lower cost alternatives to the mainframe.

With a relatively minimal additional effort and cost, the development work on the distributed CDR was recently completed. This was done by:

• Porting copies of the original Central CDR CICS-COBOL-DB2-MQSeries updater programs to the RS/6000.
• Writing a new set of data server programs that could be executed in a CICS for MVS or AIX environment and invoked by a client application using the CICS Client with an External Call Interface (ECI). This is a CICS remote procedure call or distributed program link.
• Designing and implementing the patient data directory.
• Designing and developing the Data Access Manager.
• Documenting the data server requests and result sets.

• The RS/6000 is co-located with the mainframes in a data center, not at the potential client sites.
• Data is not being distributed as management determines the cost benefits of managing distributed data.
• The data servers have been well tested by a prototype client application running under Windows NT.
• Performance tuning for mainframe-only access has been completed and appears to be very strong.

The transaction or data servers of the Distributed CDR followed a Tandem requester-server model which is classic client server remote procedure call. Modularization was the key. The transaction servers were written purely with COBOL and SQL to keep them portable for other TP monitors and as candidates for stored procedures. They were originally developed as batch modules that could be invoked by a COBOL driver program accepting input from a file and writing output to another file. The advantages to this approach were that batch programs are easy to write and debug. File input allows for scripts of test cases and the code can be tested in an environment that requires a minimum of configuration in contrast to a TP monitor environment. It also served as a check that nothing in the code was dependent on a specific TP monitor feature. Test case scripts also lend themselves well to regression testing when code must be enhanced or fixed. If the decision had been made to write the programs in C, a workstation environment would have been the natural selection for hosting the identical approach. Once the programs were thoroughly tested, they were compiled and tested in the CICS for MVS and AIX environments. This required writing a CICS for MVS driver program and a Visual Basic driver.

The development of the Distributed CDR targeted not only writing modular, reusable code, but also taking advantage of experienced, CDR-knowledgeable mainframe professionals in implementing on a UNIX machine. The result was:

• Only a handful of people had to learn UNIX in depth. Those that did prepared an environment which allowed the data server coding to be done by people who never even logged on to UNIX. They were able to run an MVS batch job that would transfer their source code to the RS/6000 and start a script to compile it for CICS-COBOL-DB2. They could then test the code using the Visual Basic driver program.
• Productivity was achieved in the best available environment. CICS Cobol code was most readily developed in the rich environment of the mainframe using ISPF and a sophisticated debugger. It was also kept very securely on the mainframe using reliable change management tools. Another example was a utility written in C with MQSeries that was developed on Windows NT. This was because of the advanced interactive tools available there. The program was then ported to the RS/6000 for re-compilation and execution with no required changes.
• Most of the developers of the Distributed CDR were only doing it on a part-time basis, supporting and enhancing production applications at the same time. This was only possible because the new tools were kept to a minimum. Specialization was also key. Individuals did what they did best. Application programming required more knowledge of the database and data. Driver programming required more knowledge of a technical environment. Testing was a dedicated function of quality assurance professionals. The CDR project manager coordinated all of the activities of the specialists.

Only a couple of people initially needed to understand the workstation technology (Windows NT) in depth. Since the CICS Client was the vehicle for accessing the CDR data servers, it was necessary to acquire experience with this software and develop two utility programs:

• an interactive data server test driver
• a performance driver

Both had the common purpose of being able to:

• allow data server developers to perform their final tests using a CICS Client;
• independently validate the results reported by the developers of client applications;
• and to serve the needs of the independent testing group.

In addition, the performance driver was meant to mirror the batch data server driver by reading and executing test scripts and writing out results and performance statistics into files. The performance driver was developed following a stepwise refinement approach (analyze a little, design a little, code a little, test a lot, reiterate) and the results were extremely useful. The project ended up with a powerful, versatile tool that was easy to modify to meet new requirements very quickly and was always being enhanced upon modular, well-tested code. What started out as a very simple C program that could be executed in a DOS window as a single process eventually became a C++ program that could execute External Call Interface calls in threads, schedule calls in the future and execute groups of calls in parallel and as a unit of work. In addition, it allowed simulating a large number of clients from a single workstation, driving a significant workload against the local and wide area networks and the middle and back end tiers. It was a significant player in our performance tuning success.

The combination of CICS for AIX, DB2 for AIX, MQSeries for AIX and Microfocus COBOL leant themselves very well to the implementation of the Distributed CDR.

The use of CICS for AIX provided the following advantages:

• The reuse of well-proven production mainframe CICS-COBOL-DB2-MQSeries programs
• The reuse of mainframe application programming and CICS systems administration skills
• High performance TCP/IP network communications between PC workstation and RS/6000
• High performance SNA LU6.2 network communications between RS/6000 and MVS mainframe
• Transparent translation between ASCII and EBCDIC
• Development of new data servers that could be used in either CICS for MVS or CICS for AIX
• Transaction coordination between DB2 and MQSeries on the RS/6000
• Load balancing capabilities that go beyond what was available in the operating system

The CICS Client provides an easy-to-use, reliable and high-performance mechanism for developing a workstation application that can access the CICS for MVS or AIX CDR data servers. Furthermore, running under a multi-tasking operating system, it is possible to issue several calls concurrently under threads and display data as it arrives, rather than waiting for all data to arrive sequentially. Both CGI and the CICS Java Gateway were explored in writing Web applications that could use the data servers. Both worked as documented.

MQSeries supports the EVE functions of routing data to the RS/6000, as well as the capture of Data Access Manager statistics and a security audit trail for transmission back to the mainframe for archival and analysis.

DB2 for AIX supported the complete physical design of the mainframe CDR database. Compatibility was key to reducing development and maintenance. DB2 was used as a replacement for the Encina Structured File System which is required by CICS for AIX. Since DB2 was needed anyway, using it for the latter eliminated the need for another data management system and its administration. Finally, Microfocus Cobol is highly compatible with VS COBOL II on the mainframe.

In addition, the following tools were invaluable for developer productivity:

• Mainframe: Xpediter for debugging
• RS/6000: SNA for AIX
• Workstation: Hummingbird eXceed for X-Windows, Visual C++, MQSeries for NT, Visual Basic, ODBC, Microsoft Query, the DB2 Client Application Enabler

It was always a target to do performance tuning of the system before going into production. Initially, it was expected to be the performance tuning of the complete Distributed CDR with concurrent updating and retrieving of data. Access to the mainframe would be the exception, rather than the rule, When the strategy changed to retrieval from the mainframe only, the target changed and there was concern about what the performance would be like. IBM was hired to help with tuning and these are some of the important parameters that were tuned:

• Splitting log files for CICS across multiple drives (Spread the work).
• Splitting log files for MQSeries across multiple drives (Spread the work).
• Modifying the application to use a parameter on the CICS call to the mainframe to eliminate. waiting for a synchpoint confirmation. This was acceptable because data on the mainframe. was only being retrieved, not updated. (Do less work).
• Modification of the application auditing function to use MQSeries non-persistent, rather than persistent messages. To compensate for the risk of losing messages, the messages are re-created later to be persistent. (Do less work or delay it).
• Adjusting the number of CICS application servers.
• Adjusting CICS shared memory.
• Adjusting DB2 agents to equal or exceed the number of CICS application servers.
• Ensuring that the number of SNA sessions equaled or exceeded the number of CICS. application servers.
• Grouping application and system transactions into transaction classes to limit the number of instances in each group that could be using CICS application servers.

The result was a very scalable mid-tier. It took more than one NT client machine to drive a workload that could saturate a 4-CPU RS/6000 R50 and at that level there was a very heavy impact on the mainframe that was well above the expected initial usage of a client system.

As stated earlier, the majority of the team were mainframe professionals who learned enough about UNIX and PC workstation technology to extend their development skills. This included:

• MVS application programmers
• MVS DBAs
• MVS CICS systems administrators

Initially, a few of the application programmers played the role of AIX, CICS and SNA system administrators until help arrived.

The CDR project showed that with the right tools, applications, programming and administration skills can be extended from the mainframe to provide additional value to customers using new distributed technology.

ABOUT THE AUTHOR:

Jerry Kong is a Project Manager for a San Francisco Bay Area Health Maintenance Organization (HMO) with over 20 years of data processing experience whose concentration over the last several years has been extending the value of and creating new legacy applications using client server technology supporting synchronous and asynchronous processing techniques in enterprise wide distributed processing solutions. He can be reached at jykong@pacbell.net.