In-Depth

Performance Implications of the Year 2000 Problem

• 11/01/1998

Computer trade journals continue to devote a great deal of space to Year 2000 (Y2K) issues, how to organize a Y2K project, the trade-offs between different solutions and all the bad things that will happen to those who don’t pay attention. Mention is made less often of an issue that may haunt Information Technology (IT) past the end of the millennium, which one will, after all, either survive or not survive. This issue concerns how well the inventory of applications will perform after their Y2K functionality has been repaired. Capers Jones of Software Productivity Research, Inc., has published an article, "The Global Economic Impact of the Year 2000 Software Problem," in which he cites both his own and other research. He concludes that software throughput (read performance) may be reduced by between 10 and 35 percent. The response in third quarter, 1999, of a chief financial officer (CFO) to the news that he needs immediately to increase available MIPS by 10 to 20 percent for the Y2K project to succeed is likely to be less than gracious.

Application performance bodes well to be a major unanticipated result of Y2K remediation projects. However, the performance challenge that Y2K raises also brings with it a unique and non-recurring opportunity. If correctly seized, it may be possible to emerge from Y2K administering application performance just as change control, testing, and (one hopes) functional quality assurance are administered now.

Y2K remediation, taken as a whole, will use a different approach from conventional Information Technology (IT) project management. By the nature of the Y2K dilemma, its resolution requires that an organization be fully upgraded in a single synchronized operation. The essential first step is to completely inventory all the applications that exist, regardless of where they are run or by whom. Each application and its components must then be evaluated and catalogued. What platform, language and operating system are employed? What file or database structure? What feeds it and where does its output go? How important is it to the organization? How often does it run, or does it run at all? It is all rather like taking on the restoration of an antique car where the engineering schematics are missing. The first step is to disassemble the vehicle, spread all the individual parts on a ground cloth, and analyze the function and condition of each part.

As the applications are inventoried, so too must the systems environment be reviewed and understood. Depending on how current a shop is on its releases, an operating system upgrade, or perhaps two, may be required. If the shop has not yet gone to the Language Environment, that will enter the planning. This may be the moment to go to DB2 because that is simpler than bringing IMS up to scratch. And thus, another pile of assemblies is laid out on the ground cloth.

When the inventory is complete, the construction of the subassemblies is understood and the requirements for repairing or refurbishing each part is defined, a plan can be prepared to reassemble the vehicle, making use of old, rebuilt, or new parts as appropriate. In doing this, decisions are made about converting to four-digit dates vs. employing a windowing technique. The requirements for bridges between applications are settled. Middleware is specified and procured. The order and timing of operating systems upgrades are established. Schedules for internal development are worked out, delivery and integration of purchased components arranged, and the installation of systems components laid on.

At this stage, too, are defined the testing, quality assurance and contingency plans. What to do if time runs out before the fixes are complete is determined. Test strategies are defined and documented since it is necessary to have assurance that the fixes actually work. Plans will be layered and tests will grow more elaborate as components are assembled into applications, which grow into systems, which are melded into a new edition of the organization’s processing inventory.

The inventory is complete, the relationships understood, a budget has been obtained, personnel lined up, schedules prepared, contingencies covered and a test apparatus put in place. With due diligence, a good deal of hard work and a measure of luck, Y2K has been put firmly in its place. What could possibly go wrong?

What could go wrong is the shopping cart problem. Groceries always fit neatly in the shopping cart until the bag boy finishes bagging them, at which point a second cart is needed to hold it all. Existing IT operations were not built entirely from scratch. They grew incrementally over long periods of time as individual applications were developed and bolted on. Even recent re-engineering projects almost never attempted reworking an entire organization. At most, financial, marketing, or manufacturing systems were done. Anything more was simply too complex to contemplate. Y2K, on the other hand, is a complete recreation of the IT operation (if not of the business itself). Everything from the oldest desktop to the largest mainframe is on that ground cloth.

When a shop emerges from Y2K, for the first time ever, everything will be simultaneously different in some way. Internal applications, operating systems, databases, network interfaces, third-party software, nothing will be as it was before. Always before, much of the environment could be held constant while one or two variables were adjusted. Now, they are all moving together. Sometime next year, if all goes well, all the coding, installation, unit and system testing, and integration will be completed. What amounts to a brand new vehicle will emerge from the development labs and be started up. That may be the first time that it becomes evident that a second shopping cart is needed. If it is, the unfortunate participants will learn that CFO’s, more than most, subscribe to that fundamental law of data processing, "There are no good surprises!"

It would be desirable to know whether IT, once reassembled for post-Y2K operations, will perform within the bounds it once occupied. If it won’t, it should either be fixed or be given as much advanced warning as possible. Achieving this will require dispensing with the habits of a lifetime. As a much younger IT staffer, I wrote BAL for a very small DOS-based machine. Later, after several years pursuing other interests, I returned to IT to write COBOL under OS and then MVS. I quickly learned not to worry about the efficiency questions that had occupied me in BAL. It was a big, fast machine and getting steadily bigger and faster. And so it has been ever since. Who among you has seen a software release that was smaller and faster than the release it replaced?

IT has always been concerned with performance. CFO’s, after all, have never liked surprises. But the concern has traditionally occupied the time and attention of systems personnel, capacity planners and other similar specialists. The tools available to monitor and tune performance have either focused on systems or needed systems-grade skills to use. Applications have been tuned for the most part in fire-fighting mode when batch window shrank, service level agreements were in jeopardy, or response time got bad and the users revolted. Otherwise, applications people, as I did, have relied on a big fast machine to deal with performance issues. Only rarely, has an IT organization created and pursued an orderly and rigorous program of tuning its application inventory.

So long as applications were created, revised and installed piecemeal, firefighting to correct obvious problems sufficed. Not Nirvana perhaps, but a rational use of scarce resources. Y2K, because it is not incremental, poses a real danger and presents a real opportunity. The danger is the shopping cart problem. It is unlikely that application performance baselines, for the most part, exist. Furthermore, all IT variables are simultaneously in a state of flux. Therefore, extrapolating post-Y2K performance from past experience probably will not avoid a difficult hour with the CFO. On the other hand, because everything has been taken apart and must be reassembled and tested, there is a unique opportunity to implement an orderly administration of application performance throughout the application life cycle, both in Y2K testing and afterwards in production.

A rigorous program of administering application performance is likely to yield two benefits. First, by ensuring that the application inventory runs as efficiently as possible, orderly administration helps to ensure that new resources are added only when they are absolutely required. Secondly, tuned applications tend to greater stability and bring with them more accurate knowledge about how they should perform. This makes identification and resolution of surprises (always bad, remember) much easier. System management becomes more orderly and capacity planning less an exercise in imagination.

Tuning applications is not unlike its cousin at the systems level in the information required. An application, like a system, uses the CPU and does I/O and, like people, it waits around a lot. The first step in tuning an application; therefore, is to collect data on how long it takes to run and to consume some gross numbers on the resources. Knowing what it does, however, is not enough. To tune and administer, it is necessary to know what the application does normally. This means that performance data for the application must be collected regularly, and often, ideally each time the application is run. A certain amount of history should be retained to allow identification of trends and a cumulative average of key performance metrics should be maintained to define what normal is. Remember at this point that normal is normal, not necessarily good.

The collected performance baselines should be maintained in a common store to allow analyzing the full collection for the best tuning opportunities. It may not make sense to tune an application that runs once a month with an elapsed time of three minutes. On the other hand, if one-tenth of a second is cut from the CPU time used by a CICS transaction, it can result in an hour of CPU gained each day if the transaction runs 36,000 times a day. And much can be done with a CPU hour on today’s bigger, faster machines.

Once a target has been identified, it becomes necessary to collect detailed data describing where (which line of code, SQL statement, or SVC call) the application uses the CPU, which files it accesses, for what purpose, how often and when, and the point at which the application has to wait for something. Fortunately, for performance tuners, applications do not generally consume resources evenly over their code. Consumption tends to be spiky and much can be accomplished by knocking down the spikes. Once corrective measures have been defined and implemented, the detailed performance data for the application should be collected again and compared with the starting data to ensure that there actually has been improvement. The process should be repeated until it appears that remaining performance improvements don’t justify the cost of finding and implementing them. Once the process has been completed, the detailed data should be preserved as a new baseline for use in future comparisons.

Tuning, however, is, at best, only a technique. Administering the performance of the application inventory must be an ongoing process if full benefit is to be gained. As much as possible, the process should be automated. It is just remotely conceivable that a small group of people possessed of vast experience, encyclopedic knowledge, and a lot of time could administer the performance of an MVS image. To administer the performance of hundreds, or thousands, of applications that run on that image is not possible. Therefore, the data collection and preliminary analysis that support the performance administration process will have to occur automatically.

The process begins with application development or, at the very least, with application testing and quality assurance. Each application should be measured and reviewed for performance to ensure that it performs as efficiently as possible prior to entering production. When it is cut over to production, its performance baseline (the picture of normal) should travel with it. If the application is new, a complete set of detailed performance data should be collected and saved. If it is a revision, the new set of performance data should be compared against the previous production history to verify that goodness continues. Thereafter, a continuous process of monitoring the application should ensue to allow warning and action when the application diverges by some margin from normal. When alerted by the automatic process that an application is causing trouble, performance tuners would fight fires as they do now. However, most of their efforts would go to tuning those applications identified by the process as the largest resource consumers.

Y2K is, for most shops, the first time in history that all of their operations have simultaneously been under review and revision. When it is over, the world will begin again in a very fundamental way. There will be new purchased code, new home brewed applications, new operating systems, new databases, and new bridges to integrate them. Much of the internally developed code may have been built under pressure, using unfamiliar compilers in the Language Environment, which may also be new to many. Internal code may have been built by outsourced contractors, the quality of whose code is uncertain. Purchased code may have been built in similar ways.

Because of these factors, it is possible that code will be more thoroughly tested than ever before. Certainly, it will be better understood by more people than ever before, if only because more people have had to look at it. If, while everything is disassembled, automated performance tools are also integrated into the testing and quality assurance process and made a part of the production environment as well, it may be possible to fit the bagged groceries into the original cart. The traditional CEO mantra, "faster, better, cheaper (pick two)," may actually produce all three. It would be unfortunate, if it were possible, to put everything back together and go back to spot firefighting and throwing iron at the rest. To be sure, Y2K is not the only good opportunity to take this step. There will be another excellent one in the run-up to the year 10,000. COBOL will undoubtedly still be an issue. But many of us who know COBOL may, by then, be pursuing other career interests. Those who lack that sort of patience ought to seize the present moment.

ABOUT THE AUTHOR: