In-Depth
Y2K Compliance Testing and Verification
With the number of days rapidly diminishing before the millennium, many organizations are faced with the dilemma of what action to take, if any, to minimize the Year 2000 problem. As consulting firms, tool vendors and Year 2000 factory firms greedily request enormous sums of monies, executives are faced with difficult decisions, which require astute, timely answers and a definitized plan of action. There is not much time until judgement day. Organizations ponder whether:
- If the idea of spending a "king's ransom" to insure that their products which work today to be functional tomorrow is a worthwhile investment?
- If the threat of jeopardizing their organization's future by ignoring the problem today will cause insurmountable future legal and technical problems.
The Year 2000 renovation decision will have a major impact on your firm's profit picture and possibly its survivability. There are four steps involved in insuring Year 2000 compliance (see Figure 1). This article discusses an aspect of the second of the four steps, evaluating potential renovation tool sets. It describes what one should consider in ascertaining what is best for their organization. The four steps are:
STEP | TITLE | BRIEF DESCRIPTION |
1 | Assessment | Determining the applications your firm is using which are affected by the Year 2000? What is the estimated cost of renovation (there are rules of thumb relative to Lines of Code, type of application, etc.)? Determining what are the legal ramifications? Determining how long will it take to remediate the problem? Determining how to assess the cost: benefit ratio of replacing or renovating existing applications? |
2 | Environment | Selecting/establishing the management structure, selecting the team, selecting the tools, creating realistic schedules. Creating secondary backup plans, creating a legal plan, providing communication with other (friendly) organizations facing the same problem. Establishing procedures for assuring there is an audit trail of the entire process. |
3 | Implementation | Renovating applications and/or replacing those applications which make economic sense to be Year 2000-compliant. Establishing testing procedures for verifying compliance. |
4 | Verification | Assuring that the renovated applications will actually be functional come January 1, 2000. If outside sources are used, assuring they warrant their products/efforts and assume significant risk where there is significant reward. Providing a comprehensive audit trail of the renovation process. |
Table 1: The Four Steps to Ensure Year 2000 Compliance
This article is going to talk about a specific aspect of Step 2, the selecting of a renovation tool set. In any complex decision-making process, making a quantitative decision (based on hard numerical evidence) as opposed to a qualitative decision (based on intuitive-gut feelings) is almost always preferable.
This article is designed to enhance an executive's (e.g., the Chief Information Officer's) ability to quantitatively recommend the "right" organization to support their renovation efforts. This is probably going to be a key decision for many organizations during the next two years, and making the "right" decision can have a major financial impact on the profitability/survivability of your firm. There are still individuals who believe the Year 2000 problem is "no big deal." These individuals believe a "few" electronic dates coded as two digits represent only a minor problem, and like ostriches sticking their heads in the sand to avoid an imminent danger, they are not cognizant of the future consequences. It is only recently that we begun to recognize how widespread the millennium problem has become.
The millennium problem is casting an "air of doom" over our electronic society. Unless we take appropriate medication rapidly, the forecast will become a reality. In Warren, Michigan, the first Year 2000 lawsuit was filed when a retailer claimed he suffered significant financial setbacks when his system could not accept credit cards expiring after the year "00." Other lawsuits are already occurring daily. Numerous law firms are gearing up to participate in this financial windfall. Nearly every major organization throughout the world needs to assess the business risks associated with the Year 2000 problem. Creating realistic schedules and a plan to alleviate/minimize the problem are now topics of nearly every boardroom. President Clinton announced at the White House Millennium Project that "We can't have the American people looking to a new century and a new millennium with their computers - the very symbol of modernity and the modern age - holding them back, and we're determined to see that it doesn't happen."
Recognizing a problem is the initial step in creating its solution. This article provides a methodology for selecting the appropriate tool set for your environment. The authors' organization has no vested interest in any renovation tool set provider. We have participated and witnessed innumerable renovations. In advising organizations as to which tool set to select, the methodology we use includes a payoff matrix. Its attributes and values can be modified by your organization to support your own unique environment and requirements.
The two authors, throughout their business life, which includes a combine total of over six decades of providing and marketing quality software tools (over four decades were at their own companies) to numerous major organizations, have used the methodology proposed in this article. The initial step is to identify what is important to your organization for a renovation tool set to support. The next step is determining the relative importance of each element. The third step determines how to evaluate each organization. The final step is a set of guidelines for the acquisition/leasing of the software. In acquiring any product, there is always certain features we perceive as being more critical to our decision making process than others. For the Year 2000, the following table describe specific features which should be part of your decision making process. We have arranged the information alphabetically (by title), to simplify its organization.
It is often necessary to adjust the weights (how important is one feature relative to another) significantly based on the renovation methodology and environment being used. For example, if your renovation plans are to employ expansion exclusively, then the importance of windowing and bridging diminish significantly. Likewise, the methodology used to determine the scoring for a specific feature would determine the creditability of your results. The authors' organization uses four different methods for "scoring" a vendor for a specific feature, which are:
- Vendor Questionnaire
- Vendor Documentation
- Client's of Vendor Questionnaire
- Verification of Vendor's Software
Not all vendors' claims are true. In assessing a specific vendor, it is not only necessary to query the vendor and study their documentation, it is required, if you want the most reliable data, that you actually audit critical capabilities by running or auditing the vendors software (see Figure 2).
TITLE | DESCRIPTION |
Audit Trail | What methodology is provided to assure auditable records of the renovation process? |
Bridging | How does the tool support interfacing with other applications? Are bridging routines provided? How about between databases, applications, different languages? |
Code Renovation | Does the tool renovate the code, or do you need to manually perform this task? How efficient is the renovation? How complex is the renovated code to maintain. |
Customer Support | Is there a customer support group, which can help solve renovation issues? How easy are they to get hold of when needed? Is there a help desk? What hours are covered? |
Date Identification | How good a job does the product do in identifying all dates in the application? Does it eliminate what looks like possible dates (false-positives)? How simple is it to find all the dates? How long does it take to identify all the dates? |
Documentation | What is the quality of the documentation? Can answers to most questions be readily found? Is their a meaningful user's guide or training manual? |
Efficiency | How efficient is the renovated code? What kind of degradation can be expected? |
Expansion | Does the tool support date expansion? Can individual variables be expanded? |
External Tools | Does the product require acquisition of external tools? What's the cost? |
Hidden Costs | What additional costs does the vendor charge |
Honesty | Does the vendor say what you want to hear, or does he tell you reality? How realistic is the vendor's schedule? Does the vendor meet his commitments? Are open problems in the product available for your perusal? |
Hot-Line | Does the vendor provide hot-line support? What hours and how competent is the support? Is there an 800 number which can be used around the clock? |
Internet Access | Is the system accessible via the Internet? How are new system upgrades provided? |
Inventory | Does the product determine what parts of the application are missing? How long does it take to determine if the inventory is complete? |
Language | What computer languages does the product support in addition to your initial needs? |
Local Representation | Does the organization provide your firm local representation? |
Mainframe | Is the product mainframe-based? Does it use legacy like features to simplify the learning curve of the staff who is legacy experts? |
Management | Is the organization's management able to support your requirements? |
Network | Does the tool support multiple users remediating an application in parallel? |
PC | Is the product PC-based? Do you need to be a legacy expert? A Windows, UNIX or Mac expert? Does the product provide an intuitive interface? |
Price of Tool | What is the cost per line of renovated code using the tool? |
Procedures | What quality procedures are in place? Is their a comprehensive test suite and procedures to assure timely releases and high quality products (no regressions)? |
Productivity | How many lines of code can your programmers renovate per hour using the tool? |
Robustness | How reliable is the product? Will the vendor provide you a list of open problems? |
References | Does the vendor have reasonable references from firms using their tool? What is the level of satisfaction of their clients with the other topics listed in this table? |
Stability | How stable is the vendor? Will they be in business tomorrow? What is their financial status, current business posture, etc.? |
Testing | What tools/support is provided to simplify actual application testing? Is there any test data generation tools? Are there any debugging tools provided as part of the tool set? |
Training | Is their initial training? Is the product so intuitive that training is not necessary? Is their follow on training? What levels of training are provided? |
Warranty | What warranty does the vendor provide for renovated code? What risk is the tool vendor willing to bear and for what period of time? |
Windowing | Does the system support "Windowing?" What level of code degradation occurs when you use windowing? Can the tool support multiple pivot years? Can the tool support both windowing and expansion within a single application's renovation? |
Table2: Vendor Checklist
When you are buying a product, a salesperson will often ask you how important is that feature, especially when the product he or she is hawking does not provide a "good" solution for the specified feature. Payoff matrices are valuable tools in assuring we make appropriate decisions from complex, varying data. Since a purchase is a decision at a given point in time, you have to recognize that when the data changes, you might very well make another decision given a different set of data. Thus, it behooves your firm to continually update its data whenever your organization is about to make a decision. Table 3 is identical to the previous table, except instead of a description, a weight has been provided. The weight is the relative importance of the feature (you will undoubtedly adjust this relative importance factor to your own firm's requirements) in comparison with the other features (see Figure 3).
TITLE | WEIGHT | | TITLE | WEIGHT |
Audit Trail | 8 | | Local Representation | 3 |
Bridging | 5* | | Mainframe | 4** |
Code Renovation | 7 | | Management | 8 |
Customer Support | 7 | | PC | 4** |
Date Identification | 9 | | Price of Tool | 9 |
Documentation | 6 | | Procedures | 9 |
Efficiency | 4 | | Productivity | 10 |
Expansion | 5 | | Robustness | 9 |
External Tools | 2 | | References | 6 |
Hidden Costs | 4 | | Stability | 7 |
Hot-Line | 6 | | Testing | 4 |
Internet Access | 1 | | Training | 6 |
Inventory | 5 | | Warranty | 9 |
Language | 7 | | Windowing | 8 |
* | Renovation tools use windowing, expansion, or some combination of the two for converting applications to be Year 2000-compliant. If the renovation methodology only uses windowing, then the importance of bridging is significantly lessen. |
** | If the renovation platform is a PC, then the importance of the mainframe diminishes significantly. If the renovation platform is a mainframe, then the importance of PCs is virtually non-existent. Figure 3: Prioritizing the Features |
We now have determined what needs to be evaluated and how important each feature is relative to each other. Now we need to look at some actual test data and determine which product is the "right" one. So, we evaluate potential vendors, by giving each a score from -100 to 10 (where -100 is so horrible that this feature alone makes it impossible to use this vendor and 10 is wonderful). We then multiply the score by the weight and add up the results to get a "Total Score" for this potential vendor. We have done this for several popular vendors and the results are not always what you might intuitively have expected. Sometimes you need to look at more than just the total score when evaluating a potential firm. For example, if a firm has a product which enables your programmers to renovate significantly more lines of code per day than other vendors, yet its procedures are so disastrous that you cannot handle 30 percent of their deliveries having regressions, than you need to use a negative figure for that firms procedure score.
In conclusion, when you prepare to spend a significant sum of your data processing budget, a small, up front investment will pay major dividends in minimizing the actual final cost. It usually requires about 3-5 days to evaluate a firm as a potential vendor. To accomplish this, you need all of the firms technical information (including quality/management procedures), its renovation tools, a completed questionnaire and the availability of the vendor's technical and management staff. The dividends of evaluating vendors and selecting the "right" one will have a major payoff to your future success in renovating your applications to be Year 2000 compliant.
ABOUT THE AUTHOR:Joel Fleiss is President and CEO of Millennium Solutions Inc. (Los Angeles), a full-spectrum consulting company which provides help to business organizations in project management, software renovation and legal dilemmas for organizations facing the Year 2000 problem. He has written and published numerous articles and been an invited speaker at several conferences throughout the United States. Go to Y2K Features