Software Quality Study Reveals Costs to Fix Software Glitches, Outages, Breaches

A new CAST Report on Application Software Health (CRASH) was released today by CAST, a software analysis and measurement firm. It’s the first I’ve seen that studies the exposure enterprises have to “fix hidden problems that remain in software and result in damaging risks in applications after they are operational” -- what it calls technical debt -- and puts it in monetary terms any boardroom executive can understand. CAST acknowledges that its estimate -- technical debt is $3.61 per line of code -- is conservative. The firm didn’t include costs to fix software so it performs its intended functionality (that is, correct logic problems).

If the survey is representative of IT applications in general, and about 15 percent of applications contain more than a million lines of code as CAST found in its sample, then a single large application may expose the enterprise to more than $3.6 million in technical debt.

That debt is problematic for another reason: it’s often not part of IT’s budget. When an application fix is needed, IT must use funds from another project, disrupting its priorities.

According to Dr. Bill Curtis, CAST’s chief scientist, senior vice president of CAST Research Labs and director of the Consortium for IT Software Quality, “The number of software glitches, outages and security breaches reported in the press this year, and the damage they have done to the reputations of organizations like Toyota, Sony and RIM, not to mention the U.S. Government and a multitude of banks and stock exchanges around the world, have made problems with structural quality in application software a boardroom issue.”

Curtis says ignoring these flaws can be dangerous. “What we found were numerous problems that should have been addressed prior to deployment. It’s little different from ignoring termites that are destroying the structure of your home.”

The study “used automated analysis to measure the structural quality of 365 million lines of code within 745 IT applications used by 160 companies throughout 10 industries.” That’s triple last year’s study, CAST’s first, which examined 288 applications with 108 million lines of code in all.

The study examined five “health factors” in judging an application’s structural soundness: security (how well it prevents unauthorized instructions), performance (application responsiveness), robustness (an application’s stability and the likelihood defects will be introduced during modifications), ease of software transferability (how easy it is for a new team to understand the application and productively work with it), and ease of changeability (how easily and quickly an application can be modified).

There were some surprises -- the survey dispelled some development myths. For example, there was no difference in structural quality between applications that were outsourced and those developed in-house, nor between onshore and development. Structural quality also was lower for applications with a smaller set of users.

Language matters: Java EE applications were the most prevalent applications in the study. Their performance scores were significantly lower and they were more costly to fix than applications written in other languages.

Using an agile methodology -- with its reputation for delivering applications (or changes) quickly -- may also bring better structural quality than custom methods. Applications built using a waterfall methodology had the highest scores for transferability and changeability; scores for those two measures continue to be the lowest in government IT departments.

COBOL applications continue to enjoy their reputation for security, scoring highest in that dimension; .NET applications had the worst security scores.

What best practices can IT follow to reduce this debt? I asked Lev Lesokhin, vice president of marketing at CAST, for recommendations.

“The first step is to recognize the problem and garner management attention,” he explained. “The very next step is to quantify and categorize the technical debt to get a handle on the problem. Most analysts will then tell you that you need to start remediation projects to pay down the debt while modernizing your systems. In our experience, while this is a valuable activity, it’s often more important to put quality gates in place to limit the production of new debt with every release to mission critical systems.”

I also asked Lesokhin if there were any surprises in the results.

“Probably the most counterintuitive finding we had in the data was that applications managed offshore, or that were outsourced, had the same quality levels as internally managed applications. More of the quality variation is explained by QA practices than sourcing method. This is a fact and we need to analyze the data further to understand the sources of variation.”

More information about the CAST survey can be found here.

-- James E. Powell
Editorial Director, ESJ

Posted by Jim Powell on 12/09/2011