In-Depth

The Perils of Modifying Software

A study of two companies shows that modified code can increase defects three-fold. It takes disciplined inspection and continuous software-development process improvements to increase software quality.

• By Rix Groenboom
• 03/09/2004

It has long been speculated that it is more difficult to modify software code than to create new code. Developers are often asked to change code they did not originally write, or may have written months (or even years) before. Lack of consistently enforced documentation standards also adds to the difficulty of modifying source code. Add to this mix how most organizations look at aggregate defect density rates when determining a product's market-readiness, and you can understand why it is difficult to determine the exact impact modifications have on code quality.

Surprisingly, there is little evidence available to determine whether developers who are asked to modify code end up introducing more defects per line then when initially creating the code. Reasoning, Inc. embarked on a study to determine if, in fact, there is evidence to support this speculation about the perils of modifying software.

A key software reliability measurement indicator is defect density, defined as the number of defects found per thousand lines of source code (KLOC). While over time the defect density for most applications declines (and may approach zero), the defect density of code that is newly introduced or changed remains constant at close to three times that of the original application.

Since March 2000, Reasoning has inspected hundreds of applications representing over 200 million lines of source code. Reasoning’s business is focused on improving software quality by locating crash-causing and security-related defects for companies that develop Java, C, and C++ applications.

The following case studies look at defect density per KLOC rates and demonstrate the need for inspection of modified software code across various stages of the development lifecycle.

Findings

Case Study A (see Figure 1) presents an embedded software company with a code base of 250,000 lines. Over a period of one year, this code was inspected for software defects multiple times. While the defect density for the entire code base went from 0.65 defects per KLOC to 0.08 defects per KLOC, the defect density for the modified code fluctuated at a rate almost three times as high as the original baseline, with a final average of 1.75 defects per KLOC.

Figure 1:

Case Study A - Defects per KLOC
Inspection:	Baseline	#1	#2	#3	#4	#5	#6
Entire Application	0.65	0.06	0.01	0.09	0.02	0.07	0.08
Modified Code	-	2.06	1.88	3.57	0.25	0.82	1.91

Case Study B (see Figure 2) is a system software company with a code base of two million lines. Over a similar one-year period of inspections and modifications, the defect density for the code base went from 0.37 defects per KLOC to 0.09 defects per KLOC. During this same period, the defect density for the modified code grew steadily to a defect density more then three times as high as the baseline—to 1.35 defects per KLOC.

Figure 2:

Case Study B - Defects per KLOC
Inspection:	Baseline	#1	#2	#3
Entire Application	0.37	0.21	0.22	0.09
Modified Code	-	0.48	0.71	1.35

Companies often measure the quality of their code at the aggregate level, and in doing so tend to develop a false sense of quality attainment. While company A saw a 50 percent improvement at the aggregate level, the quality within the updated components was stuck at three times the error rate of the original code. Company B thought they had increased software quality by 76 percent, but with the analysis provided learned that the code quality of what was being updated was stagnant at four times the original error rate.

Recommendations

What these studies support is that modifying code is a difficult and error-prone process. Companies need to further focus their resources on the effectiveness of modifying software code and the challenges it presents. Such improvements come from software inspection and software testing. Additional means of improving software quality can come from incorporating defined development processes such as CMM, as well as training and performance-based disciplines like TSP and PSP.

Each of these models introduces repeatable process and the measurement of process improvement into the development lifecycle. At their core, these represent approaches to developing better software and, therefore, can be positive steps in improving software quality.

While process is necessary, it is not sufficient to achieve quality. Practices such as software inspection and rigorous code review, coupled with thorough Q&A, will improve developers' ability to produce quality modifications to existing code bases. Companies should be sensitive and build in adequate timeframes for rigorous testing and QA procedures when assigning their developers to the particularly challenging task of modifying existing code.

Only through disciplined inspection and continuous software development process improvements can companies measure and therefore improve software quality.