Back to blogs

Software Defect Prevention – in a nutshell

Feb 07, 2019

Purushotham Narayana

 

Organizations face many problems that impede rapid development of software systems critical to their operations and growth. The challenge in any software product development lies in minimizing the number of defects. Occurrence of defects is the greatest contributor to significant increases in product costs due to correction and rework time. Most defects are caused by process failures rather than human failures. Identifying and correcting process defects will prevent many product defects from recurring.

This article will present various tools and techniques for use in creating a Defect Prevention (DP) strategy that, when introduced at all stages of a Software life cycle, can reduce the time and resources necessary to develop high quality systems. Specifically, how implementing a model-based strategy to reduce Requirement Defects, Development Rework and Manual test development efforts will lead to significant achievements in cost reduction and total productivity.

 Defect Prevention

Defect Prevention (DP) is a strategy applied to the software development life cycle that identifies root causes of defects and prevents them from recurring. It is the essence of Total Quality Management (TQM). DP, identified by the Software Engineering Institute as a level 5 Key Process Area (KPA) in the Capability Maturity Model (CMM), involves analyzing defects encountered in the past and specifying checkpoints and actions to prevent the occurrence of similar defects in the future. In general, DP activities are a mechanism

for propagating the knowledge of lessons learned between projects.

Mature IT organizations have an established software process to carry out their responsibilities. This process is enhanced when DP methodologies are implemented to improve quality and productivity and reduce development costs. Figure 1 clearly depicts that identifying defects late in the game is costly.

Figure 1: Software Defect Rate Of Discovery Versus Time

A model for an enhanced Software Process, including a DP strategy, is presented in Figure 2. Adopting a DP methodology will allow the organization to provide its clients with a product that is “of High Quality and Bug Free.”

Figure 2: Defect Prevention Strategy for Software Development Process

On a macro level defects can be classified and filtered as depicted in Figure 3.

Figure 3: Filter or Whirlpool Diagram for Software Defects

Features of Defect Prevention

 

Management must be committed to following a written policy for defect prevention at both the organization and project level. The policy should contain long-term plans for funding, resources and the implementation of DP activities across the organization including within management, to improve software processes and products. Once in place, a review of results provides identification of effective activities and lessons learned to further improve the organization’s success in applying a DP strategy.

To assist in the successful implementation of a DP strategy, members of the software-engineering group and other software-related groups should receive training to perform their DP activities. Training should include software quality assurance, configuration management and document support and focus on DP and statistical methods (e.g., cause/effect diagrams and Pareto analysis).

Creation of an Action Plan plays a key role in the implementation process. At the beginning of a software task, the members of the team meet to prepare for the task and related DP activities. A kick-off meeting is held to familiarize members of the team with details of the implementation process. Included in the meeting is information related to the software process, standards, procedures, methods, and tools applicable to the task, with an emphasis on recent changes; inputs required and available for the task; expected outputs; and methods for evaluation of outputs and of adherence to the software process. A list of common errors and recommended preventive actions are also introduced along with team assignments, a task schedule and project goals.

Periodic reviews are conducted by each of the teams assigned to coordinate DP activities. During the reviews, action items are identified and priorities set based on a causal analysis that determines:

  • the causes of defects,
  • the implications of not addressing the defects,
  • the cost to implement process improvements to prevent the defects, and
  • the expected impact on software quality.

 

A pareto analysis is helpful in setting priorities and provides direction for assignment of action items or reassignment to other teams, making changes to activities and documenting rationale for decisions.

Case Study

A case study of a real time scenario is discussed below along with statistics derived from the analysis.

The Reference Line

 As a first step, a “Defect Analysis of Past Projects” was performed to create a reference line for the PIE. As many as 1,336 defects were analyzed from the base line project (TETRA Released) and two other projects to increase statistical significance. A detailed Root Cause Analysis was performed on all defects and the Beizer [3] Taxonomy was used as the “classification vehicle”. Analysis was done for five development phases, namely: Requirement Specifications, Architectural Design, Detailed Design, Coding and System Test Case Preparation. Based on this analysis, specific Defect Prevention (DP) solutions were determined for each of the phases.

The Beizer Taxonomy included ten major categories, each of which was divided into three levels, resulting in a 4-digit number which specifies unique defects. The ten top level categories were:

0xxx Planning
1xxx Requirements and Features
2xxx Functionality as Implemented
3xxx Structural Bugs
4xxx Data
5xxx Implementation
6xxx Integration
7xxx Real-Time and Operating System
8xxx Test Definition or Execution Bugs
9xxx Other

The causes of the defects as determined by the engineers doing the classification, fell into four major categories: Communication, Education, Oversight and Transcription.

In creating the reference line, detailed interviews with 24 software engineers took place. The interviews allowed a full understanding of the reason for each defect, classification of the cause and an understanding of defect prevention activities. This data mining was performed on all defects, resulting in a series of classification tables and a Pareto analysis of the most common problems. The results of the pareto analysis according to the Beizer Taxonomy top level categories are presented below with the breakdown in descending order.

  • Requirements and Features (1xxx) 47.0%
  • Functionality as Implemented (2xxx) 13.5%
  • Structural Bugs (3xxx) 9.3%
  • Implementation (5xxx) 8.3%
  • Data (4xxx) 6.9%
  • Integration (6xxx) 5.7%
  • Real time and Operating system (7xxx) 4.9%
  • Test definition or Execution bug (8xxx) 4.3%

Within each development phase in the baseline project, the defects were further classified based on the Beizer Taxonomy. For example, in the Requirement Specifications Phase, the second level breakdown of the main defects occurred as follows:

  • Requirement Completeness (13xx) 37.5%
  • Requirement Presentation (15xx) 34.7%
  • Requirement Changes (16xx) 11.2%
  • Requirement Incorrect (11xx) 8.7%

The third level breakdown of the main Requirement Completeness defect was:

  • Incomplete Requirements (131x) 73.4%
  • Missing, unspecified requirements (132x) 11.2%
  • Overly generalized requirements (134x) 4.6%

The same type of data analysis was performed for each development phase selected for the PIE. The next step was to identify a tool-set of phase-specific improvement activities, based on the root cause analysis, that would prevent defects from recurring in the next release. Highest priority was given to the most common defect types. Extensive training and phase kickoff meetings were held to empower the development team to integrate DP activities into the existing process. The development team then applied the improvement activities determined in the analysis phase to the development phases, and ongoing defect recordings and measurements were performed.

The final step was to compare the numbers and types of TETRA Release 2 defects with those of the reference line. The effectiveness of the prevention tool-set was measured in the quantity and types of defects found in the second release of the project. The effective prevention actions could then be integrated into the OSSP to improve quality and cycle time for all the projects in MCIL. The impact on the OSSP, including changes to Review Guidelines and changes to the Phase Kickoffs, are considered part of the PIE results.

Results and Conclusion

 As a result of the project, the overall number of defects in Tetra Release 2 has decreased by 60% as compared to the number of defects detected in TETRA Release 1 (the reference line project). In part, this is attributed to the fact that Release 2 is a continuation project and not an initial project as was Release 1, and that later releases usually have less defects due to more cohesive teams, greater familiarity with the application domain, experience, and fewer undefined issues. Based on numbers from other MCIL projects, we estimate that half of the defect decrease can be attributed to the implementation of the PIE. A breakdown of defects by phase of origin shows the following results.

Table 1: Breakdown of Defects by Phase

The absolute reduction in defects, which relates to the % Improvement shown in the above table, can be observed in the following figure.

Figure 4: Reduction of Defects by Phase

The obvious observation is that a higher percentage of the defects migrated to later phases of the development process: from Requirement Specifications, Preliminary Design and Detailed Design, to Coding. In Tetra Release 1, 76.5% of the defects are in the Requirement and Design phases and only 23.4% are in Coding, while in Tetra Release 2, 45.5% of the defects are in Requirement and Design and 54.5% are in Coding. This implies that the DP methods employed in the early phases of development were very effective.

The % Improvement column shows the improvement within each development phase with respect to the absolute number of defects. This is a different view of the improvement in the number of defects, partially attributable to the Improvement Actions.

Another comparison was made in respect to the Cause category with the following results.

You can read the entire article here – https://www.isixsigma.com/tools-templates/software-defect-prevention-nutshell/

 

References

 [1] R.G. Mays, C.L. Jones, G.J. Holloway, D.P. Studinski, Experiences with Defect Prevention, IBM Systems Journal, Vol 29, No. 1, 1990.
[2] Watts S. Humphrey, Managing the Software Process, Chapter 17 – Defect Prevention, ISBN-0-201-18095-2.
[3] Beizer Boris., Software Testing Techniques, Second edition, 1990, ISBN-0-442-20672-0.