Software Defect PreventionUsing Orthogonal Defect Classification

Twin-SPIN

January 6, 2005

Presented by:

Megan Graham, ASQ CSQE

megan@metagraham.net

2

Overview

• Defect prevention is a process used to improve software quality

• Orthogonal Defect Classification is a tool that characterizes defect data used in defect analysis

3

Agenda

• About Defect Prevention

• ODC Concepts

• Applying ODC to the DP Process

• Summary

4

Agenda

• About Defect Prevention

• ODC Concepts

• Applying ODC to the DP Process

• Summary

5

Defining Defect Prevention

• What does it mean to you?• Defect prevention is a process whose

purpose is to:– identify the common causes of defects, and– change the relevant process(es) to prevent that

type of defect from recurring. (SEI)

• Take what we already know and apply it to what we think we know to produce quality software.

6

A Bug’s LifeRequirementsRequirements

DesignDesign

CodeCode

TestTest

7

A Bird’s Eye View of DP

DecreaseDecreaseDefectsDefectsInjectedInjected

IncreasIncreasee

DefectsDefectsDetecteDetecte

dd

DefectDefectPreventionPrevention

HistoricalHistoricalProjectProjectDefectsDefects

CurrentCurrentProjectProjectDefectsDefects

8

Applying Defect Prevention

• A defect in the software is also a defect in the process (injection and/or detection)

• For DP to work, we need to turn software defects into actionable process defects

Softwaredefects

Processdefects

DP

9

And then there was ODC…

• Orthogonal Defect Classification

• Developed at IBM in the 1990s by Ram Chillarege

• Methodology to characterize software defects and translate into process defects

10

Agenda

• About Defect Prevention

• ODC Concepts

• Applying ODC to the DP Process

• Summary

11

Simple ODC Classification Scheme

DefectDefect

TypeTypeTypeType

TriggerTriggerTriggerTrigger

12

Defect Types• FUNCTION: Affects significant capability, end-user interfaces, product

interfaces, interface with hardware architecture or global data structure(s).

• LOGIC: Affects the functionality of a code module and can be fixed by re-implementing an algorithm or local data structure without a need for requesting a high level design change.

• INTERFACE: Affects the interaction of components via macros, call statements and/or parameters.

• CHECKING: Affects program logic that would properly validate data and values before they are stored or used in computation.

• ASSIGNMENT: Requires change in a few lines of code, such as initialization of control blocks or data structures.

• TIMING/SERIALIZATION: Affects the proper management of shared and real-time resources.

• BUILD/PACKAGE/MERGE: Affects product version or configuration; requires a formal changes to reconfigure or rebuild the product.

13

Defect Triggers

Fault Failure

Trigger 1

Trigger 2

Trigger n

14

Review/Inspection Triggers• DESIGN CONFORMANCE: Comparing the implemented design against a

reference –design document, pattern, or guideline.• LOGIC/FLOW: Checking for correctness or flaws using knowledge of the

practice.• BACKWARD COMPATIBILITY: Examining compatibility with prior version of

the product.• LATERAL COMPATIBILITY: Examining for compatibility with other products

and platforms that need to work with this release.• CONCURRENCY: Serialization, shared resources, multi-threaded tasks, timing,

etc.• INTERNAL DOCUMENT: Inconsistencies in prologs, and sections in the same

work product.• LANGUAGE DEPENDENCY: Programming standards, specific implementation

considerations, environment restrictions, execution modes, etc.• SIDE EFFECTS: Usage behavior beyond design, but relevant in context. Do A; B

happens.• RARE SITUATION: Unusual issues related to idiosyncrasy of environment,

hardware, or software.

15

Function Test Triggers• SIMPLE PATH: White box – Straightforward usage of code path and

branches.

• COMPLEX PATH: White box – Exercising conditionals, and circuitous coverage.

• COVERAGE: Black box – Straightforward usage of function or single parameterized execution.

• VARIATION: Black box – Straightforward like coverage, but with a variety of parameters.

• SEQUENCING: Black box – Multiple functions, with a specific sequence (order is important).

• INTERACTION: Black box – When two or more bodies of code are involved (order is not important).

16

System Test Triggers• WORKLOAD STRESS: Pushing the limits of performance, resources, users,

queues, traffic, etc.

• RECOVERY: Invoke exception handling, recovery, termination, error percolation, etc.

• STARTUP/RESTART: Major events of turning on, off, or changing the degree of service availability.

• HARDWARE CONFIGURATION: Issues surfaced as a consequence of changes in hardware setup.

• SOFTWARE CONFIGURATION: Issues surfaced as a consequence of changes in software setup.

• BLOCKED TEST/NORMAL MODE: Test could not run during System Test, or customer found nonspecific trigger. Look for additional trigger.

17

Opener

Closer

ODC v5.11 Classification Scheme

Source: ODC v5.11, IBM Center for Software Engineering, www.research.ibm.com/softeng

DefectDefect

TypeType SourceSource

ImpactImpactTriggerTrigger

ActivityActivity

TargetTarget

QualifierQualifier

AgeAge

18

Agenda

• About Defect Prevention

• ODC Concepts

• Applying ODC to the DP Process

• Summary

19

When to Apply DP

• DP can be applied to one or more phases of the software lifecycle

• Depends on maturity, goals, etc.

RQTS DSGN CODE TEST

Defect Prevention

Field

Ana

lysi

s Action

Ana

lysi

s Action

Ana

lysi

s Action

Ana

lysi

s Action

Ana

lysi

s Action

20

Defect Analysis

Orthogonal Defect Orthogonal Defect ClassificationClassification

What are the attributes of the defects?

Defect CausalDefect CausalAnalysisAnalysis

When are defectsbeing injected?

Defect TriggerDefect TriggerAnalysisAnalysis

How are defectsbeing detected?

ODC provides a structure for the defect data

21

Defect Causal Analysis

• Purpose: Characterize process issues that lead to injection of defects.

Step 1: ID a set of defects.

Step 2: Identify Defect Types with team of experts.

Step 3: Plot the distribution of Defect Types.

Step 4: Map Defect Types back to development activity.

Step 5: Develop action plan to address process deficiencies.

Step 6: Monitor process to ensure changes were effective.

22

Defect Type Mapped to Phase

H/L Level DesignInterface

Configuration MgmtBuild/Package/Merge

ImplementationTiming/Serialization

ImplementationAssignment

LLD/ImplementationChecking

Low Level DesignLogic

Architecture/HLDFunction

23

Sample Distribution

0%

20%

40%

60%

80%

100%

Inter

face

B/P/M

Logic

Checki

ng

Assign

men

t

Timin

g/Ser

ial

Functio

n

24

Defect Trigger Analysis

• Purpose: Characterize process issues that allowed defects to escape to later phases.

Step 1: ID a set of defects.

Step 2: Identify Defect Trigger with team of experts.

Step 3: Plot distributions: Defect Trigger by Family, Review Triggers, Function Test Triggers, System Test Triggers.

Step 4: Map Defect Trigger family back to detection activity(ies).

Step 5: Develop action plan to increase missed Defect Triggers.

Step 6: Monitor process to ensure changes were effective.

25

Defect Trigger Family Mapped to Phase

Testing in real-world scenarios or extreme scenarios

System Test

Testing of specified scenariosFunction Test

Peer Reviews or InspectionsReview/Inspection

26

Sample Distributions

Defect Triggers by Family

0

10

20

30

40

50

60

Review/Inspection Function Test System Test

Function Test Triggers

02468

1012141618

Seq

uenc

ing

Var

iatio

n

Inte

ract

ion

Cov

erag

e

Sim

ple

Pat

h

Com

plex

Pat

h

27

Additional Use for DTA

• Purpose: Determine quality status.

Step 1: Plot distribution of Defect Triggers after each detection activity.

Step 2: Compare to historical defect profile to determine if profile is as expected.

Step 3: Develop action plan, if necessary, to get back on track.

• If your project thinks it’s one phase, but the distribution is of an earlier phase, then you’re really in the earlier phase!

28

Agenda

• About Defect Prevention

• ODC Concepts

• Applying ODC to the DP Process

• Summary

29

The Big Picture: DP & ODC

• Mission: Improve software quality by using readily available data to decrease defects injected and increase defects detected.

Defect Prevention

RQTS DSGN CODE TEST Field

Ana

lysi

s Action

Ana

lysi

s Action

Ana

lysi

s Action

Ana

lysi

s Action

Ana

lysi

s Action

ODC Classification of Software Defects

30

Important Points about ODC

• A defect in the software is a defect in the process.• Implementing ODC is very cost-effective

– Enhances data already collected (software defects)– Adding fields that are completed real-time make data

collection virtually free!– Tooled to quickly identify process defects (mapping)

• ODC can be implemented in stages– Start with field defects, then move to in-process analysis– Utilize defect profiling in-process to predict quality and

project status

• Fields can be tailored to your own organization

31

References

• Handbook of Software Reliability Engineering (Michael R. Lyu, Editor; IEEE Computer Society Press/McGraw-Hill)

• Ram Chillarege (www.chillarege.com)• IBM Research Center for Software Engineering (

www.research.ibm.com/softeng)• Soheil Khajenoori, SIAGroup (

soheil@siagroup.us)