Establishing A Defect Prediction Model Using A Combination of Product Metrics As Predictors Via Six...

[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence

Establishing A Defect Prediction Model Using A Combination of

Product Metrics As Predictors Via Six Sigma Methodology

Muhammad Dhiauddin Mohamed Suffian(M.Sc., B.Tech., Six Sigma Green Belt , CTAL-TM, CTFL)

Senior Engineer, MIMOS Test LabApril 8, 2023


( 2 )

Agenda


( 2 )

• Defect Prediction Model – An Overview• Project Background• Six Sigma Overview• Prior Studies on Defect Prediction• Findings & Discussion – Define• Findings & Discussion – Measure• Findings & Discussion – Analyze• Findings & Discussion – Design• Findings & Discussion – Verify• Conclusion & Recommendation


( 3 )

Defect Prediction Model – An Overview


( 3 )

“Build a functional defect prediction model for system testing phase in a form of mathematical equation by applying

Design for Six Sigma methodology”

Zero-KnownPost Release Defect

Defect Containment inTesting Phase

Wider Test Coverage &When to Stop Testing

Stability ofTesting Process

Test Defect Prediction

Model


( 4 )

Test Defect Prediction

Model

ISSUES/SCENARIOS• Resource planning for test

execution• Test coverage to discover

defects• Test execution time to meet

project deadline• Reliability of software

RESEARCH QUESTIONS• How to predict the total

number of defects to be found at the start of system testing phase using a model?• What are the key

contributors?• Factors contributing to

system test defect?• Relationship between

factors of defect with the total number of system defects?

• Defect type & total software defects?

• Prediction model & test process improvement?

• Type of data & how to get them?

WHAT HAS BEEN DEMONSTRATED?

• Establish a defect prediction model for software testing phase

• Demonstrate the approach in building a defect

• prediction model using Design for Six Sigma (DfSS) Methodology

• Identify the significant factors that contribute to a reliable defect prediction model

• Determine the importance of defect prediction model for improving testing process

Project Background


( 5 )

DEFINE

Identify opportunityDevelop team charter & teamProject schedulingFinancial estimationVOC analysis

MEASURE

Identify functional requirementsMeasurement system analysisPerform capability analysisGenerate concept model

ANALYZE

Quantify issues & determine significant factorsIdentify failure modes

DESIGN

Re-define significant factorsOptimize model

VERIFY

Assess reliability of selected designControl planClose project

Six Sigma Overview

A disciplined, data-driven approach and methodology for eliminating defects (driving toward six standard deviations between the mean and the nearest specification limit) in any process -- from manufacturing to transactional and from product to service

(www.isixsigma.com)

D-M-A-I-C D-M-A-D-V (DfSS)vs


( 6 )

Prior Studies on Defect Prediction

• D = 4.86 + 0.018L (Akiyama)

• D = V/3000 (Halstead)• D = 4.2 + 0.0015(L)4/3

(Gaffney)

Size/Complexity Metrics

TestingMetrics

Process Quality Data

Bayesian Belief Networks (BBN)

Product & Process Metrics


( 7 )

Findings & Discussion - Define

Requirements Design Developmen

t Testing Deploy

RequirementReview

Test Plan

DesignReview

Test CasesTest Scripts

Test ExecutionDefects Raising

Test Summary Report

Kick-Off

Software Development Life Cycle (SDLC) – Test Participation


( 8 )

Findings & Discussion – Define (cont.)

Zero-Known Post Release Defects

Defect Containment in Testing Phase

Potential # of

defects before test

Actual # of defects

after test

Customer Satisfaction

Level ofsatisfaction

Quality ofProcess

Peoplecapability

Processeffectivenes

s

ProjectManagement

Timelineallocation

Resourceallocation

High Level Schematic Diagram


( 9 )

Findings & Discussion – Define (cont.)Test Defect Prediction

Software Complexity

Requirement Pages

Design Pages

Programming

Language

Code Size

Knowledge

Developer Knowledg

e

Tester Knowledg

e

Test Process

Test Case Design

Coverage

Targeted Total Test

Cases

Test Automati

on

Test Case Execution Productivi

ty

Total Effort in

Test Design Phase

Total Effort in Phases Prior to System

Test

Errors

Requirement Error

Design Error

CUT Error

Test Plan Error

Test Cases Error

Fault

Requirement Fault

Design Fault

CUT Fault

Integration Fault

Test Case Fault

Historical Defect

DefectSeverity

Defect Type/

Category

DefectValidity

Total PR (Defects) Raised

Project

Project

Domain

Project

ThreadComponent

Application

Factors to consider

Detail Schematic Diagram – Y to X Tree

(Y)


( 10 )

Findings & Discussion – Measure

TC TC ID Actual TC Result

Tester 1

Tester 2

Tester 3

1 2 3 1 2 3 1 2 3

1 TC1 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS

2 TC2 FAIL FAIL FAIL FAIL FAIL FAIL FAIL PASS PASS PASS

3 TC3 FAIL FAIL FAIL FAIL FAIL FAIL FAIL PASS PASS PASS

4 TC4 PASS FAIL FAIL FAIL FAIL FAIL FAIL PASS PASS PASS




8 TC8 FAIL FAIL FAIL FAIL FAIL FAIL FAIL FAIL FAIL FAIL



Data for Attribute Agreement Analysis


( 11 )

Findings & Discussion – Measure (cont.)

• Result is PASS for MSA Within Appraisers as the result shows 100% assessment agreement• Kappa = 1, which shows perfect agreement exist It demonstrates strong Repeatability of TestResult within tester himself/herself

• 3 Persons (Vivek, Sandeep and Kamala) PASS the MSA based on Kappa value of 0.7 or 70%• For the All Testers, the result is acceptable for accuracy of assessment against standard as the Kappa value is greater than 0.7 or 70%

• Overall Result for MSA is PASS since Kappa value is greater than 0.7 or 70%

Result of Attribute Agreement Analysis


( 12 )

Findings & Discussion – Analyze

ProjectName

Req.Error

Design Error

CUTerror KLOC

TotalTest

Cases

TestPlanError

TestCase Error

Automation%

TestEffort

Test Execution

ProductivityFunctional

Defects

PROJECT A 5 22 12 28.8 224 0 34 0 6.38 45.80 19

PROJECT B 0 0 1 6.8 17 0 6 0 9.36 17.00 1

PROJECT C 9 10 14 5.4 24 4 6 0 29.16 5.83 4

PROJECT D 7 12 2 1.1 25 4 9 0 13.17 7.00 0

PROJECT E 11 29 3 1.2 28 4 12 0 14.26 3.40 3

PROJECT F 0 2 7 6.8 66 1 7 0 32.64 31.00 16

PROJECT G 3 25 11 4.0 149 5 0 0 7.15 74.50 3

PROJECT H 4 9 2 0.2 24 4 0 0 18.78 7.67 0

PROJECT I 7 0 1 1.8 16 1 3 0 9.29 2.68 1

PROJECT J 1 7 2 2.1 20 1 4 0 6.73 1.95 0

PROJECT K 17 0 3 1.4 13 1 4 0 8.44 6.50 1

PROJECT L 3 0 0 1.3 20 1 7 0 14.18 9.75 1

PROJECT M 2 3 16 2.5 7 1 6 0 8.44 1.75 0

Data for Regression Analysis


( 13 )

Findings & Discussion – Analyze (cont.)

• P-value for selected individual predictors shows significant relationship with the Y (defects) which has value of less than 0.05

• R-sq and R-sq shows strong value which is more than 90%

Best Multiple Regression Result


( 14 )

Findings & Discussion – Design• Revise the predictors to generate prediction model that is logical and makes sense from the viewpoint of software practitioner• Filter the metrics to contain only valid data in order to generate the model• Reduced the model to have only coefficients that have logical correlation to the response (Defect)• Revisit the model to identify suitable response of the regression equation: Functional Defects or All Defects

PROJECT REQ .ERROR

DESIGNERROR

CUT ERROR

KLOC REQ. PAGE

DESIGN PAGE

TOTAL TEST

CASES

TEST CASES ERROR

TOTAL EFFORT

TEST DESIGN EFFORT

FUNCTIONAL DEFECTS

ALL DEFECTS

PROJECT A 5 22 12 28.8 81 121 224 34 16.79 15.20 19 19PROJECT B 0 0 1 6.8 171 14 17 6 45.69 40.91 1 1PROJECT C 9 10 14 5.4 23 42 24 6 13.44 13.44 4 4PROJECT D 7 12 2 1.1 23 42 25 9 4.90 4.90 0 0PROJECT E 11 29 3 1.2 23 54 28 12 4.72 4.59 3 3PROJECT F 0 2 7 6.8 20 70 66 7 32.69 16.00 16 27PROJECT G 3 25 11 4.0 38 131 149 0 64.00 53.50 3 3PROJECT H 4 9 2 0.2 26 26 24 0 5.63 5.63 0 0PROJECT I 17 0 3 1.4 15 28 13 4 9.13 7.88 1 1PROJECT J 61 34 24 36 57 156 306 16 89.42 76.16 25 28PROJECT K 32 16 19 12.3 162 384 142 0 7.00 7.00 12 12PROJECT L 0 2 3 3.8 35 33 40 3 8.86 8.86 6 6PROJECT M 15 18 10 26.1 88 211 151 22 30.99 28.61 39 57PROJECT N 0 4 0 24.2 102 11 157 0 41.13 28.13 20 33

Revised Data for Regression Analysis


( 15 )

Findings & Discussion – Design (cont.)

YFunctional Defects

XsRequirement Error,

CUT Error, KLOC, Requirement Pages,

Design Pages, Targeted Total Test Cases, Total Effort

Days

YAll Defects




Days

YFunctional Defects




Days in Test Design

YAll Defects




Days in Test Design


( 16 )

Findings & Discussion – Verify• Verification was done by applying the selected model on new/fresh project that have yet to go for System Test• The actual defects found after System Test completed are compared against the predicted defects• Verify that actual defects fall between 95% prediction interval of the model

Y Effort Predictors

Project Predicted Defects

Actual Defects 95% CI(min, max)

95% PI(min, max)

Functional Defect

All Tester Effort Prior to System Test

Agrimall Beta Release 182 187 (155, 210) (155, 210)

Grid Workflow 6 1 (0, 12) (0, 14)

TTS-English 1 1 (0, 3) (0, 6)

All DefectAll Tester

Effort Prior to System Test


Grid Workflow 9 9 (0, 21) (0, 24)

TTS-English 2 1 (0, 6) (0, 12)

Functional Defect

All Tester Effort in Test

Design Prior to System Test


Grid Workflow 8 1 (0, 17) (0, 19)

TTS-English 2 1 (0, 5) (0, 9)

All DefectAll Tester

Effort in Test Design Prior to

System Test


Grid Workflow 11 9 (0, 32) (0, 37)

TTS-English 3 1 (0, 10) (0, 19)

Revised Data for Regression Analysis


( 17 )

Findings & Discussion – Verify (cont.)

No Req. Error CUT Error KLOC Req. Page Des. Page Total TC Effort Days

1. 12 6 84.03 75 293 102 69.75

2. 49 15 8.69 64 38 65 91.9

3. 0 1 0.4 10 10 18 1.13

No Predicted Functional Defects Actual Functional Defects 95% CI (min, max) 95% PI (min, max)

1. 182 187 (155, 209) (154, 209)

2. 6 1 (0, 12) (0, 14)

3. 1 1 (0, 3) (0, 6)

Verify against new project:

Prediction result *:

* The actual functional defects were raised after both functional and ad-hoc testing completed

PROPOSED MODELFunctional Defects (Y) = 4.00 - 0.204 Req. Error - 0.631 CUT Error + 1.90 KLOC - 0.140 Req. Page + 0.125 Design Page - 0.169 Total Test Cases + 0.221 Effort Days


( 18 )

Findings & Discussion – Verify (cont.)


( 19 )

Conclusion & Recommendation

CONCLUSION•

Defect prediction model for testing phase can be constructed using identified factors via multiple regression

• Significant factors contributed to defects found in testing phase have been discovered

• Six Sigma can be used to develop test defect prediction model

• Test defect prediction model does contribute to zero-known post release defects and test process improvement

RECOMMENDATION• T

o consider other factors that contribute in predicting defects

• To improve model by being able to predict defect based on different severity

• To predict defects to be found against time

• To predict defects based on different type of software

THANK YOU


[email protected]

Experience our Testing Excellence in

“End-to-End Testing Strategies” Workshop

See you again on 17th June 2010, 2.00 pm, Room 301

Establishing A Defect Prediction Model Using A Combination of Product Metrics As Predictors Via Six...

Education

Transcript of Establishing A Defect Prediction Model Using A Combination of Product Metrics As Predictors Via Six...