Methods for Estimating Defects
description
Transcript of Methods for Estimating Defects
![Page 1: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/1.jpg)
Methods for Estimating Defects
Catherine V. Stringfellow Mathematics and Computer Science Department
New Mexico Highlands University October 20, 2000
![Page 2: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/2.jpg)
Problem and Questions
To improve the efficiency of system testing by helping testers determine when to stop testing and release software.
When should testers stop testing without sacrificing effectiveness?
Can defect estimations be used to make release decisions?
![Page 3: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/3.jpg)
Data
available in defect database release name phase defect reported component name date defect reported and much more
![Page 4: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/4.jpg)
Approach
Estimate number of remaining components with defects using: Capture-Recapture Methods Curve-Fitting Methods Experience-Based Methods
Estimate number of remaining failures using: SRGM Selection Method
Compare estimates to acceptability thresholds to make release decisions
![Page 5: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/5.jpg)
Estimating Components with Defects in Post-Release, but not in Test
Capture/recapture Methods
Curve Fitting Models
Experience-based Estimation Method
(Uses historical data in the estimation)
![Page 6: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/6.jpg)
Capture/Recapture Methods Derived from Wildlife Biological Models Software Defect Estimation Models
Use multiple independent reviewers to count faults (instead of animals)
A reviewer captures a certain fault and other reviewers who identify same fault are said to “recapture it”
![Page 7: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/7.jpg)
Capture/Recapture
x x
x x x
x x x x x x x x x x
x x
xx x
Model requires an overlap if most overlap, few remaining (unless MANY found)
if few overlap, many remaining
![Page 8: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/8.jpg)
Five CRC Estimation Methods M0ML MTML MTChpm (for two reviewers) MHJK MthChao
![Page 9: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/9.jpg)
Models’ AssumptionsModel 1, M0Test Group 1
Test Group 2
Test Group 3
Model 3, MtTest Group 1
Test Group 2
Test Group 3 Detection probabilities
Defect number
Model 2, MhTest Group 1
Test Group 2
Test Group 3 Detection probabilities
Defect number
Model 4, MthTest Group 1
Test Group 2
Test Group 3 Detection probabilities
Defect number
![Page 10: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/10.jpg)
CRC with MtML Example
Sample (CRC with MLE method)n m n1 n2 N >= n L(N)12 2 8 7 12 -6.86
13 -6.5414 -6.4115 -6.3416 -6.32
MAX -> 17 -6.3218 -6.3319 -6.3420 -6.37
Estimated Remaining is 17-12 = 5
Correct remaining = 8
MLE method tends to underestimate
![Page 11: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/11.jpg)
0
2
4
6
8
10
1 2 3 4 5 6 7 8 9 10
Defect #
# d
efe
ct
oc
cu
rre
nc
es
Curve Fitting Models Detection Profile Method (fit with decreasing exp curve)
Cumulative Method (fit with increasing exponential curve)
0
10
20
30
40
1 2 3 4 5 6 7 8 9 10
Defect #
Cu
mu
lati
ve #
de
fec
t o
cc
urr
en
ce
s
Estimated # total defects is smallest x-value with y-value <= 0.5
Estimate is x-value at which expfunction is asymptotic
![Page 12: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/12.jpg)
Sample Datadefect # rev1 rev2 # occurrences Exp Regression
1 1 1 2 1.06762 1 1 2 1.05843 1 1 2 1.03854 1 0 1 1.00005 1 0 1 1.00006 1 0 1 1.00007 1 0 1 1.00008 1 0 1 1.00009 0 1 1 1.0000
10 0 1 1 1.000011 0 1 1 1.0000
n = 12 0 1 1 1.0000
Total n1= 8 n2= 7m = 2 reviewers
![Page 13: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/13.jpg)
Detection Profile Method
y = 1.8196e-0.0654x
0
0.5
1
1.5
2
2.5
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
defect #
# o
ccu
rren
ces
Look for x-value w here y-value becomes 0.5DPM Estimate approximately 20 total, 20-12=8 remainingCorrect Answ er is 8 remaining
DPM Curve Fitting Example
DPM Estimate: approximately 8 remaining
Correct Answer: 8
![Page 14: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/14.jpg)
Results (3 sites)
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
2
m0ml mtml mhjk mthChao cumulative dpm (exp) dpm (linear) exp-based
Estimators (3 sites)
Rel
. Err
or
Release 1 Release 2 Release 3 Mean Abs.
4 2 5 8 7 6 3 1Ranks:
![Page 15: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/15.jpg)
SRGMs
Some Models’ Assumptions Test according to operational profile. Failure rate proportional to number of failures
remaining. Defect repair is immediate. Defect repair is perfect. No new code is introduced during the test
period.
![Page 16: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/16.jpg)
SRGMs in practice Assumptions violated.
Previous studies show SRGMs are robust: SRGMs perform well in predicting failure rates.
![Page 17: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/17.jpg)
SRGMs Exponential Model (Musa’s) Delayed S-shaped Gompertz curve Yamada Exponential Yamada Raleigh
![Page 18: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/18.jpg)
Release 1
![Page 19: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/19.jpg)
Release 2
![Page 20: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/20.jpg)
Release 3
![Page 21: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/21.jpg)
SRGM Selection Method
![Page 22: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/22.jpg)
Integrating Estimation Methods
Logical AND: all methods must say stop Sequential
If mhjk says stop, stop. Else if SRGM selection method and one other
method says stop, stop. Else if at least one of the following methods,
m0ml, mtml, dpm(linear), AND the experienced-based method say stop, stop.
Else continue.
![Page 23: Methods for Estimating Defects](https://reader034.fdocuments.us/reader034/viewer/2022050909/568149fe550346895db72dbb/html5/thumbnails/23.jpg)
Integrating Estimation Methods Majority
Group m0ml, mtml, dpm(linear) together. If one says stop, stop.
In case of tie: a) continue test for another week; or b) compare number of defects in last week to acceptability threshold (5)