Extension of E(Θ) metric for Evaluation of Reliability
description
Transcript of Extension of E(Θ) metric for Evaluation of Reliability
![Page 1: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/1.jpg)
Extension of E(Θ) Extension of E(Θ) metric for Evaluation metric for Evaluation
of Reliabilityof Reliability
![Page 2: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/2.jpg)
PresentationPresentation
►Background on Reliability and E(Θ) Background on Reliability and E(Θ) ►Testing MethodologiesTesting Methodologies►Present formulae for calculating E(Θ)Present formulae for calculating E(Θ)►Need of a new formulaNeed of a new formula►Formula DerivationFormula Derivation►Validation of Derivation TechniqueValidation of Derivation Technique►Summary and Future WorkSummary and Future Work
![Page 3: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/3.jpg)
E(Θ)E(Θ) ? ?
►E(Θ)E(Θ) - Reliability of a software is - Reliability of a software is defined in terms of its E(Θ) value.defined in terms of its E(Θ) value.
►E(Θ) refers to the probability of the E(Θ) refers to the probability of the software encountering a failure software encountering a failure subsequent to testing subsequent to testing
![Page 4: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/4.jpg)
Why measure reliabilityWhy measure reliability
► Cost of Designing a Faulty S/W in less Cost of Designing a Faulty S/W in less
time >> Cost of Designing a Reliable time >> Cost of Designing a Reliable S/WS/W
► Safety Critical ApplicationsSafety Critical Applications
![Page 5: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/5.jpg)
Faults or BugsFaults or Bugs
► Failure – Unexpected O/PFailure – Unexpected O/P
► Faults – I/Ps causing FailuresFaults – I/Ps causing Failures
![Page 6: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/6.jpg)
How to measure reliabilityHow to measure reliability
Testing:Testing:
►Means of Probing S/W for bugsMeans of Probing S/W for bugs
►Means of gaining confidence about its Means of gaining confidence about its reliabilityreliability
Source : Source : Evaluating Testing Methods by Delivered Reliability -Phyllis G. Frankl, Richard G. Hamlet, Member, IEEE, Bev Littlewood, Member, IEEE,and Lorenzo Strigini, Member, IEEE
![Page 7: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/7.jpg)
Testing MethodsTesting Methods
►Operational – Operational –
Test Case = Statistical Test Case = Statistical Distribution or Distribution or Random (I/P domain)Random (I/P domain)
►Debug –Debug –
Test Case = Selective Sampling Test Case = Selective Sampling (I/P (I/P Domain)Domain)
![Page 8: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/8.jpg)
Operational Vs. DebugOperational Vs. Debug
►Debug – Probes for bugs actively Debug – Probes for bugs actively (assumes S/W is faulty)(assumes S/W is faulty)
►Operational – Waits for Failures to Operational – Waits for Failures to occur.occur.
![Page 9: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/9.jpg)
Sub-Domain TestingSub-Domain Testing
► A debug testing technique.A debug testing technique.
► Concentrates test-cases to specific sub-Concentrates test-cases to specific sub-domains with high failure probabilitydomains with high failure probability
► Based on Debugger’s IntuitionBased on Debugger’s Intuition
► Eg. Boundary Testing – Bugs lie in the Eg. Boundary Testing – Bugs lie in the cornerscorners
![Page 10: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/10.jpg)
ExampleExample
0- ve x
+ ve x
- ve y
+ ve y
Failure Region
Input domain
Failure Region
![Page 11: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/11.jpg)
Sub-Domain Testing So Sub-Domain Testing So Far…Far…
![Page 12: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/12.jpg)
Terminologies usedTerminologies used
►q = failure probabilityq = failure probability►d = detection rated = detection rate►T = Number of test cases per sub-T = Number of test cases per sub-
domaindomain►n = n number of sub-domains/failure n = n number of sub-domains/failure
regions regions
![Page 13: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/13.jpg)
Single Domain - Single Failure Single Domain - Single Failure RegionRegion
A, 20Failure Region
B, 30Domain nature:
Single Domain = 1 – 50 (50 points)
Failure Region A = 21 – 40 (20 points)
E(Θ) = q( 1 - d)E(Θ) = q( 1 - d)TT
![Page 14: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/14.jpg)
Single Failure Region – Multiple Single Failure Region – Multiple Sub-DomainsSub-Domains
Ai,5 Aj, 15 Ak,10
B, 20 D, 25Domain nature:
Sub-Domain 1= 1 – 25 (25 pt.s) Sub-Domain 2= 25 – 50 (50 pt.s) Sub-Domain 3= 50 – 85 (35 pt.s) Failure Region A = 40-70 (30 pt.s)
E(Θ) = q E(Θ) = q nnΠΠi=1i=1 ( 1 - ( 1 - ddii))T T
C, 10
![Page 15: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/15.jpg)
Multiple Failure Regions – Single Multiple Failure Regions – Single Sub-DomainSub-Domain
A, 10
B, 20
D, 55Domain nature:
Domain = 1 – 100 (100 pt.s)
Failure Region A = 20-30 (20 pt.s)Failure Region B = 50-70 (20 pt.s)Failure Region C = 80- 95(20 pt.s)E(Θ) = E(Θ) = nnΣΣi=1 i=1 q qii( 1 - d( 1 - dii))T T
C, 15
![Page 16: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/16.jpg)
Multiple Failure Regions – Multiple Failure Regions – Multiple Sub-DomainsMultiple Sub-Domains
A, 20
B, 20
C, 30 D, 30Domain nature:
Sub-Domain 1= 1 – 50 (50 pt.s) Sub-Domain 2= 50 – 100 (50 pt.s) Failure Region A = 30-50 (10 pt.s)Failure Region A = 70-90 (20 pt.s)E(Θ) = ?E(Θ) = ?
No definitive Formula. Empirical Testing only SolutionNo definitive Formula. Empirical Testing only Solution
![Page 17: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/17.jpg)
And Now……And Now……
![Page 18: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/18.jpg)
New Set of TerminologiesNew Set of Terminologies
►q = Total failure probabilityq = Total failure probability►ddAA = Detection rate of a failure region = Detection rate of a failure region
in a in a sub-domain sub-domain►FFii = Number of points in a failure = Number of points in a failure
regionregion►M = Total number of failure points in M = Total number of failure points in
the the sub-domain sub-domain
![Page 19: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/19.jpg)
Multiple Failure Regions – Multiple Failure Regions – Multiple Sub-DomainsMultiple Sub-Domains
A, 20
B, 20
C, 30 D, 30Domain nature:
Sub-Domain 1= 1 – 50 (50 pt.s) Sub-Domain 2= 50 – 100 (50 pt.s) Failure Region A = 20-40 (20 pt.s)Failure Region A = 70-90 (20 pt.s)
E(Θ) = q – ( dE(Θ) = q – ( dAAFF11/M+ d/M+ dBBFF22/M)/M)
![Page 20: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/20.jpg)
Formula DerivationFormula Derivation
A, 20 B, 20
D, 30 Fault Case Regions Detected Faults Remaining
00 None A, B
01 B A
10 A B
11 A, B None
E (Θ) = (Probability that the particular case will occur) * (probability that the rest of the faults are undetected)
C, 30
![Page 21: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/21.jpg)
Contd.Contd.Assuming 1 test-case per sub-domainAssuming 1 test-case per sub-domain
Test Case 1 Test Case 2 Fault A Fault B Case Number
Region D Region C undetected
Region D
Region A
Region A
Region B
Region C
Region B
undetected
undetected detected
detected undetected
detected detected
Seq. 1
Seq. 2
Seq. 3
Seq. 4
E (Θ) = E(Θ) (seq1) + E(Θ) (seq2) + E(Θ) seq3) + E(Θ) seq4) contd.
![Page 22: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/22.jpg)
E (Θ) = E(Θ) (seq1) + E(Θ) (seq2) + E(Θ) seq3) + E(Θ) seq4)
= {(1- dA)* (1- dB)* (q)} + {(1-dA)* (dB)* (q –F1/M)} + {(dA)* (1- dB)* (q –F2/M)} + {(dA)* (dB)* (q –F1/M – F2/M)}
= {(30/50 * 30/50) * 40/100} + {(30/50 * 20/50) * 20/100} + {(20/50 * 30/50) * 20/100} + {(20/50 * 20/50) * 0/100}
= 0.144 + 0.048 + 0.048 = 0.24
![Page 23: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/23.jpg)
Verification of the TechniqueVerification of the Technique
A, 20B, 20
C, 60Domain nature:
Domain = 1 – 100 (100 pt.s)
Failure Region A = 20-40 (20 pt.s)Failure Region A = 70-90 (20 pt.s)
Approach I:Approach I:E(Θ) = E(Θ) = nnΣΣi=1i=1 q qii( 1 - d( 1 - dii))TT
= 20/100(1- 20/100) + 20/100(1- 20/100)= 20/100(1- 20/100) + 20/100(1- 20/100) = 0.32
Approach II: E(Θ)E(Θ) = {(1- dA)* (1- dB)* (q)} + {(1-dA)* (dB)* (q –F1/M)} + {(dA)* (1- dB)* (q –F2/M)} + {(dA)* (dB)* (q –F1/M – F2/M)} contd.
![Page 24: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/24.jpg)
= {(80/100 * 80/100)(40/100)} + {(80/100 * 20/100)(20/100)}+ {(20/100 * 80/100)(20/100)} + {(20/100 * 20/100)(0/100)}
= 0.32
Both approaches yield the same result
Similar tests done for different scenarios validate the 2nd technique of calculating E(Θ)E(Θ) with respect to Frankl’s Method
![Page 25: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/25.jpg)
Getting back to Formula Getting back to Formula DerivationDerivation
Upon further solving the method just described we get ….
![Page 26: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/26.jpg)
Method ReductionMethod Reduction
E (Θ) = { (1- dA)* (1- dB)* (q) } + { (1-dA)* (dB)* (q –F1/M) } + { (dA)*(1- dB)* (q –F2/M) } +
{ (dA)* (dB)* (q –F1/M – F2/M) }
= q [ {1–dB – dA + dAdB} + { (dB – dAdB) (1–F2/M*q) } + { (dA – dAdB) (1-F1/M*q) } + { dAdB (1–F1/M*q – F2/M*q) } ] Contd.
![Page 27: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/27.jpg)
= q [ {1–dB –dA + dAdB} + { dB – dB*F2/M*q – dAdB + dAdBF2/M*q}+
{ dA – dAF1/M*q – dAdB + dAdBF1/M*q} + { dAdB–dAdB*F1/M*q – dAdBF2/M*q}
]
= q [ 1– dBF2/M*q – dAF1/M*q]
= q – ( dAF1/M + dBF2/M )
![Page 28: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/28.jpg)
Three Failure Regions &Three Failure Regions & Three Three Sub-domainsSub-domains
E (Θ) = {(1- dA)* (1- dB)* (1- dC)*(q)} + {(1-dA)* (1-dB)* (dC)*(q –F3/M)} +
{(1-dA)* (dB)* (1-dC)*(q –F2/M)} + {(1-dA)* (dB)* (dC)*(q –F2/M–F3/M)} + {(dA)* (1-dB)* (1-dC)*(q –F1/M)} +
{(dA)* (1-dB)* (dC)*(q –F1/M–F3/M )} + {(dA)* (dB)* (1-dC)*(q –F1/M–F2/M)} + {(dA)* (dB)* (dC)*(q –F1/M–F2/M–F3/M)}
= q[ 1 – dCF3/ M*q – dBF2/M*q – dAF1/M*q]
= q – { dAF1/M + dBF2/M + dCF3/M}q – { dAF1/M + dBF2/M + dCF3/M}
![Page 29: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/29.jpg)
SimulatorSimulator
►Based on the above mentioned & Based on the above mentioned & verified technique a simulator was verified technique a simulator was designed for empirical calculation of designed for empirical calculation of E (Θ)
►Coding Language : JavaCoding Language : Java
►Graphical User Interface : Java SwingsGraphical User Interface : Java Swings
![Page 30: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/30.jpg)
Simulator screen shotsSimulator screen shots Step I:
Specify the total number of Sub-Domains & Failure Regions
![Page 31: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/31.jpg)
Step II:Step II:
A, 20 B, 20
C, 30D, 30
Specify the number of points per Sub-Domain, Failure Specify the number of points per Sub-Domain, Failure RegionRegion
![Page 32: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/32.jpg)
The result got from running the simulator is similar to the one calculated theoretically (above). This proves validity of the Simulator Design.
![Page 33: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/33.jpg)
SummarySummary
►Value of E (Θ) for software with multiple failure regions spread across multiple sub-domains can be calculated using: E (Θ) = q – ( dAF1/M + dBF2/M)
E (Θ) = q – ( dAF1/M + dBF2/M + q – ( dAF1/M + dBF2/M +
dCF3/M)dCF3/M)
![Page 34: Extension of E(Θ) metric for Evaluation of Reliability](https://reader035.fdocuments.us/reader035/viewer/2022062807/5681506d550346895dbe68c3/html5/thumbnails/34.jpg)
Future WorkFuture Work
►The number of test-cases per sub-The number of test-cases per sub-domain was limited to only 1domain was limited to only 1
►We would like to generalize the present We would like to generalize the present formula further by considering the formula further by considering the effect of ‘n’ (multiple) test cases per effect of ‘n’ (multiple) test cases per sub-domain.sub-domain.
►No generalization doesn’t merely No generalization doesn’t merely involve putting ‘T’ as a exponential involve putting ‘T’ as a exponential term in the equation ( have tried that )term in the equation ( have tried that )