SENG 521 Software Reliability & Testing
description
Transcript of SENG 521 Software Reliability & Testing
![Page 1: SENG 521 Software Reliability & Testing](https://reader036.fdocuments.us/reader036/viewer/2022070408/568143cc550346895db05a7a/html5/thumbnails/1.jpg)
SENG521 (Fall 2002) [email protected] 1
SENG 521SENG 521Software Reliability & Software Reliability & TestingTesting
Cleanroom Development
(Part 12a)Department of Electrical & Computer Engineering, University of Calgary
B.H. Far ( [email protected] )http://www.enel.ucalgary.ca/~far/Lectures/SENG521/12a/
![Page 2: SENG 521 Software Reliability & Testing](https://reader036.fdocuments.us/reader036/viewer/2022070408/568143cc550346895db05a7a/html5/thumbnails/2.jpg)
SENG521 (Fall 2002) [email protected] 2
What is it ? /1What is it ? /1 A software engineering methodology with the emphasis on
design with no defects and test based on statistical quality control.
Analysis and design models are based on incremental software model and created using box structure representation.
A “box” encapsulates the system (or some aspect of the system) at a specific level of abstraction.
Correctness verification is applied once the box structure design is complete.
![Page 3: SENG 521 Software Reliability & Testing](https://reader036.fdocuments.us/reader036/viewer/2022070408/568143cc550346895db05a7a/html5/thumbnails/3.jpg)
SENG521 (Fall 2002) [email protected] 3
What is it ? /2What is it ? /2 The software is tested by defining a set of usage scenarios
(i.e., operational modes), determining the probability of use
for each scenario (i.e., operational profile), and then defining
random tests that conform to the probabilities.
The error records are checked. No corrective actions are
taken. Only certification test is conducted to check whether
errors (i.e., failure intensity) meet the projected reliability
for the software component.
![Page 4: SENG 521 Software Reliability & Testing](https://reader036.fdocuments.us/reader036/viewer/2022070408/568143cc550346895db05a7a/html5/thumbnails/4.jpg)
SENG521 (Fall 2002) [email protected] 4
Cleanroom Process ModelCleanroom Process Model
SESESESE
RGRGRGRGBSSBSSBSSBSS FDFDFDFD CVCVCVCV CGCGCGCG CICICICI
TPTPTPTPSUTSUTSUTSUT CCCC
Increment Increment 11
RGRGRGRGBSSBSSBSSBSS FDFDFDFD CVCVCVCV CGCGCGCG CICICICI
TPTPTPTPSUTSUTSUTSUT CCCC
Increment Increment 22
SE: System engineeringRG: Requirement gatheringBSS: Box structure specificationFD: Formal designCV: Correctness verification
CG: Code GenerationCI: Code inspectionSUT: Statistical use testingC: Certification testTP: Test planning
![Page 5: SENG 521 Software Reliability & Testing](https://reader036.fdocuments.us/reader036/viewer/2022070408/568143cc550346895db05a7a/html5/thumbnails/5.jpg)
SENG521 (Fall 2002) [email protected] 5
Cleanroom Strategy /1Cleanroom Strategy /1 Requirement gathering (RG)Requirement gathering (RG)
A detailed description of customer level requirements for each increment.
Box structure specification (BSS)Box structure specification (BSS) Functional specification using box structure to separate
behavior, data and procedures. Formal design (FD)Formal design (FD)
Specifications (black boxes) are refined to become analogous to architectural (state boxes) and procedural (clear boxes) design.
![Page 6: SENG 521 Software Reliability & Testing](https://reader036.fdocuments.us/reader036/viewer/2022070408/568143cc550346895db05a7a/html5/thumbnails/6.jpg)
SENG521 (Fall 2002) [email protected] 6
Cleanroom Strategy /2Cleanroom Strategy /2 Correctness verification (CV)Correctness verification (CV)
A set of correctness verification activities on the design and moves later to code. First level verification is via application of a set of “correctness questions”.
Code generation, inspection & verification (CG Code generation, inspection & verification (CG & CI)& CI) The box structure transformed to a programming
language. Walkthrough and code inspection techniques are used to ensure semantic conformance with the box structure.
![Page 7: SENG 521 Software Reliability & Testing](https://reader036.fdocuments.us/reader036/viewer/2022070408/568143cc550346895db05a7a/html5/thumbnails/7.jpg)
SENG521 (Fall 2002) [email protected] 7
Cleanroom Strategy /3Cleanroom Strategy /3 Statistical test planning (TP)Statistical test planning (TP)
Planning the test based on operational modes, operational profiles and reliability.
Statistical use testing (SUT)Statistical use testing (SUT) Creating test case, execute them and collecting error data.
Certification (C)Certification (C) Conducting certification test rather than reliability growth
to accept/reject developed software components (using reliability demo chart, etc).
![Page 8: SENG 521 Software Reliability & Testing](https://reader036.fdocuments.us/reader036/viewer/2022070408/568143cc550346895db05a7a/html5/thumbnails/8.jpg)
SENG521 (Fall 2002) [email protected] 8
Box Structure /1Box Structure /1 Black boxBlack box
Specifies the behavior of a system or a part of a system. The system responds to specific stimuli (events) by applying a set of transition rules that map the stimuli to response.
State boxState box Encapsulates state data and services (operations). Input to
the state box and outputs are represented. Clear boxClear box
Transition function that are implied by the state box. It contains the procedural design of the state box.
![Page 9: SENG 521 Software Reliability & Testing](https://reader036.fdocuments.us/reader036/viewer/2022070408/568143cc550346895db05a7a/html5/thumbnails/9.jpg)
SENG521 (Fall 2002) [email protected] 9
Box Structure /2Box Structure /2CBCB1.1.11.1.1CBCB1.1.11.1.1
CBCB1.1.21.1.2CBCB1.1.21.1.2
CBCB1.1.31.1.3CBCB1.1.31.1.3
BBBB11BBBB11
BBBB1.11.1BBBB1.11.1
BBBB1.21.2BBBB1.21.2
BBBB1.n1.nBBBB1.n1.n
BBBB1.1.11.1.1BBBB1.1.11.1.1
BBBB1.1.21.1.2BBBB1.1.21.1.2
BBBB1.1.31.1.3BBBB1.1.31.1.3
SBSB1.1.11.1.1SBSB1.1.11.1.1
Black box
State box
clear box
![Page 10: SENG 521 Software Reliability & Testing](https://reader036.fdocuments.us/reader036/viewer/2022070408/568143cc550346895db05a7a/html5/thumbnails/10.jpg)
SENG521 (Fall 2002) [email protected] 10
Box Structure /3Box Structure /3
f: S* → Rf: S* → RS R
Black box
f: S* → Rf: S* → RS R
State
TState
T
State box
S R
State
TState
T
g11g11
g12g12
g13g13
cg1cg1
clear box
![Page 11: SENG 521 Software Reliability & Testing](https://reader036.fdocuments.us/reader036/viewer/2022070408/568143cc550346895db05a7a/html5/thumbnails/11.jpg)
SENG521 (Fall 2002) [email protected] 11
Design RefinementDesign Refinement If a function f is expanded into a sequence g and h, the
correctness rule for all input to f is: Does g followed by h do f?
If a function f is expanded into a condition if-then-else, the correctness rule for all input to f is:
Whenever condition <c> is true does g do f and whenever <c> is false does h do f?
When function f is refined as a loop, the correctness rule for all input to f is:
Is termination guaranteed? Whenever <c> is true, does g followed by f do f? and whenever <c>
is false, does skipping the loop still do f?
![Page 12: SENG 521 Software Reliability & Testing](https://reader036.fdocuments.us/reader036/viewer/2022070408/568143cc550346895db05a7a/html5/thumbnails/12.jpg)
SENG521 (Fall 2002) [email protected] 12
Advantages of VerificationAdvantages of Verification Design verification has the following advantages:
Verification is reduced to a finite process. Every step of design and every line of code can be
verified. Near zero defect level is achieved. Scalability is possible. Better code than unit testing can be generated.
![Page 13: SENG 521 Software Reliability & Testing](https://reader036.fdocuments.us/reader036/viewer/2022070408/568143cc550346895db05a7a/html5/thumbnails/13.jpg)
SENG521 (Fall 2002) [email protected] 13
Cleanroom TestingCleanroom Testing Using “statistical use testing”. Determine a “usage probability distribution” via
the following steps:1) Analyze the specification to identify a set of stimuli
(direct and indirect input variables).
2) Create usage scenarios (operational modes).
3) Assign probability to use of each stimuli (operational profile).
4) Generate test cases for each stimuli according to the usage probability distribution.
![Page 14: SENG 521 Software Reliability & Testing](https://reader036.fdocuments.us/reader036/viewer/2022070408/568143cc550346895db05a7a/html5/thumbnails/14.jpg)
SENG521 (Fall 2002) [email protected] 14
Certification TestCertification Test Cleanroom approach DOES NOT emphasize on
Unit or integration testing. Bug fixing as a result of test and regression.
Certification procedure involves the followings: Create usage scenarios. Specify a usage profile. Generate test cases from the profile. Execute test cases and record failure data. Compute reliability and certify the component or system
using reliability demo chart, etc.
![Page 15: SENG 521 Software Reliability & Testing](https://reader036.fdocuments.us/reader036/viewer/2022070408/568143cc550346895db05a7a/html5/thumbnails/15.jpg)
SENG521 (Fall 2002) [email protected] 15
Reliability Demo Chart /1Reliability Demo Chart /1 An efficient way of
checking whether the failure intensity objective (λF) is met or not based on collecting failure data at time points.
Vertical axis: failure number
Horizontal axis: normalized failure data, i.e., failure time x λF
![Page 16: SENG 521 Software Reliability & Testing](https://reader036.fdocuments.us/reader036/viewer/2022070408/568143cc550346895db05a7a/html5/thumbnails/16.jpg)
SENG521 (Fall 2002) [email protected] 16
Certification ExampleCertification Example
Failure number
Measure
(million transactions)
Normalized
Measure
(MTTF)
1 0.1875 0.75
2 0.3125 1.25
3 1.25 5
4 failures/million transactions
0.1 Supplier risk
0.1 Consumer risk
2 Discrimination ratio
F
![Page 17: SENG 521 Software Reliability & Testing](https://reader036.fdocuments.us/reader036/viewer/2022070408/568143cc550346895db05a7a/html5/thumbnails/17.jpg)
SENG521 (Fall 2002) [email protected] 17
Conclusions Conclusions Cleanroom approach is a rigorous approach to
software engineering that has emphasis on: Mathematical verification of correctness of design. Certification of software reliability.
Cleanroom approach is yet to become a common practice in software development industry because of emphasis on the above two points.