SENG521 (Fall 2002) far@enel.ucalgary.ca 1

SENG 521SENG 521Software Reliability & Software Reliability & TestingTesting

Cleanroom Development

(Part 12a)Department of Electrical & Computer Engineering, University of Calgary

B.H. Far （ far@enel.ucalgary.ca ）http://www.enel.ucalgary.ca/~far/Lectures/SENG521/12a/

SENG521 (Fall 2002) far@enel.ucalgary.ca 2

What is it ? /1What is it ? /1 A software engineering methodology with the emphasis on

design with no defects and test based on statistical quality control.

Analysis and design models are based on incremental software model and created using box structure representation.

A “box” encapsulates the system (or some aspect of the system) at a specific level of abstraction.

Correctness verification is applied once the box structure design is complete.

SENG521 (Fall 2002) far@enel.ucalgary.ca 3

What is it ? /2What is it ? /2 The software is tested by defining a set of usage scenarios

(i.e., operational modes), determining the probability of use

for each scenario (i.e., operational profile), and then defining

random tests that conform to the probabilities.

The error records are checked. No corrective actions are

taken. Only certification test is conducted to check whether

errors (i.e., failure intensity) meet the projected reliability

for the software component.

SENG521 (Fall 2002) far@enel.ucalgary.ca 4

Cleanroom Process ModelCleanroom Process Model

SESESESE

RGRGRGRGBSSBSSBSSBSS FDFDFDFD CVCVCVCV CGCGCGCG CICICICI

TPTPTPTPSUTSUTSUTSUT CCCC

Increment Increment 11

RGRGRGRGBSSBSSBSSBSS FDFDFDFD CVCVCVCV CGCGCGCG CICICICI

TPTPTPTPSUTSUTSUTSUT CCCC

Increment Increment 22

SE: System engineeringRG: Requirement gatheringBSS: Box structure specificationFD: Formal designCV: Correctness verification

CG: Code GenerationCI: Code inspectionSUT: Statistical use testingC: Certification testTP: Test planning

SENG521 (Fall 2002) far@enel.ucalgary.ca 5

Cleanroom Strategy /1Cleanroom Strategy /1 Requirement gathering (RG)Requirement gathering (RG)

A detailed description of customer level requirements for each increment.

Box structure specification (BSS)Box structure specification (BSS) Functional specification using box structure to separate

behavior, data and procedures. Formal design (FD)Formal design (FD)

Specifications (black boxes) are refined to become analogous to architectural (state boxes) and procedural (clear boxes) design.

SENG521 (Fall 2002) far@enel.ucalgary.ca 6

Cleanroom Strategy /2Cleanroom Strategy /2 Correctness verification (CV)Correctness verification (CV)

A set of correctness verification activities on the design and moves later to code. First level verification is via application of a set of “correctness questions”.

Code generation, inspection & verification (CG Code generation, inspection & verification (CG & CI)& CI) The box structure transformed to a programming

language. Walkthrough and code inspection techniques are used to ensure semantic conformance with the box structure.

SENG521 (Fall 2002) far@enel.ucalgary.ca 7

Cleanroom Strategy /3Cleanroom Strategy /3 Statistical test planning (TP)Statistical test planning (TP)

Planning the test based on operational modes, operational profiles and reliability.

Statistical use testing (SUT)Statistical use testing (SUT) Creating test case, execute them and collecting error data.

Certification (C)Certification (C) Conducting certification test rather than reliability growth

to accept/reject developed software components (using reliability demo chart, etc).

SENG521 (Fall 2002) far@enel.ucalgary.ca 8

Box Structure /1Box Structure /1 Black boxBlack box

Specifies the behavior of a system or a part of a system. The system responds to specific stimuli (events) by applying a set of transition rules that map the stimuli to response.

State boxState box Encapsulates state data and services (operations). Input to

the state box and outputs are represented. Clear boxClear box

Transition function that are implied by the state box. It contains the procedural design of the state box.

SENG521 (Fall 2002) far@enel.ucalgary.ca 9

Box Structure /2Box Structure /2CBCB1.1.11.1.1CBCB1.1.11.1.1

CBCB1.1.21.1.2CBCB1.1.21.1.2

CBCB1.1.31.1.3CBCB1.1.31.1.3

BBBB11BBBB11

BBBB1.11.1BBBB1.11.1

BBBB1.21.2BBBB1.21.2

BBBB1.n1.nBBBB1.n1.n

BBBB1.1.11.1.1BBBB1.1.11.1.1

BBBB1.1.21.1.2BBBB1.1.21.1.2

BBBB1.1.31.1.3BBBB1.1.31.1.3

SBSB1.1.11.1.1SBSB1.1.11.1.1

Black box

State box

clear box

SENG521 (Fall 2002) far@enel.ucalgary.ca 10

Box Structure /3Box Structure /3

f: S* → Rf: S* → RS R

Black box

f: S* → Rf: S* → RS R

State

TState

T

State box

S R

State

TState

T

g11g11

g12g12

g13g13

cg1cg1

clear box

SENG521 (Fall 2002) far@enel.ucalgary.ca 11

Design RefinementDesign Refinement If a function f is expanded into a sequence g and h, the

correctness rule for all input to f is: Does g followed by h do f?

If a function f is expanded into a condition if-then-else, the correctness rule for all input to f is:

Whenever condition <c> is true does g do f and whenever <c> is false does h do f?

When function f is refined as a loop, the correctness rule for all input to f is:

Is termination guaranteed? Whenever <c> is true, does g followed by f do f? and whenever <c>

is false, does skipping the loop still do f?

SENG521 (Fall 2002) far@enel.ucalgary.ca 12

Advantages of VerificationAdvantages of Verification Design verification has the following advantages:

Verification is reduced to a finite process. Every step of design and every line of code can be

verified. Near zero defect level is achieved. Scalability is possible. Better code than unit testing can be generated.

SENG521 (Fall 2002) far@enel.ucalgary.ca 13

Cleanroom TestingCleanroom Testing Using “statistical use testing”. Determine a “usage probability distribution” via

the following steps:1) Analyze the specification to identify a set of stimuli

(direct and indirect input variables).

2) Create usage scenarios (operational modes).

3) Assign probability to use of each stimuli (operational profile).

4) Generate test cases for each stimuli according to the usage probability distribution.

SENG521 (Fall 2002) far@enel.ucalgary.ca 14

Certification TestCertification Test Cleanroom approach DOES NOT emphasize on

Unit or integration testing. Bug fixing as a result of test and regression.

Certification procedure involves the followings: Create usage scenarios. Specify a usage profile. Generate test cases from the profile. Execute test cases and record failure data. Compute reliability and certify the component or system

using reliability demo chart, etc.

SENG521 (Fall 2002) far@enel.ucalgary.ca 15

Reliability Demo Chart /1Reliability Demo Chart /1 An efficient way of

checking whether the failure intensity objective (λF) is met or not based on collecting failure data at time points.

Vertical axis: failure number

Horizontal axis: normalized failure data, i.e., failure time x λF

SENG521 (Fall 2002) far@enel.ucalgary.ca 16

Certification ExampleCertification Example

Failure number

Measure

(million transactions)

Normalized

Measure

(MTTF)

1 0.1875 0.75

2 0.3125 1.25

3 1.25 5

4 failures/million transactions

0.1 Supplier risk

0.1 Consumer risk

2 Discrimination ratio

F

SENG521 (Fall 2002) far@enel.ucalgary.ca 17

Conclusions Conclusions Cleanroom approach is a rigorous approach to

software engineering that has emphasis on: Mathematical verification of correctness of design. Certification of software reliability.

Cleanroom approach is yet to become a common practice in software development industry because of emphasis on the above two points.