White Box Testing (Introduction to)

Università degli Studi dell’Aquila

L20: White Box/Structural Testing

Henry Muccini

DISIM, University of L’Aquila

www.henrymuccini.com, [email protected]

The material in these slides may be freely reproduced and distributed, partially or totally, as far as an explicitreference or acknowledge to the material author ispreserved.

Partially based on material from Alex Orso, Mauro Pezze’, Michal Young, and Andreas Zeller

Recap

Fault, Error, Failure chain

White Box Testing

Statement Coverage

Branch Coverage

AGENDA

Branch Coverage

Basic Condition Coverage

MC/CD Coverage

Recap

�“Systematic” testing vs. “ad hoc”testing

→Repeatable, Measurable

�Software Testing Process

→Test Selection, Test Execution, Oracle, Adequacy

�Type of testing

→Unit, Integration, System

→Performance, Stress

→Regression

→WB, BB

Fundamental Testing Questions

Test Case:How is test described / recorded?

Test Criteria: What should we test?

Test Oracle: Is the test correct?

Test Adequacy: Test Adequacy: How much is enough?

Test Process: Is testing complete and effective?

Test Driver: used to execute the system on the SUT

Stubs: to reproduce missing objects, for integration testing

Fault, Error, Failure

Fault: anomaly in source code

→ may or may not produce a failure

→ a “bug”

Error: inappropriate development action

→ action introduced by a fault→ action introduced by a fault

→ Cause of a fault

Failure: incorrect/unexpected behavior or output

→ incident is the symptoms revealed by execution

→ failures are usually classified

THE FAULT-FAILURE MODEL

Fault

must be activated

Error

sw internal stateis corrupted(latent or manifest)

Failure

affects thedelivered service

must be activated

is corrupted(latent or manifest)

A fault will manifest itself as a failure if all of the 3 following conditions hold:

1) EXECUTION: the faulty piece of code is executed

2) INFECTION: the internal state of the program is corrupted (error state)

3) PROPAGATION: the error propagates to program output

PIE MODEL (Voas, IEEE TSE Aug. 1992)

WHITE BOX/STRUCTURAL TESTING

How to test this

Java program?

Selection of test suite is based on some elements in the code

Assumption: Executing the faulty “element” is a necessary condition for revealing a fault Source

Code

(Test) Inputs

There exist several examples

→ Control flow (statement, branch, basis path, path)

→ Condition (simple, multiple)

→ Loop

→ Dataflow (all-uses, all-du-paths)

→ Fault based (mutation)

Code

Output

Internal behavior

Idea:

→ A program is a graph

→ A graph can be traversed, by using some inputs

→ A test suite is adequate if I guarantee certain

coveragecoverage

─ Metric: percentage of coverage achieved

Objective: Cover the software structure

Control Flow

→ The partial order of statement execution, as defined by the

semantics of the language

Data Flow

→ The flow of values from definitions of a variable to its uses→ The flow of values from definitions of a variable to its uses

Graph representation of control flow and

data flow relationships

1234567

function P return INTEGER isbegin

X, Y: INTEGER;READ(X); READ(Y);while (X > 10) loop

X := X – 10;exit when X = 10;7

89

101112131415

exit when X = 10;end loop;if (Y < 20 and then X mod 2 = 0) then

Y := Y + 20;else

Y := Y – 20;end if;return 2*X + Y;

end P;

6 10

T T

7

TF

2,3,4 5 9´

12

14

T T

F

F 9 T

F

T

9a 9b

printSum(int a, int b) {

int result = a + b;

if (result > 0)

Test this case

if (result > 0)

printcol(“red”, result);

else if (result < 0)

printcol(“blue”, result);

[else do nothing]

}

and this one

and this one!

Defined in terms of

→ test requirements

Result in

→ test specifications→ test specifications

→ test cases

Selection VS adequacy/evaluation criteria

test specifications


int result = a + b;

if (result > 0)

a + b > 0

if (result > 0)




[else do nothing]

}

a + b < 0

a + b == 0

test cases


int result = a + b;

if (result > 0)

a == 3

b == 9

a == -5if (result > 0)




[else do nothing]

}

a == -5

b == -8

a == 0

b == 0

Test requirements: Statements in program

Cstmts = (number of executed statements)

(number of statements)


int result = a + b;

if (result > 0)if (result > 0)




}

Coverage: 0%


int result = a + b;

if (result > 0)

a == 3

b == 9

if (result > 0)




}

Coverage: 71%


int result = a + b;

if (result > 0)

a == 3

b == 9

a == -5

b == -8

if (result > 0)




}

Coverage: 100%

Test hypothesis:

→By executing a path once, potential faults related to it will

be revealed (independently from the input)

Ideally:Ideally:

→To exhaustively cover all possible paths along the program

graph representation

Approximations to path coverage

→Coverage criteria

Only about 1/3 of NASA statements were executed under test before software was released (Stucki 1973)

Microsoft reports 80-90% statement coverage

Boeing must get 100% statement coverage (feasible) for all softwarefor all software

Usually can about 85% coverage; 100% is harder

→ Unreachable code; dead code

→ Complex sequences

→ Not enough resources


int result = a + b;

if (result > 0)

a == 3

b == 9

a == -5

b == -8

if (result > 0)




}

Coverage: 100%


int result = a + b;

if (result > 0)

a == 3

b == 9

a == -5

b == -8

if (result > 0)




[missing branch]

}

Coverage: 100%

Test requirements: Branches in the program

Cbranches = (number of traversed branches)

(number of branches)


int result = a + b;

if (result > 0)

a == 3

b == 9

a == -5

b == -8

if (result > 0)




[missing branch]

}

Coverage: ?


int result = a + b;

if (result > 0)

a == 3

b == 9

a == -5

b == -8

if (result > 0)




[missing branch]

}

Coverage: 88%


int result = a + b;

if (result > 0)

a == 3

b == 9

a == -5

b == -8

a == 0

b == 0

if (result > 0)




[missing branch]

}

Coverage: 100%

1. void main() {

2. float x, y;

3. read(x);

4. read(y);

33

entryentryentryentry

44

X=0 ||!(X=0

• Consider test cases {(x=5,y=5), (x=5, y=-5)}

4. read(y);

5. if(x==0)||(y>0)

6. y = y/x;

7. else x = y+2;

8. write(x);

9. write(y);

10.}

77

88

exitexitexitexit

X=0 ||

y>0

99

66

!(X=0

||y>0)

1. void main() {

2. float x, y;

3. read(x);

4. read(y);

33


44

X=0 ||!(X=0

• Consider test cases {(x=5,y=5), (x=5, y=-5)}

• The test suite is adequate for branch coverage, but does not reveal the fault at statement 6

4. read(y);

5. if(x==0)||(y>0)

6. y = y/x;

7. else x = y+2;

8. write(x);

9. write(y);

10.}

77

88

exitexitexitexit

X=0 ||

y>0

99

66

!(X=0

||y>0)at statement 6

• Predicate 5 can be true or false operating on only one condition

⇓

Basic condition coverageBasic condition coverage

Test requirements: Truth values assumed by

basic conditions

Cbc = (number of boolean values assumed by all basic conditions)

(number of boolean values of all basic conditions)(number of boolean values of all basic conditions)

x y

T T

F F

1. void main() {

2. float x, y;

3. read(x);

4. read(y);

33


44

X=0 ||!(X=0

• Consider test cases {(x=0,y=-5), (x=5, y=5)}

• The test suite is adequate for basic condition coverage

x y4. read(y);

5. if(x==0)||(y>0)

6. y = y/x;

7. else x = y+2;

8. write(x);

9. write(y);

10.}

77

88

exitexitexitexit

X=0 ||

y>0

99

66

!(X=0

||y>0)

• but is not adequate for branch coverage.

⇓Branch and condition Branch and condition

coveragecoverage

x y

T T

F F

Test requirements: Branches and truth values

assumed by basic conditions

if ( ( a || b ) && c ) { … }

a b c Outcome

T T T T

F F F F

Key idea: Test important combinations of

conditions, avoiding exponential blowup

A combination is “important” if each basic

condition is shown to independently affect the

outcome of each decision

MC/DC criterion: For each basic condition C, there are

two test cases in which the truth values of all two test cases in which the truth values of all

conditions except C are the same, and the compound

condition as a whole evaluates to True for one of those

test cases and False for the other

( a && b && c )Test case a b c outcome

1 True True True True

2 True True False False

3 True False True False3 True False True False

4 True False False False

5 False True True False

6 False True False False

7 False False True False

8 False False False False




3 True False True False

MC/DC criterion: For

each basic condition

C, there are two test

cases in which the

truth values of all

conditions except C 3 True False True False






conditions except C

are the same, and the

compound condition

as a whole evaluates

to True for one of

those test cases and

False for the other

Tradeoff between number of required test

cases and thoroughness of the test

Required by both US and European quality

standards in aviation

Path coverage: cover each path in the program

Loop coverage (given n): cover all paths that contain at

most n iterations of any loop in the program

Data-flow coverage: cover data-flow relations in the

program (du pairs)program (du pairs)

Mutation coverage: cover (kill) mutated versions of the

program

�Most widely used as adequacy criteria because fully

automatable!

�Not all graph paths correspond to actual program paths

→some paths could be not executable

→Conditions may not be satisfiable due to interdependent →Conditions may not be satisfiable due to interdependent

conditions

→Paths may not be executable due to interdependent decisions

�The testing output is strongly related to the testing

inputs

White Box Testing (Introduction to)

Education

Transcript of White Box Testing (Introduction to)