Post on 09-May-2015
description
Università degli Studi dell’Aquila
L20: White Box/Structural Testing
Henry Muccini
DISIM, University of L’Aquila
www.henrymuccini.com, henry.muccini@univaq.it
The material in these slides may be freely reproduced and distributed, partially or totally, as far as an explicitreference or acknowledge to the material author ispreserved.
Partially based on material from Alex Orso, Mauro Pezze’, Michal Young, and Andreas Zeller
Recap
Fault, Error, Failure chain
White Box Testing
Statement Coverage
Branch Coverage
AGENDA
Branch Coverage
Basic Condition Coverage
MC/CD Coverage
Recap
�“Systematic” testing vs. “ad hoc”testing
→Repeatable, Measurable
�Software Testing Process
→Test Selection, Test Execution, Oracle, Adequacy
�Type of testing
→Unit, Integration, System
→Performance, Stress
→Regression
→WB, BB
Fundamental Testing Questions
Test Case:How is test described / recorded?
Test Criteria: What should we test?
Test Oracle: Is the test correct?
Test Adequacy: Test Adequacy: How much is enough?
Test Process: Is testing complete and effective?
Test Driver: used to execute the system on the SUT
Stubs: to reproduce missing objects, for integration testing
Fault, Error, Failure
Fault: anomaly in source code
→ may or may not produce a failure
→ a “bug”
Error: inappropriate development action
→ action introduced by a fault→ action introduced by a fault
→ Cause of a fault
Failure: incorrect/unexpected behavior or output
→ incident is the symptoms revealed by execution
→ failures are usually classified
THE FAULT-FAILURE MODEL
Fault
must be activated
Error
sw internal stateis corrupted(latent or manifest)
Failure
affects thedelivered service
must be activated
is corrupted(latent or manifest)
A fault will manifest itself as a failure if all of the 3 following conditions hold:
1) EXECUTION: the faulty piece of code is executed
2) INFECTION: the internal state of the program is corrupted (error state)
3) PROPAGATION: the error propagates to program output
PIE MODEL (Voas, IEEE TSE Aug. 1992)
WHITE BOX/STRUCTURAL TESTING
How to test this
Java program?
Selection of test suite is based on some elements in the code
Assumption: Executing the faulty “element” is a necessary condition for revealing a fault Source
Code
(Test) Inputs
There exist several examples
→ Control flow (statement, branch, basis path, path)
→ Condition (simple, multiple)
→ Loop
→ Dataflow (all-uses, all-du-paths)
→ Fault based (mutation)
Code
Output
Internal behavior
Idea:
→ A program is a graph
→ A graph can be traversed, by using some inputs
→ A test suite is adequate if I guarantee certain
coveragecoverage
─ Metric: percentage of coverage achieved
Objective: Cover the software structure
Control Flow
→ The partial order of statement execution, as defined by the
semantics of the language
Data Flow
→ The flow of values from definitions of a variable to its uses→ The flow of values from definitions of a variable to its uses
Graph representation of control flow and
data flow relationships
1234567
function P return INTEGER isbegin
X, Y: INTEGER;READ(X); READ(Y);while (X > 10) loop
X := X – 10;exit when X = 10;7
89
101112131415
exit when X = 10;end loop;if (Y < 20 and then X mod 2 = 0) then
Y := Y + 20;else
Y := Y – 20;end if;return 2*X + Y;
end P;
6 10
T T
7
TF
2,3,4 5 9´
12
14
T T
F
F 9 T
F
T
9a 9b
printSum(int a, int b) {
int result = a + b;
if (result > 0)
Test this case
if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
[else do nothing]
}
and this one
and this one!
Defined in terms of
→ test requirements
Result in
→ test specifications→ test specifications
→ test cases
Selection VS adequacy/evaluation criteria
test specifications
printSum(int a, int b) {
int result = a + b;
if (result > 0)
a + b > 0
if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
[else do nothing]
}
a + b < 0
a + b == 0
test cases
printSum(int a, int b) {
int result = a + b;
if (result > 0)
a == 3
b == 9
a == -5if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
[else do nothing]
}
a == -5
b == -8
a == 0
b == 0
Test requirements: Statements in program
Cstmts = (number of executed statements)
(number of statements)
printSum(int a, int b) {
int result = a + b;
if (result > 0)if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
}
Coverage: 0%
printSum(int a, int b) {
int result = a + b;
if (result > 0)
a == 3
b == 9
if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
}
Coverage: 71%
printSum(int a, int b) {
int result = a + b;
if (result > 0)
a == 3
b == 9
a == -5
b == -8
if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
}
Coverage: 100%
Test hypothesis:
→By executing a path once, potential faults related to it will
be revealed (independently from the input)
Ideally:Ideally:
→To exhaustively cover all possible paths along the program
graph representation
Approximations to path coverage
→Coverage criteria
Only about 1/3 of NASA statements were executed under test before software was released (Stucki 1973)
Microsoft reports 80-90% statement coverage
Boeing must get 100% statement coverage (feasible) for all softwarefor all software
Usually can about 85% coverage; 100% is harder
→ Unreachable code; dead code
→ Complex sequences
→ Not enough resources
printSum(int a, int b) {
int result = a + b;
if (result > 0)
a == 3
b == 9
a == -5
b == -8
if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
}
Coverage: 100%
printSum(int a, int b) {
int result = a + b;
if (result > 0)
a == 3
b == 9
a == -5
b == -8
if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
[missing branch]
}
Coverage: 100%
Test requirements: Branches in the program
Cbranches = (number of traversed branches)
(number of branches)
printSum(int a, int b) {
int result = a + b;
if (result > 0)
a == 3
b == 9
a == -5
b == -8
if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
[missing branch]
}
Coverage: ?
printSum(int a, int b) {
int result = a + b;
if (result > 0)
a == 3
b == 9
a == -5
b == -8
if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
[missing branch]
}
Coverage: 88%
printSum(int a, int b) {
int result = a + b;
if (result > 0)
a == 3
b == 9
a == -5
b == -8
a == 0
b == 0
if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
[missing branch]
}
Coverage: 100%
1. void main() {
2. float x, y;
3. read(x);
4. read(y);
33
entryentryentryentry
44
X=0 ||!(X=0
• Consider test cases {(x=5,y=5), (x=5, y=-5)}
4. read(y);
5. if(x==0)||(y>0)
6. y = y/x;
7. else x = y+2;
8. write(x);
9. write(y);
10.}
77
88
exitexitexitexit
X=0 ||
y>0
99
66
!(X=0
||y>0)
1. void main() {
2. float x, y;
3. read(x);
4. read(y);
33
entryentryentryentry
44
X=0 ||!(X=0
• Consider test cases {(x=5,y=5), (x=5, y=-5)}
• The test suite is adequate for branch coverage, but does not reveal the fault at statement 6
4. read(y);
5. if(x==0)||(y>0)
6. y = y/x;
7. else x = y+2;
8. write(x);
9. write(y);
10.}
77
88
exitexitexitexit
X=0 ||
y>0
99
66
!(X=0
||y>0)at statement 6
• Predicate 5 can be true or false operating on only one condition
⇓
Basic condition coverageBasic condition coverage
Test requirements: Truth values assumed by
basic conditions
Cbc = (number of boolean values assumed by all basic conditions)
(number of boolean values of all basic conditions)(number of boolean values of all basic conditions)
x y
T T
F F
1. void main() {
2. float x, y;
3. read(x);
4. read(y);
33
entryentryentryentry
44
X=0 ||!(X=0
• Consider test cases {(x=0,y=-5), (x=5, y=5)}
• The test suite is adequate for basic condition coverage
x y4. read(y);
5. if(x==0)||(y>0)
6. y = y/x;
7. else x = y+2;
8. write(x);
9. write(y);
10.}
77
88
exitexitexitexit
X=0 ||
y>0
99
66
!(X=0
||y>0)
• but is not adequate for branch coverage.
⇓Branch and condition Branch and condition
coveragecoverage
x y
T T
F F
Test requirements: Branches and truth values
assumed by basic conditions
if ( ( a || b ) && c ) { … }
a b c Outcome
T T T T
F F F F
Key idea: Test important combinations of
conditions, avoiding exponential blowup
A combination is “important” if each basic
condition is shown to independently affect the
outcome of each decision
MC/DC criterion: For each basic condition C, there are
two test cases in which the truth values of all two test cases in which the truth values of all
conditions except C are the same, and the compound
condition as a whole evaluates to True for one of those
test cases and False for the other
( a && b && c )Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False
( a && b && c )Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False
( a && b && c )Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False
MC/DC criterion: For
each basic condition
C, there are two test
cases in which the
truth values of all
conditions except C 3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False
conditions except C
are the same, and the
compound condition
as a whole evaluates
to True for one of
those test cases and
False for the other
( a && b && c )Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False
1 True True True True
5 False True True False
( a && b && c )Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False
1 True True True True
5 False True True False
( a && b && c )Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False
1 True True True True
5 False True True False
( a && b && c )Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False
1 True True True True
5 False True True False
3 True False True False
( a && b && c )Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False
1 True True True True
5 False True True False
3 True False True False
( a && b && c )Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False
1 True True True True
5 False True True False
3 True False True False
( a && b && c )Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False
1 True True True True
5 False True True False
3 True False True False
2 True True False False
Tradeoff between number of required test
cases and thoroughness of the test
Required by both US and European quality
standards in aviation
Path coverage: cover each path in the program
Loop coverage (given n): cover all paths that contain at
most n iterations of any loop in the program
Data-flow coverage: cover data-flow relations in the
program (du pairs)program (du pairs)
Mutation coverage: cover (kill) mutated versions of the
program
�Most widely used as adequacy criteria because fully
automatable!
�Not all graph paths correspond to actual program paths
→some paths could be not executable
→Conditions may not be satisfiable due to interdependent →Conditions may not be satisfiable due to interdependent
conditions
→Paths may not be executable due to interdependent decisions
�The testing output is strongly related to the testing
inputs