Post on 29-Jul-2018
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
MTAT.03.094
Software Engineering
Lecture 11:
Verification and Validation
(Part 2) Dietmar Pfahl
email: dietmar.pfahl@ut.ee Fall 2015
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Exam Registration – Status as of Nov 19
Exam 1:
• Friday, 08-Jan-2015, 14:15-16:15, room 2-405
• Capacity limit: 90 students
• Registered: 47 – Free seats: 43
Exam 2:
• Friday, 15-Jan-2015, 14:15-16:15, room 2-405
• Capacity limit: 90 students
• Registered: 9 – Free seats: 81
You must register: First-come-first-serve (FIFO) principle
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Schedule of Lectures
Week 01: Introduction to SE
Week 02: Requirements Engineering I
Week 03: Requirements Engineering II
Week 04: Analysis
Week 05: Development Infrastructure I
Week 06: Development Infrastructure II
Week 07: Architecture and Design
Week 08: No lecture
Week 09: Refactoring
Week 10: Verification and Validation I
Week 11: Industry Lecture (Testing)
Week 12: Verification and Validation II
Week 13: Agile/Lean Methods
Week 14: Industry Lecture (Agile)
Week 15: Measurement / Course
wrap-up, review and exam
preparation
Week 16: no lecture
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Structure of Lectures 09 & 11
• Fundamental Definitions and Concepts
• Test Strategies
• Black-Box
• White-Box
• Assessing the Quality of Test Suites
• Test Coverage
• Mutation Testing
• Static Analysis
• Quality Prediction
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
How Do We Test This? 34 switches = 234 = 1.7 x 1010 possible inputs = 1.7 x 1010 tests
If only 3-way interactions, need only 33 tests
For 4-way interactions, need only 85 tests
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Combinatorial Testing
What is it?
• Methods for systematically testing t-way interaction effects of input (or configuration parameter) values.
Why do it?
• The interaction of specific combinations of input values may trigger failures that won’t be triggered if testing input (or configuration parameter) values only in isolation.
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Two Scopes of Combinatorial Testing
Pizza Ordering
System under test (SUT)
Test inputs
Test case OS CPU Protocol
1 Windows Intel IPv4
2 Windows AMD IPv6
3 Linux Intel IPv6
4 Linux AMD IPv4
Test Configurations
Test Inputs Size Topp Addr Phone
Sm Custom Val Val
Sm Preset Inv Inv
Med Custom Inv Val
Med Preset Val Inv
Lg Custom Val Inv
Lg Preset Inv Val
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Combinatorial Testing – Example 2
Platform configuration parameters:
• OS: Windows XP, Apple OS X, Red Hat Linux
• Browser: Internet Explorer, Firefox
• Protocol: IPv4, IPv6
• CPU: Intel, AMD
• DBMS: MySQL, Sybase, Oracle
Total number of combinations: 3*2*2*2*3 = 72
Do we need 72 test cases?
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Pair-Wise Testing (2-Way Interaction)
Only 10 tests needed, if we want to test all interactions of one parameter with one other parameter (pair-wise interaction)
# of parameter pairs:
# of 2-way interactions N=57:
Test OS Browser Protocol CPU DBMS
1 XP IE IPv4 Intel MySQL
2 XP Firefox IPv6 AMD Sybase
3 XP IE IPv6 Intel Oracle
4 OS X Firefox IPv4 AMD MySQL
5 OS X IE IPv4 Intel Sybase
6 OS X Firefox IPv4 Intel Oracle
7 RHL IE IPv6 AMD MySQL
8 RHL Firefox IPv4 Intel Sybase
9 RHL Firefox IPv4 AMD Oracle
10 OS X Firefox IPv6 AMD Oracle
10)!25(!2
!5
2
5
9032
52
2
540 22
N
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Pair-Wise Testing (2-Way Interaction)
# of parameter pairs:
# of 2-way interactions N=57:
a:b 3 x 2 = 6
a:c 3 x 2 = 6 a:d 3 x 2 = 6 a:e 3 x 3 = 9 b:c 2 x 2 = 4 b:d 2 x 2 = 4 b:e 2 x 3 = 6 c:d 2 x 2 = 4 c:e 2 x 3 = 6 d:e 2 x 3 = 6 ------ 57
Test OS
(a)
Browser
(b)
Protocol
(c) CPU
(d)
DBMS
(e)
1 XP IE IPv4 Intel MySQL
2 XP Firefox IPv6 AMD Sybase
3 XP IE IPv6 Intel Oracle
4 OS X Firefox IPv4 AMD MySQL
5 OS X IE IPv4 Intel Sybase
6 OS X Firefox IPv4 Intel Oracle
7 RHL IE IPv6 AMD MySQL
8 RHL Firefox IPv4 Intel Sybase
9 RHL Firefox IPv4 AMD Oracle
10 OS X Firefox IPv6 AMD Oracle
10)!25(!2
!5
2
5
9032
52
2
540 22
N
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Pair-Wise Testing (2-Way Interaction)
Only 10 tests needed, if we want to test all interactions of one parameter with one other parameter (pair-wise interaction)
Parameter pair 1:
3 x 2 = 6 value combinations
Test OS Browser Protocol CPU DBMS
1 XP IE IPv4 Intel MySQL
2 XP Firefox IPv6 AMD Sybase
3 XP IE IPv6 Intel Oracle
4 OS X Firefox IPv4 AMD MySQL
5 OS X IE IPv4 Intel Sybase
6 OS X Firefox IPv4 Intel Oracle
7 RHL IE IPv6 AMD MySQL
8 RHL Firefox IPv4 Intel Sybase
9 RHL Firefox IPv4 AMD Oracle
10 OS X Firefox IPv6 AMD Oracle
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Pair-Wise Testing (2-Way Interaction)
Only 10 tests needed, if we want to test all interactions of one parameter with one other parameter (pair-wise interaction)
Pair 2:
2 x 2 = 4 value combinations
Test OS Browser Protocol CPU DBMS
1 XP IE IPv4 Intel MySQL
2 XP Firefox IPv6 AMD Sybase
3 XP IE IPv6 Intel Oracle
4 OS X Firefox IPv4 AMD MySQL
5 OS X IE IPv4 Intel Sybase
6 OS X Firefox IPv4 Intel Oracle
7 RHL IE IPv6 AMD MySQL
8 RHL Firefox IPv4 Intel Sybase
9 RHL Firefox IPv4 AMD Oracle
10 OS X Firefox IPv6 AMD Oracle
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Pair-Wise Testing (2-Way Interaction)
Only 10 tests needed, if we want to test all interactions of one parameter with one other parameter (pair-wise interaction)
Pair 3:
3 x 3 = 9 value combinations
Test OS Browser Protocol CPU DBMS
1 XP IE IPv4 Intel MySQL
2 XP Firefox IPv6 AMD Sybase
3 XP IE IPv6 Intel Oracle
4 OS X Firefox IPv4 AMD MySQL
5 OS X IE IPv4 Intel Sybase
6 OS X Firefox IPv4 Intel Oracle
7 RHL IE IPv6 AMD MySQL
8 RHL Firefox IPv4 Intel Sybase
9 RHL Firefox IPv4 AMD Oracle
10 OS X Firefox IPv6 AMD Oracle
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Is Testing 2-Way Interactions Enough?
• Analyses of failure-triggering conditions showed this:
• Medical device (dark blue)
• NASA distrib. DB (light blue)
• Browser (green)
• Web server (magenta)
• Network security (orange)
• TCAS* module (purple)
* Traffic Collision Avoiding System
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Combinatorial Testing Links
http://cse.unl.edu/~citportal/
http://csrc.nist.gov/groups/SNS/acts/index.html
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
One problem is left …
Creating test (input) data is the (relatively) easy part!
How do we check that the code worked correctly on the test input?
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
How to (Automatically) Generate Test Oracles? Creating test (input) data is the (relatively) easy part!
How do we check that the code worked correctly on the test input?
• Using existing test sets (with known oracles) – easy if test sets exist
• Crash testing code to ensure it does not crash for randomly generated test input (‘fuzz testing’) – easy but of limited value
• Embedded assertions – incorporate assertions in code to check critical states at different points in the code – will they cover all possible incorrect states?
• Model-checking using a mathematical model of the system under test and a model checker to generate expected results for each input – expensive but tractable
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
State-Transition Testing
Create a set of test cases that triggers each state-transition at least once
Input (Event)
Action State-Transition Graph
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
State Table
State
Input (Event)
Wait for Card
(S1)
Wait for PIN
(S2)
Next
(S3)
Card inserted S2 (Ask for PIN) - -
Invalid PIN - S2 (Beep) -
Valid PIN - S3 (Ask amount) -
Cancel - S1 (Return card) -
... ... ... ...
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
State Table
State
Input (Event)
Wait for Card
(S1)
Wait for PIN
(S2)
Next
(S3)
Card inserted S2 (Ask for PIN) - -
Invalid PIN - S2 (Beep) -
Valid PIN - S3 (Ask amount) -
Cancel - S1 (Return card) -
... ... ... ...
4 Test Cases: - Input ’Card inserted’ in S1 - Input ’Invalid PIN’ in S2 - Input ’Valid PIN’ in S2 - Input ’Cancel’ in S2
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Error Guessing
• Exploratory testing, happy testing, ...
• Always worth including
• Can trigger failures that systematic techniques miss
• Consider
• Past failures
• Intuition
• Experience
• Brain storming
• ”What is the craziest thing we can do?”
• Lists in literature
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Exploratory Testing
• Inventors:
• Cem Kaner, James Bach (1990s)
• Definition:
• “Exploratory testing is simultaneous learning, test design, and test execution.”
• Elements / Variants
• Charter: defines mission (and sometimes tactics to use)
• Example: “Check UI against Windows interface standards”
• Session-based test management: Defects + Notes + Interviews of the testers
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Testing Strategies
requirements
input
events
output
Black Box Testing White Box Testing
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
There are many possible paths!
loop < 20x
If-then-else
Selective Testing
White-Box Testing
520 (~1014 ) different paths
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
a selected path
Control flow testing
Data flow testing
Selective:
Selective Testing
MTAT.03.159 / Lecture 03 / © Dietmar Pfahl 2015
empty Blocks (=Nodes): 4
Edges: 4
Control Flow Graph (CFG)
MTAT.03.159 / Lecture 03 / © Dietmar Pfahl 2015
empty Blocks: 4
Edges: 4
e3 e4
s1
s2
d1 s5
s6
s3
s4
Nodes: 8
Edges: 8
e7 e8
e1
e2
e5 e6
Control Flow Graph (CFG)
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
CFG Construction – Example
If (c1) then {
if (c2) then {s1}
s2
while (c3) do {s3}
}
else {
if (c4) then {
repeat {s4} until (c5)
}
}
d1 d2
d3
d5
d4
d6 d7
d8 d9
d10
d11
d12 d13
d14
c1
c4 c2
c5 c3
s1
s2
s4
s3
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
CFG Construction – Example
If (c1) then {
if (c2) then {s1}
s2
while (c3) do {s3}
}
else {
if (c4) then {
repeat {s4} until (c5)
}
}
d1 d2
c1
CFG(f) CFG(t)
If (c1) then {
CFG(c1=true)
}
else {
CFG(c1=false)
}
}
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
CFG Construction – Example
If (c1) then {
if (c2) then {s1}
s2
while (c3) do {s3}
}
else {
if (c4) then {
repeat {s4} until (c5)
}
}
d1 d2
d4
d6
d9
d10
d11
d14
c1
c4 CFG(if)
s2
If (c1) then {
CFG(if)
s2
CFG(while)
}
else {
if (c4) then {
CFG(repeat)
}
}
CFG (while)
CFG (repeat)
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
CFG Construction – Example
If (c1) then {
if (c2) then {s1}
s2
while (c3) do {s3}
}
else {
if (c4) then {
repeat {s4} until (c5)
}
}
d1 d2
d3
d5
d4
d6 d7
d8 d9
d10
d11
d12 d13
d14
c1
c4 c2
c5 c3
s1
s2
s4
s3
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Overview of Control Flow Criteria
Statement (or Block) Coverage – all nodes
Decision (or Branch) Coverage – all edges
Condition Coverage
Condition/Decision Coverage
Multiple Condition Coverage
Modified Condition Decision Coverage (MC/DC)
Linearly Independent Paths
Simple Paths
Visit-Each Loop
All Paths
…
d1 d2
d3
d5
d4
d6 d7
d8 d9
d10
d11
d12 d13
d14
c1
c4 c2
c5 c3
s1
s2
s4
s3
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Life Insurance Example
boolean AccClient(int age; gtype gender)
if (gender = female && age < 85)
return(TRUE);
if (gender = male && age < 80)
return(TRUE);
return(FALSE);
S
E
3
1
5
1:
2:
3:
4:
5:
d1 = c1 && c2
d2 = c3 && c4
2 4
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Statement Coverage
Criterion:
• Execute each statement at least once
• Tools can be used to monitor execution ( coverage)
Possible concern:
• Dead code
• Example: assume that due to some previous calculations, AccClient can only be invoked with parameter value gender = female
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Statement Coverage /1
boolean AccClient(int age; gtype gender)
if (gender = female && age < 85)
return(TRUE);
if (gender = male && age < 80)
return(TRUE);
return(FALSE);
S
E
3
1
5
1:
2:
3:
4:
5:
d1 = c1 && c2
d2 = c3 && c4
2 4
Test:
AccClient(83, female)->accept
AccClient(83, male)->reject
0 %
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Statement Coverage /2
boolean AccClient(int age; gtype gender)
if (gender = female && age < 85)
return(TRUE);
if (gender = male && age < 80)
return(TRUE);
return(FALSE);
S
E
3
1
5
1:
2:
3:
4:
5:
d1 = c1 && c2
d2 = c3 && c4
2 4
Test:
AccClient(83, female)->accept
AccClient(83, male)->reject
40 %
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Statement Coverage /3
boolean AccClient(int age; gtype gender)
if (gender = female && age < 85)
return(TRUE);
if (gender = male && age < 80)
return(TRUE);
return(FALSE);
S
E
3
1
5
1:
2:
3:
4:
5:
d1 = c1 && c2
d2 = c3 && c4
2 4
Test:
AccClient(83, female)->accept
AccClient(83, male)->reject
80 %
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Statement Coverage /4
boolean AccClient(int age; gtype gender)
if (gender = female && age < 85)
return(TRUE);
if (gender = male && age < 80)
return(TRUE);
return(FALSE);
S
E
3
1
5
1:
2:
3:
4:
5:
d1 = c1 && c2
d2 = c3 && c4
2 4
Test:
AccClient(83, female)->accept
AccClient(83, male)->reject
AccClient(25, male)->accept
100 %
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Decision (Branch) Coverage /1
boolean AccClient(int age; gtype gender)
if (gender = female && age < 85)
return(TRUE);
if (gender = male && age < 80)
return(TRUE);
return(FALSE);
S
E
3
1
5
1:
2:
3:
4:
5:
d1 = c1 && c2
d2 = c3 && c4
2 4
Test:
AccClient(83, female)->accept
AccClient(83, male)->reject
0 %
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Decision (Branch) Coverage /2
boolean AccClient(int age; gtype gender)
if (gender = female && age < 85)
return(TRUE);
if (gender = male && age < 80)
return(TRUE);
return(FALSE);
S
E
3
1
5
1:
2:
3:
4:
5:
d1 = c1 && c2
d2 = c3 && c4
2 4
Test:
AccClient(83, female)->accept
AccClient(83, male)->reject
AccClient(25, male)->accept
25 %
T
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Decision (Branch) Coverage /3
boolean AccClient(int age; gtype gender)
if (gender = female && age < 85)
return(TRUE);
if (gender = male && age < 80)
return(TRUE);
return(FALSE);
S
E
3
1
5
1:
2:
3:
4:
5:
d1 = c1 && c2
d2 = c3 && c4
2 4
Test:
AccClient(83, female)->accept
AccClient(83, male)->reject
AccClient(25, male)->accept
75 %
T/F
F
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Decision (Branch) Coverage /4
boolean AccClient(int age; gtype gender)
if (gender = female && age < 85)
return(TRUE);
if (gender = male && age < 80)
return(TRUE);
return(FALSE);
S
E
3
1
5
1:
2:
3:
4:
5:
d1 = c1 && c2
d2 = c3 && c4
2 4
Test:
AccClient(83, female)->accept
AccClient(83, male)->reject
AccClient(25, male)->accept
100 %
T/F
T/F
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Decision (Branch) Coverage /?
boolean AccClient(int age; gtype gender)
if (gender = female){
if (age < 85)
return(TRUE);
return(FALSE);}
if (gender = male){
if (age < 80)
return(TRUE);
return(FALSE);}
return(FALSE);
S
E
5
1
8
1:
2:
3:
4:
5:
6:
7:
8:
9:
d1 = c1
d3 = c3
4 6
Test:
AccClient(83, female)->accept
AccClient(83, male)->reject
AccClient(25, male)->accept
75 %
2
3
7
d2 = c2
9
Dead code!
d4 = c4
T/F
T
T/F
T
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Decision (Branch) Coverage /?
boolean AccClient(int age; gtype gender)
if (gender = female){
if (age < 85)
return(TRUE);
return(FALSE);}
if (gender = male){
if (age < 80)
return(TRUE);
return(FALSE);}
return(FALSE);
S
E
5
1 1:
2:
3:
4:
5:
6:
7:
8:
9:
d1 = c1
d3 = c3
6
Test:
AccClient(83, female)->accept
AccClient(83, male)->reject
AccClient(25, male)->accept
100 %
2
3
7
d2 = c2
9
d4 = c4
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Condition Coverage
• Test all conditions (in all predicate nodes)
• Each condition must evaluate at least once (or: once to ’true’ and once to ’false’)
• A (simple) condition may contain:
• Relational operators
• Arithmetic expressions
• ...
• A predicate may contain several (simple) conditions connected via Boolean operators
If (A<10 and B>250) then …
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Condition Coverage /1
boolean AccClient(int age; gtype gender)
if (gender = female && age < 85)
return(TRUE);
if (gender = male && age < 80)
return(TRUE);
return(FALSE);
S
E
3
1
5
1:
2:
3:
4:
5:
d1 = c1 && c2
d2 = c3 && c4
2 4
Test:
AccClient(83, female)->accept
AccClient(83, male)->reject
0 %
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Condition Coverage /2
boolean AccClient(int age; gtype gender)
if (gender = female && age < 85)
return(TRUE);
if (gender = male && age < 80)
return(TRUE);
return(FALSE);
S
E
3
1
5
1:
2:
3:
4:
5:
d1 = c1 && c2
d2 = c3 && c4
2 4
Test:
AccClient(83, female)->accept
AccClient(83, male)->reject
50 % or 25 %
T T T
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Condition Coverage /3
boolean AccClient(int age; gtype gender)
if (gender = female && age < 85)
return(TRUE);
if (gender = male && age < 80)
return(TRUE);
return(FALSE);
S
E
3
1
5
1:
2:
3:
4:
5:
d1 = c1 && c2
d2 = c3 && c4
2 4
Test:
AccClient(83, female)->accept
AccClient(83, male)->reject
100 % or 62.5 %
T/F T T/F
T F F
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Condition Coverage /4
boolean AccClient(int age; gtype gender)
if (gender = female && age < 85)
return(TRUE);
if (gender = male && age < 80)
return(TRUE);
return(FALSE);
S
E
3
1
5
1:
2:
3:
4:
5:
d1 = c1 && c2
d2 = c3 && c4
2 4
100 % or 75 %
T/F T T/F
T F/T F/T
Test:
AccClient(83, female)->accept
AccClient(83, male)->reject
AccClient(25, male)->accept
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Advanced Condition Coverage
• Condition/Decision Coverage (C/DC)
• as DC plus: every (simple) condition in each decision is tested in each possible outcome
• Modified Condition/Decision coverage (MC/DC)
• as above plus, every (simple) condition shown to independently affect a decision outcome (by varying that condition only)
• a (simple) condition independently affects a decision when, by flipping that condition and holding all the others fixed, the decision changes
• this criterion was created at Boeing and is required for aviation software according to RCTA/DO-178B
• Multiple-Condition Coverage (M-CC)
• all possible combinations of (simple) conditions within each decision taken
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
CC, DC, C/DC, M-CC, MC/DC Examples
Condition:
(TF) A = 2; B = 200 (D: False)
[(FT) A = 12; B = 300 (D: False)]
Decision:
(TT) A = 2; B = 300 (D: True)
(FT) A = 12; B = 300 (D: False)
Condition/Decision:
(TT) A = 2; B = 300 (D: True)
(FF) A = 12; B = 200 (D: False)
Multiple Condition:
(TT) A = 2; B = 300 (D: True)
(FT) A = 12; B = 300 (D: False)
(TF) A = 2; B = 200 (D: False)
(FF) A = 12; B = 200 (D: False)
Modified Condition/Decision:
(TT) A = 2; B = 300 (D: True)
(FT) A = 12; B = 300 (D: False)
(TF) A = 2; B = 200 (D: False)
If (A<10 and B>250) then …
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Independent Path Coverage
• McCabe cyclomatic complexity estimates number of test cases needed
• The number of independent paths needed to cover all simple paths at least once in a program
• Visualize by drawing a CFG
• CC = #(edges) – #(nodes) + 2
• CC = #(decisions) + 1
if-then-else
while-loop
case-of
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Independent Paths Coverage – Example
• Independent Paths Coverage
• Requires that a minimum set of linearly independent paths through the control flow-graph be executed
• This test strategy is the rationale for McCabe’s cyclomatic number (McCabe 1976) …
• … which is equal to the number of test cases required to satisfy the strategy.
1 2
3
5
4
6 7
8 9
10
11 12
13
14
Cyclomatic Complexity = 5 + 1 = 6
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Independent Paths Coverage – Example
Edges: 1-2-3-4-5-6-7-8-9-10-11-12-13-14
Path1: 1-0-0-1-0-1-0-0-1-0---0---0---0---0
Path2: 1-0-1-0-1-1-1-1-1-0---0---0---0---0
Path3: 1-0-0-1-0-1-1-1-1-0---0---0---0---0
Path4: 0-1-0-0-0-0-0-0-0-1---0---1---0---1
Path5: 0-1-0-0-0-0-0-0-0-1---0---1---1---1
Path6: 0-1-0-0-0-0-0-0-0-0---1---0---0---0
1 2
3
5
4
6 7
8 9
10
11 12
13
14
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Independent Paths Coverage – Example
Edges: 1-2-3-4-5-6-7-8-9-10-11-12-13-14
Why no need to cover Path7 ???
Path7: 1-0-1-0-1-1-0-0-1-0---0---0---0---0
Because it equals Path1+Path2-Path3 !!!
Path1: 1-0-0-1-0-1-0-0-1-0---0---0---0---0
Path2: 1-0-1-0-1-1-1-1-1-0---0---0---0---0
P1+P2: 2-0-1-1-1-2-1-1-2-0---0---0---0---0
Path3: 1-0-0-1-0-1-1-1-1-0---0---0---0---0
-P3: 1-0-1-0-1-1-0-0-1-0---0---0---0---0
1 2
3
5
4
6 7
8 9
10
11 12
13
14
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Control-Flow Coverage Relationships
Subsumption:
a criterion C1 subsumes another criterion C2, if any test set {T} that satisfies C1 also satisfies C2
Statement
Decision
Complete Path
Linearly Indep. Path
Multiple Condition
MC/DC
Condition
C/DC
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Data Flow Testing
• Identifies paths in the program that go
• from the assignment of a value to a variable
• to the use of such variable,
to make sure that the variable is properly used.
X:=14; ….. Y:= X-3;
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Data Flow Testing – Definitions
• Def – assigned or changed
• Uses – utilized (not changed)
• C-use (Computation) e.g. right-hand side of an assignment, an index of an array, parameter of a function.
• P-use (Predicate) branching the execution flow, e.g. in an if statement, while statement, for statement.
• Example: All def-use paths (DU) requires that each DU chain is covered at least once
• Why interesting?
• E.g., consider: def-def-def (and no use) or use (without def)
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Data Flow Testing – Example
Considering age, there are two DU paths:
(a)[1]-[4]
(b)[1]-[6]
Test case data:
AccClient(*, female)-> *
AccClient(*, male)-> *
[1] bool AccClient(agetype
age; gndrtype gender)
[2] bool accept
[3] if(gender=female)
[4] accept := age < 85;
[5] else
[6] accept := age < 80;
[7] return accept
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Data Flow Testing – Example
Considering gender, there is one DU path:
(a)[1]-[3]
Test case data:
AccClient(*, *)-> *
[1] bool AccClient(agetype
age; gndrtype gender)
[2] bool accept
[3] if(gender=female)
[4] accept := age < 85;
[5] else
[6] accept := age < 80;
[7] return accept
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Data Flow Criteria
Weaker
Stronger
# tests All uses
All p-uses, some c-uses
All c-uses All p-uses All defs
All def-use paths
All c-uses, some p-uses
MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015
(c) 2008 Mauro Pezzè & Michal Young OO-Testing Approaches
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2015
Class Testing and State
push(s, elem1)
Show_top(s) -> elem1
pop(s, x) -> x=elem1
pop(s, x) -> x=?
The result of a method call
depends on the state of the
stack
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2015
Stack Example
How many test cases
needed to cover all
feasible branches?
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2015
Stack Example
How many test cases
needed to cover all
feasible branches?
4 decisions
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2015
Stack Example
How many test cases
needed to cover all
feasible branches?
4 decisions
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2015
Stack Example
2 test cases are enough:
D1
D2
D3
D4
D3: false
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2015
Stack Example
2 test cases are enough:
D1
D2
D3
D4
D1: false
D2: true
D1: false
D2: true
D1: false
D2: true
D1: true
D4: true ... false
D3: true
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2015
Stack Example – CFGs
if
if
if
true
true
false
false
true false
for true false
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2015
Stack Example – CFG (Class)
push(x)
if
if
if
true
true
false
false
true false
for true false
resize()
pop()
Stack
Stack()
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2015
State Diagram for Stack Example
non_exist empty (size = 0)
error
c
Stack() pop()
not_empty (0 < size < length)
[size = 1] pop() push(x)
[size < length - 1 ]
push(x) [size > 1] pop()
[size = length - 1] push(x) / resize()
push(x): size++ / pop(): size—
resize(): length := length * 2
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Structure of Lectures 09 & 11
• Fundamental Definitions and Concepts
• Test Strategies
• Black-Box
• White-Box
• Assessing the Quality of Test Suites
• Test Coverage
• Mutation Testing
• Static Analysis
• Quality Prediction
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Code Coverage (Test Coverage)
• Measures the extent to which certain code items related to a defined test adequacy criterion have been executed (covered) by running a set of test cases (= test suites)
• Goal: Define test suites such that they cover as many (disjoint) code items as possible
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Code Coverage Measure – Example
• Statement Coverage (CVs)
• Portion of the statements tested by at least one test case.
: number of statements tested
: total number of statemen
100%
ts
ts
p
t
p
SCV
S
S
S
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Code Coverage Analysis
• Code coverage analysis is the process of:
• Finding areas of a program not exercised by a set of test cases
• Creating additional test cases to increase coverage
• Determining a quantitative measure of code coverage, which is (believed to be) a predictor of code quality
• Code coverage analyzers automate this process
• Additional aspect of code coverage analysis:
• Identifying redundant test cases that do not increase coverage
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Main Classes of Test Adequacy Criteria
• Control Flow Criteria:
• Statement, decision (branch), condition, and path coverage are examples of control flow criteria
• They rely solely on syntactic characteristics of the program (ignoring the semantics of the program computation)
• Data Flow Criteria:
• Require the execution of path segments that connect parts of the code that are intimately connected by the flow of data
• Other Criteria:
• Requirements ’covered’, ...
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Structure of Lectures 09 & 11
• Fundamental Definitions and Concepts
• Test Strategies
• Black-Box
• White-Box
• Assessing the Quality of Test Suites
• Test Coverage
• Mutation Testing
• Static Analysis
• Quality Prediction
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Static Analysis
• Document Review (manual)
• Different types
• Static Code Analysis (automatic)
• Structural properties / metrics
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Static Code Analysis – Example 1 Bug finding is about pointing out programming mistakes: bad practice, coding errors, unexpected behavior. One interesting example of bugs that static analysis can find is null pointer dereference.
int i=0;
String s = null;
If (i>0) {
s = "positive";
}
If (s.contains("pos")) {
System.out.println(s);
}
This code will compile but at
runtime a null pointer exception
will be thrown, the String s being
null and calling the method
contains.
Static analysis tools, for instance
FindBugs can find this bug and
report it.
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Static Code Analysis – Example 2 Bug finding is about pointing out programming mistakes: bad practice, coding errors, unexpected behavior. One interesting example of bugs that static analysis can find is security vulnerabilities.
Static analysis can be applied to find security flaws in code. With dataflow analysis, it is possible to follow the
propagation of input data, and thus detect possible code injection.
Example:
public static void main(String args[]) {
File f = new File(args[0]);
f.open();
//...
}
This program opens a file with an argument entered in command line. The fact that we use this argument to
open a file is just an example, the important fact is that we use directly an input without validation, which
constitutes a serious security vulnerability. Dataflow analysis can detect this kind of flaw by finding the source
of data inputs, following their propagation until their use in a sensitive instruction (like creating a File object).
Source: http://blog.zenika.com/index.php?post/2012/08/23/Static-analysis-(part-1/4)
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Reviews - Terminology
• Static testing – testing without software execution
• Review – meeting to evaluate software artifact
• Inspection – formally defined review
• Walkthrough – author guided review
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Relative Cost of Faults Maintenance
200
Source: Davis, A.M., “Software Requirements: analysis and specification” (1990)
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Reading Techniques
• Ad hoc
• Checklist-based
• Defect-based
• Scenario-based
• Usage-based
• Perspective-based
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Perspective-based Reading
• Scenarios
• Purpose • Decrease overlap
(redundancy)
• Improve
effectiveness
Designer
Tester
User
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Structure of Lectures 09 & 11
• Fundamental Definitions and Concepts
• Test Strategies
• Black-Box
• White-Box
• Assessing the Quality of Test Suites
• Test Coverage
• Mutation Testing
• Static Analysis
• Quality Prediction
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Quality Prediction
• Based on product and process properties
• Quality = Function(Code Size | Complexity)
• Quality = Function(Code Changes)
• Quality = Function(Detected #Defects)
• Quality = Function(Test Effort)
• Based on detected defects
• Capture-Recapture Models
• Reliability Growth models
Quality def.: Undetected #Defects
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Capture-Recapture – Defect Estimation
• Situation: Two inspectors are assigned to inspect the same product
• d1: #defects detected by Inspector 1
• d2: #defects detected by Inspector 2
• d12: #defects by both inspectors
• Nt: total #defects (detected and undetected)
• Nr: remaining #defects (undetected)
12
21
d
ddNt )( 1221 dddNN tr
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Capture-Recapture – Example
• Situation: Two inspectors are assigned to inspect the same product
• d1: 50 defects detected by Inspector 1
• d2: 40 defects detected by Inspector 2
• d12: 20 defects by both inspectors
• Nt: total defects (detected and undetected)
• Nr: remaining defects (undetected)
10020
4050
12
21
d
ddNt 30)204050(100 rN
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Advanced Capture-Recapture Models
• Four basic models used for inspections
• Degree of freedom
• Prerequisites for all models
• All reviewers work independently of each other
• It is not allowed to inject or remove faults during inspection
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Advanced Capture-Recapture Models
Model
Probability of defect being
found is equal across ...
Estimator Defect Reviewer
M0 Yes Yes Maximum-likelihood
Mt Yes No Maximum-likelihood
Chao’s estimator
Mh No Yes Jackknife
Chao’s estimator
Mth No No Chao’s estimator
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Mt Model
Maximum-likelihood: • Mt = total marked animals
(=faults) at the start of the t'th sampling interval
• Ct = total number of animals (=faults) sampled during interval t
• Rt = number of recaptures in the sample Ct
• An approximation of the maximum likelihood estimate of population size (N) is: SUM(Ct*Mt)/SUM(Rt)
First resampling:
M1=50 (first inspector)
C1=40 (second inspector)
R1=20
N=40*50/20=100
Second resampling:
M2=70 (first and second inspector)
C2=40 (third inspector)
R2=30
N=(40*50+40*70)/(20+30)=4800/50=96
Third resampling:
M3=80
C3=30 (fourth inspector)
R3=30
N=(2000+2800+30*80)/(20+30+30)=7200/80=90
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Reliability Growth Models
• To predict the probability of future failure occurrence based on past (observed) failure occurrence
• Can be used to estimate
• the number of residual (remaining) faults or
• the time until the next failure occurs
• the remaining test time until a reliability objective is achieved or
• Application typically during system test
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Reliability Growth Models (RGMs)
Purpose:
Stop testing when
a certain percentage (90%, 95%, 99%, 99.9%, …) of estimated total number of failures has been reached
a certain failure rate has been reached
Cumulative #Failures (m)
Test Intensity (t)
(CPU time, test effort, test days, calendar days, …)
100%
95%
(estimated n0)
dtet
0
0
0
0)(n
t
tm
MTAT.03.094 / Lectures 09 & 11 / © Dietmar Pfahl 2015
Model Selection
Many different RGMs have been proposed (>100)
To choose a reliability model, perform the following steps:
1. Collect failure data
2. Examine data (failure data vs. test time/effort)
3. Select a set of candidate models
4. Estimate model parameters for each candidate model
Least squares method
Maximum likelihood method
5. Customize model using the estimated parameters
6. Compare models with goodness-of-fit test and select the best
7. Make reliability predictions with selected model(s)