1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition,...
-
Upload
hayden-hornbuckle -
Category
Documents
-
view
216 -
download
1
Transcript of 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition,...
![Page 1: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/1.jpg)
1
Unit Testing
CS 4311
Hans Van Vliet, Software Engineering, Principles and Practice, 3rd edition, John Wiley & Sons, 2008. Chapter 13.
![Page 2: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/2.jpg)
2
Hierarchy of TestingTesting
Program Testing
Top Down
Bottom Up
Integration TestingUnit Testing
System Testing
Big Bang
Sandwich
Black Box
White Box
Function
Performance
Reliability
Availability
AcceptanceTesting
Properties
Security
Equivalence
Boundary
Decision Table
State Transition
Use Case
Domain Analysis
Control Flow Data Flow
Usability
Documentation
Portability
Capacity
Ad Hoc
Benchmark
Pilot
Alpha
Beta
Black Box
Equivalence
Black Box
Boundary
Equivalence
Boundary
Black Box
Equivalence
Boundary
Black Box
Equivalence
Decision Table
Boundary
Black Box
Equivalence
State Transition
Use Case
State Transition
Domain Analysis
Use Case
State Transition
White Box
Domain Analysis
Use Case
State Transition
![Page 3: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/3.jpg)
3
Outline
Basic concepts Black box testing White box testing
![Page 4: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/4.jpg)
4
Basic Concepts
What is unit testing? Test focusing on the smallest units of code, such as
Functions, procedures, subroutines, subprograms Methods, classes
Component tested in isolation from the rest of the system and in a controlled environment:
Uses appropriately chosen input data Uses component-level design description as guide
Often the target of testing frameworks such as JUnit
![Page 5: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/5.jpg)
5
Basics
Why unit testing? Foundation for other testing like integration and
system testing What to test?
Data transformations across the unit are tested Data structures are tested to ensure data integrity
When and who? Frequently done (at least partially) during code
development by code developers
![Page 6: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/6.jpg)
6
Unit Test Procedures
Driver
Module to Be Tested
Stub Stub
Results
Testcases
![Page 7: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/7.jpg)
7
Two Different Techniques
Black box Based on specification Inner structure of test object is not considered
White box Based on source code Inner structure of test object is the basis of test case selection
Often complementary Effectiveness of black box is similar to white box, but the
mistakes found are different (Hetzel 1976, Myers 1978) Use in combinations
![Page 8: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/8.jpg)
8
Outline
Basic concepts Black box testing
Basic conceptsEquivalence classes…
White box testing
![Page 9: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/9.jpg)
9
What Is Black Box Testing?
Unit (code, module) seen as a black box No access to the internal or logical structure Determine if given input produces expected
output
Input Output
![Page 10: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/10.jpg)
10
Black Box Testing Test set is derived from specifications or requirements Goal is to cover the input space Lots of approaches to describing input space:
Equivalence classes Boundary value analysis Decision tables State transitions Use cases . . .
![Page 11: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/11.jpg)
11
Advantage and Disadvantage
(Dis)Advantages It does not require access to the internal logic of a
component However, in most real-world applications, impossible
to test all possible inputs
Need to define an efficient strategy to limit number of test cases
![Page 12: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/12.jpg)
12
General Process
Analyze specifications or requirements Select valid and invalid inputs (i.e., positive and
negative tests) Determine expected outputs Construct tests Run tests Compare actual outputs to expected outputs
![Page 13: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/13.jpg)
13
Outline
Basic concepts Black box testing
Basic conceptsEquivalence classesBoundary values…
White box testing
![Page 14: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/14.jpg)
14
Motivation Assume you test a sign function, int sign(int x),
whose result is defined as:
1 if x > 0
0 if x = 0
-1 otherwise (i.e., x < 0) One way to reduce the number of test cases is:
to partition the input into three subsets
{ x | x > 0}, { x | x = 0}, { x | x < 0} to pick one from each subset, e.g., 10, 0, and -10.
![Page 15: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/15.jpg)
15
Basic Strategy of Equivalence Classes
Partition the input into equivalence classes This is the tricky part. It’s an equivalence class if:
Every test using one element of the class tests the same thing that any other element of the class tests
If an error is found with one element, it should be found with any element
If an error is not found with some element, it is not found by any element
Test a subset from each class
![Page 16: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/16.jpg)
16
Exercise
Consider a factorial function, fact(n): if n < 0 or n >= 200, errorif 0 n 20, exact valueif 20 < n < 200, approximate value within 0.1%.
Q: What equivalence classes can you see?
![Page 17: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/17.jpg)
17
Simple Example
Suppose you are building an airline reservation system. A traveler can be a child, an adult, or a senior. The price depends on the type of traveler. The seat reservation does not depend on the type of traveler.
Q: How many test cases can you identify for the reservation component and the billing component?
![Page 18: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/18.jpg)
18
Heuristics for Finding Equivalence Classes
Identify restrictions for inputs and outputs in the specification.
![Page 19: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/19.jpg)
19
Heuristics for Finding Equivalence Classes
Identify restrictions for inputs and outputs in the specification.
If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN).
![Page 20: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/20.jpg)
20
Heuristics for Finding Equivalence Classes
Identify restrictions for inputs and outputs in the specification.
If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN).
If a number of values is required, create one valid and one or more invalid classes.
![Page 21: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/21.jpg)
21
Heuristics for Finding Equivalence Classes
Identify restrictions for inputs and outputs in the specification.
If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN).
If a number of values is required, create one valid and one or more invalid classes.
If a set of values is specified where each may be treated differently, create a class for each element of the set and one more for elements outside the set.
![Page 22: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/22.jpg)
22
Heuristics for Finding Equivalence Classes
Identify restrictions for inputs and outputs in the specification.
If there is a continuous numerical domain, create one valid and two or three invalid classes (above, below, and NaN).
If a number of values is required, create one valid and one or more invalid classes.
If a set of values is specified where each may be treated differently, create a class for each element of the set and one more for elements outside the set.
If there is a condition, create two classes, one satisfying and one not satisfying the condition.
![Page 23: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/23.jpg)
23
Exercise
Define test cases for a program that reserves a golf tee time. The standard green fee is $65 on weekdays (Monday-Friday) and
$80 on weekend (Saturday and Sunday). However, an El Paso resident pays a reduced green fee of $45 and
$60 on weekdays and weekend, respectively. A senior (of age 60+) pays only $40 and $50 on weekdays and
weekend, respectively. A junior (of age <17) pays only $20 and $30 on weekdays and
weekend, respectively.
Q: How many equivalence classes? Define them.Q: Define one test case per equivalence class.
![Page 24: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/24.jpg)
24
Outline
Basic concepts Black box testing
Basic concepts Equivalence classes Boundary values Decision tables …
White box testing
![Page 25: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/25.jpg)
25
Boundary Values
Observations Programs that fail at interior elements of an
equivalence class usually fail at the boundaries too. Programmers often make errors on boundary
conditions (e.g., branch/loop condition x <= 10 instead of x < 10).
Test the boundaries if it should work for 1-99, test 0, 1, 99, 100. if it works for A-Z, try @, A, Z, [, a, and z
The hard part is identifying boundaries.
![Page 26: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/26.jpg)
26
Hints
If a domain is a restricted set, check the boundaries. e.g., D=[1,10], test 0, 1, 10, 11 It may be possible to test the boundaries of outputs, also.
For ordered sets, check the first and last elements. For complex data structures, the empty list, full lists, the
zero array, and the null pointer should be tested. Extremely large data sets should be tested. Check for off-by-one errors.
![Page 27: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/27.jpg)
27
More Hints
Some boundaries are not obvious and may depend on the implementation (use gray box testing if needed) Numeric limits (e.g., test 255 and 256 for 8-bit values) Implementation limits (e.g., max array size)
![Page 28: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/28.jpg)
28
Example
Consider a simple voltage stabilizer with an input range of 190-240 volts 50-60 hertz.
![Page 29: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/29.jpg)
29
Example
Consider a simple voltage stabilizer with an input range of 190-240 volts 50-60 hertz.
Voltage boundaries 189 volts (below the lowest boundary) 190 volts (lowest boundary) 240 volts (highest boundary) 241 volts (above the highest boundary)
Cycles (hertz) boundaries 49 hertz (below) 50 hertz (lowest) 60 hertz (highest) 61 hertz (above)
![Page 30: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/30.jpg)
30
Example
Consider a simple voltage stabilizer with an input range of 190-240 volts 50-60 hertz.
Voltage boundaries 189 volts (below the lowest boundary) 190 volts (lowest boundary) 240 volts (highest boundary) 241 volts (above the highest boundary)
Cycles (hertz) boundaries 49 hertz (below) 50 hertz (lowest) 60 hertz (highest) 61 hertz (above)
How to combine?Exhaustive,
Pairwise (later),One extremes with
others fixed.
![Page 31: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/31.jpg)
31
Exercise
Determine the boundary values for US Postal Service ZIP codes (5 digits such as 79912 or 5 digits hyphen 4 digits such as 79912-1818).
Determine the boundary values for a 15-character last name entry.
Hints: Two kinds of boundaries---size boundary and value boundary.
![Page 32: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/32.jpg)
32
Outline
Basic concepts Black box testing
Basic concepts Equivalence classes Boundary values Decision tables Pairwise testing State transitions Use cases
White box testing
![Page 33: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/33.jpg)
33
Decision Tables
Construct a table (to help organize the testing) Identify each rule or condition in the system that
depends on some input For each input to one of these rules, list the
combinations of inputs and the expected results
![Page 34: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/34.jpg)
34
Example
Test Case C1
Student?
C2
Senior?
Result
Discount?
1 T T T
2 T F T
3 F T T
4 F F F
Theater ticket prices are discounted for senior citizens and students.
![Page 35: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/35.jpg)
35
Exercise
Construct a decision table for a program that reserves a golf tee time. The standard green fee is $65 on weekdays (Monday-Friday) and
$80 on weekend (Saturday and Sunday). However, an El Paso resident pays a reduced green fee of $45 and
$60 on weekdays and weekend, respectively. A senior (of age 60+) pays only $40 and $50 on weekdays and
weekend, respectively. A junior (of age <17) pays only $20 and $30 on weekdays and
weekend, respectively.
![Page 36: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/36.jpg)
36
Consider a website that must: operate correctly with different browsers: IE 5, IE 6, and IE 7;
Mozilla 1.1; Opera 7; FireFox 2, 3, and 4, and Chrome. work using RealPlayer, MediaPlayer, or no plug-in. run under Windows ME, NT, 2000, XP, and Vista, 7.0, and 8.0 accept pages from IIS, Apache and WebLogic running on
Windows NT, 2000, and Linux servers.
How many different configurations are there?
Pairwise Testing: Motivation
![Page 37: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/37.jpg)
37
Consider a website that must: operate correctly with different browsers: IE 5, IE 6, and IE 7;
Mozilla 1.1; Opera 7; FireFox 2, 3, and 4, and Chrome. work using RealPlayer, MediaPlayer, or no plug-in. run under Windows ME, NT, 2000, XP, and Vista, 7.0, and 8.0 accept pages from IIS, Apache and WebLogic running on
Windows NT, 2000, and Linux servers.
How many different configurations are there?A: 9 browsers x 3 plug-ins x 7 OS’s x 3 web servers x 3
server OS’s = 1701 combinations
Motivation
![Page 38: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/38.jpg)
38
Many such examples (i.e., finite sets of discrete values).
A bank is ready to test a data processing system Customer types: Gold, Platinum, Business, Non profits Account types: Checking, Savings, Mortgages, Consumer loans,
Commercial loans States (with different rules): CA, NV, UT, ID, AZ, NM
How many different configurations are there?
Motivation
![Page 39: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/39.jpg)
39
Approaches?
When given a large number of combinations: Give up and don’t test Test all combinations: miss targets, delay product
launch, and go out of business Choose one or two cases Choose a few that the programmers already ran Choose the tests that are easy to create List all combinations and choose first few List all combinations and randomly choose some “Magically” choose a small subset with high probability of
revealing defects
![Page 40: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/40.jpg)
40
Pairwise Testing
Combinatorial testing technique in which every pair of input parameters of software is tested.
Reasonable cost-benefit compromise Much faster than exhaustive testing More effective than less exhaustive methods that fail to exercise
all possible pairs of input parameters Why? Majority of software failures are caused by a single input
parameter or a combination of two input parameters. Each pair of input parameter values should be captured
at least by one test case. However, finding the least number of test cases is an
NP-complete problem.
![Page 41: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/41.jpg)
41
Example
Consider software that takes three input parameters, say x, y, and z.
If each parameter can have three different values, then there will be 27 different pairs: (x1, y1), (x1, y2), …, (y3, z3).
A test case (x1, y3, z2), for example, captures three of these 27 pairs: (x1, y3), (x1, z2), and (y3, z3).
By selecting test cases judiciously, all pairs of input parameters can be exercised with a minimum number of test cases; e.g., a set of 9 test cases can capture all 27 pairs of three parameters, each with three different values.
![Page 42: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/42.jpg)
42
State Transitions
Use state transitions (e.g., UML state machine diagrams) to derive test inputs
Cover a state transition diagram, e.g., Visit each state at least once Trigger each event at least once Exercise each transition at least once Exercise each path at least once*
* Might not be possible.
![Page 43: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/43.jpg)
43
Example and Exercise
Reserved
Paid
Ticketed
Used
paidprinted
deliveredcanceled
Customer makes a reservation and has a limited time to pay. May cancel at any time.
Groups: How many tests to visit each state, trigger each event, exercise each transition, and exercise each path?
Cancelled
Unpaid
ordered/start timer
canceled/refund
timerexpired
canceled/refund
![Page 44: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/44.jpg)
44
Use Cases
Use the use cases to specify test cases Use case specifies both normal, alternate, and
exceptional operations However, use cases may not have sufficient
details, esp. for unit testing
![Page 45: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/45.jpg)
45
Outline
Basic concepts Black box testing White box testing
Control flow graph (CFG)Statement coverage…
![Page 46: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/46.jpg)
46
White Box Testing
Test set is derived from structure of (source) code Also known as:
Glass box testing Structural testing
Often use a graph called a control flow graph (CFG) To represent code structure To cover it (CFG), e.g., all statements
Input Output
![Page 47: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/47.jpg)
47
Control Flow Graphs Programs are made of three kinds of statements:
Sequence (i.e., series of statements) Condition (i.e., if statement) Iteration (i.e., while, for, repeat statements)
![Page 48: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/48.jpg)
48
Control Flow Graphs Programs are made of three kinds of statements:
Sequence (i.e., series of statements) Condition (i.e., if statement) Iteration (i.e., while, for, repeat statements)
CFG: visual representation of flow of control Node represents a sequence of statements with single entry
and single exit Edge represents transfer of control from one node to another
![Page 49: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/49.jpg)
49
Control Flow Graph (CFG)
Sequence If-then-else If-then Iterative
![Page 50: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/50.jpg)
50
Control Flow Graph (CFG)
Join Join
Sequence If-then-else If-then Iterative
When drawing CFG, ensure that there is one exit: include the join node if
needed
![Page 51: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/51.jpg)
51
Example
1. read (result);2. read (x,k)3. while result < 0 {4. ptr false5. if x > k then6. ptr true7. x x + 18. result result + 1 }9. print result
3
5
6
7
9
8
2
1
4
![Page 52: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/52.jpg)
52
Example
1. read (result);2. read (x,k)3. while result < 0 {4. ptr false5. if x > k then6. ptr true7. x x + 18. result result + 1 }9. print result
3
5
6
7
9
8
2
1
4
3
4,5
6
Join
9
7,8
1,2
![Page 53: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/53.jpg)
53
Exercise: Draw CFGs
Example 1
1. if (a < b) then 2. while (a < n)3. a a + 1;4. else 5. while (b < n)6. b b + 1;
Example 2
1. read (a, b); 2. if (a 0 && b 0) then {3. c a + b;4. if c > 10 then 5. c max;6. else c min; }7. else print ‘Done’;
Example 3
1. read (a, b); 2. if (a 0 || b 0) then {3. c a + b;4. while( c < 100)5. c a + b; }6. c a * b;
![Page 54: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/54.jpg)
54
Outline
Basic concepts Black box testing White box testing
Control flow graph (CFG) Statement coverage Branch coverage Condition coverage Path coverage …
![Page 55: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/55.jpg)
55
Coverage
Coverage-based testingAdequacy of testing in terms of the coverage of the product to be
tested, e.g., the percentage of statements executed or the percentage of functional requirements tested.
Control-flow coverage Statement Branch Condition Path
Data-flow coverage Def-use
![Page 56: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/56.jpg)
56
Statement Coverage
Every statement gets executed at least once Every node in the CFG gets visited at least
once, also known as all-nodes coverage
![Page 57: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/57.jpg)
57
Example
Q: Number of paths needed for statement coverage?
2
3
4
7
5
1
8
9
6
2
3
4
7
5
1
8
9
6
![Page 58: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/58.jpg)
58
Branch Coverage
Every decision is made true and false. Every edge in a CFG of the program gets traversed at
least once. Also known as
Decision coverage All edges coverage Basis path coverage
Branch is finer than statement. C1 is finer than C2 if
T1 C1 (T2 C2 T2 T1)
true false
![Page 59: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/59.jpg)
59
Example
Q: Number of paths needed for branch coverage?
2
3
4
7
5
1
8
9
6
2
3
4
7
5
1
8
9
6
![Page 60: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/60.jpg)
60
Condition Coverage
Make each Boolean sub-expression (called a condition) of a decision evaluated both to true and false
Example:
if (x > 0 || y > 0) { … } else { … } C1: x > 0, x 0 C2: y > 0, y 0
Q: How many tests?
![Page 61: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/61.jpg)
61
Many Variances of Condition Coverage
Condition coverage is not finer than branch coverage. There are pathological cases where you can achieve
condition and not branch. E.g., if (x > 0 || y > 0) { … } (x = 10, y = 0), (x = 0, y = 10): condition but not branch
Under most circumstances, achieving condition achieves branch
Many variances, e.g., Multiple condition Condition/decision Modified condition/decision (MC/DC)
![Page 62: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/62.jpg)
62
CFG for Condition Coverage One way to determine the number of paths is to break the
compound conditional into atomic conditionals Suppose you were writing the CFG for the assembly language
implementation of the control construct if (A AND B) then
Cendif
(short circuit eval) (no short circuit eval) LD A LD A ; in general, lots of BZ :endif LAND B ; code for A and B LD B BZ :endif BZ :endif JSR C JSR C :endif nop
:endif nop
![Page 63: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/63.jpg)
63
AND Condition
1. read (a, b); 2. if (a == 0 && b == 0) then {3. c a + b; }4. else c a * b;
paths:1, 2A,2B,3, J1, 2A, 2B, 4, J1, 2A, 4, J
2A
2B
Join
3
4
1
true false
true false
![Page 64: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/64.jpg)
64
OR Condition
1. read (a,b); 2. if (a == 0 || b == 0) then {3. c a + b;4. while( c < 100)5. c a + b;}
Paths:1, 2A, 3, 4, 5, 4 … J1, 2A, 3, 4, J1, 2A, 2B, 3, 4, 5, 4, … J1, 2A, 2B, J
2A
3
Join
4
2B
5
1
true false
false
true
![Page 65: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/65.jpg)
65
Outline
Basic concepts Black box testing White box testing
Control flow graph (CFG) Statement coverage Branch coverage Condition coverage Path coverage Def-use coverage
![Page 66: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/66.jpg)
66
Path Coverage
Every distinct path through code is executed at least once.
All-paths coverage similar to exhaustive testing A path is a sequence of:
Statements Branches Edges
![Page 67: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/67.jpg)
67
Example
1. read (x)2. read (z)3. if x 0 then {4. y x * z;5. x z; }6. else print ‘Invalid’;7. if z > 1 then8. print z;9. else print ‘Invalid’;
Test Paths:1, 2, 3, 4, 5, J1, 7, 8, J21, 2, 3, 4, 5, J1, 7, 9, J21, 2, 3, 6, J1, 7, 8, J21, 2, 3, 6, J1, 7, 9, J2
1,2,3
4,5
Join1
6
7
8
Join2
9
![Page 68: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/68.jpg)
68
Linearly Independent Paths
It is not feasible to calculate the total number of paths. It is feasible to calculate the number of linearly
independent paths: A complete path which, disregarding back tracking (such as
loops), has an unique set of decisions in a program. Any path through the program that introduces at least one new
edge that is not included in any other linearly independent paths.
The number of linearly independent paths is the number of end-to-end paths required to touch every path segment at least once.
![Page 69: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/69.jpg)
69
McCabe’s Cyclomatic Complexity Software metric for the logical complexity of a program Defines the number of independent paths in the basis
set of a program (basis set: maximal linearly independent set of paths).
Provides the upper bound for the number of tests that must be conducted to ensure that all statements have been executed at least once.
For edges (E) and nodes (N), cyclomatic complexity number V is:V = E – N + 2
![Page 70: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/70.jpg)
70
Example: Complexity of CFG
1
2
3
1 3,4
5 6
Join
3,4
5
Join
N = 3E = 2V = E - N + 2 = 2 – 3 + 2 = 1
N = 1E = 0V = E - N + 2 = 0 - 1 + 2 = 1
N = 4E = 4V = E - N + 2 = 4 - 4 + 2 = 2
N = 3E = 3V = E - N + 2 = 3 - 3 + 2 = 2
![Page 71: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/71.jpg)
71
Example
1
2,3
6
10
8
4,5
9
7
11
V = 11 – 9 + 2 = 4
Independent paths:1-111-2-3-4-5-10-1-111-2-3-6-8-9-10-1…-111-2-3-6-7-9-10-1…-11
![Page 72: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/72.jpg)
72
Exercise: Cyclomatic Complexity
3
4,5
6
Join
9
7,8
1,2
1
2
3
Join
5
6
1,2
Join
3,4
5 6
Join
7
1, 2
3
Join
4
5
6
![Page 73: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/73.jpg)
73
Linearly Independent Path Coverage
Cyclomatic-number criterion: construct a test set such that all linearly independent paths are covered.
1. read (x)2. read (z)3. if x 0 then {4. y x * z;5. x z; }6. else print ‘Invalid’;7. if z > 1 then8. print z;9. else print ‘Invalid’;
Cyclomatic complexity: 3 (= 9 – 8 + 2)Test Paths:
1, 2, 3, 4, 5, J1, 7, 8, J21, 2, 3, 4, 5, J1, 7, 9, J21, 2, 3, 6, J1, 7, 8, J2,
1,2,3
4,5
Join1
6
7
8
Join2
9
![Page 74: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/74.jpg)
74
Outline
Basic concepts Black box testing White box testing
Control flow graph (CFG) Statement coverage Branch coverage Condition coverage Path coverage Def-use coverage
![Page 75: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/75.jpg)
75
Data Flow Coverage
Consider how variables are treated along the various paths, a.k.a. data flow analysis, e.g., definitions and uses.
Many variations All-defs-uses All-defs All-uses All-C-uses/Some-P-uses All-P-uses/Some-C-uses All-P-uses
* P-uses are predicate uses, e.g., in conditions; all others are C-uses.
![Page 76: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/76.jpg)
76
Def-Use Coverage Def-use coverage: every path
from every definition of every variable to every use of that definition is exercised in some test.
1. read (x);2. read (z);3. if x 0 then {4. y x * z;5. x z; }6. else print ‘Invalid’;7. if y > 1 then8. print y;9. else print ‘Invalid’;
Test Path: 1, 2, 3, 4, 5, 7, 8, J
1,2,3
4,5
Join
6
7
8
Join
9
D: x, zU: x
U: x, z D: y, x
U: none
U: y
U: y U: none
![Page 77: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/77.jpg)
77
Strength of Coverage
Statement
Branch
Def-Use
Path
Arrows point from weaker to stronger coverage.
Stronger coverage requires more test cases.
Condition
![Page 78: 1 Unit Testing CS 4311 Hans Van Vliet, Software Engineering, Principles and Practice, 3 rd edition, John Wiley & Sons, 2008. Chapter 13.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649c7b5503460f9492f301/html5/thumbnails/78.jpg)
78
Limitation of Paths
What they don’t tell you: Timing errors Unanticipated error conditions User interface inconsistency (or anything else) Configuration errors Capacity errors