White-box Testing: When Quality Really Matters
-
Upload
techwellpresentations -
Category
Technology
-
view
135 -
download
1
description
Transcript of White-box Testing: When Quality Really Matters
T14 Test Techniques
5/2/2013 1:30:00 PM
White-box Testing: When Quality Really
Matters
Presented by:
Jamie Mitchell
Jamie Mitchell Consulting, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Jamie Mitchell
Jamie Mitchell has more than thirty years of experience developing and testing both hardware and software. In 1991, Jamie moved from hardware to the dark side - software. He was a pioneer in the test automation field, working with a variety of vendor, open source, and custom-built tools since Windows 3.0. Jamie's specialty is increasing the productivity of automation and test teams through innovative ideas and custom tool extensions. In addition, Jamie provides training, mentoring, and expert technical support in all aspects of testing and automation. Jamie is the coauthor (with Rex Black) of Advanced Software Testing, Volume 3: Guide to the ISTQB Advanced Certification as an Advanced Technical Test Analyst.
4/19/2013
1
Structural TestingStructural TestingWhen Quality Really Matters
Jamie L Mitchell
Jamie Mitchell Consulting Inc.
When Software Really Needs to WorkWhen Software Really Needs to Work
Copyright (c) Jamie Mitchell Consulting Inc. 2
4/19/2013
2
Why StructureWhy Structure--based Testing?based Testing?
� Experience-based techniques are not thorough enough:◦ Error-guessing◦ Exploratory testing◦ Defect- and taxonomy-based
� Black-box testing depends on how good your test basis model is:◦ Requirements◦ Use cases◦ Functional specifications
� How well have you tested?
Is testing less than half of your code sufficient?
3Copyright (c) Jamie Mitchell Consulting Inc.
WhiteWhite--Box Test BasisBox Test Basis
� Design documents and models
◦ Architecture
◦ Functional design
◦ Low-level design
� Object models
� Class interface documents
� Code
Copyright (c) Jamie Mitchell Consulting Inc. 4
4/19/2013
3
Today’s Discussion TargetsToday’s Discussion Targets
� Control-flow analysis: selecting certain paths through the software to test, based on desired coverage
� Data-flow analysis: a control-flow technique that scrutinizes the life-cycle of data variables
Copyright (c) Jamie Mitchell Consulting Inc. 5
ControlControl--flow Testingflow Testing
� Select certain paths through the software
◦ Based on desired level of coverage
� Based on statements
� Based on decisions
� Based on loops
� Based on paths
Copyright (c) Jamie Mitchell Consulting Inc. 6
4/19/2013
4
Statement CoverageStatement Coverage
� Execute every line of code
� Tools available for measurement
◦ Run tool while executing tests
◦ Evaluate coverage
◦ Create more tests to reach areas missed
� Also called instruction or code coverage
Copyright (c) Jamie Mitchell Consulting Inc. 7
Decision CoverageDecision Coverage
� Statement coverage may miss bugs◦ One test needed for statement coverage
(a = 5, b = 4) � Passes
z = 0;
if (a > b) then
z = 12;
x = 72 / z
� Select inputs to force each decision to execute both possible ways (T/F)◦ Now two test cases are needed for coverage
(a = 5, b = 4) � Passes
(a = 4, b = 5) � Finds bug
Copyright (c) Jamie Mitchell Consulting Inc. 8
4/19/2013
5
Condition CoverageCondition Coverage
� Decisions may be arbitrarily complex
if (a > b) then…
if (A || B) && (C == D) || (!E) && (F) then…
� Condition testing requires that each atomic condition must be evaluated both true and false
� Note that condition coverage – by itself – may not be as strong as decision coverage
if (a AND b) then
(a = F, b = T � F)
(a = T, b = F � F)
Copyright (c) Jamie Mitchell Consulting Inc. 9
Decision / Condition CoverageDecision / Condition Coverage
� A combination of Condition and Decision1. Each atomic condition must be evaluated
both ways AND
2. Decision coverage must be satisfied
if (a AND b) then
(a = F, b = T � F)
(a = T, b = F � F)
(a = T, b = T �T)
� Is that enough coverage?
Copyright (c) Jamie Mitchell Consulting Inc. 10
4/19/2013
6
Modified Condition / DecisionModified Condition / Decision
� Usually called MC/DC◦ For each condition in the predicate, there is at least one test where the outcome is TRUE only because of that condition being TRUE◦ For each condition in the predicate, there is at least one test where the outcome is FALSE only because of that condition being FALSE
� Usually results in N+1 test cases where N is number of conditions in predicate
� Required by some safety standards to ensure that enough testing has been done
Ex: FAA DO/178B Level A (catastrophic) software
Copyright (c) Jamie Mitchell Consulting Inc. 11
MC/DC Coverage (2)MC/DC Coverage (2)
if ((a || b) && c) then …
� Decision/Condition can be achieved with
(a = T, b = T, c = T) �T
(a = F, b = F, c = F) � F
� Note that (b) never independently affects the outcome of the condition
Copyright (c) Jamie Mitchell Consulting Inc. 12
4/19/2013
7
MC/DC Coverage (3)MC/DC Coverage (3)
if ((a || b) && c) then …
� Following test set will achieve MC/DC coverage
1. (a = T, b = F, c = T) �T
2. (a = F, b = F, c = T) � F
3. (a = F, b = T, c = T) �T
4. (a = T, b = F, c = F) � F
Copyright (c) Jamie Mitchell Consulting Inc. 13
Coverage Based on LoopsCoverage Based on Loops
� Loops make testing …interesting…
� The number of possible paths may approach infinite depending on how many loops are executed
� Basic coverage criteria
◦ Test when no loops are made
◦ Test when a single loop is made
◦ Test when n loops are made (where n is either the expected max or a large number)
Copyright (c) Jamie Mitchell Consulting Inc. 14
4/19/2013
8
Loops (2)Loops (2)
� This loop can calculate a factorial (n!)
for (i = 1; i <= n; i++)
f *= i;
� Use (n = 0) for zero loops
� Use (n = 1) for one time through
� Determining the max requires thought
◦ In this case, (f) gets very large
◦ Use (n = 12); 13 would cause an overflow
Copyright (c) Jamie Mitchell Consulting Inc. 15
Nested LoopsNested Loops
� Nested loops are really hard to test
� Boris Beizer* had some advice1. Start at the innermost loop, setting all outer loops to
their minimum iteration setting
2. Test boundaries for innermost loop while keeping outer loops at their minimum
3. If you have done outermost loop goto 5. Else move one loop outward and test as in step 2 with all inner loops set to their typical values
4. Continue outward until all loops done
5. Test the boundaries for all loops simultaneously
* Software Testing Techniques, 2nd Ed. by Boris Beizer
Copyright (c) Jamie Mitchell Consulting Inc. 16
4/19/2013
9
Path TestingPath Testing
� Full path testing in a non-trivial system is impossible◦ The number of paths approaches infinite◦ This is due to loops as seen previously
� We can identify independent, non-looping paths which cover all the edges and nodes of a control-flow graph◦ These are called basis paths◦ Testing the basis paths can guarantee decision (and statement) coverage◦ McCabe cyclomatic complexity uses this method◦ Tools are available for calculating this value
Copyright (c) Jamie Mitchell Consulting Inc. 17
Cyclomatic Complexity ExampleCyclomatic Complexity Example
� Complexity = 3
� Tests needed
◦ 1-2-6-7-8
◦ 1-3-4-7-8
◦ 1-3-5-8
Example from “A Complexity Measure” by Thomas McCabe, IEEE Transactions on SW Engineering, Vol. SE-2, No 4, December 1976
Copyright (c) Jamie Mitchell Consulting Inc. 18
4/19/2013
10
DataData--flow Analysis (1)flow Analysis (1)
� Programs create, set, read, evaluate and destroy data
� The life cycle of data can be analyzed to find defects that control-flow techniques miss
� Some defect examples◦ Incorrect assignment of a value◦ Incorrect input statement ◦ Failure to define a variable before use◦ Incorrect path taken due to incorrect value in a control predicate ◦ Use of a variable after it is destroyed or out of scope◦ Redefinition of a variable before it can be used.
� Look for consecutive touches to each variable
Copyright (c) Jamie Mitchell Consulting Inc. 19
DataData--flow Analysis (2)flow Analysis (2)
� Dependent on the language used
� We look for anomalies in data life-cycle ◦ Not all anomalies are defects
◦ While this is nominally a static technique, it will not find all data defects (often data is manipulated or assigned dynamically)
� Sometimes called DUK testing◦ D � Define
◦ U � Use
◦ K � Kill
Copyright (c) Jamie Mitchell Consulting Inc. 20
4/19/2013
11
DataData--flow Testing Combinations (1)flow Testing Combinations (1)
Copyright (c) Jamie Mitchell Consulting Inc. 21
Anomaly Explanation
1. ~d first define Allowed
2. du define-use Allowed, normal case
3. dk define-kill Potential bug; data was never used
4. ~u first use Potential bug; data was used without definition. It may be a global variable, set outside the routine
5. ud use-define Allowed: data used and then re-defined
6. uk use-kill Allowed
7. ~k first kill Potential bug; data is killed before definition
8. ku kill-use Serious defect; data is used after being killed
DataData--flow Testing Combinations (2)flow Testing Combinations (2)
Copyright (c) Jamie Mitchell Consulting Inc. 22
Anomaly Explanation9. kd kill-define Usually allowed. Data is killed and then re-
defined. Some theorists believe this should be disallowed
10. dd define-define Potential bug; double definition
11. uu use-use Allowed; normal case. Some do not bother testing this pair since no redefinition occurred
12. kk kill-kill Potential bug
13. d~ define last Potential bug; dead variable? May be a global variable which is used in other context
14. u~ use last Allowed. Variable was used in this routine but not killed off
15. k~ kill last Allowed; normal case
4/19/2013
12
Function to TestFunction to Test1. public static double calculateBill (int Usage) {
2. double Bill = 0;
3. if(Usage > 0) {
4. Bill = 40;
5.
6. if(Usage > 100) {
7. if(Usage <= 200) {
8. Bill = Bill + (Usage - 100) * 0.5;
9. }
10. else {
11. Bill = Bill + 50 + (Usage - 200) * 0.1;
12. if(Bill >= 100) {
13. Bill = Bill * 0.9;
14. }
15. }
16. }
17. }
18. return Bill;
19. } Copyright (c) Jamie Mitchell Consulting Inc. 23
Variable DUK PatternsVariable DUK Patterns
Copyright (c) Jamie Mitchell Consulting Inc. 24
Pattern Explanation
1. ~d (1) normal case
2. du (1-3) normal case
3. uu (3-6)(6-7)(7-8)(7-11)
normal case
4. uk (6-19)(8-19)(11-19)
normal case
5. k~ (19) normal case
Pattern Explanation
1. ~d (2) normal case
2. dd (2-4) suspicious
3. du (2-18)(4-8)(4-11)(11-12)
normal case
4. ud (8-8)(11-11)(13-13) acceptable
5. uu (12-13)(12-18) normal case
6. uk (18-19) normal case
7. k~ (19) normal case
� Usage data-flow information
� Bill data-flow information
4/19/2013
13
SummarySummary
� Many testers simply use black-box and experience-based techniques
� Some systems require far deeper testing
� This presentation barely touches the many advanced techniques that can be used in testing
� Learning these techniques can give you an advantage in your career
Copyright (c) Jamie Mitchell Consulting Inc. 25