Software Testing
-
Upload
kadeem-mullen -
Category
Documents
-
view
45 -
download
0
description
Transcript of Software Testing
Software Testing
Sudipto GhoshCS 406 Fall 99November 9, 1999
11/09/99 CS 406 Testing 2
Learning objectives
• Functional testing• Equivalence class partitioning• Boundary value analysis
• Structural testing• Control flow-based
11/09/99 CS 406 Testing 3
Testing for correctness
• Identify the input domain of P.• Execute P against each element of the
input domain.• For each execution of P, check if P
generates the correct output as per its specification S.
11/09/99 CS 406 Testing 4
Input domain
• The set of all valid inputs that a program P can expect is the input domain of P.
• The size of an input domain is the number of elements in it.
• An input domain could be finite or infinite.• Finite input domains may be large!
11/09/99 CS 406 Testing 5
Identifying input domains
• For the sort program:• N: size of the sequence, N < 3• Sequence of N integers, such that for any
integer K, 0 K (e –1), e = 3• K: each element in the sequence• Some sequences are:
• [ ] : An empty sequence (N = 0)• [0] : A sequence of size 1 (N = 1)• [2 1] : A sequence of size 2 (N = 2)
11/09/99 CS 406 Testing 6
Size of input domain
• Suppose that 0 N 106
• The size of the input domain is the number of all sequences of size 0, 1, 2…
• The size is computed as:
• To test for correctness, P needs to be tested on all inputs.
• How many years would that take?
610
0i
ie
11/09/99 CS 406 Testing 7
Exhaustive testing
• Previous example• P is executed on all elements in the input
domain.• For most programs, exhaustive testing is
not feasible• What is the alternative?
11/09/99 CS 406 Testing 8
Partition testing
• Input domain is partitioned into a finite number of sub-domains.
• P is then executed on a few elements in each sub-domain.
• Four sub-domains for our
example:
[], [2], [2 0], [2 1 0]
Reduced from 85 to 4 !
N=01
N=12
N=34
N=23
11/09/99 CS 406 Testing 9
Confidence in your program
• Confidence is a measure of one’s belief in the correctness of the program
• Correctness is not measured in binary terms: a correct or incorrect program
• Measured as a probability of correct operation of a program when used in various scenarios• Reliability• Test completeness: Extent to which a program has
been tested and errors have been found and removed
11/09/99 CS 406 Testing 10
Types of module testing
• “Informal testing” performed by programmer while developing module
• Methodical testing performed by SQA group• Nonexecution-based testing• Execution-based testing
• Haphazard data as input• Systematically constructed test cases
• Testing to specifications
• Testing to code
11/09/99 CS 406 Testing 11
Testing to specifications
• Black-box testing• Data-driven testing• Input/output-driven testing• Functional testing• Code is ignored, i.e. the internal structure
of the code is not important• The specification document is the only
information used for creating test cases
11/09/99 CS 406 Testing 12
Testing to code
• White-box testing• Glass-box testing• Logic-driven testing• Path-oriented testing• Specifications are not used to generate
the test cases• Test “to code,” i.e. the internal structure of
the code is important
11/09/99 CS 406 Testing 13
What is functional testing?
• Functional testing tests how well a program meets the functionality requirements.
• When test inputs are generated using program specifications, we say that we are doing functional testing.
11/09/99 CS 406 Testing 14
Feasibility of functional testing
• Problem with exhaustive testing• Very large (possibly infinite) number of test
cases.
• Need to devise:• Small, manageable set of test cases.• Maximize the chances of detecting faults.• Minimize the chances of wasting test cases by
having more than one test case detecting the same fault.
11/09/99 CS 406 Testing 15
Equivalence class partitioning
• Divide the domain of all inputs into a set of equivalence classes.
• If any test in an equivalence class succeeds, then every test in that class will succeed.
• How would you get ideal equivalence classes?• Difficult without looking at the internal structure• Difficult even with the internal structure
11/09/99 CS 406 Testing 16
Equivalence class partitioning
Set of all inputs Specifications
Put those inputs together for which the behaviorpattern of the module is specified to be different
into similar groups
Equivalence classes
11/09/99 CS 406 Testing 17
Rationale behind equivalence class partitioning
We assume that if the specifications require exactly the same behavior from each element in a class of values, then the program is likely to be constructed so that it either succeeds or fails for each of the values in that class.
11/09/99 CS 406 Testing 18
Guidelines for partitioning
• For robust software, we must test for incorrect inputs too.
• For each equivalence class of valid inputs, we have equivalence classes of invalid inputs.• Input condition specifies a range a X b.
• a X b (Valid case)• X < a, and X > b (invalid cases)
• Input specifies a value• create one for the valid value• create two for incorrect (one above, one below)
11/09/99 CS 406 Testing 19
Guidelines for partitioning
• Input specifies a value (contd.)• For boolean, only one value
• Input condition specifies a member of a set• Create one for the valid value• Create one for invalid value (not in the set)
• Example:• Factorial (n), where n 0• Valid classes {0}, {x | x 1}• Invalid class {x | x < 0}
11/09/99 CS 406 Testing 20
Non-overlapping partitions
• In the previous example, the equivalence classes were non-overlapping, i.e., the sub-domains were disjoint.
• It is sufficient to pick one test from each equivalence class to test the program.
• An equivalence class is considered covered if at least one test has been selected from it.
• Our goal is to cover all equivalence classes.
11/09/99 CS 406 Testing 21
Overlapping partitions
• Suppose a program P takes three integers, X, Y and Z. It is known that:• X < Y• Z > Y
Z > Y
X < YZ > Y
X < Y
X YZ Y
11/09/99 CS 406 Testing 22
Overlapping partition:test selection
• Select 4 test sets as• X=4, Y=7, Z=1 (satisfies X < Y)• X=4, Y=2, Z=1 (satisfies X Y)• X=1, Y=7, Z=9 (satisfies Z > Y)• X=1, Y=7, Z=2 (satisfies Z Y)
• Can also reduce the number of test cases to 2• X=4, Y=7, Z=1 (satisfies X < Y and Z Y)• X=4, Y=2, Z=3 (satisfies X Y and Z > Y)
11/09/99 CS 406 Testing 23
Boundary Value Analysis (BVA)
• Observation:• Programs that work correctly for a set of
values in an equivalence class, fail on some special values.
• These values lie on the boundary of the equivalence class.
• Choose an input from a test case from an equivalence class such that the input lies at the edge of the equivalence class.
• These test cases are “extreme cases”
11/09/99 CS 406 Testing 24
BVA examples
• Suppose the range is 0.0 X 1.0.• 0.0 (valid input)• 1.0 (valid input)• -0.1 (invalid input)• 1.1 (invalid input)
• For a list• first and last element of the list
• A program takes a string S and integer X as input such that • a X b, and length(S) 100. Derive tests.
11/09/99 CS 406 Testing 25
BVA analysis
• A program takes two integers X and Y, such that a X b, c Y d.
• How many test cases do we get?
a b
c
d
11/09/99 CS 406 Testing 26
Output variables
• Equivalence class partitioning can be applied to output variables
• BVA also can be applied to output data
11/09/99 CS 406 Testing 27
Testing functions
• Identify each function implemented in a module
• Devise test data to test each function separately
• Module may contain a hierarchy of lower level functions connected by program control structures• Perform functional testing recursively• Higher level: black-box• Lower level: glass-box
11/09/99 CS 406 Testing 28
Problems with previous approach
• Usually higher level functions are not as structured• Lower level functions are intertwined
• Functionality does not coincide with module boundaries• Distinction between module testing and
integration testing is blurred• Testing one module is not possible without
simultaneously testing other modules whose functionality is used
11/09/99 CS 406 Testing 29
Structural testing
• Intent is to exercise the different programming structures and data structures used in the program
• Intent is not to exercise all the different input and output conditions• though this may be a by-product
• Achieve test cases that force the desired “coverage” of different structures
• Criteria are formal and precise
11/09/99 CS 406 Testing 30
Control-flow based criteria
• Statement coverage• Run a series of test cases and ensure that
every statement is executed at least once• Simplest form of glass box testing• What is the weakness? Consider the example:
int abs (int x) {
if (x >= 0)
x = 0 - x;
return x;
}
Error Test inputs:
x = 5: What is coverage?x = 0: What is coverage?
11/09/99 CS 406 Testing 31
Statement coverage
• Not very strong, may leave errors undetected.
• Examples:• if statement without else part• conjunctions of predicates
• In all these cases, all branches were probably not exercised.
• Can you think of a better criterion based on the above observation?
11/09/99 CS 406 Testing 32
Branch coverage
• Require that every decision is evaluated to true and false values at least once during testing
• Branch coverage implies statement coverage• Each statement is part of some branch
11/09/99 CS 406 Testing 33
Control flow graph of a program
• Let G be the graph of a program P.• Node:
• Represents a block of statements that are always executed together
• Edge (i, j) from node i to node j:• Represents a possible transfer of control after
executing the last statement of the block represented by node i to the first statement in the block represented by node j.
11/09/99 CS 406 Testing 34
Control flow graph of a program
• Start node:• Node corresponding to a block whose first
statement is the start statement of P.
• Exit node:• Node corresponding to a block whose last
statement is an exit statement of P.
• Path:• Finite sequence of nodes (n1, n2, …, nk), k>1
such that there is an edge (ni, ni+1) for all nodes ni in the sequence, except the last node nk.
11/09/99 CS 406 Testing 35
Control flow graph of a program
• Complete Path:• Path whose first node is a start node and the
last node is an exit node.
• All-nodes criterion (statement coverage)• All-edges criterion (branch coverage)
11/09/99 CS 406 Testing 36
An example (xy)
1. scanf(x, y); if(y < 0)
2. pow = 0 – y;
3. else pow = y;
4. z = 1.0;
5. while(pow != 0)
6. { z = z * x; pow = pow – 1; }
7. if ( y < 0 )
8. z = 1.0/z;
9. printf(z);
11/09/99 CS 406 Testing 37
Control Flow Graph of example
1
2 3
6
4
5
7
8 9
11/09/99 CS 406 Testing 38
Problems with branch coverage
• What if a decision has many conditions (using and, or)
• Decision may evaluate to true or false without actually exercising all the conditionsint check (int x) {
if ((x >= 5) && (x <= 200))
return TRUE;
return FALSE;
}
Error (should be 100)
Test inputs:x = 5:x = -5:
11/09/99 CS 406 Testing 39
Solution?
• Require all individual conditions to evaluate to true and false
• Problem:• Even if individual conditions evaluate to true
and false, the decision may not get both true and false values
• Solution:• Require both decision / condition coverage!!
• Still there is a problem.
11/09/99 CS 406 Testing 40
Path testing
• Some errors are related to some combinations of branches.
• Presence revealed by an execution of a path that includes those branches.
• Solution:• Require all possible paths in the CFG to be
executed during testing.• Path-coverage criterion, or all-paths criterion• Path coverage implies branch coverage