Practical Testing Techniques. Verification and Validation Validation –does the software do what...
-
Upload
isis-bludworth -
Category
Documents
-
view
214 -
download
0
Transcript of Practical Testing Techniques. Verification and Validation Validation –does the software do what...
Verification and Validation
• Validation– does the software do what was wanted?
• “Are we building the right system?”
– This is difficult to determine and involves subjective judgments (reviews, etc.)
• Verification– does the software meet its specification?
• “Are we building the system right?”
– This can only be objective if the specifications are sufficiently precise
– Implications?
• Everything must be verified– …including the verification process itself
Testing
• Testing is a verification technique– critically important for quality software
• Industry averages:– 30-85 errors per 1000 lines of code;– 0.5-3 errors per 1000 lines of code not detected before
delivery.
• The ability to test a system depends on a thorough, competent requirements document.
Goals of Testing
• Goal: show a program meets its specification– But: testing can never be complete for non-
trivial programs
• What is a successful test?– One in which no errors were found?– One in which one or more errors were
found?
Goals of Testing - 2 • Testing should be:
– repeatable• if you find an error, you want to repeat the test to show
others• if you correct an error, you want to repeat the test to check
that you fixed it
– systematic• random testing is not enough• select test sets that are representative of real uses• select test sets that cover the range of behaviors of the
program
– documented• keep track of what tests were performed, and what the
results were
Goals of Testing - 3.
• Therefore you need a way to document test cases showing:– the input– the expected output– the actual result
• These test plans/scripts are critical to project success!
Errors/Bugs
• Errors of all kinds are known as “bugs”.• Bugs come in two main types:
– compile-time (e.g., syntax errors)
which are cheap to fix– run-time (usually logical errors)
which are expensive to fix.
Testing Strategies
• Never possible for designer to anticipate every possible use of system. Systematic testing is therefore essential.
• Offline strategies:1. syntax checking & “lint” testers;
2. walkthroughs (“dry runs”);
3. inspections
• Online strategies:1. black box testing;
2. white box testing.
Syntax Checking
• Detecting errors at compile time is preferable to having them occur at run time!
• Syntax checking will simply determine whether a program “looks” acceptable
• “lint” programs try to do deeper tests on code:– will detect “this line will never be executed”
– “this variable may not be initialized”
• Compilers do a lot of this in the form of “warnings”.
Inspections
• Formal procedure, where a team of programmers read through code, explaining what it does.
• Inspectors play “devils advocate”, trying to find bugs.
• Time consuming process!• Can be divisive/lead to interpersonal
problems.• Often used only for critical code.
Walkthroughs
• Similar to inspections, except that inspectors “mentally execute” the code using simple test data.
• Expensive in terms of human resources.• Impossible for many systems.• Usually used as discussion aid.
• Inspections/walkthroughs typically take 90-120 minutes.
• Can find 30%-70% of errors.
Black Box Testing
• Generate test cases from the specification – i.e. don’t look at the code
• Advantages:– avoids making the same assumptions as
the programmer– test data is independent of the
implementation– results can be interpreted without knowing
implementation details
Consider this Function
function max_element ($in_array){ /* Returns the largest element in
$in_array */ … return $max_elem;}
function max_element ($in_array){ /* Returns the largest element in
$in_array */ … return $max_elem;}
A Test Set
• Is this enough testing?
input output OK?3 16 4 32 9 32 yes9 32 4 16 3 32 yes22 32 59 17 88 1 88 yes1 88 17 59 32 22 88 yes1 3 5 7 9 1 3 5 7 9 yes7 5 3 1 9 7 5 3 1 9 yes9 6 7 11 5 11 yes5 11 7 6 9 11 yes561 13 1024 79 86 222 97 1024 yes97 222 86 79 1024 13 561 1024 yes
“Black Box” Testing
• In black box testing, we ignore the internals of the system, and focus on the relationship between inputs and outputs.
• Exhaustive testing would mean examining system output for every conceivable input.– Clearly not practical for any real system!
• Instead, we use equivalence partitioning and boundary analysis to identify characteristic inputs.
Black Box Testing
• Three ways of selecting test cases:– Paths through the specification
• e.g. choose test cases that cover each part of the preconditions and postconditions
– Boundary conditions• choose test cases that are at or close to boundaries for
ranges of inputs
– Off-nominal cases• choose test cases that try out every type of invalid input
(the program should degrade gracefully, without loss of data)
Equivalence Partitioning
• Suppose system asks for “a number between 100 and 999 inclusive”.
• This gives three equivalence classes of input:– less than 100– 100 to 999– greater than 999
• We thus test the system against characteristic values from each equivalence class.
• Example: 50 (invalid), 500 (valid), 1500 (invalid).
Boundary Analysis
• Arises from the observation that most programs fail at input boundaries.
• Suppose system asks for “a number between 100 and 999 inclusive”.
• The boundaries are 100 and 999.• We therefore test for values:
99 100 101 998 999 1000
lower boundary upper boundary
White (Clear) Box Testing
• In white box testing, we use knowledge of the internal structure to guide development of tests.
• The ideal: examine every possible run of a system.– Not possible in practice!
• Instead: aim to test every statement at least once
White Box Testing
• Examine the code and test all paths
…because black box testing can never guarantee we exercised all the code
• Path completeness:– A test set is path complete if each path
through the code is exercised by at least one case in the test set
White Box Testing - Example
if ($signal > 5) {
echo “hello”;
} else {
echo ‘‘goodbye’’;
}
• There are two possible paths through this code, corresponding to $signal > 5 and $signal <= 5.
• Aim to execute each one.
White Box Testing - Example 2
Consider an extremely simplified version of the code to authorize withdrawals from an ATM:
$balance = get_account_balance($user);
if ($amount_requested <= 300) {
if ($amount_requested < $balance) {
dispense_cash($amount_requested);
} else {
echo “You don’t have enough to withdraw $amount_requested”;
}
} else {
echo “You can not withdraw more than $300”;
}
Mini-exercise: Document the white-box tests you need to create to test this code snippet.
Weaknesses of Path Completeness
• Path completeness is usually infeasible– e.g.
– there are 2100 paths through this program segment (!)
… and even if you test every path, it doesn’t mean your program is correct
for ($j=0, $i=0; $i<100; $i++)if ($a[i]==True) $j=$j+1;
for ($j=0, $i=0; $i<100; $i++)if ($a[i]==True) $j=$j+1;
Loop Testing
Another kind of boundary analysis:
1. skip the loop entirely
2. only one iteration through the loop
3. two iterations through the loop
4. m iterations through the loop (m < n)
5. (n-1), n, and (n+1) iterations
where n is the maximum number of iterations through the loop
Test Planning
• Testing must be taken seriously, and rigorous test plans and test scripts developed.
• These are generated from requirements analysis document (for black box) and program code (for white box).
• Distinguish between:1. unit tests;
2. integration tests;
3. system tests.
Alpha and Beta Testing
• In-house testing is usually called alpha testing.
• For software products, there is usually an additional stage of testing, called beta testing.
• Involves distributing tested code to “beta test sites” (usually prospective/friendly customers) for evaluation and use.
• Typically involves a formal procedure for reporting bugs.
Review: Triangle Classification
Question: How many test cases are enough?
Answer: A lot!• expected cases (one for each type of triangle): (3,4,5), (4,4,5),
(5,5,5)• boundary cases (only just not a triangle): (1,2,3)• off-nominal cases (not valid triangle): (4,5,100)• vary the order of inputs for expected cases: (4,5,4), (5,4,4), …• vary the order of inputs for the boundary case: (1,3,2), (2,1,3),
(2,3,1), (3,2,1), (3,1,2)• vary the order of inputs for the off-nominal case: (100,4,5),
(4,100,5), …• choose two equal parameters for the off-nominal case: (100,4,4), …• bad input: (2, 3, q)• vary the order of bad inputs: …
Documenting Test Cases
• Describe how to test a system/module/function
• Description must identify– short description (optional)– system state before executing the test– function to be tested– input (parameter) values for the test– expected outcome of the test
Test Automation
• Testing is time consuming and repetitive
• Software testing has to be repeated after every change (regression testing)
• Write test drivers that can run automatically and produce a test report
Exercise: Create a Test Plan
• What are you going to test? – functions, features, subsystems, the entire
system• What approach are you going to use?
– white box, black box, in-house, outsourced, inspections, walkthroughs, etc.
– what variant of each will you employ?• When will you test?
– after a new module is added? after a change is made? nightly? ….
– what is your testing schedule?