Post on 14-Dec-2015
Software Reliability
Risks of faulty software
Example: – Therak 25, – AT&T network failure– Airport traffic control
Costs of software errors can be huge– Y2K problem detection, fix, litigation
Testing
Testing is the process of searching for errors, with the goal of finding as many as possible so that they can be turned over to those responsible for fixing them
Types of errors
Syntax errors - typos, easy to fix Validity errors - wrong result Verification errors - undetected bad input Run-time errors - General Protection Fault Maintenance errors
– one fix produces another problem
Cost to repair errors – tied to ease of identification
Two approaches to testing
Black-box testing– Only inputs and outputs are tested– Code is ignored (black box)
Glass-box testing– Testing based on enumerated special cases
and anticipated possible inputs– Must examine code to see what conditions it
expects to see
Unit vs. System Testing
Design->implementation-> unit testing-> system testing
Unit testing– Testing individual components of the system– Easy to isolate bugs this way– Based on the program design
System testing– After units check out do a ‘system build’– Based on the program specifications
Unit testing
Unit testing handles the individual components one at a time.
In OOD the units to be tested are the various classes
It is important to catch bugs at this level, rather than later on after the system has been integrated.
System integration
After unit testing is finished you are ready for a ‘system build’
You are looking for errors in the overall operation of the system.
Follow the specifications that preceded the design.
Acceptance testing
This form of system testing is overseen by the client to insure that the program meets their specs.
Functional testing - testing to demonstrate that the function of each specification is met.
System testing components
Look at specifications for the end product Some errors may require redesign Functional testing
– Specifying required functions which must work
Beta testing– Software release to ‘guinea pigs’
Test plans
Used by trained testing teamsScientific method
– (hypothesis verification)Test cases
– 1. Typical ones– 2. Extreme cases– 3. Invalid input
Example
Specifications:– Write a program called Max that will
read in a sequence of integers and print out the largest one entered.
A test plan for Max should handle all types of errors.
Test plan for Max program
Test Input Output Type
1 2 38 2 938 20 47 2 -109 0 38 938 1
2 100 29 28 10 4 -3 84 17 -9 100 2
3 938 2847 17273 2867 238 19288 19288 2
4 -1 -2 -3 -4 -5 -6 -7 -1 2
5 1 1 1 1 1 1 1 1 1 1 1 2
6 2.5 0.9 1 2 3 error 3
7 error 3
Unit testing techniques
Drivers– Small programs designed to test a function– A driver program calls the function with a
set of test data, and checks to see if the function processes it correctly.
– A suitable value is returned by the function, but the driver program really does not need it
Test driver for max function(Part 1 of 2)
#include <iostream.h> int max(int a[], int n); // defined in cx3-3.cpp; link with the driver
int main() { int a[100], i; cout << "Max driver” << endl; cout << "Enter numbers terminated by -9999”; cout << endl; cout << "Length of input must be <= 100”; cout << endl;
Test driver for max function(Part 2 of 2)
for (i = 0; i < 100; i++) { int val; cin >> val; if (val == -9999) // sentinel break; else a[i] = val; } cout << "\nMax is " << max(a, i) << endl; cout << endl << endl; return 0;}
Stubs
A stub is– a function that does nothing more than give
calling routines what they ask for.
– Used to test a function that depends on other functions that are not yet available.
Sample stub for function median
#include <iostream.h> int median(int a[], int n) { cout << "function median called with n = " << n <<; cout << “a[] = "; int i; for (i = 0; i < n; i++) cout << a[i] << '\t'; cout << "\nType in a value to return:"; int return_value; cin >> return_value; return return_value; }
Alternatives to testing
The problem with testing is that you cannot test all possible situations.
So you can never ‘prove’ that the program works.
A better way of testing would be using a mathematical proof of the programs’ correctness
Problems with proofs
However, proving that a program will perform correctly is not the same as a guarantee that it will do what you want it to.
We can show that the program specifications are being met, but we cannot show that a specification itself is correct.
Example
We could show that a vending machine program meets the spec. of dispensing the fewest number of coins possible as change.
We cannot show that this specification is the correct strategy (I.e. postal machines that give you Susan B. Anthony dollars)
Program Correctness Techniques
Proving program correctness is a primary goal of computer science, but a very broad topic
There is one aspect we can use.Assertions
– Precise statements of the behavior of the program at a particular spot.
Assertions
An assertion is a statement of the specifications of a contract.
The program can be evaluated to see if it lives up to the terms of the contract at any given point.
There are many key types of assertions
Assertions
Preconditions– Statements about what we expect to be
true before the function begins Loop invariants
– Statements about conditions that are true for each iteration of the loop
Postconditions– What we expect the result of the function to
be
Proof of correctness
A proof of correctness uses the loop invariant to show that– given a certain set of preconditions– the invariant guarantees that the
postcondition will always be met.
Pre and post conditions
These are special assertions that describe how the program performs.
“If when function f is called A is true, then, when function f returns, B will be true.”
Example
If when the max function is called, it is true that ‘a’ is an array ranging from 0 to n-1, then, when max returns, the value returned will be equal to the largest value in ‘a’.
Contracts
Preconditions and postconditions are contracts between client code and the function.
If the preconditions are met by the client, then the function guarantees the specified result upon its return.
Broken contracts
The program has a bug in it if it cannot fulfill its contracts
One common way of breaking a contract is for client code to fail to meet the precondition
This happens whenever there is failure to correctly exchange data through the function interface.
Loop invariants
Loop invariants are statements that are true at the beginning and ending of each iteration of the loop.
Loops are another major source of broken contracts– they do not terminate at the right time– they do not terminate at all– they do not execute
Code Example Function max
int max (int a[], int n) { // assertion 1: THE PRECONDITION
// a is an array with subscripts from 0 to n-1 int max_val(a[0]), i; for (i = 1; i < n; i++) // assertion 2: THE LOOP INVARIANT
// (max_val >= a[k] for 0 <= k < i) and // (max_val = a[j] for some j, 0 <= j < i) if (max_val < a[i]) max_val = a[i]; // assertion 3: THE POST CONDITION
// (max_val >= a[k] for 0 <= k < n) and // (max_val == a[j] for some j,0 <= j< n) // i.e., max_val is the value of largest int in a return max_val; }
Assertions are like contracts
Guarantees between program segments
Function interfaces are prime spots for bugs– General Protection Faults (Turbo)– Segmentation faults (unix)
Assert.h
#include <iostream.h> #include <assert.h> int max (int a[], int); // See next slide int main() { int a[] = { 45,21,76,34,62,58,92,34,10,98 }; int n = 10; cout << "The largest is: " << max(a,n) << endl; return 0; }
Example: assertions
int max (int a[], int n) {// assertion 1: a is an array with subscripts ranging from 0 to n-1 assert (0 < n); // n is positive int max_val(a[0]), i; for (i = 1; i < n; i++) { if (max_val < a[i]) max_val = a[i]; // assertion 2: (max_val >= a[k] for 0 <= k < i) and // (max_val = a[j] for some j, 0 <= j < i) assert (max_val >= a[i]); } // assertion 3: (max_val >= a[k] for 0 <= k < n) and // (max_val == a[j] for some j, 0 <= j < n) // i.e., max_val is equal the value of largest int in array a return max_val; }
Loop invariants
Anything that is true for every iteration is a loop invariant
There are many possible invariantsi == i is true but uninterestingi < n is also true but uninteresting,
even though it is the termination condition
Important invariants
Operant condition– (max_val >= a[k] for 0 <= k < i) and– (max_val == a[j] for some j, 0 <= j < i)
After the loop is over we can substitute n for the value of i– (max_val >= a[k] for 0 <= k < n) and– (max_val == a[j] for some j, 0 <= j < n)
Now we have the post condition!
Establishing a loop invariant
We must show that two things are true– The invariant is true the first time the
loop is entered.– If it is true for one interation it is true
for the next as well.
The invariant is true the first time
When we enter the loop i is 1, therefore we have:– (max_val >= a[k] for 0 <= k < 1) and– (max_val == a[j] for some j, 0 <= j < 1)
This is true since k and j are 0
The invariant is true for each successive time
For all i, a[i] is either greater that max_val or not
If it is, max_val becomes a[i]If it is not, max_val stays unaltered
Illustrated loop invariant for function max
max_val >= everything in here this part is unknown
i
Proving termination
How do we know the loop will end?Because i starts out less than n and is
incremented in each iteration - it will eventually get to n no matter what n is.
Steps for analyzing a loop
Establish the preconditionFind the invariantEstablish the postconditionProve termination
sum function
int sum(int a[], int n) { // Precondition: a is an array with // subscripts from 0 to n-1
int i; total(0); for (i = 0; i < n; i++) // Loop invariant:
// total = a[0] + a[1] + … + a[i] total += a[i]; // Postcondition: // total = a[0] + a[1] + … + a[n-1]
return total; }
Example
The insertion sort
Basic idea of Insertion Sort
7 11 17 26 33
7 11 17 26 3323
23
Unprocessed Items
Insertion Sort
void insertNextItem(int a[], int i); void insertionSort(int a[], int n) {// Precondition:
// a is an array with subscripts from 0 to n-1 for (int i = 1; i < n; i++) // Loop invariant: // items in range from 0 to i-1 are sorted;
// items from i to n-1 haven’t been examined. insertNextItem(a, i); // Postcondition: array a is sorted }
InsertNextItem
void insertNextItem(int a[], int i) {// Precondition: array a is sorted from 0 to i-1 int newItem(a[i]), insertPos(i); for (;insertPos && newItem < a[insertPos-
1];insertPos--) // Loop Invariant: newItem <= a[insertPos+1] .. a[i] && // a[insertPos+1] .. a[i] are sorted a[insertPos] = a[insertPos-1]; // slide item right a[insertPos] = newItem; // Postcondition: array a is sorted from 0 to i }
Proof of termination
Two ways to exit the loop– insertPos <= 0– newItem < a[insertPos - 1]
We must show that one of these eventually happens
If insertPos starts out greater than 0 then the first will always be met since we subtract 1 each iteration.