Testing

• Testing is a critical process within any software development and maintenance life cycle model.

• Testing is becoming increasingly important to the software development industry because of its implications on the quality of the produced software and the reputation of the producing company.

• Testing activities include testing specifications, designs and implementations.

• Testing implementations is often referred to as software testing, or simply testing.

Test coverage

• Only exhaustive testing can guarantee full coverage.

• Exhaustive testing is impractical since it may take years to complete even for testing small size programs.

• Consider testing the software when a first name text field of 20 characters maximum is filled by a user. Exhaustive testing requires that we test the software for all possible correct and incorrect field input values. – Assume 80 possible correct and incorrect characters, the number of

possible combinations for a valid or invalid first name would exceed 8020. Using a computer that takes, say, 50 nanoseconds to process one combination, we would need about 1011 years to complete the testing for all possible combinations. It is indeed impractical!

The reality

• With the absence of exhaustive testing, there are no formal guarantees that software is error-free.

• In fact, it may happen in practice that even when all errors have been detected during testing, they may not all be fixed due to the lack of time.

• Unfixed errors may be fixed in future releases of the software.

• The difficulty in software testing is the inability to quantify the errors existing in software, and hence predict how many errors are left undiscovered.

Test coverage experiment

• Random testing experiment: errors of different levels of complexity were inserted in the software under test

• All errors except for one were discovered progressively.

• No indication on how much more testing is needed

• Finding errors is an asymptotic nonlinear function of time.

Dynamic and static testing

• Dynamic testing

– Test the software by executing it and applying test inputs

– Observe the output and make a judgment to take a verdict

• Static testing

– Check code without executing it

– Automatic static testing: Code as an input to a static analysis tool

– Non-automatic static testing: review, walkthrough, inspection

Test case

• Test plan includes a test suite

• A test suite is a set of organized test cases

– Organize around use cases for example

• Template to describe a test case

– Id, name, reference, priority, preconditions, preamble, input, expected output, observed output, verdict and remarks

• Test cases are obtained using known testing techniques and strategies

Test case design vs execution

• Test case is developed first – verdict and remarks are empty

• Test case is executed when the actual testing is performed – the verdict is obtained

• Typically test cases are developed (designed) before even writing any code

• Acceptance test cases in the acceptance test plan (ATP) can be developed after requirements specification phase

• Unit and integration test plans can be developed after the design is completed and prior to coding

OOP Development 15

Test Case (TC) Example

TC#: S-221

TC Description: This test is to check the response to an invalid input selection.

TC Precondition: Go to screen Flight_Management

TC Input: Enter <F7>

TC Expected Output: Error message: “Invalid input, Enter “1” to “4” or ESC”

White-box, black-box and grey-box testing techniques

• White-box: module or unit testing

• Black-box: module testing and system testing

Black-box unit/module testing techniques

• Boundary value analysis

• Equivalence class testing

• Decision table / decision tree based testing

• State based testing

• Robustness testing

Boundary value analysis

• BVA: test inputs obtained from the boundaries

– Serious errors hide at the boundaries.

• For example, simple boundary values for an input of integer data type would include -INT_MAX, -INT_MAX + 1, -1, 0, +1, INT_MAX - 1 and INT_MAX.

• For complex and composite data types, boundary values become more difficult to generate and may require more ingenuity to obtain them.

• Moreover, if interface specification includes more than one input, different test cases considering possible feasible combinations of BVs for each input must be considered

BVA Example

• Suppose you are given an executable code including a function called digitCount whose functional interface specification states that digitCount takes an integer n as an input parameter, and returns an integer representing the number of digits in n.

• Test cases whose test inputs are selected based on boundary value analysis are partially shown in Table 8.3.

• Assume that for a 32-bit integer, INT_MAX is +2147483647.

Testing function digitCount based on boundary value analysis

Example 8.2 – Testing search function

Equivalence class testing

• Partitioning input domain into disjoint subsets or partitions

• Partition includes related values

• Consider one test input from each partition

• Equivalence relation among partition elements

– ‘having the same number of digits’

– ‘all positive integers’

– ‘all even numbers’

– ‘all prime numbers’

– ‘all powe or 2 numbers’

– …..

Decision table based testing

• Decision table: each column denoting a rule becomes the basis of a test case

– Conditions produce inputs

– Actions produce expected outputs

– In table 4.1: input variables are age, history and speed, and output variables are penalty and points

Decision tree-based testing

• Each distinct path from the root node to a leaf node is the basis of a test case

• If along the path, a variable input value does not exist, the test input for that variable can be assigned any valid value.

• For example, in fig 4.23, for the test case corresponding to the leftmost path, the speed value can be any illegal speed

State-based testing

• An implemented finite state machine conforms to its specification, if the observed output sequence matches the expected output according to the specification

• Test sequence must cover all states transitions

• Used to test do black box testing of systems

– GUI, protocols, web page connectivity, …

• Transition tour: a sequence covering all transitions and returning back to the initial state

– Can only cover output errors

Black box-based robustness testing

• Test a system using one or preferably more black box based testing method

• Test for incorrect and illegal inputs also

• In equivalence class testing, select one representative bad input from every partition of the bad input values

• For FSM based systems, apply unexpected inputs at each state – the system should survive

• For boundary value testing, test for illegal values beyond the boundaries – the system should survive

White-box testing techniques

• Given the source code of the module to test

• Control flow based testing techniques

• Data flow based testing techniques

• Path testing

Control flow based testing

• Given source code of a module, transform it to a control flowgraph

• Nodes and edges – decision nodes, joining nodes

Example: digitCount

1 int digitCount(int n)

2 {

3 int count = 0;

4 while (n != 0)

5 { n = n / 10;

6 count++;

7 }

8 return count;

9 }

OOP Development 50

White-box testing

• Statement coverage

• Graph based

– Branch coverage

– Condition coverage

– Path coverage

OOP Development 51

Example: Statement Coverage

1 int maxPositive(int n1, int n2) {

2 if (n1<=1 && n2<=1)

3 n1=1;

4 if (n1<=n2)

5 n1=n2;

6 return n1; }

TC Inputs: n1=n2=1

OOP Development 52

Example: Branch Coverage

1 int maxPositive(int n1, int n2) {

2 if (n1<=1 && n2<=1)

3 n1=1;

4 if (n1<=n2)

5 n1=n2;

6 return n1; }

1,2

3

4

5

6

TC#2 Inputs: n1=2, n2=0

TC#1 Inputs: n1=n2=1

OOP Development 53

Example: Condition Coverage

1 int maxPositive(int n1, int n2) {

2 if (n1<=1 && n2<=1)

3 n1=1;

4 if (n1<=n2)

5 n1=n2;

6 return n1; }

TC#1 Inputs: n1=n2=1

TC#2 Inputs: n1=2, n2=0

1,2

3

4

5

6

TC#3 Inputs: n1=0, n2=2

OOP Development 54

Example: Path Coverage

1 int maxPositive(int n1, int n2) {

2 if (n1<=1 && n2<=1)

3 n1=1;

4 if (n1<=n2)

5 n1=n2;

6 return n1; }

TC#1 Inputs: n1=n2=1

TC#2 Inputs: n1=2, n2=0

TC#3 Inputs: n1=0, n2=2

1,2

3

4

5

6

TC#4 Inputs: n1=n2=0

Path expression

• Representing in a generic way an execution sequence

• Can represent the complete program

• Use operators: + for choice

. For a concatenation

{} * for zero or more repetition

{}+ for one or more repetition

() for grouping

Path expression

• Useful to study properties of test sequences

• Annotate the transitions with desirable property then analyze the resulting expression

• For example: assign a probability to each transition – allows you to evaluate the probability of each path; helps you decide on testing the most probable paths

Integration testing

• After testing all modules individually using both white and black box testing techniques

• Top-down integration

• Bottom-up integration

• Sandwich / hybrid integration

Top down integration

• Start with top level modules and integrate left to rigth

• Good for testing GUI heavy modules

• Increase of confidence to see GUI at the beginning

• Less time to test low level utility modules – high reusability modules

• Variation: integrate i/o modules to get real input and outputs as early as possible

• Need stubs – fake modules to represent called modules

Bottom up integration

• Start from low level modules

• Good testing of low level utility modules

• Can see the GUI late in the process

• Require driver modules – to call the integrated modules

Sandwich integration testing

• Two teams – one starting bottom up and the other starting top down

• They meet in the middle

• Good testing of both high level GUI modules and low level utility modules

• Requires coordination and good management to ensure middle level modules also get a good share of the testing time

OOP Development 64

System Testing

• Performed in terms of Inputs and outputs of the system.

• Performed on the targeted platform.

• Goals:

– Reveal bugs that are represented only at system level.

– Demonstrate that the system implements all required capabilities.

– Answer the question: Can we ship it yet?

OOP Development 65

Regression Testing

• Ensuring that modifications have not caused unintended effects.

• Retesting is required