Software Testing Techniques

2

Introduction• Many aspects to achieving software quality

– Formal reviews (of both the software process and the various stages of development), audits, documentation, etc.

– Unit testing– Integration testing– Verification

• Does the module meet the specifications

– Validation• Does the product meet the requirements

3

Introduction

• A Critical element of the Software Quality Assurance

• Represents a critical review of Specifications, Design and Coding

• Destructive rather than Constructive (try to break the system)

• Major objective is to find errors not to show the absence of errors (as distinct from Verification and Validation)

4

Objectives

• A good test case is one that has a high probability of finding an as-yet undiscovered error

• A Successful test is one that uncovers an as-yet undiscovered error

5

Principles• All tests should be traceable to customer

requirements• Tests should be planned long before testing begins• The Pareto principle applies to Testing

– Typically, 80% of the errors come from 20% of the modules

• Testing should begin ‘‘in the small’’ and progress towards “in the large”

• Exhaustive Testing is not possible, but, – if time permits, conduct multiple failure mode testing

• Test plans must receive independent review

6

Testability

• The ease with which a computer program can be tested .

7

Characteristics for Testability

• Operability– the better it works ,

the more efficiently it can be tested

• The system has few bugs• No bugs block the execution of tests• The product evolves in functional stages

8

Characteristics for Testability

• Observability– what you see is what you test

• Distinct output for each input• System states and variables visible during

execution• Past system states and variables are visible• All factors affecting the output are visible• Incorrect output is easily identified• Internal errors are automatically detected and

reported

9

Characteristics for Testability• Controllability

– the better we can control the software ,the more testing can be automated

• All possible outputs can be generated through some combination of input

• All code is executable through some combination of input

• Input and Output formats are consistent and structured

• All sequences of task interaction can be generated• Tests can be conveniently specified and reproduced

10

Characteristics for Testability

• Decomposability– By controlling the scope of testing , isolate

problems and perform smarter retesting

• The Software system is built from independent modules

• Software modules can be tested independently

– While this is very important, it does not obviate the need for integration testing

11

Characteristics for Testability

• Simplicity– the less there is to test ,

the more quickly we can test it

• Functional simplicity• Structural simplicity• Code simplicity

12

Characteristics for Testability

• Stability– the fewer the changes , the

fewer disruptions to testing• Changes are infrequent

• Changes are controlled

• Changes do not invalidate existing tests

• The software recovers well from failures

13

Characteristics for Testability

• Understandability– the more information we have ,

the smarter we will test

• The design is well understood• Dependencies between internal, external and

shared components well understood• Changes to design are well communicated• Technical documentation is instantly accessible,

well-organized, specific and accurate

14

Types of Testing

• White-Box Testing– Knowing the internal workings of a product,

tests are conducted to ensure that “all all gears mesh”gears mesh”

• Black-Box Testing– Knowing the specified function that a

product has been designed to perform , tests are conducted to demonstrate that each function is fully operational (note: this is still different from validation)

15

White Box Testing

• Uses the control structure of the procedural design to derive test cases

• Guarantees that all independent paths within a module have been exercised at least once

• Exercises all loops at their boundaries and within their operational bounds

• Exercises internal data structures to assure their validity - again, at their boundaries and with their operational bounds

16

Control Structure Testing

• Attacks the control flow of the program

• Provides us with a logical complexity measure of a procedural design

• Use this measure as a guide for defining a Basis set of execution paths

• Test cases derived to exercise the Basis set are guaranteed to execute every statement in the program at least once

17

Basis Path Testing• Basis path testing is a white-Box testing

technique first proposed by Tom McCabe.

• The basis path method enables the test case designer to derive a logical complexity measure of a procedural design use this measure as a guide for defining a basis set of execution paths.

• Test cases which derived to exercise, are guaranteed to execute every stmt in the program at least one time during testing.

18

Basis path testing(contd..)

• A Flow Graph notation(Simple notation for representationof control flow)

– represents the control flow of the program – each node in the graph represents one or

more procedural statements– Any procedural design representation can be

translated into a flow graph

19

Flow Graph Notation

20

Basis Path Testing ( contd.)• Flow Graph

1

2

4 3

6 5

7a

7b

8

21

Basis Path Testing ( contd.)

• Cyclomatic Complexity

– Quantitative measure of the complexity of a program

– is the number of independent paths in the basis set of a program

– Upper bound for the number of tests that must be conducted to ensure that all statements have been executed at least once

22

Basis Path Testing ( contd.)

• Cyclomatic Complexity calculation

V (G) = E -N+ 2 = P + 1 = No. of regions in the graph

where E = no. of edges, N = no. of nodes, and P = no. of predicate nodes

• For the previous example

– Independent paths

path 1 : 1 - 8path 2 : 1 - 2

- 3 - 7b - 1 - 8 path 3 : 1 - 2 - 4 - 6 - 7a - 7b - 1 - 8 path 4 : 1- 2 - 4 - 5 - 7a - 7b - 1 - 8

– Cyclomatic complexity = 11 - 9 + 2 = 3 + 1 = 4

23

Basis Path Testing ( contd.)

• Prepare test cases that will force execution of each independent path in the Basis Path set

• Each test case is executed and compared to expected results

24

Example

25

Example

26

Condition Testing

• Exercises all the logical conditions in a module

• Types of possible errors – Boolean variable error– Boolean Parenthesis error– Boolean Operator error– Arithmetic expression error

27

Types of Condition Testing

• Branch Testing – the TRUE and FALSE branches of the

condition and every simple condition in it are tested

• Domain Testing– for every Boolean expression of n

variables , all of 2n possible tests are required

28

Data Flow Testing• Assume functions do not modify their

arguments or global variables. Then define– DEF ( S ) = { X | Statement S contains a

definition of X }– USE ( S ) = { X | Statement S contains a

use of X }– Definition - Use chain ( DU chain )

• [ X , S , S ‘ ] , where X DEF ( S ) and X USE ( S ‘ ) and the definition of X in S is live at S ’

• Every DU chain to be covered at least once

29

Kinds of Loops

30

Loop Testing• Focus is on the validity of loop constructs• Simple loop ( n is the max. no. of allowable passes )

– Skip the loop entirely– Only one pass through the loop– Two passes– m passes , where m < n – n-1 , n , n+1 passes

• Nested loop– Start at innermost loop– Conduct simple loop test for this loop– Move outwards one loop at a time

31

Loop Testing ( contd.)

• Concatenated loops– Multiple simple loop tests if independent– Nested loop approach if dependent

• Unstructured loops– Should be restructured into a combination of

simple and nested loops

32

Black Box Testing• Focus is on the functional requirements of the

software• Uncovers errors such as

– Incorrect or missing functions– Interface errors– Errors in data structures – Performance errors– Initialization and Termination errors

• Unlike White Box Testing , this is performed at later stages of testing

33

• How is functional validity tested?• How is system behavior and performance tested?• What classes of input will make good test cases?• Is the system particularly sensitive to certain input

values?• How are the boundaries of a data class isolated?• What data rates and data volume can the system

tolerate?• What effect will specific combinations of data have on

system operation?

34

Graph Based Testing

• Identify all objects modeled by the software• Identify the relationships that connect these

objects• Create an Object-Relationship graph

– node– node weights– links– link weights

35

Graph Testing ( contd.)

• Example graph

newfile

Document window

Document text

is represented as containsAttributes :

start dimension Background color

text color

allowsediting

of

menu select generates

generation < 1 sec

36

Equivalence Partitioning• It is a Black box testing method that divides,

Input domain of a program into classes of data from which test cases are derived

• Goal is to design a single test case that uncovers classes of errors , thereby reducing the total number of test cases to be developed

• Each class represents a set of valid or invalid states for input conditions

37

Equivalence Partitioning ( contd.)

• Test case design is based on an evaluation of equivalence classes for an input condition– range specified , one valid and two invalid

equivalence classes– requires a specific value , one valid and two invalid

equivalence classes– specifies a member of a set , one valid and one

invalid equivalence classes– is boolean , one valid & one invalid equivalence

class

38

Equivalence Partitioning ( cont. )• Example

Automatic Banking– area code : input condition , boolean

input condition , range [200,999]– prefix : input condition , range >200, no 0’s, < 1000– suffix : input condition , value -- 4 digits– password : input condition , boolean

input condition , value -- 6 char str– command : input condition , set

39

Boundary Value Analysis• Greater number of errors tend to occur at the

boundaries of the input domain• Select test cases that exercise bounding values• Input condition

– range , test cases are just below min and just above max

– set , test cases are minimum and maximum values, if ordered

• The above guidelines are also applied to output conditions– example

• outputs that produce minimum and maximum values in the output range

40

Comparison Testing• Multiple copies of the software are constructed

in case of critical applications– Example: Shuttle Flight Control Software

• Each version is independently built from the specs by different teams

• Test cases derived from other BB Testing techniques are provided as input to each version

• Outputs are compared and versions validated• Not fool proof

41

Other Cases• GUI testing

– See text for partial list of things to test• Client Server

– Often distributed -- complicates testing– Additional emphasis on non-interference among

clients• Documentation

– Use black box techniques• Real-time

– Beyond the scope of this course

42

Summary• Destructive rather than constructive• Main goal is to find errors not to prove the

absence of errors• White Box Testing

– control structure testing– Condition testing– Data flow testing– Loop testing

• Black Box Testing - Functional requirements– Graph based testing – Equivalence partitioning– Boundary Value testing– Comparison testing

43

CLCS Example

Software Vendor Version Platform Facility

IRIX (UNIXoperating system)

Silicon GraphicsIncorporated (SGI)

6.2 SGI Indigo 2,SGI Indy,SGI Challenge

SDE,LCC-X

IRIX (UNIXoperating system)

Silicon GraphicsIncorporated (SGI)

6.3 SGI O2 SDE,LCC-X

VxWorks (GatewayOS)

VxWorks 5.2 SDS Gateway,CS Gateways

SDE,LCC-X

Table 1.1: Juno Baselined COTS Software

44

CLCS Example

data.

Step Description Expected Results1. Turn on SDE1 network hardware and PC’s Blinky lights start blinking on the

network devices, PC’s execute poweron self tests, boots OS

2. Turn on the sde1net workstation, wait for it to finishbooting (login screen will be displayed), then turn onthe sde1boot workstation and wait for it to finishbooting.

POST (Power On Self Test) testsoccur, Operating system startprocedures initiate, Login screensappear.

3. Turn on all remaining HCI workstations and thesde1ddp1 machine.

POST (Power On Self Test) testsoccur, Operating system startprocedures initiate, Login screensappear.

45

CLCS Example

16. Initiate data acquisition at the sde1hci7 workstation.In the Dnav master menu, select “Global Apps”, thenselect “Start receive process”, then select “GW toHCI JUNO_DDP_8”

The System messages windowindicates that the Start receiveprocess is executing, no unexpectederrors are displayed in the consolewindow.

17. Start data display. In the Dnav master menu, select“Shuttle”, then select any of the following:

Wind SpeedWind Direction PAD AWind Direction PAD BTemperature

The command is accepted (as shownin the System messages and consolewindows), the appropriate display(s)are started and are regularly updated.

18. Stop data display at the workstation. Select quit fromdisplay menu(s)

Display windows are closed.

46

Test Results

Number Title Opened

During

Criticality Date

Opened

Current

Status

Juno-11 Telnet from sde1hci1 to

sde1ddp-r failed

System

Test

Major 4/14/97 Open

Juno-12 Remote delog written

into wrong directory

System

Integration

Major 4/15/97 Open

Juno-13 Application displays

CPU intensive

System

Integration

Minor 4/15/97 Open

Juno-14 Telnet to SDS Gateway

not working

System

Test

Minor 4/22/97 Open

Juno-15 Error received when

attempting to start

receive process

System

Test

Major 4/22/97 Open

47

Lessons Learned• The configuration management process

was not complete, multiple baselines of some software components existed.

• A CLCS software sustaining process needs to be defined and implemented as soon as possible.

• Requirements buy-off process needs to be refined.

• Hardware identification could be improved.