1

Software EngineeringSoftware EngineeringSoftware Testing TechniquesSoftware Testing Techniques

TestabilityTestability

OperabilityOperability——it operates cleanlyit operates cleanly

ObservabilityObservability——the results of each test case are readilythe results of each test case are readilyobservedobserved

ControllabilityControllability——the degree to which testing can be automatedthe degree to which testing can be automatedand optimizedand optimized

DecomposabilityDecomposability——testing can be targetedtesting can be targeted

SimplicitySimplicity——reduce complex architecture and logic to simplifyreduce complex architecture and logic to simplifyteststests

StabilityStability——few changes are requested during testingfew changes are requested during testing

UnderstandabilityUnderstandability——of the designof the design

What is aWhat is a ““GoodGood”” Test?Test?

A good test has a high probability of findingA good test has a high probability of findingan erroran error

A good test is not redundant.A good test is not redundant.

A good test should beA good test should be ““best of breedbest of breed””

A good test should be neither too simple norA good test should be neither too simple nortoo complextoo complex

Test Case DesignTest Case Design

"Bugs lurk in corners"Bugs lurk in cornersand congregate atand congregate atboundaries ..."boundaries ..."

BorisBoris BeizerBeizer

OBJECTIVEOBJECTIVE

CRITERIACRITERIA

CONSTRAINTCONSTRAINT

to uncover errorsto uncover errors

in a complete mannerin a complete manner

with a minimum of effort and timewith a minimum of effort and time

Exhaustive TestingExhaustive Testing

loop < 20 Xloop < 20 X

There are 10 possible paths! If we execute oneThere are 10 possible paths! If we execute onetest per millisecond, it would take 3,170 years totest per millisecond, it would take 3,170 years totest this program!!test this program!!

1414

Selective TestingSelective Testing

loop < 20 Xloop < 20 X

Selected pathSelected path

Software TestingSoftware Testing

Methods

Strategies

white-boxmethods

black-boxmethods

WhiteWhite--Box TestingBox Testing

... our goal is to ensure that all... our goal is to ensure that allstatements and conditions havestatements and conditions havebeen executed at least once ...been executed at least once ...

Why Cover?Why Cover?

logic errors and incorrect assumptionslogic errors and incorrect assumptionsare inversely proportional to a path'sare inversely proportional to a path'sexecution probabilityexecution probability

we oftenwe oftenbelievebelievethat a path is notthat a path is notlikely to be executed; in fact, reality islikely to be executed; in fact, reality isoften counter intuitiveoften counter intuitive

typographical errors are random; it'stypographical errors are random; it'slikely that untested paths will containlikely that untested paths will containsomesome

Basis Path TestingBasis Path Testing

First, we compute the cyclomaticcomplexity:

number of simple decisions + 1

or

number of enclosed areas + 1

In this case, V(G) = 4

CyclomaticCyclomatic ComplexityComplexity

A number of industry studies have indicatedA number of industry studies have indicatedthat the higher V(G), the higher the probabilitythat the higher V(G), the higher the probabilityor errors.or errors.

V(G)V(G)

modulesmodules

modules in this range aremodules in this range aremore error pronemore error prone

Basis Path TestingBasis Path TestingNext, we derive theNext, we derive theindependent paths:independent paths:

Since V(G) = 4,Since V(G) = 4,there are four pathsthere are four paths

Path 1: 1,2,3,6,7,8Path 1: 1,2,3,6,7,8Path 2: 1,2,3,5,7,8Path 2: 1,2,3,5,7,8Path 3: 1,2,4,7,8Path 3: 1,2,4,7,8Path 4: 1,2,4,7,2,4,...7,8Path 4: 1,2,4,7,2,4,...7,8

Finally, we derive testFinally, we derive testcases to exercise thesecases to exercise thesepaths.paths.

11

22

3344

55 66

77

88

Basis Path Testing NotesBasis Path Testing Notes

you don't need a flow chart,you don't need a flow chart,but the picture will help whenbut the picture will help whenyou trace program pathsyou trace program paths

count each simple logical test,count each simple logical test,compound tests count as 2 orcompound tests count as 2 ormoremore

basis path testing should bebasis path testing should beapplied to critical modulesapplied to critical modules

Graph MatricesGraph Matrices A graph matrix is a square matrix whose sizeA graph matrix is a square matrix whose size

(i.e., number of rows and columns) is equal to(i.e., number of rows and columns) is equal tothe number of nodes on a flow graphthe number of nodes on a flow graph

Each row and column corresponds to anEach row and column corresponds to anidentified node, and matrix entries correspond toidentified node, and matrix entries correspond toconnections (an edge) between nodes.connections (an edge) between nodes.

By adding aBy adding a link weightlink weight to each matrix entry, theto each matrix entry, thegraph matrix can become a powerful tool forgraph matrix can become a powerful tool forevaluating program control structure duringevaluating program control structure duringtestingtesting

Control Structure TestingControl Structure Testing

Condition testingCondition testing —— a test case design method thata test case design method thatexercises the logical conditions contained in a programexercises the logical conditions contained in a programmodulemodule

Data flow testingData flow testing —— selects test paths of a programselects test paths of a programaccording to the locations of definitions and uses ofaccording to the locations of definitions and uses ofvariables in the programvariables in the program

Loop TestingLoop Testing

NestedNestedLoopsLoops

ConcatenatedConcatenatedLoopsLoops UnstructuredUnstructured

LoopsLoops

SimpleSimplelooploop

Loop Testing: Simple LoopsLoop Testing: Simple Loops

Minimum conditionsMinimum conditions——Simple LoopsSimple Loops

1. skip the loop entirely1. skip the loop entirely

2. only one pass through the loop2. only one pass through the loop

3. two passes through the loop3. two passes through the loop

4. m passes through the loop m < n4. m passes through the loop m < n

5. (n5. (n--1), n, and (n+1) passes through1), n, and (n+1) passes throughthe loopthe loop

where n is the maximum numberwhere n is the maximum numberof allowable passesof allowable passes

Loop Testing: Nested LoopsLoop Testing: Nested Loops

Start at the innermost loop. Set all outer loops to theirStart at the innermost loop. Set all outer loops to theirminimum iteration parameter values.minimum iteration parameter values.

Test the min+1, typical, maxTest the min+1, typical, max--1 and max for the1 and max for theinnermost loop, while holding the outer loops at theirinnermost loop, while holding the outer loops at theirminimum values.minimum values.

Move out one loop and set it up as in step 2, holding allMove out one loop and set it up as in step 2, holding allother loops at typical values. Continue this step untilother loops at typical values. Continue this step untilthe outermost loop has been tested.the outermost loop has been tested.

If the loops are independent of one anotherIf the loops are independent of one anotherthen treat each as a simple loopthen treat each as a simple loopelse* treat as nested loopselse* treat as nested loops

endifendif**

for example, the final loop counter value of loop 1 isfor example, the final loop counter value of loop 1 isused to initialize loop 2.used to initialize loop 2.

Nested LoopsNested Loops

Concatenated LoopsConcatenated Loops

BlackBlack--Box TestingBox Testing

requirementsrequirements

eventseventsinputinput

outputoutput

BlackBlack--Box TestingBox Testing

How is functional validity tested?How is functional validity tested?

How is system behavior and performance tested?How is system behavior and performance tested?

What classes of input will make good test cases?What classes of input will make good test cases?

Is the system particularly sensitive to certain inputIs the system particularly sensitive to certain inputvalues?values?

How are the boundaries of a data class isolated?How are the boundaries of a data class isolated?

What data rates and data volume can the systemWhat data rates and data volume can the systemtolerate?tolerate?

What effect will specific combinations of data haveWhat effect will specific combinations of data haveon system operation?on system operation?

GraphGraph--Based MethodsBased Methods

new

file

menu select generates

(generation time 1.0 sec)

document

window

document

text

is represented as

contains

Attributes:

background color: whitetext color: default color

or preferences

(b)

object#1

Directed link

(link weight)

object#2

object

#3

Undirected link

Parallel links

Node weight(value

)

(a)

allows editingof

To understand theTo understand theobjects that areobjects that aremodeled inmodeled insoftware and thesoftware and therelationships thatrelationships thatconnect theseconnect theseobjectsobjects

In this context, weIn this context, weconsider the termconsider the term““objectsobjects”” in the broadestin the broadestpossible context. Itpossible context. Itencompasses dataencompasses dataobjects, traditionalobjects, traditionalcomponents (modules),components (modules),and objectand object--orientedorientedelements of computerelements of computersoftware.software.

Equivalence PartitioningEquivalence Partitioning

useruserqueriesqueries

mousemousepickspicks

outputoutputformatsformats

promptsprompts

FKFKinputinput

datadata

Sample EquivalenceSample EquivalenceClassesClasses

user supplied commandsuser supplied commands

responses to system promptsresponses to system prompts

file namesfile namescomputational datacomputational data

physical parametersphysical parametersbounding valuesbounding valuesinitiation valuesinitiation values

output data formattingoutput data formattingresponses to error messagesresponses to error messagesgraphical data (e.g., mouse picks)graphical data (e.g., mouse picks)

data outside bounds of the programdata outside bounds of the programphysically impossible dataphysically impossible dataproper value supplied in wrong placeproper value supplied in wrong place

Valid dataValid data

Invalid dataInvalid data

Boundary Value AnalysisBoundary Value Analysis

useruserqueriesqueries

mousemousepickspicks

outputoutputformatsformats

promptsprompts

FKFKinputinput

datadata

outputoutputdomaindomaininput domaininput domain

Comparison TestingComparison Testing

Used only in situations in which the reliability of softwareUsed only in situations in which the reliability of softwareis absolutely critical (e.g., humanis absolutely critical (e.g., human--rated systems)rated systems)

Separate software engineering teams develop independentSeparate software engineering teams develop independentversions of an application using the same specificationversions of an application using the same specification

Each version can be tested with the same test data to ensureEach version can be tested with the same test data to ensurethat all provide identical outputthat all provide identical output

Then all versions are executed in parallel with realThen all versions are executed in parallel with real--timetimecomparison of results to ensure consistencycomparison of results to ensure consistency

Orthogonal Array TestingOrthogonal Array Testing

Used when the number of input parameters is small andthe values that each of the parameters may take areclearly bounded

One input item at a time L9 orthogonal array

XY

Z

XY

Z

OOTOOT——Test Case DesignTest Case Design

BerardBerard [BER93] proposes the following approach:[BER93] proposes the following approach:

1.1. Each test case should be uniquely identified and should be expliEach test case should be uniquely identified and should be explicitlycitlyassociated with the class to be tested,associated with the class to be tested,

2.2. The purpose of the test should be stated,The purpose of the test should be stated,

3.3. A list of testing steps should be developed for each test and shA list of testing steps should be developed for each test and shouldouldcontain [BER94]:contain [BER94]:

a.a. a list of specified states for the object that is to be testeda list of specified states for the object that is to be tested

b.b. a list of messages and operations that will be exercised asa list of messages and operations that will be exercised asa consequence of the testa consequence of the test

c.c. a list of exceptions that may occur as the object is testeda list of exceptions that may occur as the object is tested

d.d. a list of external conditions (i.e., changes in the environmenta list of external conditions (i.e., changes in the environment externalexternalto the software that must exist in order to properly conduct theto the software that must exist in order to properly conduct the test)test)

e.e. supplementary information that will aid in understanding orsupplementary information that will aid in understanding orimplementing the test.implementing the test.

Testing MethodsTesting Methods

FaultFault--based testingbased testing

The tester looks for plausible faults (i.e., aspects of the impThe tester looks for plausible faults (i.e., aspects of the implementation of thelementation of thesystem that may result in defects). To determine whether these fsystem that may result in defects). To determine whether these faults exist, testaults exist, testcases are designed to exercise the design or code.cases are designed to exercise the design or code.

Class Testing and the Class HierarchyClass Testing and the Class Hierarchy

Inheritance does not obviate the need for thorough testing of alInheritance does not obviate the need for thorough testing of all derived classes.l derived classes.In fact, it can actually complicate the testing process.In fact, it can actually complicate the testing process.

ScenarioScenario--Based Test DesignBased Test Design

ScenarioScenario--based testing concentrates on what the user does, not what thebased testing concentrates on what the user does, not what theproduct does. This means capturing the tasks (via useproduct does. This means capturing the tasks (via use--cases) that the user hascases) that the user hasto perform, then applying them and their variants as tests.to perform, then applying them and their variants as tests.

OOT Methods: Random TestingOOT Methods: Random Testing

Random testingRandom testing

identify operations applicable to a classidentify operations applicable to a class

define constraints on their usedefine constraints on their use

identify aidentify a miminummiminum test sequencetest sequence

an operation sequence that defines the minimum lifean operation sequence that defines the minimum lifehistory of the class (object)history of the class (object)

generate a variety of random (but valid) test sequencesgenerate a variety of random (but valid) test sequences

exercise other (more complex) class instance lifeexercise other (more complex) class instance lifehistorieshistories

OOT Methods: Partition TestingOOT Methods: Partition Testing

Partition TestingPartition Testing reduces the number of test cases required to test a class inreduces the number of test cases required to test a class in

much the same way as equivalence partitioning formuch the same way as equivalence partitioning forconventional softwareconventional software

statestate--based partitioningbased partitioning categorize and test operations based on their ability to changecategorize and test operations based on their ability to change

the state of a classthe state of a class

attributeattribute--based partitioningbased partitioning categorize and test operations based on the attributes that theycategorize and test operations based on the attributes that they

useuse

categorycategory--based partitioningbased partitioning categorize and test operations based on the generic functioncategorize and test operations based on the generic function

each performseach performs

OOT Methods: InterOOT Methods: Inter--Class TestingClass Testing

InterInter--class testingclass testing For each client class, use the list of class operators toFor each client class, use the list of class operators to

generate a series of random test sequences. The operatorsgenerate a series of random test sequences. The operatorswill send messages to other server classes.will send messages to other server classes.

For each message that is generated, determine theFor each message that is generated, determine thecollaborator class and the corresponding operator in thecollaborator class and the corresponding operator in theserver object.server object.

For each operator in the server object (that has been invokedFor each operator in the server object (that has been invokedby messages sent from the client object), determine theby messages sent from the client object), determine themessages that it transmits.messages that it transmits.

For each of the messages, determine the next level ofFor each of the messages, determine the next level ofoperators that are invoked and incorporate these into the testoperators that are invoked and incorporate these into the testsequencesequence

OOT Methods: Behavior TestingOOT Methods: Behavior Testing

emptyacctopen setupAccnt

set upacct

deposit(initial)

workingacct

withdrawal(final)

deadacct close

nonworkingacct

deposit

withdrawbalance

creditaccntInfo

Figure 14.3 St at e diagram f or Account class (adapt ed f rom [ KIR94] )

The tests to beThe tests to bedesigned shoulddesigned shouldachieve all stateachieve all statecoverage [KIR94].coverage [KIR94].That is, theThat is, theoperationoperationsequences shouldsequences shouldcause thecause theAccount class toAccount class tomake transitionmake transitionthrough allthrough allallowable statesallowable states

Testing PatternsTesting Patterns

Pattern name:Pattern name: pair testingpair testing

Abstract:Abstract: A processA process--oriented pattern, pair testing describes a technique that isoriented pattern, pair testing describes a technique that isanalogous to pair programming (Chapter 4) in which two testers wanalogous to pair programming (Chapter 4) in which two testers work togetherork togetherto design and execute a series of tests that can be applied to uto design and execute a series of tests that can be applied to unit, integrationnit, integrationor validation testing activities.or validation testing activities.

Pattern name:Pattern name: separate test interfaceseparate test interface

Abstract:Abstract: There is a need to test every class in an objectThere is a need to test every class in an object--oriented system,oriented system,includingincluding ““internal classesinternal classes”” (i.e., classes that do not expose any interface(i.e., classes that do not expose any interfaceoutside of the component that used them). The separate test inteoutside of the component that used them). The separate test interface patternrface patterndescribes how to createdescribes how to create ““a test interface that can be used to describe specifica test interface that can be used to describe specifictests on classes that are visible only internally to a componenttests on classes that are visible only internally to a component..”” [LAN01][LAN01]

Pattern name:Pattern name: scenario testingscenario testingAbstract:Abstract: Once unit and integration tests have been conducted, there is aOnce unit and integration tests have been conducted, there is a needneedto determine whether the software will perform in a manner thatto determine whether the software will perform in a manner that satisfies users.satisfies users.The scenario testing pattern describes a technique for exercisinThe scenario testing pattern describes a technique for exercising the softwareg the softwarefrom the userfrom the user’’s point of view. A failure at this level indicates that the softs point of view. A failure at this level indicates that the softwarewarehas failed to meet a user visible requirement. [KAN01]has failed to meet a user visible requirement. [KAN01]