1

COVERAGE CRITERIA FOR TESTING

2

ill-defined terms in Testingcomplete testing exhaustive testingfull coverage Are poorly defined terms because of theoretical limitation of software. The number of potential inputs for most

programs is so large as to be effectively infinite.The only limitation is the size of the file that

can be read by the parser. the number of inputs is effectively infinite

3

Formal Coverage Criteria

We cannot test with all inputCoverage criteria are used to decide which

test inputs to use.Effective use of coverage criteria makes it

more likely Test engineers will find faults in a program

This provides informal assurance

Result : High quality and Reliability of the Software

4

Advantage of Coverage Criteria

According to the practical perspective Coverage criteria provide rules for when to stop testing.

5

Traditional Testing Levels

Class A

method mA1()

method mA2()

Class B

method mB1()

method mB2()

main Class P Acceptance testing : Is

the software acceptable to the user?

Integration testing : Test how modules interact with each other

System testing : Test the overall functionality of the system

Module testing (developer testing) : Test each class, file, module or component

Unit testing (developer testing) : Test each unit (method) individuallyThis view obscures underlying similarities

6

Object-Oriented Testing Levels

Class A

method mA1()

method mA2()

Class B

method mB1()

method mB2()

Intra-class testing : Test an entire class as sequences of calls

Inter-class testing : Test multiple classes together

Inter-method testing : Test pairs of methods in the same class

Intra-method testing : Test each method individually

Coverage Criteria• Even small programs have too many inputs to fully test them all

– private static computeAverage (int A, int B, int C)– On a 32-bit machine, each variable has over 4 billion possible values– Over 80 octillion possible tests!!– Input space might as well be infinite

• Testers search a huge input space– Trying to find the fewest inputs that will find the most problems

• Coverage criteria give structured, practical ways to search the input space– Search the input space thoroughly– Not much overlap in the tests

7

Advantages of Coverage Criteria

• Provide traceability from software artifacts to tests– Source, requirements, design models, …

• Make regression testing easier

• Gives testers a “stopping rule” … when testing is finished

• Can be well supported with powerful tools

8

Test Requirements and CriteriaTest Criterion : A collection of rules and a

process that define test requirements̶� Cover every statement̶� Cover every functional requirement

Test Requirements : Specific things that must be satisfied or covered during testing– Each statement is a test requirement

– Each functional requirement is a test requirement

9

Testing researchers have defined dozens of criteria, but they are all really just a few criteria on four types of structures …

1. Graphs

2. Logic expressions

3. Input domains

4. Syntax descriptions

10

Graph Coverage Criteria

Structural Coverage Criteria : Defined on a graph just in terms of nodes and edgesData Flow Coverage Criteria : Requires a graph to be annotated with references to variables

11

Coverage Criterion

A coverage criterion is simply a recipe for generating test requirements in a systematicway:Coverage Criterion: A coverage criterion is a rule or collection of rules that impose test requirements on a test set.

12

Coverage Criterion

A coverage criterion is a rule or collection of rules that impose test requirements on a test set.That is, the criterion describes the test requirements in a complete and unambiguous manner.

The “flavor criterion” yields a simple strategy for selecting jellybeans. In this case, the set of test requirements, TR, can be formally written out as

TR = {flavor = Lemon, flavor = Pistachio, flavor = Cantaloupe,flavor = Pear, flavor = Tangerine, flavor = Apricot}

13

Coverage

Given a set of test requirements TRfor a coveragecriterion C, a test set T satisfies C if and only if for every test requirement trin TR, at least one test t in T exists such that t satisfies tr .

14

Coverage Level

Given a set of test requirements TR and atest set T, the coverage level is simply the ratio of the number of test requirementssatisfied by T to the size of TR.

Graph CoverageFour Structures for Modeling Software

Graphs Logic Input Space Syntax

Use cases

Specs

Design

Source

Applied to

DNFSpecs

FSMsSource

Applied to

Input

Models

Integ

Source

Applied to

15

16

1

2

0

Node Coverage (Formal Definition): For each node n ∈ reachG(N0), TR contains the predicate “visit n.”

Node Coverage (NC): TR contains each reachable node in G.

Structural Coverage Criteria -I

Node Coverage (NC) (Standard Definition) Test set T satisfies node coverage on graph G if and only if for every syntactically reachable node n in N, there is some path p in path(T) such that p visits n.TR = { 0, 1, 2 }path(t1) = [0, 1 , 2] T1={t1} T1 satisfies node coverage on the graph

17

Structural Coverage Criteria - II Edge coverage is slightly stronger than node coverage Edge Coverage (EC) : TR contains each reachable

path of length up to 1, inclusive, in G.The “length up to 1” allows for graphs with one node and no

edges

Edge Coverage : TR = { (0,1), (0, 2), (1, 2) }path (t1 ) = [ 0,1,2]

path (t2 ) = [ 0, 2 ]

T2={t1, t2} Test Paths = [ 0, 1, 2 ] [ 0, 2 ]

T2 satisfies edge coverage on the graph1

2

0

18

Paths of Length 1 and 0

0

A graph with only one node will not have any edges

• It may be boring, but formally, Edge Coverage needs to require Node Coverage on this graph

Edge Coverage needs to require Node Coverage on this graph

Otherwise, Edge Coverage will not subsume Node CoverageSo we define “length up to 1” instead of simply “length 1”

We have the same issue with graphs that only have one edge for Edge Pair Coverage …

1

0

19

Structural Coverage Example

6

0

2

1

3 4

5

Node CoverageTR = { 0, 1, 2, 3, 4, 5, 6 }Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ]

Edge CoverageTR = { (0,1), (0,2), (1,2), (2,3), (2,4), (3,6), (4,5), (4,6), (5,4) }

Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 2, 4, 5, 4, 6 ]

path (t1 )= [ 0, 1, 2, 3, 6 ] path (t2 )=[ 0, 2, 4, 5, 4, 6 ]T={t1 , t2}