1

Software Testing & Quality Assurance

Lecture 14

Created by: Paulo AlencarModified by: Frank Xu

2

OverviewStructural Testing

Introduction – General ConceptsFlow Graph TestingData Flow Testing

DefinitionsSome Basic Data Flow Analysis AlgorithmsDefine/use TestingSlice Based TestingGuidelines and Observations

Hybrid MethodsRetrospective on Structural Testing

3

Issues on Structural Testing

• An important question relates to the point where we stop to test and release the system for beta testing. Some possible answers could be: – When all faults have been removed!– When we run out of time– When continued testing produces no new failures– When continued testing reveals no new faults– When we have exhausted the testing criteria and the test cases we

have designed– When we reach a point of diminishing returns– When the mandated coverage has been achieved– When the failure intensity (failures / CPU hour) level drops below

a certain point

4

Metrics for Testing Method Comparison (1)

• Functional testing methods may (eventually) provide a high level of path coverage, but at the cost of high redundancy

• The question is to define metrics that relate the effectiveness of a functional technique with that of a structural technique (path coverage)

• To start addressing this question we first observe that functional testing methods are measured according to the number of test cases they produce, while structural testing methods are measured according to the level of the corresponding path coverage they achieve.

5

Metrics for Testing Method Comparison (2)

• Let’s assume a functional testing method M, and a structural metric S that identifies s coverage elements in the unit under test

• Furthermore, let us assume that the functional testing method M produces m test cases, and these m test cases traverse n of the s structural coverage elements. We then define the following:

– Coverage of methodology M with respect to metric S is defined as C(M,S) = n/s

– Redundancy of methodology M with respect to metric S is defined as R(M,S) = m/s

– Net redundancy of methodology M with respect to metric S is defined as NR(M,S) = m/n

6

Sample Comparison of Functional Testing Methods with respect to a Structural Metric

Method m n s C(M,S) R(M,S) NR(M,S)

nominal 15 7 11 0.64 1.36 2.14

Worst case 125 11 11 1.00 11.36 11.36

Method m n s C(M,S) R(M,S) NR(M,S)

Output bva 25 11 11 1.00 2.27 2.27

Decision Table

3 11 11 1.00 0.27 0.27

S: Program Pathss: total number of pathsn: number of paths traversed using method Mm: number of test cases generated by method MM: Functional method

Ref: “Software testing: A Craftsman’s Approach”, P. Jorgensen

Triangle Problem

Commission Problem

7

Sample Comparison of a Functional Testing Method (Output BVA) with respect to Structural Metrics (DD-Path, DU-Path,

Slices)

Method m n s C(M,S) R(M,S) NR(M,S)

DD-Path 25 11 11 1.00 2.27 2.27

DU-Path 25 33 33 1.00 0.76 0.76

Slices 25 40 40 1.00 0.63 0.63

S: Structural Metrics: total number of paths in structural metric Sn: number of paths traversed using method Mm: number of test cases generated by method MM: Functional method

Ref: “Software testing: A Craftsman’s Approach”, P. Jorgensen

Commission Problem

8

Testing Sophistication vs. Number of Test Coverage Items

DD-Path Basis Path DU-Path Slice

low

high

Sophistication

Number of Test Coverage Items

Ref: “Software testing: A Craftsman’s Approach”, P. Jorgensen

9

Test Coverage Items

DD-Path Basis Path DU-Path Slice

11

40

Number of Test Coverage Items

5

33

Ref: “Software testing: A Craftsman’s Approach”, P. Jorgensen

10

Test Case Sophistication vs. Test Case Identification Effort

DD-Path Basis Path DU-Path Slice

low

high

Sophistication

Effort to Identify Test Coverage Items

Ref: “Software testing: A Craftsman’s Approach”, P. Jorgensen

11

Remarks

• Functional testing is prone to gaps and redundancies

• Structural path-coverage testing considers infeasible paths

• Data flow testing considers program semantics on Control Flow Graph paths

• Hybrid approaches that combine functional-structural methods are useful

• Structural testing techniques are best suited for unit testing

12

Subsumption

• A test criterion C1 subsumes a test criterion C2 if every test suite that satisfies C1 also satisfies C2

• Two test criteria C1 and C2 are equivalent if C1 subsumes C2 and vice-versa

• What is the subsumption relation among coverage criteria?

13

Subsumption Hierarchy – Control Flow

14

Subsumption Hierarchy – Data Flow