Applications ofMetamorphic Testing

Chris MurphyUniversity of Pennsylvaniacdmurphy@cis.upenn.edu

November 17, 2011

About Me

Lecturer, University of Pennsylvania

PhD-Computer Science, Columbia Univ, 2010 Advisor: Prof. Gail Kaiser Research: software testing, CS education

Seven years experience in software development industry

Problem:When testing a piece of software, how can we know that we’ve createdenough test cases?

Problem:When testing a piece of software, how can we create more test cases?

Solution:We use properties of the software tocreate new test cases from existing ones(particularly those that have not failed).

Result:This approach, known as metamorphictesting, is more effective at testing certaintypes of software than other approaches.

Today's Talk

What is metamorphic testing?

How is metamorphic testing used to find bugs in software?

How can metamorphic testing be applied to applications that do not have test oracles?

What are the open research questions related to metamorphic testing?

UPenn CIS 573 Software Engineering

Graduate-level software engineering course Just over 100 students Focuses on software maintenance issues:

– Testing

– Formal verification

– Debugging

– Fault Localization

– Refactoring

It's your first day of work as asoftware engineer at BloobleSoft.

Your boss gives you 6,000 lines of code and a specification and says “find the bugs”.

Where would you start?

SpecificationCode

Test CaseGeneration

Strategy

TestCasesTest

CasesTest

CasesTest

Cases

You've created 4,837 test cases.

All of them pass.

How do you know when you'redone creating test cases?

Testing Requirements

TestCases

Desired Adequacy Level

MeasurableAdequacy Criteria

Acceptable?CoverageLevel

Your test cases are achieving100% coverage.

None of them havefound any bugs.

Are those test cases useful?

Maybe those test cases can beused to create new test cases.

The more test cases, the better.Right?

This is the idea behind“metamorphic testing”. [Chen et al., HKUST TR CS-98-01, 1998]

A(really, really, really, really)

simple example

Let's say you're testing acosine function.

(I know, I know...)

How could we use this test caseto create new test cases?

You have a test case {45º, 0.7071}, i.e.cos(45º) = 0.7071

We know that the cosine functionexhibits certain properties.

That is, if we make certain changesto the input, we can predict theeffect on the output.

These are referred to as“metamorphic properties”.

What are the metamorphic propertiesof the cosine function?

cos(x + 360º) = cos(x)

cos(x + 180º) = -1 * cos(x)

cos(x - 360º) = cos(x)

That is, if we add 360 to the input,the output should not change.

Given our original test case {45°, 0.7071}, we can create three follow-on test cases.

Property: cos(x + 360º) = cos(x)Input: 45º + 360º = 405ºOutput: cos(45º) = 0.7071

Property: cos(x - 360º) = cos(x)Input: 45º - 360º = -315ºOutput: cos(45º) = 0.7071

Property: cos(-1 * x) = -1 * cos(x)Input: -1 * 45º = -45ºOutput: -1 * cos(45º) = -0.7071

x f f(x)

Initial test case{x, f(x)}

t(x) f f(t(x))Follow-on test case

{t(x), f(t(x))}

t

A metamorphic property of a function fis a pair of functions (t, g)such that f(t(x)) = g(f(x))for all inputs x

g(f(x))

g

=

but wait, isn’t that the same as…

Program invariants: -1 ≤ cos(x) ≤ 1

Algebraic properties: cos²(x) = 1 – sin²(x)

Describe legal ranges/values of a function,but not how it should react when theinput is changed.

Describe the relationships between multiple functions, but not a single function.

simple categories of properties

a b c d e fInitialtest case

c e b a f dPermute

2a 2b 2c 2d 2e 2fMultiply

ssum

sum

sum

s

sumAdd a+2b+2c+2d+2e+2f+2 s+12

2s

sum s+ga b c d e fInclude g

a b c d eExclude s-fsum

a b c d e fInitialtest case #1 ssum

g h i j k lInitialtest case #2 tsum

Compose s+tsuma b c d e f

g h i j k l

Combinationof properties

2s+2tsum2h 2d 2a 2k 2e 2g

2i 2c 2l 2f 2b 2j

Common Metamorphic Properties

• Additive: Increase (or decrease) numerical values by a constant

• Multiplicative: Multiply numerical values by a constant

• Permutative: Randomly permute the order of elements in a set

• Invertive: Create the “opposite” of a set

• Inclusive: Add a new element to a set

• Exclusive: Remove an element from a set

• Compositional: Compose a set[Murphy et al., SEKE’08]

Other Types of Properties

• Noise-based: include input values that will not affect the output

• Semantically Equivalent: create inputs that are have the same “meaning” as the original

• Heuristic: create inputs that are “close” to the original

• Statistical: create inputs that exhibit the same statistical properties

one more example

Consider a function that takes a setof Points (x-y coordinates) and calculatesthe total distance from the first to the last,via the rest.

What are that function’s metamorphicproperties?

Okay, I think I get it.

But does it really work?!?!

In order to find bugs…

But how can this be?!?!

1. The original test case must pass, even though there is a bug.

2. The follow-on test case must fail.

/* Return the smallest value in the array */int findMin(int A[]) { int min = A[0]; for (int i = 1; i < A.length-1; i++) { if (A[i] < min) min = A[i]; } return min;}

/* Return the smallest value in the array */int findMin(int A[]) { int min = A[0]; for (int i = 1; i < A.length-1; i++) { if (A[i] < min) min = A[i]; } return min;}

Test case { {2, 1, 4, 3}, 1} PASS!

100% statement coverage!

100% branch coverage!

/* Return the smallest value in the array */int findMin(int A[]) { int min = A[0]; for (int i = 1; i < A.length-1; i++) { if (A[i] < min) min = A[i]; } return min;}

Test case { {2, 1, 4, 3}, 1}

Metamorphic property: If we permute theinput, the output remains the same.

Follow-on test case: { {4, 2, 3, 1}, 1} FAIL!

metamorphic testing inthe real world

Bioinformatics [Chen et al., BMC Bioinf., 2009]

Machine Learning [Xie et al., JSS, 2011]

Network Simulation [Chen et al., FTDS, 2009]

Computer Graphics [Guderlei et al., QSIC, 2007]

what types of applicationsis metamorphic testing good for?

Applications that deal primarily withnumerical input and numerical output.

Applications that do not have test oracles.

Applications that use graph-basedalgorithms.

Compilers.[Zhou et al., ISFST’04]

SpecificationProgram

TestInput

ActualOutput

ExpectedOutput

Oracle

what if there is no oracle?

Machine Learning

Length of Stay versus Utilization

0

50

100

150

200

250

300

0 2 4 6 8 10 12

number of beds

unit

s of

tim

e

0

2

4

6

8

10

12

14

16

perc

ent

utiliz

ation

LOS

DoctorUtilizationNurseUtilizationTriageUtilizationClerkUtilization

Discrete Event Simulation

x f f(x)

t(x) f f(t(x))Actual

t

If f(t(x)) = g(f(x)) that does notmean that the output is correct.

g

=?

But if f(t(x)) != g(f(x)) thenone (or both) must be incorrect.

g(f(x))Expected

example: RapidMiner

RapidMiner is a suite of machine learningalgorithms implemented in Java.

That is:c = Classify(M(T), e)

In its NaïveBayes implementation, aconfidence level c is reported wheneverit classifies an example e using a modelM created from a training data set T.

We expect that if we modify T to includean extra instance of e, then the confidencelevel should double, since we are twiceas certain about the classification. That is:Classify(M(T+e), e) = 2 * Classify(M(T), e)

Our testing detected violations of this property, thus revealing a bug.[Murphy et al., ICST’09]

empirical study

Goal:Show that metamorphic testing is moreeffective than other techniques at findingbugs in applications without test oracles.

Approach:Use mutation analysis to insert faultsinto the applications, and see how manyare detected using various techniques.

Application domains investigated:

1. Machine Learning (C4.5, MartiRank, Support Vector Machines, PAYL)

2. Discrete Event Simulation (JSim)

3. Information Retrieval (Lucene)

4. Optimization (gaffitter)

Techniques investigated:

1. Metamorphic Testing

2. Runtime Assertion Checking

3. Partial Oracle

Experimental Results

0 20 40 60 80 100 120

TOTAL

gaffitter

Lucene

JSim

PAYL

SVM

MartiRank

C4.5

% of Mutants Killed

Partial Oracle Runtime Assertion Checking Metamorphic Testing

[Murphy et al., ISSTA’09]

can we do better?

That experiment used application-level metamorphic properties.

What if we test at the function level, too?

This is known asMetamorphic Runtime Checking. [Murphy et al., TR CUCS-042-09, 2009]

And continuously conduct those tests while the software is running?

0 20 40 60 80 100 120

TOTAL

gaffitter

Lucene

JSim

PAYL

SVM

MartiRank

C4.5

Partial Oracle Runtime Assertion Checking Metamorphic Testing MT + MRC

Experimental Results

research directions

When I run my test, I see that themetamorphic property is violated.

Does that mean there's a bug?

How can we know whether themetamorphic properties are sound?

Well, not necessarily....

I've used the guidelines to identify asmany metamorphic properties as I could.

Does that mean that's all of them?

How can we know whether the set ofmetamorphic properties is complete?

Well, not necessarily....

I've used the guidelines to identify asmany metamorphic properties as I could.

Does that mean that's all of them?

Could we detect (likely) metamorphic properties automatically?

Well, not necessarily....

I have a function for which I expect that,if I double the input, the output shouldbe doubled.

Could I verify that property without actually executing the code?

Can metamorphic properties be verified statically?

Well, probably....

summary

Metamorphic testing is a method ofcreating new test cases from existing ones.

It depends heavily on the software’smetamorphic properties, which areoften numerical.

Metamorphic testing is particularlyeffective at finding bugs in applications that do not have test oracles.

thanks!

Applications ofMetamorphic Testing

Chris MurphyUniversity of Pennsylvaniacdmurphy@cis.upenn.edu