Dr Andrew Brooks - MS thesis/project ideas 1

MS thesis/project ideasfor MS students in the Department of Computer Science

at the University of Minnesota Duluth

suggestions by Dr Andrew Brooks, Heller Hall 334http://www.d.umn.edu/~abrooks/

abrooks@d.umn.edu

10/17/2014

Dr Andrew Brooks - MS thesis/project ideas 2

background research article

10/17/2014

Kanewala, U.; Bieman, J.M., "Techniques for testing scientific programs without an oracle," Software Engineering for Computational Science and Engineering (SE-CSE), 2013 5th International Workshop on , pp.48-57, 2013doi: 10.1109/SECSE.2013.6615099

http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6615099

Some presentation material drawn directly from article(s).

Dr Andrew Brooks - MS thesis/project ideas 310/17/2014

A Taxonomy for Test Oracles, Douglas Hoffman,Software Quality Week (QW’98), 1998, 8pp.http://www.softwarequalitymethods.com/Papers/OracleTax.pdf

A gentle introduction to oracles:

Some presentation material drawn directly from article(s).

Chapter 6 Assertion facility, in JDK 1.4 Tutorial, Greg Travis, Manning Publications Company, 2002.http://www.oracle.com/technetwork/articles/javase/javapch06.pdf

A gentle introduction to assertions:

the difficulty of testing software

Dr Andrew Brooks - MS thesis/project ideas 4

Test case identifier Test case values Expected results Actual results

1 [2,7,0] 1 1

2 [0,7,2] 1 0

How do we calculate the expected results?(and know that they are correct)

The oracle is the mechanism used to generate expected results.

10/17/2014

Dr Andrew Brooks - MS thesis/project ideas 5

Oracle methods used by Hoffman• A human calculates the expected results.• A separate program implementing the same algorithm is used to

calculate the expected results.• A simulation is used to calculate the expected results. • A hardware simulator is used to calculate the expected results.• An earlier version of the software is used to calculate the expected

results.• The same version of the software on a different hardware platform

is used to calculate the expected results.• A check is made on the consistency of generated values and end

points.• A sample of values can be checked against independently generated

expected results from existing commercial or open-source software.

10/17/2014

Dr Andrew Brooks - MS thesis/project ideas 6

the difficulty of testing scientific software

10/17/2014

“The existence of an oracle is often assumed in software testing.”

“But in many situations, especially for scientific programs, oracles do not exist or they are too hard to implement.”

Kanewala and Bieman, 2013

Scientists often write programs to discover answers that are previously unknown.

Dr Andrew Brooks - MS thesis/project ideas 7

Metamorphic Testing(a way around the oracle problem)

• A metamorphic relation specifies how a particular change to the input of the program would change the output.

10/17/2014

failure

Dr Andrew Brooks - MS thesis/project ideas 8

examples of metamorphic relations

10/17/2014

Is it possible to replicate an experiment?

• A report should contain enough detail so that another researcher can repeat the experiment and obtain similar results with similar conclusions.

10/17/2014 Dr Andrew Brooks - MS thesis/project ideas 9

Dr Andrew Brooks - MS thesis/project ideas 10

MS thesis/project idea 1.

• First begin by replicating in part or in full one of the previous experiments on metamorphic testing.– References to several emprical studies are given

by Kanewala and Bieman.

10/17/2014

Dr Andrew Brooks - MS thesis/project ideas 11

MS thesis/project idea 1.

• Then tackle one of the problems described by Kanewala and Bieman.– automatic detection of metamorphic relations

• some recent work has been done by Kanewala and Bieman

– identifying the most effective metamorphic relations

– identifying the most effective combinations of metamorphic relations

– understanding why some program mutations could not be killed using metamorphic testing

10/17/2014

deliberately introduced mistakes

Dr Andrew Brooks - MS thesis/project ideas 12

Assertion Testing(a way around the oracle problem)

• Assertions are embedded into source code and evaluated when a test case is executed.

10/17/2014

An assertion is a predicate placed in a program to indicate that the developer thinks that the predicate is always true at that place.

Dr Andrew Brooks - MS thesis/project ideas 13

package asserting;

public class First {

public static void main(String[] args) { int j = 10; assert j>10; }}

simple Java example

Exception in thread "main" java.lang.AssertionErrorat asserting.First.main(First.java:7)

What happens when you run this program?console

10/17/2014

Dr Andrew Brooks - MS thesis/project ideas 14

Assertion basics(Greg Travis chapter)

• Preconditions should be true at the start of a method.

• Postconditions should be true at the end of a method.

• Invariants should always be true.– A speed should not be faster than light speed...

10/17/2014

Dr Andrew Brooks - MS thesis/project ideas 15

MS thesis/project idea 2.

• First begin by replicating in part or in full one of the previous experiments on assertion testing.– References to several empirical studies are given

by Kanewala and Bieman.

10/17/2014

Dr Andrew Brooks - MS thesis/project ideas 16

MS thesis/project idea 2.• Then tackle one of the problems described by

Kanewala and Bieman.– automatic invariant detection can yield spurious

invariants • can ways be found to reduce or eliminate these?

– the effectiveness of assertion checking seems all or nothing depending on the method • can we understand why?

– many more experiments are needed• working with specifications to identify assertions• rewriting program comments as assertions

10/17/2014