Automation of Test Case Generation. 2 Testing is expensive We have talked about 3 different kinds...

Automation of Test Case Generation

2

Testing is expensive

We have talked about 3 different kinds of coverage

However, the three coverage are mainly used As measure for progress

Not as target to be achieved

Why? Money coverage

Time coverage

The most used targets…

3

Testing is expensive

Numbers How many lines of test code

About 40% in Microsoft code base

How much is spent each year on software testing It is not easy to even estimate the time/effort spent on

testing inside a IT company (Recall words of Bill Gates) The software testing market (out source testing) is

about $17B in year 2012

How many new testers More than 30k each year in US recently Well more than 150K each year globally

4

Automation is the way to reduce cost

Good or bad news, no way to fully automate testing in predictable future

Three phase Execution automation

Mostly done for unit testing: xUnit Somewhat done for system testing: scripts

Test case generation automation Quite hot topic in academia Great progress made recently

Test oracle check automation Some good trials

5

Automating different steps

Automating test execution Unit Testing Framework Record and Replay for GUI testing Scripts for Command-line tools and configuration

testing

Automating test case generation (this lecture)

6

Automatic test case generation

Random testing Pure Random

Rule-based

Combinatory

Feedback-guided

Exploration testing

Systematical testing Symbolic execution

Chaining

Search-based testing

7

Random Testing

Randomly generate inputs to feed in a software Easiest way to do automatic test case

generation

Do not require any preparation and easy to implement

Problems Semantically redundant inputs

E.g., for a simple program 10/x, providing any input except 0 means the same

8

Rule-based Random Testing

Try to reduce redundancy

Use rules Try boundary cases

0, 1, -1, and other known boundaries

Use distributions Generate random values following certain distribution

Very effective if the distribution of input is known e.g, scores of a final exam

9

Combinatory Testing

Recall input combination coverage

Try to achieve better coverage until reach 100% Add a new test case that covers uncovered value

combinations

Give priority to the test case that covers multiple new possible values

For multiple features of a given input, use constraint solvers to generate a value that satisfies feature combinations (e.g., length = 2 && starts-with A)

10

Adaptive random testing

Intuition Input patterns for bug revealing

Strips: bug is revealed when x + y = 4 Blocks: bug is revealed when x > 3 & y < 4

Trying two very similar test cases does not make much sense

Adaptive random testing

Approach When choosing the next test case, choose the test

case having the largest minimum distance from existing test cases

{t1, t2,…tn} are existing test cases, choose ch so that

Example: one input, int16 x

0T2:32767T1: -20T3: -32768 T4: 16394T5: -16374 T6:...

12

Feedback-Guided Random Testing

Question: It is easy to generate random values for

primitive types, how to generate a random object?

You may generate a random object id = -1, content = “”, child = null May not make sense… For a class, id may be always positive, content

may not be empty, and child cannot be null…

13

Feedback-Guided Random Testing Consequence:

Randomly generated objects may cause a lot of bugs

But they are not real bugs…class BugClass{ int id; String content; Object child; public BugClass(String str){ this.id = Global.id ++; this.content = "bug:" + str; this.child = Bug.createDefaultChild(); } public do(){ String type = Global.type[this.id]; String content = this.content.substring(4); this.child.do(); }}

id = -1;content = "";child = null;

Index array < 0!!!Index out of string length!!!Null pointer reference!!!

14

Feedback-Guided Random Testing

Solution: Try from the methods taking primitive types

first

When an object of a class is generated, use the object in following test cases

Do not use duplicate and null objects

Do not use objects generated with exception

15

Exploration Testing

Very commonly used in testing of GUI & Web software Explore the user interface

Click all clickable

Feed in random value to input boxes

May often achieve good coverage One problem is the login part

Or any inputs that requires some semantics

16

Exploration Testing

Example Menu Window

Records Setting AboutPlay

Select Level

Play

Game SettingAudio Setting

Confirm

Login

17

Exploration Testing

Return to the last window Return button, go back, …

Not always available

Record the state of the software at the previous window Sometimes high overhead

Usually Require instrumentation of the code To check outputs (sent to the screen) with oracles

To tell the explorer that the window is ready (the drawing of the window is done)

18

Exploration Testing

Signal based exploration testing

Software under testDriver

Click event

Event handler finishes

Click event

Event handler finishes

Analyze new GUI

19

Tool Introduction-Randoop

Feedback-based random testing

Joint work by MIT and Microsoft Research

The most robust and effective random testing tool for Java

Have both stand-alone tool and eclipse plug-in

Tool website:

https://code.google.com/p/randoop/

20

Tool Introduction - Randoop

Process Install with eclipse update site:

http://randoop.googlecode.com/hg/plugin.updateSite/

Right click on the class to be tested

Choose as Randoop Input

Configuration How long should randoop run Size of test methods Timeout of certain thread #Test methods per file

21

Tool Introduction - Randoop

Results Add support Jar file

Use Junit to re-execute the test suite

Remove some invalid test cases

Use EclEmma to check results

22

Search-based Testing

Deem test case generation as a optimization problem

Somewhat between random testing and systematical testing Based on random testing, and focus on the input

domains

Use code coverage as guidance

23

Optimization

Try to find the maximal or minimal value of a certain function

Numerous practical problems can be viewed as optimization problem Least cost to travel to a number of cities

Least camera to cover an area

Distribution of stores to attract most customers

Design of pipe systems with least material

Put items into a backpack (with limited volume) with highest value

24

Illustration of Optimizationglobal peak

local peak

25

Solution of Optimization

Hill climbing Start from a random point

Try all neighboring points, and go to the point with highest value, until all neighboring points has a value lower than the current point

Easy to find a local peak

Random-restart Hill climbing Restart hill climbing for many times

To avoid local peaks

26

Solution of Optimization

Annealing simulation Adaptation of hill climbing

Has a probability to move after reached local peak

The probability drops as time goes by

Genetic algorithm Simulate the process of evolution

Start with random points

Select a number of best points

Combine and mutate these points

Until no more improvements can be made

27

Transform Testing to Optimization

Using code coverage as criterion: Try to generate a test case that covers certain code

element (method, statement, branch, …)

Measure how well we have solve the problem A simple fitness function

How far is the already covered elements from the target code elements

Try to make the distance 0

28

Go from random testing

A list of random test cases as the start point

Each test case is a point in the input domain

Using various optimization algorithms to find the best test case

29

An example with hill climbing

Target is st2

Start from 0, 0

f(0, 0) = 2 steps

Try (0,1) (1,0), (0,-1), (-1,0)

No improvement

Go all directions equally

Until reach (5,5)

f(5,5) = 1 step

Increase x and y

Until reach (7,5)

Done!

read x, y;

if(x+y>=10){ st1; if(x>=7){ st2; // target }}st3;

read x, y

if x+y >= 10

st3st1

if x >= 7

st2

yN

y

N

y

30

Problem

The fitness function is too simple

Requires a lot of random walk before getting to the correct place (5,5)

Better algorithms may help, but not much, because all algorithms require to compare the value of fitness functions

A better fitness function: Consider the value gap at all branches

31

Rerun example with hill climbing

Target is st2

Start from 0, 0

f(0, 0) = 10 value gap

Try (0,1) (1,0), (0,-1), (-1,0)

Go to (0,1) or (1,0)

Until reach (0,10)

f(0,10) = 7 value gap

Increase x

Until reach (7,10)

Done!

read x, y;

if(x+y>=10){ st1; if(x>=7){ st2; // target }}st3;

read x, y

if x+y >= 10

st3st1

if x >= 7

st2

yN

y

N

y

32

EvoSuite: Demo

Search based software testing tool

Work in around 2010 by Gordon Fraser and colleagues at U Saarland, Germany

Use genetic algorithm as the optimization algorithm

Use combination of step + value gap as the fitness function

33

Process

Their eclipse plug-in is rather instable

Have a relative stable command-line tool for mac os X

Very easy to setup Download the jar

Package your own code to jar file

Run the command with options Coverage criterion

Timeout Random seeds

34

Results

Junit test cases Even with comments

Very high coverage on relatively simple examples

35

Systematical testing

Target at certain code coverage

Try to understand how the code works

Analyze the code structure to find out a path to go to a certain statement

Analyze the code structure to find out the constraint of the inputs to let the program follow the path

36

Symbolic execution

Target at better statement coverage

Basic idea If a statement is not covered yet, try to provide an

input to go over that statement

A statement is covered only when a path goes to it is covered

Then, what input will cause the program to go through certain paths?

Only when the input satisfy all if-conditions along the path!

Symbolic execution: example

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}


a==nulla==null

a.Length>0a.Length>0

a[0]==123…a[0]==123…T

TF

T

F

F

38

How to know what input to feed in

Static symbolic execution Construct a constraint for each statement, the

statement will be executed when the constraint is satisfied

The parameters (variables) in the constraint are inputs of the software

Solve the constraint

The variables used in conditions may not be inputs

So they must be represented by expressions of inputs

39

Static symbolic execution

Basic Example

y = read(); y = 2 * y; if (y <= 12) fails(); else success();print ("OK");

T (y=s), s is a symbolic variable for input

Here T is the condition for the statement to be executed, (y=s) is the relationship of all variables to the inputs after the statement is executed

T (y=2*s)T (y=2*s)T^y<=12 (y=2*s)

T^!(y<=12) (y=2*s)

T^y<=12 (y=2*s) | T^!(y<=12) (y=2*s)

40

Static symbolic execution

Generating test casesy = read(); T (y=s), s is the input

y = 2 * y; T (y=2*s)

if (y <= 12) T (y=2*s)

fails(); T^y<=12 (y=2*s) s<=6 -> 8

else

success(); T^!(y<=12) (y=2*s) s>6 ->3

print ("OK"); T^y<=12 (y=2*s) | T^!(y<=12) (y=2*s)

41

Constraint Solver

Solve constraints -> provide a value set satisfying the constraint Float values

Linear programming if the constraints are all linear Boolean values

SAT problem Integer values

Can be deduced to SAT String values

SMT problem

Mostly NPC problems Can be deduced from/to SAT problem

42

Problems of static symbolic execution

Path explosion Remember n branches will cause 2n paths

Infinite paths for unbounded loops

Calculate constraints on all paths is infeasible for real software

Too complex constraint The constraint gets very complex for large programs

Not to mention the resolving part is NPC

43

Chaining and Goal-oriented Approach

Not all branches are actually relevant If the outcome of a branch will not affect whether a

statement will be executed

We should not waste time on the branch

A classification of branches Critical branches

The branch must not be taken if target is executed

Non-essential branches Other branches

44

Example of branch classification

Read a, b;if (a > 0){ a = a + b;}if (b >= 10){ st2; //target}

Read a, b

if a > 10

a = a+b;

if b >= 10

st2

N-E

N-E

C

End

45

Chaining

Find a path to reach the target code

Start with the start node and the target node

<read a,b>, <target>

Avoid critical branches:

b>=10

Choose b = 10

46

A different example

Read a; b = 0;if (a > 0){ b = a + b;}if (b >= 10){ st2; //target}

b=0

if a > 0

b = a+b;

if b >= 10

st2

N-E

N-E

C

End

47

Chaining

Find a path to reach the target code

Start with the start node and the target node

<read a>, <target>


b>=10 , no solution

Go to definition statement of b:

b = 0; no solution

b = a + b; -> a + b>=10

48

Chaining: next steps

<read a>, <b=a+b, a+b>=10>, <target>


a>0, a+b>=10

go to definition of a, b

b = 0;

read a; -> solve the constraint: a = 10

49

Chaining

Compared to static symbolic execution: Reduce the branches to be considered

Reduce the definitions to be considered

More scalable than static symbolic execution

Problems Still not scalable enough

Spend much time to generating only 1 test case to cover a certain statement

50

Dynamic symbolic execution

Koushik Sen et al. 2005

One of the most important progress in software engineering in the 21st century

Basic idea Actually run the software

Generate constraints as the program runs

Flip constraints to reach other statements

51


Code to generate inputs for:

Constraints to solve

a!=null a!=null &&a.Length>0 a!=null &&a.Length>0 &&a[0]==1234567890



Observed constraints

a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890

a!=null &&a.Length>0 &&a[0]==1234567890

Data

null

{}

{0}

{123…}

a==null

a==null

a.Length>0a.Length>0

a[0]==123…a[0]==123…T

TF

T

F

F

Execute&MonitorSolve

Choose next path

Done: There is no statement left.Done: There is no statement left.

Negated conditionNegated condition

52


Advantages No need to analysis the whole system before perform

testing

All constraints and expressions can be executed along one execution path

Can handle library method calls

Testing as the analysis taking place

Disadvantages Overhead in testing

53

Review of automatic test case generation

Random testing Rule-based

Adaptive

Feedback-based

Exploration

Search-based testing Optimization algorithms

Fitness function


Chaining and goal-oriented

54

Review of testing

General guidelines

Levels of testing Unit testing, higher level testing, GUI testing

Test coverage Code, input, mutation

Regression testing

Non-function testing Security testing, performance testing

Test automation

55

Review: General guidelines

Test is the practical choice: the best affordable approach

Concepts: test case, test oracle, test suite, test driver, test script, test coverage

Granularity: unit, integration, system, acceptance

Type by design principle: black-box, white-box

Black-box-testing: boundary, equivalence, decision table

White-box-testing: branch coverage, complexity

56

Review of Unit Testing

Using unit test framework to do unit testing It does all the common things, reduce test

interference

Inside test framework setUp -> test -> assert -> tearDown

Writing informative assertions

Writing complete tear downs

Dependencies are evil! Dependency injection

Remove dependencies

Test doubles

57

Review: Higher level testing

Integration Testing Strategies: big bang, top-down, bottom-up, sandwich,

path-based

System Testing Environment issues: building, platforms, web services,

environment failures, distribution issues

Acceptance Testing GUI testing

Event-based, Screen-based, CV-based Record and replay

58

Review of test coverage

Code coverage Target: code Adequacy: no -> 100% code coverage != no bugs Overhead: low (instrumentation cause some overhead)

Input combination coverage Target: inputs Adequacy: yes -> 100% input coverage == no bugs Overhead: none

Mutation coverage Target: bugs Adequacy: no -> 100% mutant coverage != no bugs Overhead: very high (execution on instrumented mutated versions)

59

Review of Regression Testing

Test Prioritization Try only the most important test cases

Test Relevant Code Try the most relevant test cases

Record and Replay Reuse the execution results of previous test cases

60

Review of Non-Functional Testing

Performance Testing Test whether the efficiency (time and space)

of a software meets requirements

Security Testing Test whether the software is vulnerable to

attacks (special invalid inputs designed to control the software or reveal info from the software)

61

Review of automatic test case generation

Random testing Rule-based

Adaptive

Feedback-based

Exploration

Search-based testing Optimization algorithms

Fitness function


Chaining and goal-oriented

Automation of Test Case Generation. 2 Testing is expensive We have talked about 3 different kinds...

Documents

Transcript of Automation of Test Case Generation. 2 Testing is expensive We have talked about 3 different kinds...