Algorithms For Optimizing Test Cases

Presented by team 4Jim Kile

Don LittleSamir Shah

Software Testing – So What? July 28 1962 – Mariner I space probe

Mission control destroys the rocket

1985-1987 – Therac-25 medical accelerator At least five patients die Others seriously injured

November 2000 – National Cancer Institute Panama City At least eight patients die Another 20 receive significant overdoses Physicians indicted for murder

What Is Test Case Optimization? Typically applies to unit test cases where

coverage approaches 100% Implies ordering execution such that:

Rate of fault detection is increasedAmount of time to perform regression is

reduced Elimination of unnecessary test cases

during regression runs

What Are Goals For Test Case Execution? Increase the rate of early fault detection

and correctionFind bugs early so they can be corrected

early Regression test only those areas that have

changed Reduce the amount of time to execute full

unit regression test suites

Why Is Test Case Optimization Important? Execution complexity

Non-linear problem Elapsed time to execute Computing resources Human resources

Wait time for test completionLate defect detection and correction

Who Develops Test Cases?

Developers

Dedicated quality assurance staff

Automated test generation techniques

What Are Benefits Of Automated Test Generation Techniques? No cognitive bias Better able to:

Generate test cases that concentrate on error prone areas

Produce highly novel test cases Better coverage overall

Why Automated Test Optimization?

Generating set of basic test cases - easyEasily cover 50–70% of faults

Improving a test set’s quality - hard Improving to 90–100%

Time consuming and expensive

What Is Difference Between Optimization And Test Case Generation? Optimization problem

Single goal is sought Test case generation

No single goalOptimized coverage of code under test

What Are Benefits Of GA Vs. Random Test Generation? Random test generation

Uniform distribution GA-based search process

More focused test setSet focuses on identified flawsSome highly novel test cases

How Do Genetic Algorithms Work?

A GA operates on strings of digits called chromosomes

Each digit that makes up the chromosome is called a gene

Collection of such chromosomes makes up a population

How Do Genetic Algorithms Work?

Each chromosome has a fitness value Fitness value determines probability of

survival in next generation

How Do Genetic Algorithms Work? Algorithm begins with random population Algorithm evolves incrementally – generation Produces a structure from iterative development

Reproduction Combine with another chromosome (crossover) Adjusted slightly (mutation)

Original chromosome may have a poor/ low fitness Create offspring with much higher fitness

Uses of GA’s In Testing 1) Generate a range of effective test data

with fault revealing power

Both papers used this technique

2) Introduce faults in software under test to determine effectives of test cases

Only one paper used this technique

How Is Quality Of Test Cases Evaluated? Number of detected injected faults

Killed by the test caseOtherwise, it’s alive

Develop a scoreErrors killed by test caseDivided by test set

How Does GA Work Specifically?Example

Genetic algorithmGenerates random population of binary digitsFor example each chromosome may be 36

bits long Each twelve bit segment representing one of the

sides of a triangle

Chromosomes cross between gene 5 and 32

How Does GA Work Specifically? Crossover

Before crossover between gene 5 and 32

1) 111001110101 100101100110 001010111000

2) 111101011010 100101101010 101110110100

After crossover

1) 111001011010 100101101010 101110111000

2) 111101110101 100101100110 001010110100

How Does Mutation Work Specifically?

Mutation will randomly switch genes in population

Gene 23 in chromosome 1 was switched from a 1 to 0

How Does Mutation Work?

Before mutation of gene 23 chromosome 1

1) 111001110101 100101100110 001010111000

After mutation

1) 111001110101 100101100100 001010111000

Generation Of Next Population

Based on a “roulette wheel” Where fitness determines the probability of

selectionThose with higher fitness more chance of

offspring in the next generation in comparison to their less fit companions

Dynamic Software Testing Techniques Structural – first paper

Code coverage Boundary conditions Individual or combined statement traversal Path coverage

Functional – second paperConfirms that a function from specification is

correctly implementedNo analysis of the structure of the program

First paper

Automatic test case optimization: A bacteriologic algorithmBenoit Baudry, Franck Fleurey, Jean-marc Jézéquel And Yves Le Traon

Contribution

Finding an optimal set of test cases through revealing a test case’s “fault revealing power”

Building confidence in the test suite through “mutation analysis”

Bacteriologic AlgorithmTheoretical Basis

Adapted from genetic algorithms Inspired by evolutionary ecology and

bacteriologic adaptation Similarities in this problem domain

Can’t generate a single perfect test suite

Bacteriologic AlgorithmFramework

Bacteriologic AlgorithmBasic Functions

Initialization Iterate incrementally creating new

generation Limitation

Only works on test cases of similar size

Bacteriologic AlgorithmInitialization

Initial test cases either written by hand or automatically generated

For the experiment test cases were randomly generated

Initial size set to 25 nodes

Bacteriologic AlgorithmComputing Fitness

Tool used to generate test case mutants Uses the mutation score of a set of test

cases as that set’s fitness functionMS(T) = 100(d/(m - equiv))

Test cases are executed to determine how many mutants they can killGlobal mutation score computed

Bacteriologic AlgorithmMemorization

Used to compute relative fitness Test case mutation score relative to the solution set’s

mutation score

Test cases are selected whose relative score exceeds the memorization threshold

)(}){(),( TCSMStcTCSMStcTCSrelMS

Bacteriologic AlgorithmMutation

Randomly selects test casesSelection is weighted by relative fitness of the

test case Selected cases and code are mutated to

create new cases for the next generation Code is represented by an abstract syntax

treeNodes in the tree are replaced

Bacteriologic AlgorithmMutation – Abstract Syntax Tree

A finite, labeled directed treeNodes are labeled by operatorsEdges represent operandsLeaves contain variables or constants

Used in a parser Range of all possible structures defined by

the syntax

Bacteriologic AlgorithmMutation – Abstract Syntax Tree Example

x = a + b;

y = a * b;

while (y > a) {

a++;

x = a + b;

}

Bacteriologic AlgorithmFiltering

Filtering = removing Two different implementations

Delete any test case whose relative mutation score is equal to 0

That is the function kills no mutant that the test cases in the solution set haven’t killed

Reduce the coverage matrix by deleting redundant test cases

Bacteriologic AlgorithmResults

Bacteriologic AlgorithmResults

Comparison with genetic algorithmBoth ran 50 times

Genetic algorithm results200 generations createdAverage mutation score of 85Required executing an average of 480,000

test cases

Bacteriologic AlgorithmResults

Bacteriologic algorithm results30 generations createdAverage mutation score of 96Required executing an average of 46,375 test

cases

Second paper

Breeding software test cases with genetic algorithmsD. Berndt, J. Fisher, L. Johnson, J. Pinglikar And A. Watkins

Focus Breeding software test cases using genetic

algorithms as part of a software testing cycle

Uses automated test generation techniques

Evolving fitness function Relies on fossil record of organisms Search behaviors

Novelty Proximity Severity

Genetic AlgorithmSimple Triangle Classification Program (TRITYP)

Classify triangle by type Three sides of the triangle

Parameters x, y, and zRange 0 – 2000

Search space Illegal / legal triangle

Genetic AlgorithmApproach Flaws were intentionally introduced into

data for testing purposes Errors introduced for specific ranges of x

and y parametersX coordinate between 500 - 1000Y coordinate between 0 – 500Result in error

Genetic AlgorithmSearch Space Illegal/Legal Triangle

Genetic AlgorithmHow does it work specifically? Generates random population of x, y and z

coordinates as binary digits Each chromosome is 36 bits long

Each twelve bit segment representing one of the sides of a triangle

Genetic AlgorithmFitness

Relative fitness function Compares

Particular chromosome’s fitnessHistorical information from the fossil record

Genetic AlgorithmGenerating Software Test Cases

Variety of sources into test case breeding with genetic algorithms Powerful evolutionary Naturally parallel computational engine

Balance fitness with diversity Wide variety of test cases can be bred

Concepts of novelty, proximity and severity Used to create a relative or changing fitness function

Genetic AlgorithmBreeding Software Test Cases Using genetic algorithms Evolving fitness function

Fossil record of organisms Interesting search behaviors

Novelty Proximity Severity

Genetic AlgorithmNovelty Measure of the uniqueness of particular test

case Quantified by measuring distance in

parameter space from previous invocations stored in the fossil record

2* ijijn fck

Genetic AlgorithmProximity Measure of closeness to other test cases that

resulted in system failures

2* ijijp eck

Genetic AlgorithmSeverity

Measure of the seriousness of a system error

Genetic AlgorithmDiversity Used to avoid being trapped by local

maxima Generation of test cases diversity means

Emphasizing noveltyDownplaying proximity

Simple rules complex behaviorExplorersProspectorsMiners

Genetic AlgorithmExplorer

Highly novel test case Spread across the lightly populated

regions of the test space Once an error is discovered - fitness

function encourages more thorough testing of the region

Genetic AlgorithmProspectors

Somewhat unique test cases Near newfound errors Both novelty and proximity Prospectors uncover additional errors Fitness function will reward points that are

simply near other errors

Genetic AlgorithmMiners

More fully probe error areas

Genetic AlgorithmExplorer, Prospectors & Miners

Genetic AlgorithmFossil Record Contains all previously generated test

cases Information about type of error generated,

if any Provides an context for changing notions

of fitness Relative rather than absolute fitness

function

Genetic AlgorithmFitness Functions and Fossil Records

Genetic algorithm is used to generate good test cases

Compared to fossil records Points awarded for novelty & proximity Causes population to evolve

Genetic AlgorithmVisualizing the fossil record Three dimensional representation Using 200-by-200 cells to divide up the x-y

parameter space Bars represent the number of organisms

Genetic AlgorithmHigh Novelty & High Proximity

Fairly high novelty Forces the search to reward exploratory

behavior High proximity

Fosters the collection of points where errors are detected

Three seeded range errors

Genetic AlgorithmFossil Record

Genetic AlgorithmLow Novelty & Low Proximity

Weights are reversed Little reward for novelty or proximity More uniformly distribution

Genetic AlgorithmFossil Record

Genetic AlgorithmResults Describes preliminary results from genetic algorithm

based approach to software test case breeding

Project’s reliance on relative or changing fitness function using fossil record

Fossil record records past organisms allowing any current fitness calculations to be influenced by past generations

Three factors are developed for the fitness function: novelty proximity and severity

Interplay of these factors produces fairly complex search behaviors

Conclusions Structural versus functional

Structural - Code coverage – First Paper Boundary conditions Individual or combined statement traversal Path coverage

Functional – specification – Second Paper Confirms that a function from specification is correctly

implemented No analysis of the structure of the program

GA algorithm generation Generate effective test data – Both Papers Introduction of software faults – First Paper