AUSTIN: A tool for Search based software testing
description
Transcript of AUSTIN: A tool for Search based software testing
![Page 1: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/1.jpg)
AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTINGAnd its Evaluation on Deployed Automotive Systems
Kiran Lakhotia, Mark Harman and Hamilton Gross in
![Page 2: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/2.jpg)
Publication Trend in SBSE
![Page 3: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/3.jpg)
Spread of Activities in SBSE
![Page 4: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/4.jpg)
Yet Another Search Based Testing Tool?
IGUANA - for C (Phil
McMinn)
eTOC – for JAVA (Paolo
Tonella)
Others?
![Page 5: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/5.jpg)
Yet Another Search Based Testing Tool?
IGUANA - for C (Phil McMinn)
eTOC – for JAVA (Paolo
Tonella)Others?
AUSTIN
![Page 6: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/6.jpg)
OutlineAUSTIN •Introduction to Austin
ETF •Overview of ETF
EmpiricalStudy
•Hypotheses, Test Subjects•Results and Analysis
Conclusion •ETF or AUSTIN?
![Page 7: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/7.jpg)
AUgmented Search based TestINg
CIL
AST, CFG
instrument
gcc -o sut.exe
runAUSTIN
gcc -P -Ec sourcefiles
preprocessed
c sourcefiles
instrumented
c source
![Page 8: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/8.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
![Page 9: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/9.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 87, key2 = 33, p = 0, q = 0
![Page 10: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/10.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 88, key2 = 33, p = 0, q = 0
![Page 11: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/11.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 86, key2 = 33, p = 0, q = 0
![Page 12: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/12.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 84, key2 = 33, p = 0, q = 0
![Page 13: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/13.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 80, key2 = 33, p = 0, q = 0
![Page 14: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/14.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 80, key2 = 33, p = 0, q = 0Fitness Function:approach_level + norm(branch_distance)Search Method:Alternating Variable Method
![Page 15: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/15.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0, q = 0
![Page 16: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/16.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0, q = 0
Symbolic Path Condition:key1 == key2 && p == 0
![Page 17: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/17.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0, q = 0
Symbolic Path Condition:key1 == key2 && p == 0
Simplified Path Condition:p == 0
![Page 18: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/18.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0, q = 0
Symbolic Path Condition:key1 == key2 && p == 0
Simplified Path Condition:p == 0
Solve: p ≠ 0
![Page 19: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/19.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0, q = 0
Symbolic Path Condition:key1 == key2 && p == 0
Simplified Path Condition:p == 0
Solve: p ≠ 0
p 0
![Page 20: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/20.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0x..., q = 0
![Page 21: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/21.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0x..., q = 0
Symbolic Path Condition:key1 == key2 && p ≠ 0 && p ≠ q
Simplified Path Condition:p ≠ 0 && p ≠ q
Solve: p ≠ 0 && p == qp 0
q
p,q 0
![Page 22: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/22.jpg)
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0x..., q = 0x...
![Page 23: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/23.jpg)
Evolutionary Testing Framework (ETF)rZSlJ3hb
ETF was developed as part of the EvoTest project Integrated into the Eclipse IDE Supports white & black box testing Uses Evolutionary Algorithms (through the
GUIDE, EO evolutionary library interface) to generate test data
ETF
Selection
Recombinatio
n
Mutation
ReinsertionGUIDE + EO
Send Fitness Values
Send Individuals
![Page 24: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/24.jpg)
ETF cont.rZSlJ3hb
Screenshots taken from a talk by Peter M. Kruse from Berner & Mattner, April 2009
![Page 25: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/25.jpg)
ETF cont.rZSlJ3hb
Screenshots taken from a talk by Peter M. Kruse from Berner & Mattner, April 2009
![Page 26: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/26.jpg)
Fitness Function and Algorithm Both ETF and AUSTIN use the standard
fitness function for branch coverage in Search Based Testing apporach_level + norm(branch_distance)
normalization function: 1 - 1.001-branch_distance
ETF uses a Genetic Algorithm AUSTIN uses the Alternating Variable
Method
![Page 27: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/27.jpg)
Pointers in ETF and AUSTIN AUSTIN
Construct an equivalence of memory locations based on = and ≠ operators in symbolic path condition
Initialize (pointer) input variables based on the graph Assign new memory location via malloc or use existing memory
location On-demand pointer handling
ETF Generate an index and auxiliary variable for each pointer input Assign all input variables of the same type (including globals)
to a pool Optimize indices and values of auxiliary variables along with the
inputs. Indices are used to select variables from pools and assign their address to the pointer input
![Page 28: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/28.jpg)
Pointers in ETF
pool of input parameters and auxiliary variables
pCell
globDbl
val_p, val_q,globX
struct cell{ int value; struct cell* next;};int globX;double globDbl;struct cell* pCell;
void testme(int* p, int* q){ ...}
p = &globX;q = &val_p;
![Page 29: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/29.jpg)
Hypothesis
Null Hypotheses Alternate HypothesesAUSTIN is as effective as the ETF in achieving branch coverage.
AUSTIN is more effective than the ETF in achieving branch coverage.
AUSTIN is equally as efficient as the ETF in achieving branch coverage of a function.
AUSTIN is more efficient than the ETF in achieving branch coverage of a function.
![Page 30: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/30.jpg)
Empirical Study 3 case studies (comprising 8 functions)
Embedded Software Modules from Automotive Industry
Case Study
Description LOC - Tested
Total Branches
Nr. Inputs
B Adaptive Headlight Control Software
1,236 598 132
C Door-Lock Control Software
383 315 71
D Electric Windows Control Software
339 248 31
![Page 31: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/31.jpg)
Coverage ETF vs AUSTIN
![Page 32: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/32.jpg)
![Page 33: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/33.jpg)
Efficiency ETF vs AUSTIN
![Page 34: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/34.jpg)
![Page 35: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/35.jpg)
Coverage Random vs AUSTIN
![Page 36: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/36.jpg)
Efficiency Random vs AUSTIN
![Page 37: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/37.jpg)
ETF or AUSTIN? Coverage is about the same AUSTIN is overall a little bit more
efficient than the ETF in terms of fitness evaluations. In terms of wall clock time the difference is greater (due to the overhead of the ETF-GUIDE setup)
AUSTIN does not construct random memory graphs
ETF is not able generate inputs such asstruct cell{ int mem;};void testme(int* p, struct cell* c) { if( p == &c->mem ) //target}
![Page 38: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/38.jpg)
Summary
![Page 39: AUSTIN: A tool for Search based software testing](https://reader036.fdocuments.us/reader036/viewer/2022062501/56816743550346895ddbf895/html5/thumbnails/39.jpg)
Summary
http://www.cs.ucl.ac.uk/staff/K.Lakhotia/software/austin.htm