Search Congress Valencia Multiplica Enric Quintero Analitica Web + Testing
Search-Based Testing · Applications in Mutation Testing Test suite prioritisation....
Transcript of Search-Based Testing · Applications in Mutation Testing Test suite prioritisation....
![Page 1: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/1.jpg)
Search-Based TestingPhil McMinn
University of Sheffield, UK
![Page 2: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/2.jpg)
OverviewHow and why Search-Based Testing Works
Examples
Search-Based Test Data Generation
Future Directions
Testability Transformation
Input Domain Reduction
Empirical and Theoretical Studies
Temporal, Functional, Structural
Applications in Mutation Testing
Test suite prioritisation
![Page 3: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/3.jpg)
Acknowledgements
The material in some of these slides has kindly been provided by:
Mark Harman (KCL/UCL)
Joachim Wegener (Berner & Matner)
![Page 4: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/4.jpg)
manual design of test cases / scenarios
Conventional Testing
![Page 5: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/5.jpg)
manual design of test cases / scenarios
laborious,time-consuming
Conventional Testing
![Page 6: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/6.jpg)
manual design of test cases / scenarios
laborious,time-consuming
tedious
Conventional Testing
![Page 7: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/7.jpg)
manual design of test cases / scenarios
laborious,time-consuming
tedious
difficult!(where are the faults ???)
Conventional Testing
![Page 8: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/8.jpg)
the hard part...
manual design of test cases / scenarios
laborious,time-consuming
tedious
difficult!(where are the faults ???)
Conventional Testing
![Page 9: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/9.jpg)
Random Test Data Generation
Input
![Page 10: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/10.jpg)
Random Test Data Generation
Input
![Page 11: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/11.jpg)
Random Test Data Generation
Input
![Page 12: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/12.jpg)
Search-Based Testing is an automated search of a
potentially large input space
the search is guided by a problem-specific ‘fitness function’
![Page 13: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/13.jpg)
Search-Based Testing is an automated search of a
potentially large input space
the search is guided by a problem-specific ‘fitness function’
the fitness function guides the search to the test goal
![Page 14: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/14.jpg)
Fitness Function
The fitness function scores different inputs to the system according to the test goal
![Page 15: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/15.jpg)
Fitness Function
The fitness function scores different inputs to the system according to the test goal
which ones are ‘good’ (that we should develop/evolve further)
![Page 16: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/16.jpg)
Fitness Function
The fitness function scores different inputs to the system according to the test goal
which ones are ‘good’ (that we should develop/evolve further)
which ones are useless (that we can forget about)
![Page 17: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/17.jpg)
Fitness-guided search
Input
Fitn
ess
![Page 18: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/18.jpg)
Fitness-guided search
Input
Fitn
ess
![Page 19: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/19.jpg)
Fitness-guided search
Input
Fitn
ess
![Page 20: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/20.jpg)
First publication on SBST
Webb Miller David Spooner
Automatic Generation of Floating-Point Test DataIEEE Transactions on Software Engineering, 1976
Winner of the 2009 Accomplishment by a Senior Scientist Award of the International Society for Computational Biology
2009 Time100Scientists and Thinkers
![Page 21: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/21.jpg)
Publications since 1976
source: SEBASE publications repository http://www.sebase.org
![Page 22: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/22.jpg)
International Search-Based Testing Workshop
co-located with ICST
http://sites.google.com/site/icst2011workshops/
http://sites.google.com/site/icst2011/
4th event at ICST 2011 next year
check websites for submission deadlines etc.:
![Page 23: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/23.jpg)
International Symposium on Search-Based Software
Engineering
Benevento, Italy.7th - 9th September 2010
www.ssbse.org
![Page 24: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/24.jpg)
International Symposium on Search-Based Software Engineering
2011: Co-location with FSE. Szeged, Hungary
www.ssbse.org
Phil McMinnGeneral Chair
Myra Cohen and Mel Ó Cinnéide Program Chairs
![Page 25: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/25.jpg)
Fitness Functions
Often easy
We often define metrics
Need not be complex
![Page 26: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/26.jpg)
Daimler Temporal Testing
tmin
tmax
t
dece
lera
tion
optimal point in time for
triggering the airbag igniter
Fitness = duration
![Page 27: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/27.jpg)
J. Wegener and M. Grochtmann. Verifying timing constraints of real-time systems by means of evolutionary testing. Real-Time Systems, 15(3):275– 298, 1998.
![Page 28: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/28.jpg)
Conventional testingmanual design of test cases / scenarios
laborious,time-consuming
tediousdifficult!
(where are the faults ???)
![Page 29: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/29.jpg)
Conventional testingmanual design of test cases / scenarios
laborious,time-consuming
tediousdifficult!
(where are the faults ???)
Search-Based Testing:automatic - may sometimes be time consuming, but it is not a human’s time being consumed
![Page 30: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/30.jpg)
Conventional testingmanual design of test cases / scenarios
laborious,time-consuming
tediousdifficult!
(where are the faults ???)
Search-Based Testing:automatic - may sometimes be time consuming, but it is not a human’s time being consumed
Search-Based Testing:a good fitness function will lead the search to
the faults
![Page 31: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/31.jpg)
Generating vs CheckingConventional Software Testing Research
Write a method to construct test cases
Search-Based Testing
Write a method to determine how good a test case is
![Page 32: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/32.jpg)
Generating vs CheckingConventional Software Testing Research
Write a method to construct test cases
Search-Based Testing
Write a fitness functionto determine how good a test case is
![Page 33: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/33.jpg)
psi
gap
dist2space
space length
space width
Daimler Autonomous Parking System
Input
![Page 34: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/34.jpg)
Daimler Autonomous Parking System
Fitness
![Page 35: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/35.jpg)
![Page 36: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/36.jpg)
Generation 0
![Page 37: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/37.jpg)
Generation 0 Generation 10critical
![Page 38: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/38.jpg)
Generation 0 Generation 10
Generation 20
collision
critical
![Page 39: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/39.jpg)
Test setupUsual approach to testing
Software under test
Simulation environment
Manually generated input situations
Output of test
![Page 40: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/40.jpg)
Test setup
Software under test
Simulation environment
Search Algorithm
Automatically generated inputs
outputs (converted to fitness values)
Search-Based Testing approach
Feedback loop
![Page 41: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/41.jpg)
O Bühler and J Wegener: Evolutionary functional testing, Computers & Operations Research, 2008 - Elsevier
O. Buehler and J. Wegener. Evolutionary functional testing of an automated parking system. In International Conference on Computer, Communication and Control Technologies and The 9th. International Conference on Information Systems Analysis and Synthesis, Orlando, Florida, USA, 2003.
![Page 42: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/42.jpg)
Structural testing
![Page 43: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/43.jpg)
Structural testing
fitness function analysesthe outcome of
decision statements and the values of variables in
predicates
![Page 44: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/44.jpg)
Structural testing
fitness function analysesthe outcome of
decision statements and the values of variables in
predicates
More later ...
![Page 45: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/45.jpg)
Assertion testing
![Page 46: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/46.jpg)
Assertion testing
assertion condition speed < 150mph
![Page 47: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/47.jpg)
Assertion testing
assertion condition speed < 150mph
fitness function:f =150 - speed
![Page 48: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/48.jpg)
Assertion testing
assertion condition speed < 150mph
fitness function:f =150 - speed
fitness minimised If f is zero or less a fault is found
![Page 49: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/49.jpg)
Bogdan Korel, Ali M. Al-Yami: Assertion-Oriented Automated Test Data Generation. ICSE, 1996: pp. 71-80
![Page 50: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/50.jpg)
Search Techniques
![Page 51: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/51.jpg)
Hill Climbing
Input
Fitn
ess
![Page 52: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/52.jpg)
Hill Climbing
Input
Fitn
ess
![Page 53: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/53.jpg)
Hill Climbing
Input
Fitn
ess
![Page 54: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/54.jpg)
Hill Climbing
Input
Fitn
ess
![Page 55: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/55.jpg)
Hill Climbing
Input
Fitn
ess
No better solution in neighbourhood Stuck at a local optima
![Page 56: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/56.jpg)
Hill Climbing - Restarts
Input
Fitn
ess
![Page 57: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/57.jpg)
Hill Climbing - Restarts
Input
Fitn
ess
![Page 58: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/58.jpg)
Hill Climbing - Restarts
Input
Fitn
ess
![Page 59: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/59.jpg)
Hill Climbing - Restarts
Input
Fitn
ess
![Page 60: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/60.jpg)
Hill Climbing - Restarts
Input
Fitn
ess
![Page 61: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/61.jpg)
Hill Climbing - Restarts
Input
Fitn
ess
![Page 62: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/62.jpg)
Hill Climbing - Restarts
Input
Fitn
ess
![Page 63: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/63.jpg)
Hill Climbing - Restarts
Input
Fitn
ess
![Page 64: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/64.jpg)
Simulated Annealing
Input
Fitn
ess
![Page 65: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/65.jpg)
Simulated Annealing
Input
Fitn
ess
![Page 66: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/66.jpg)
Simulated Annealing
Input
Fitn
ess
![Page 67: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/67.jpg)
Simulated Annealing
Input
Fitn
ess
![Page 68: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/68.jpg)
Simulated Annealing
Input
Fitn
ess
![Page 69: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/69.jpg)
Simulated Annealing
Input
Fitn
ess
![Page 70: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/70.jpg)
Simulated Annealing
Input
Fitn
ess
Worse solutions temporarily
accepted
![Page 71: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/71.jpg)
Evolutionary Algorithm
Input
Fitn
ess
![Page 72: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/72.jpg)
Evolutionary Algorithm
Input
Fitn
ess
![Page 73: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/73.jpg)
Evolutionary Algorithm
Input
Fitn
ess
![Page 74: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/74.jpg)
Evolutionary Algorithms
inspired by Darwinian Evolution and concept of survival of the fittest
Crossover
Mutation
![Page 75: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/75.jpg)
Crossover
a b c10 10 20 40
d
a b c20 -5 80 80
d
![Page 76: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/76.jpg)
Crossover
a b c10 10 20 40
d
a b c20 -5 80 80
d
![Page 77: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/77.jpg)
Crossover
a b c10 10 20 40
d
a b c20 -5 80 80
d
![Page 78: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/78.jpg)
Crossover
a b c10 10 20 40
d
a b c20 -5 80 80
d
a10
b10
c20 40
d
![Page 79: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/79.jpg)
Crossover
a b c10 10 20 40
d
a b c20 -5 80 80
d
c80 80
d
a20
b-5
a10
b10
c20 40
d
![Page 80: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/80.jpg)
d40
Mutation
a b c10 10 20
![Page 81: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/81.jpg)
d40
Mutation
a b c10 10 20 20
d
![Page 82: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/82.jpg)
d40
Mutation
a b c10 10 20 20
d40
d
![Page 83: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/83.jpg)
d40
Mutation
a b c10 10 20 20
d40
da20
![Page 84: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/84.jpg)
Evolutionary Testing
![Page 85: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/85.jpg)
Evolutionary Testing
Mutation
Crossover
Selection
Insertion
Fitness Evaluation
End?
![Page 86: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/86.jpg)
Evolutionary Testing
Mutation
Crossover
Selection
Insertion
Fitness Evaluation
End?
Test cases
Monitoring
Execution
![Page 87: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/87.jpg)
Which search method?Depends on characteristics of the search landscape
![Page 88: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/88.jpg)
Which search method?Depends on characteristics of the search landscape
![Page 89: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/89.jpg)
Which search method?Depends on characteristics of the search landscape
![Page 90: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/90.jpg)
Which search method?Depends on characteristics of the search landscape
![Page 91: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/91.jpg)
Which search method?Depends on characteristics of the search landscape
![Page 92: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/92.jpg)
Which search method?Some landscapes are
hard for some searches but easy for
others
...and vice versa...
![Page 93: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/93.jpg)
Which search method?Some landscapes are
hard for some searches but easy for
others
...and vice versa...
![Page 94: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/94.jpg)
Which search method?Some landscapes are
hard for some searches but easy for
others
...and vice versa...
![Page 95: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/95.jpg)
Which search method?Some landscapes are
hard for some searches but easy for
others
more on this later...
...and vice versa...
![Page 96: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/96.jpg)
Representation
Fitness Function
Neighbourhood
Ingredients for an optimising search
algorithm
![Page 97: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/97.jpg)
Representation
Fitness Function
Neighbourhood
Ingredients for Search-Based Testing
![Page 98: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/98.jpg)
Representation A method of encoding all possible inputs
Usually straightforward
Inputs are already in data structuresFitness Function
Neighbourhood
Ingredients for Search-Based Testing
![Page 99: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/99.jpg)
Part of our understanding of the problem
We need to know our near neighbours
Representation
Fitness Function
Neighbourhood
Ingredients for Search-Based Testing
![Page 100: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/100.jpg)
Transformation of the test goal to a numerical function
Numerical values indicate how ‘good’ an input is
Ingredients for Search-Based Testing
Representation
Fitness Function
Neighbourhood
![Page 101: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/101.jpg)
More search algorithms
![Page 102: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/102.jpg)
More search algorithms
Tabu Search
Estimation of Distribution Algorithms
Particle Swarm Optimisation
Ant Colony Optimisation
Genetic Programming
![Page 103: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/103.jpg)
SBST Surveys & ReviewsPhil McMinn: Search-based software test data generation: a survey. Software Testing, Verification and Reliability 14(2): 105-156, 2004
Wasif Afzal, Richard Torkar and Robert Feldt: A Systematic Review of Search-based Testing for Non-Functional System Properties. Information and Software Technology, 51(6):957-976, 2009
Shaukat Ali, Lionel Briand, Hadi Hemmati and Rajwinder Panesar-Walawege: A Systematic Review of the Application and Empirical Investigation of Search-Based Test-Case Generation. IEEE Transactions on Software Engineering, To appear, 2010
![Page 104: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/104.jpg)
Getting started in SBSEM. Harman and B. Jones: Search-based software engineering. Information and Software Technology, 43(14):833–839, 2001.
M. Harman: The Current State and Future of Search Based Software Engineering, In Proceedings of the 29th International Conference on Software Engineering (ICSE 2007), 20-26 May, Minneapolis, USA (2007)
D. Whitley: An overview of evolutionary algorithms: Practical issues and common pitfalls. Information and Software Technology, 43(14):817–831, 2001.
![Page 105: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/105.jpg)
More applications of Search-Based Testing
![Page 106: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/106.jpg)
Mutation Testing
original
![Page 107: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/107.jpg)
Mutation Testing
original
mutant
mutant
![Page 108: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/108.jpg)
Mutation Testing
mutant
mutant
![Page 109: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/109.jpg)
Mutation Testing
mutant
mutant
higher order mutant
![Page 110: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/110.jpg)
Mutation Testing
mutant
mutant
higher order mutant
Fewer Equivalent MutantsA. J. Offutt. Investigations of the software testing coupling effect. ACM
Transactions on Software Engineering and Methodology 1(1):5–20, Jan. 1992.
![Page 111: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/111.jpg)
Finding Good HOMsDue to the large number of potential HOMs finding the
ones that are most valuable is hard
We want:
Subtle mutants
Reduced Test Effort
![Page 112: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/112.jpg)
Finding Good HOMsDue to the large number of potential HOMs finding the
ones that are most valuable is hard
We want:
Subtle mutants
Reduced Test Effort
HOMs that are hard to kill, corner cases where
undiscovered faults reside
![Page 113: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/113.jpg)
Finding Good HOMsDue to the large number of potential HOMs finding the
ones that are most valuable is hard
We want:
Subtle mutants
Reduced Test Effort
HOMs that are hard to kill, corner cases where
undiscovered faults reside
HOMs that subsume as many first-order mutants as
possible
![Page 114: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/114.jpg)
Subtle Mutants -Fitness Function
ratio of the killability of the HOM to the killability of the constituent FOMs
> 1 HOM is weaker than FOMs
< 1 HOM is stronger than FOMs
= 0, potential equivalent mutant
Y. Jia and M. Harman. Constructing subtle faults using higher order mutation testing. In 8th International Working Conference on Source Code Analysis and
Manipulation (SCAM 2008), Beijing, China, 2008. IEEE Computer Society.
![Page 115: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/115.jpg)
Time aware test suite prioritisation
K. R. Walcott, M. L. Soffa, G. M. Kapfhammer, and R. S. Roos. Time aware test suite prioritization. In Proc. ISSTA, pages 1–11, 2006.
![Page 116: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/116.jpg)
Time aware test suite prioritisation
Suppose a 12 minute time budget
![Page 117: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/117.jpg)
Time aware test suite prioritisation
Order by considering the number of faults that can be
detected
![Page 118: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/118.jpg)
Time aware test suite prioritisation
Order by considering the time only
![Page 119: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/119.jpg)
Time aware test suite prioritisation
Average % of faults detected
![Page 120: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/120.jpg)
Time aware test suite prioritisation
Intelligent heuristic search
![Page 121: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/121.jpg)
Time aware test suite prioritisation
Intelligent heuristic search
Same no. of faults found,
less time
![Page 122: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/122.jpg)
Fitness FunctionThe tester is unlikely to know the location of faults
Need to estimate how likely a test is to find defects
% of code coverage is used to estimate a suite’s potential
the time taken to execute the test suitex
![Page 123: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/123.jpg)
Results
Mutations used to seed faults
![Page 124: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/124.jpg)
Multi-objective Search
Instead of combining the objectives into one fitness function, handle them as distinctive goals
Time Faults
![Page 125: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/125.jpg)
Multi-objective SearchFitness Function A
Fitness Function B
![Page 126: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/126.jpg)
Multi-objective SearchFitness Function A
Fitness Function B
![Page 127: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/127.jpg)
Multi-objective SearchFitness Function A
Fitness Function B
![Page 128: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/128.jpg)
Multi-objective SearchFitness Function A
Fitness Function B
![Page 129: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/129.jpg)
Multi-objective SearchFitness Function A
Fitness Function B
![Page 130: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/130.jpg)
Multi-objective SearchFitness Function A
Fitness Function B
![Page 131: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/131.jpg)
Multi-objective SearchFitness Function A
Fitness Function B
![Page 132: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/132.jpg)
Multi-objective SearchFitness Function A
Fitness Function B
![Page 133: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/133.jpg)
Multi-objective SearchFitness Function A
Fitness Function B
![Page 134: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/134.jpg)
Multi-objective SearchFitness Function A
Fitness Function B
![Page 135: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/135.jpg)
Multi-objective SearchFitness Function A
Fitness Function B
![Page 136: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/136.jpg)
Multi-objective SearchFitness Function A
Fitness Function B
![Page 137: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/137.jpg)
Multi-objective SearchFitness Function A
Fitness Function B
![Page 138: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/138.jpg)
Multi-objective SearchFitness Function A
Fitness Function B
![Page 139: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/139.jpg)
Multi-objective SearchFitness Function A
Fitness Function B
![Page 140: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/140.jpg)
Multi-objective SearchFitness Function A
Fitness Function B
The Pareto Front
![Page 141: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/141.jpg)
Some ResultsPareto Efficient Multi-Objective Test Case
Selection, Shin Yoo and Mark Harman. Proceedings of ACM International Symposium on Software Testing and Analysis 2007 (ISSTA
2007), p140-150
![Page 142: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/142.jpg)
Three objectives
![Page 143: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/143.jpg)
Other applications of Search-Based Testing
Combinatorial Interaction Testing
GUI Testing
M.B. Cohen, M.B. Dwyer and J. Shi, Interaction testing of highly-configurable systems in the presence of constraints, International Symposium on Software Testing and Analysis
(ISSTA), London, July 2007, pp. 129-139.
X. Yuan, M.B. Cohen and A.M. Memon, GUI Interaction Testing: Incorporating Event Context, IEEE Transactions on Software Engineering, to appear
![Page 144: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/144.jpg)
Other applications of Search-Based Testing
Stress testing
State machine testing
L. C. Briand, Y. Labiche, and M. Shousha. Stress testing real-time systems with genetic algorithms. In Proceedings of the Genetic and Evolutionary Computation Conference
(GECCO 2005), pages 1021–1028, Washington DC, USA, 2005. ACM Press.
K. Derderian, R. Hierons, M. Harman, and Q. Guo. Automated unique input output sequence generation for conformance testing of FSMs. The Computer Journal,
39:331–344, 2006.
![Page 145: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/145.jpg)
Search-Based Structural Test Data
Generation
![Page 146: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/146.jpg)
Covering a structure
TARGET
![Page 147: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/147.jpg)
Fitness evaluation
TARGET
![Page 148: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/148.jpg)
Fitness evaluation
TARGET
The test data executes the ‘wrong’ path
![Page 149: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/149.jpg)
Analysing control flow
TARGET
![Page 150: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/150.jpg)
Analysing control flow
TARGET
The outcomes at key decision statements
matter.
These are the decisions on
which the target is control dependent
![Page 151: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/151.jpg)
Approach Level
TARGET
![Page 152: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/152.jpg)
Approach Level
TARGET
= 2
![Page 153: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/153.jpg)
Approach Level
TARGET
= 2
= 1
![Page 154: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/154.jpg)
Approach Level
TARGET
= 2
= 1
= 0
![Page 155: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/155.jpg)
Approach Level
TARGET
= 2
= 1
= 0
minimisation
![Page 156: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/156.jpg)
Approach LevelRoy P. Pargas, Mary Jean Harrold, Robert Peck: Test-Data Generation
Using Genetic Algorithms. Softw. Test., Verif. Reliab. 9(4): 263-282 (1999)
Joachim Wegener, André Baresel, Harmen Sthamer: Evolutionary test environment for automatic structural testing. Information & Software Technology 43(14): 841-854 (2001)
![Page 157: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/157.jpg)
Analysing predicatesApproach level alone gives us coarse values
a = 50, b = 0 a = 45, b = 5 a = 40, b = 10 a = 35, b = 15 a = 30, b = 20 a = 25, b = 25 !
![Page 158: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/158.jpg)
Analysing predicatesApproach level alone gives us coarse values
a = 50, b = 0 a = 45, b = 5 a = 40, b = 10 a = 35, b = 15 a = 30, b = 20 a = 25, b = 25 !
getting ‘closer’ to being true
![Page 159: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/159.jpg)
Branch distanceAssociate a distance formula with different
relational predicates
a = 50, b = 0 branch distance = 50a = 45, b = 5 branch distance = 40a = 40, b = 10 branch distance = 30a = 35, b = 15 branch distance = 20a = 30, b = 20 branch distance = 10a = 25, b = 25 branch distance = 0
getting ‘closer’ to being true
![Page 160: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/160.jpg)
Branch distances for relational predicates
Nigel Tracey, John Clark & Keith Mander. The Way Forward for Unifying Dynamic Test-Case Generation:
The Optimisation-Based Approach.1998
Bogdan Korel. Automated Software Test Data Generation. IEEE Trans. Software Eng. 1990
Webb Miller & D. Spooner. Automatic Generation of Floating-Point Test Data. IEEE Trans. Software Eng. 1976
![Page 161: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/161.jpg)
Putting it all together
true
true
if a >= b
if b >= c
TARGET
false
false
true if c >= d false
Fitness = approach Level + normalised branch distance
TARGET
![Page 162: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/162.jpg)
Putting it all together
true
true
if a >= b
if b >= c
TARGET
TARGET MISSEDApproach Level = 2
Branch Distance = b - a
false
false
true if c >= d false
Fitness = approach Level + normalised branch distance
TARGET
normalised branch distance between 0 and 1indicates how close approach level is to being penetrated
![Page 163: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/163.jpg)
Putting it all together
true
true
if a >= b
if b >= c
TARGET
TARGET MISSEDApproach Level = 1
Branch Distance = c - b
TARGET MISSEDApproach Level = 2
Branch Distance = b - a
false
false
true if c >= d false
Fitness = approach Level + normalised branch distance
TARGET
normalised branch distance between 0 and 1indicates how close approach level is to being penetrated
![Page 164: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/164.jpg)
Putting it all together
true
true
if a >= b
if b >= c
TARGET
TARGET MISSEDApproach Level = 1
Branch Distance = c - b
TARGET MISSEDApproach Level = 2
Branch Distance = b - a
false
false
true if c >= d false
TARGET MISSEDApproach Level = 0
Branch Distance = d - c
Fitness = approach Level + normalised branch distance
TARGET
normalised branch distance between 0 and 1indicates how close approach level is to being penetrated
![Page 165: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/165.jpg)
Normalisation FunctionsSince the ‘maximum’ branch distance is generally unknown we need a non-standard normalisation function
Baresel (2000), alpha = 1.001
![Page 166: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/166.jpg)
Normalisation FunctionsSince the ‘maximum’ branch distance is generally unknown we need a non-standard normalisation function
Arcuri (2010), beta = 1
![Page 167: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/167.jpg)
void fn( input1, input2, input3 .... )
Alternating Variable Method‘Probe’ moves
![Page 168: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/168.jpg)
decrease
void fn( input1, input2, input3 .... )
increase
Alternating Variable Method‘Probe’ moves
![Page 169: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/169.jpg)
decrease decrease
void fn( input1, input2, input3 .... )
increase increase
Alternating Variable Method‘Probe’ moves
![Page 170: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/170.jpg)
decrease decrease decrease
void fn( input1, input2, input3 .... )
increase increase increase
Alternating Variable Method‘Probe’ moves
![Page 171: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/171.jpg)
Alternating Variable Method
Input variable value
Fitn
ess
Accelerated hill climb
![Page 172: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/172.jpg)
Alternating Variable Method
Input variable value
Fitn
ess
Accelerated hill climb
![Page 173: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/173.jpg)
Alternating Variable Method
Input variable value
Fitn
ess
Accelerated hill climb
![Page 174: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/174.jpg)
Alternating Variable Method
Input variable value
Fitn
ess
Accelerated hill climb
![Page 175: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/175.jpg)
Alternating Variable Method
Input variable value
Fitn
ess
Accelerated hill climb
![Page 176: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/176.jpg)
Alternating Variable Method
Input variable value
Fitn
ess
Accelerated hill climb
![Page 177: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/177.jpg)
Alternating Variable Method
Input variable value
Fitn
ess
Accelerated hill climb
![Page 178: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/178.jpg)
Alternating Variable Method1. Randomly generate start point
a=10, b=20, c=30
![Page 179: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/179.jpg)
Alternating Variable Method1. Randomly generate start point
a=10, b=20, c=30
2. ‘Probe’ moves on a
a=9, b=20, c=30 no effect
a=11, b=20, c=30
![Page 180: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/180.jpg)
Alternating Variable Method1. Randomly generate start point
a=10, b=20, c=30
2. ‘Probe’ moves on a
a=9, b=20, c=30 no effect
a=11, b=20, c=30
3. ‘Probe’ moves on b
a=10, b=19, c=30 improved
branch distance
![Page 181: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/181.jpg)
Alternating Variable Method1. Randomly generate start point
a=10, b=20, c=30
2. ‘Probe’ moves on a
a=9, b=20, c=30 no effect
a=11, b=20, c=30
3. ‘Probe’ moves on b
a=10, b=19, c=30
4. Accelerated moves in direction of improvement
improved
branch distance
![Page 182: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/182.jpg)
Key PublicationsBogdan Korel. Automated Software Test Data Generation.
IEEE Trans. Software Eng. 1990
Alternating variable method
Evolutionary structural test data generationS. Xanthakis, C. Ellis, C. Skourlas, A. Le Gall, S. Katsikas, and K. Kara- poulios.
Application of genetic algorithms to software testing (Applica- tion des algorithmes g ́en ́etiques au test des logiciels). In 5th International Conference on Software
Engineering and its Applications, pages 625–636, Toulouse, France, 1992
![Page 183: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/183.jpg)
A search-based test data generator tool
![Page 184: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/184.jpg)
Input Generation Using Automated Novel Algorithms
![Page 185: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/185.jpg)
IGUANA(Java)
Test object
(C code compiled to a DLL )
![Page 186: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/186.jpg)
IGUANA(Java)
Test object
(C code compiled to a DLL )
inputs
![Page 187: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/187.jpg)
IGUANA(Java)
Test object
(C code compiled to a DLL )
inputs
information from test object
instrumentation
fitness computation
![Page 188: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/188.jpg)
IGUANA(Java)
Test object
(C code compiled to a DLL )
inputs
information from test object
instrumentation
fitness computation
search algorithm
![Page 189: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/189.jpg)
IGUANA(Java)
Test object
(C code compiled to a DLL )
inputs
information from test object
instrumentation
Java Native Interface
fitness computation
search algorithm
![Page 190: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/190.jpg)
A function for testing
![Page 191: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/191.jpg)
1. Parse the code and extract control dependency graph
TARGET
“which decisions are key for the execution of individual structural
targets” ?
if a == b
if b == c
return 1
Test Object Preparation
![Page 192: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/192.jpg)
2. Instrument the code
TARGET
if a == b
if b == c
return 1
for monitoring control flow and variable values in
predicates
Test Object Preparation
![Page 193: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/193.jpg)
3. Map inputs to a vector
Test Object Preparation
a b c
![Page 194: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/194.jpg)
3. Map inputs to a vector
Test Object Preparation
a b c
Straightforward in many cases
![Page 195: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/195.jpg)
3. Map inputs to a vector
Test Object Preparation
a b c
Straightforward in many casesInputs composed of dynamic data structures are harder to compose
Kiran Lakhotia, Mark Harman and Phil McMinn.Handling Dynamic Data Structures in Search-Based Testing.
Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2008), Atlanta, Georgia, USA, July 12-16 2008, pp. 1759-1766, ACM Press.
![Page 196: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/196.jpg)
Instrumentation
![Page 197: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/197.jpg)
Instrumentation
![Page 198: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/198.jpg)
Each branching condition is
replaced by a call to the function
node(...)
the instrumentation should only observe the program and not alter its behaviour
![Page 199: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/199.jpg)
The first parameter is the control flow
graph node ID of the decision statement
The second parameter is a boolean condition that replicates the structure in the original program
(i.e. including short-circuiting)
![Page 200: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/200.jpg)
Relational predicates are replaced with functions that compute branch distance.
![Page 201: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/201.jpg)
The instrumentation tells us:
Which decision nodes were executed
and their outcome (branch distances)
Therefore we can find which decision control flow diverged from a target for an input....
... and compute the approach level from the control dependence graph
... and lookup the branch distance
fitness value for an input
![Page 202: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/202.jpg)
TARGET
if a == b
if b == c
return 1
1nput: <20, 20, 30>NODE T F
1 0 1
2 10 0
4 20 0
....
Diverged at node 2approach level: 0
branch distance: 10
fitness = 0.009945219
![Page 203: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/203.jpg)
TestabilityTransformation
![Page 204: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/204.jpg)
The ‘Flag’ Problem
![Page 205: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/205.jpg)
Program Transformation
![Page 206: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/206.jpg)
Program Transformation
![Page 207: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/207.jpg)
Programs will inevitably have features that heuristic searches handle less well
Mark Harman, Lin Hu, Rob Hierons, Joachim Wegener, Harmen Sthamer, Andre Baresel and Marc Roper. Testability Transformation.
IEEE Transactions on Software Engineering. 30(1): 3-16, 2004.
![Page 208: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/208.jpg)
Testability transformation: change the program to improve test data generation
Programs will inevitably have features that heuristic searches handle less well
Mark Harman, Lin Hu, Rob Hierons, Joachim Wegener, Harmen Sthamer, Andre Baresel and Marc Roper. Testability Transformation.
IEEE Transactions on Software Engineering. 30(1): 3-16, 2004.
![Page 209: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/209.jpg)
Testability transformation: change the program to improve test data generation
... whilst preserving test adequacy
Programs will inevitably have features that heuristic searches handle less well
Mark Harman, Lin Hu, Rob Hierons, Joachim Wegener, Harmen Sthamer, Andre Baresel and Marc Roper. Testability Transformation.
IEEE Transactions on Software Engineering. 30(1): 3-16, 2004.
![Page 210: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/210.jpg)
Nesting
Phil McMinn, David Binkley and Mark Harman Empirical Evaluation of a Nesting Testability Transformation for Evolutionary Testing
ACM Transactions on Software Engineering and Methodology, 18(3), Article 11, May 2009
![Page 211: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/211.jpg)
Testability Transformation
![Page 212: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/212.jpg)
Testability Transformation
Note that the programs are no longer equivalent
![Page 213: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/213.jpg)
Testability Transformation
Note that the programs are no longer equivalent
But we don’t care - so long as we get the test data is still adequate
![Page 214: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/214.jpg)
Nesting & Local Optima
![Page 215: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/215.jpg)
Nesting & Local Optima
![Page 216: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/216.jpg)
Nesting & Local Optima
![Page 217: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/217.jpg)
Nesting & Local Optima
![Page 218: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/218.jpg)
-100
-80
-60
-40
-20
0
20
40
60
80
100
Nested branches
Chan
ge in
suc
cess
rate
afte
r app
lyin
g tr
ansf
orm
atio
n (%
)Results - Industrial & Open source code
![Page 219: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/219.jpg)
Dependent & Independent Predicates
Predicates influenced by disjoint sets of input variables
IndependentCan be optimised
in parallel
e.g. ‘a == 0’ and ‘b == 0’
![Page 220: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/220.jpg)
Dependent & Independent Predicates
Predicates influenced by non-disjoint sets of input
variables
DependentInteractions
between predicates inhibit
parallel optimisation
e.g. ‘a == b’ and ‘b == c’
![Page 221: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/221.jpg)
-100
-80
-60
-40
-20
0
20
40
60
80
100
Nested branches
Chan
ge in
suc
cess
rate
afte
r app
lyin
g tr
ansf
orm
atio
n (%
)
Dependent predicates
![Page 222: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/222.jpg)
-100
-80
-60
-40
-20
0
20
40
60
80
100
Nested branches
Chan
ge in
suc
cess
rate
afte
r app
lyin
g tr
ansf
orm
atio
n (%
)
Dependent predicates
Independent and some dependent predicates
![Page 223: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/223.jpg)
![Page 224: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/224.jpg)
When not preserving program equivalence can go wrong
![Page 225: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/225.jpg)
we are testing to cover structure
... but the structure is the problem
so we transform the program
... but this alters the structure
![Page 226: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/226.jpg)
we are testing to cover structure
... but the structure is the problem
so we transform the program
... but this alters the structure
![Page 227: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/227.jpg)
so we need to be careful:are we still testing according to the same
criterion?
we are testing to cover structure
... but the structure is the problem
so we transform the program
... but this alters the structure
![Page 228: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/228.jpg)
Input Domain Reduction
![Page 229: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/229.jpg)
Mark Harman, Youssef Hassoun, Kiran Lakhotia, Phil McMinn and Joachim Wegener.The Impact of Input Domain Reduction on Search-Based Test Data Generation.
The 6th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/
FSE 2007), Cavtat, Croatia, September 3-7 2007, pp. 155-164, ACM Press.
![Page 230: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/230.jpg)
Mark Harman, Youssef Hassoun, Kiran Lakhotia, Phil McMinn and Joachim Wegener.The Impact of Input Domain Reduction on Search-Based Test Data Generation.
The 6th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/
FSE 2007), Cavtat, Croatia, September 3-7 2007, pp. 155-164, ACM Press.
![Page 231: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/231.jpg)
Mark Harman, Youssef Hassoun, Kiran Lakhotia, Phil McMinn and Joachim Wegener.The Impact of Input Domain Reduction on Search-Based Test Data Generation.
The 6th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/
FSE 2007), Cavtat, Croatia, September 3-7 2007, pp. 155-164, ACM Press.
![Page 232: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/232.jpg)
Effect of Reduction-100,000 ... 100,000-100,000 ... 100,000
-100,000 ... 100,000
approx 1016
![Page 233: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/233.jpg)
Effect of Reduction-100,000 ... 100,000-100,000 ... 100,000
-100,000 ... 100,000
![Page 234: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/234.jpg)
Effect of Reduction-100,000 ... 100,000-100,000 ... 100,000
-100,000 ... 100,000
![Page 235: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/235.jpg)
Effect of Reduction-100,000 ... 100,000-100,000 ... 100,000
-100,000 ... 100,000
200,001
![Page 236: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/236.jpg)
Variable Dependency Analysis
![Page 237: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/237.jpg)
Variable Dependency Analysis
![Page 238: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/238.jpg)
Variable Dependency Analysis
![Page 239: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/239.jpg)
Variable Dependency Analysis
![Page 240: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/240.jpg)
Empirical Study
Random Search Alternating Variable Method Evolutionary Testing
Defroster F2
Gimp Spice
Tiff
Case studies:
Studied the effects of reduction with:
![Page 241: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/241.jpg)
Effect on Random Testing-49 ... 50
Probability of executing target:
-49 ... 50-49 ... 50
100 x 100 x 1
100 x 100 x 100
![Page 242: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/242.jpg)
Effect on Random Testing-49 ... 50
Probability of executing target:
-49 ... 50-49 ... 50
100 x 100 x 1
100 x 100 x 100
![Page 243: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/243.jpg)
Results with Random Testing
![Page 244: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/244.jpg)
Results with AVM
![Page 245: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/245.jpg)
Effect on AVMSaves probe moves (and thus wasteful fitness evaluations) around irrelevant variables
void fn( irrelevant1, irrelevant2, irrelevant3, required1 .... )
increase increase increase increase
decrease decrease decrease decrease
![Page 246: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/246.jpg)
Effect on AVMSaves probe moves (and thus wasteful fitness evaluations) around irrelevant variables
void fn( irrelevant1, irrelevant2, irrelevant3, required1 .... )
increase
decrease
![Page 247: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/247.jpg)
Effect on ET
Saves mutations on irrelevant variables
Mutations concentrated on the variables that matter
Likely to speed up the search
![Page 248: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/248.jpg)
Results with ET
![Page 249: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/249.jpg)
Conclusions for Input Domain Reduction
Variable dependency analysis can be used to reduce input domains
This can reduce search effort for the AVM and ET
Perhaps surprisingly, there is no overall change for random search
![Page 250: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/250.jpg)
Which searchalgorithm?
![Page 251: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/251.jpg)
Empirical Study Bibclean Defroster F2 Eurocheck Gimp Space Spice Tiff Totinfo
Mark Harman and Phil McMinn.A Theoretical and Empirical Study of Search Based Testing: Local, Global and Hybrid Search.
IEEE Transactions on Software Engineering, 36(2), pp. 226-247, 2010.
760 branches in ~5 kLOC
![Page 252: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/252.jpg)
Interesting branches
20 98
Alternating Variable Method
Evolutionary Testing
![Page 253: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/253.jpg)
0
10
20
30
40
50
60
70
80
90
100
f2 F2 (
11F)
spac
e spa
ce_s
eqrot
rg (17
T)
spac
e spa
ce_s
eqrot
rg (22
T)
spice
clip_
to_cir
cle (4
9T)
spice
clip_
to_cir
cle (6
2T)
spice
clip_
to_cir
cle (6
8F)
spice
clipa
rc (22
T)
spice
clipa
rc (24
T)
spice
clipa
rc (63
F)
totinf
o Info
Tbl (14
F)
totinf
o Info
Tbl (21
F)
totinf
o Info
Tbl (29
T)
totinf
o Info
Tbl (29
F)
totinf
o Info
Tbl (35
T)
totinf
o Info
Tbl (35
F)
Suc
cess
Rat
e (%
)
Evolutionary Testing
Hill Climbing
Wins for the AVM
![Page 254: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/254.jpg)
1
10
100
1,000
10,000
100,000
gimp_
rgb_to
_hsl
(4T)
gimp_
rgb_to
_hsv
(5F)
gimp_
rgb_to
_hsv
4 (11
F)
gimp_
rgb_to
_hsv
_int (1
0T)
gradie
nt_ca
lc_bil
inear_
factor
(8T)
gradie
nt_ca
lc_co
nical_
asym
_facto
r (3F
)
gradie
nt_ca
lc_co
nical_
sym_fa
ctor (
3F)
gradie
nt_ca
lc_sp
iral_f
actor
(3F)
clipa
rc (13
F)
clipa
rc (15
T)
clipa
rc (15
F)
TIFF_SetS
ample
(5T)
Aver
gage
num
ber o
f fitn
ess
eval
uatio
ns
Evolutionary Testing
Hill Climbing
Wins for the AVM
![Page 255: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/255.jpg)
When does the AVM win?fit
ness
inputinput
![Page 256: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/256.jpg)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
chec
k_IS
BN (23F
)
chec
k_IS
BN (27T
)
chec
k_IS
BN (29T
)
chec
k_IS
BN (29F
)
chec
k_IS
SN (23F
)
chec
k_IS
SN (27T
)
chec
k_IS
SN (29T
)
chec
k_IS
SN (29F
)
Suc
cess
Rat
e
Evolutionary Testing
Hill Climbing
Wins for Evolutionary Testing
![Page 257: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/257.jpg)
When does ET win?
The branches in question were part of a routine for validating ISBN/ISSN strings
When a valid character is found, a counter variable is incremented
![Page 258: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/258.jpg)
When does ET win?Evolutionary algorithms incorporate
a population of candidate solutions
crossover
Crossover enables valid characters to be crossed over into different ISBN/ISSN strings
![Page 259: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/259.jpg)
Schemata
1010100011110000111010
1111101010000000101011
0001001010000111101011
Subsets of useful genes
e.g. substrings of
1’s in a binary all
ones problem
![Page 260: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/260.jpg)
Schemata
1010100011110000111010
1111101010000000101011
0001001010000111101011
The schema theory predicts that schema of above average fitness will proliferate in subsequent
generations of the evolutionary search
Subsets of useful genes
e.g. substrings of
1’s in a binary all
ones problem
![Page 261: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/261.jpg)
Schemata
1**1 *11*
1111
crossover of fit schemata can lead to even fitter, higher order schemata
![Page 262: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/262.jpg)
Royal Roads
landscape structures where there is a ‘red carpet’ for crossovers to follow
![Page 263: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/263.jpg)
The Genetic AlgorithmRoyal Road
S1: 1111****************************S2: ****1111************************S3: ********1111********************S4: ************1111****************S5: ****************1111************S6: ********************1111********S7: ************************1111****S8: ****************************1111S9: 11111111************************S10: ********11111111****************S11: ****************11111111********S12: ************************11111111S13: 1111111111111111****************S14: ****************1111111111111111S15: 11111111111111111111111111111111
![Page 264: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/264.jpg)
When Crossover Helps
![Page 265: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/265.jpg)
When Crossover Helps Executes the target
R1 R2
Q1
P1 P2
Q2
P3 P4
Q1
P1 P2
Q2
P3 P4
![Page 266: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/266.jpg)
When Crossover Helps
![Page 267: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/267.jpg)
When Crossover Helps Executes the target
Contains 2 valid characters
Contains1 valid
character
Contains1 valid
character
Contains1 valid
character
Contains1 valid
character
Contains1 valid
character
Contains1 valid
character
Contains1 valid
character
Contains1 valid
character
Contains 2 valid characters
Contains 2 valid characters
Contains 2 valid characters
Contains 4 valid characters
Contains 4 valid characters
![Page 268: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/268.jpg)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
chec
k_IS
BN (23F
)
chec
k_IS
BN (27T
)
chec
k_IS
BN (29T
)
chec
k_IS
BN (29F
)
chec
k_IS
SN (23F
)
chec
k_IS
SN (27T
)
chec
k_IS
SN (29T
)
chec
k_IS
SN (29F
)
Suc
cess
Rat
e
Evolutionary Testing
Headless Chicken Test
Headless Chicken TestT. Jones, “Crossover, macromutation and population-based search,” Proc. ICGA ’95. Morgan Kaufmann, 1995, pp. 73–80.
![Page 269: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/269.jpg)
Investigations into Crossover
Royal Roads
HIFF
Real Royal Roads
Ignoble Trails
M. Mitchell, S. Forrest, and J. H. Holland, “The royal road for genetic algorithms: Fitness landscapes and GA performance,” Proc. 1st European Conference on Artificial Life. MIT Press, 1992, pp. 245–254.
R. A. Watson, G. S. Hornby, and J. B. Pollack, “Modeling building-block interdependency,” Proc. PPSN V. Springer, 1998, pp.97–106.
T. Jansen and I. Wegener, “Real royal road functions - where crossover provably is essential,” Discrete Applied Mathematics, vol. 149, pp. 111–125, 2005.
J. N. Richter, A. Wright, and J. Paxton, “Ignoble trails - where crossover is provably harmful,” Proc. PPSN X. Springer, 2008, pp. 92–101.
![Page 270: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/270.jpg)
Evolutionary Testing Schemata
{(a, b, c) | a = b}
{(a, b, c) | a > 0}
(50, 50, 25)(100, 100, 10)
...
(50, 10, 25)(100, -50, 10)
...
![Page 271: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/271.jpg)
Crossover of good schemata
{(a, b, c) | a = b} {(a, b, c) | b ≥ 100}
{(a, b, c) | a = b ∧ b ≥ 100}
![Page 272: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/272.jpg)
Crossover of good schemata
{(a, b, c) | a = b} {(a, b, c) | b ≥ 100}
{(a, b, c) | a = b ∧ b ≥ 100}
subschema subschema
superschema
![Page 273: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/273.jpg)
Crossover of good schemata
{(a, b, c) | a = b} {(a, b, c) | b ≥ 100}
{(a, b, c) | a = b ∧ b ≥ 100}
subschema subschema
superschema
building block building block
![Page 274: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/274.jpg)
Crossover of good schemata
{(a, b, c) | a = b ∧ b ≥ 100} {(a, b, c) | c ≤ 10}
{(a, b, c) | a = b ∧ b ≥ 100 ∧ c ≤ 10}
![Page 275: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/275.jpg)
Crossover of good schemata
{(a, b, c) | a = b ∧ b ≥ 100} {(a, b, c) | c ≤ 10}
{(a, b, c) | a = b ∧ b ≥ 100 ∧ c ≤ 10} coveringschema
![Page 276: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/276.jpg)
What types of program and program structure enable Evolutionary Testing to perform well, through crossover and how?
P. McMinn, How Does Program Structure Impact the Effectiveness of the Crossover Operator in Evolutionary Testing? Proc. Symposium on Search-Based Software Engineering, 2010
1. Large numbers of conjuncts in the input condition
{(a, b, c...) | a = b ∧ b ≥ 100 ∧ c ≤ 10 ...
each represents a ‘sub’-test data generation problem that can be solved independently and
combined with other partial solutions
![Page 277: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/277.jpg)
P. McMinn, How Does Program Structure Impact the Effectiveness of the Crossover Operator in Evolutionary Testing? Proc. Symposium on Search-Based Software Engineering, 2010
2. Conjuncts should reference disjoint sets of variables
{(a, b, c, d ...) | a = b ∧ b = c ∧ c = d ...
the solution of each conjunct independently does not necessarily result in an overall solution
What types of program and program structure enable Evolutionary Testing to perform well, through crossover and how?
![Page 278: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/278.jpg)
Progressive Landscapefit
ness
input input
![Page 279: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/279.jpg)
Crossover - Conclusions
Crossover lends itself to programs/units that process large data structures (e.g. strings, arrays) resulting in input condition conjuncts with disjoint variables
1. Large numbers of conjuncts in the input condition 2. Conjuncts should reference disjoint sets of variables
![Page 280: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/280.jpg)
Crossover - Conclusions
Crossover lends itself to programs/units that process large data structures (e.g. strings, arrays) resulting in input condition conjuncts with disjoint variables
1. Large numbers of conjuncts in the input condition 2. Conjuncts should reference disjoint sets of variables
... or units that require large sequences of method calls to move an object into a required state
![Page 281: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/281.jpg)
Crossover - Conclusions
Crossover lends itself to programs/units that process large data structures (e.g. strings, arrays) resulting in input condition conjuncts with disjoint variables
1. Large numbers of conjuncts in the input condition 2. Conjuncts should reference disjoint sets of variables
... or units that require large sequences of method calls to move an object into a required state
e.g. testing for a full stack - push(...), push(...), push(...)
![Page 282: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/282.jpg)
Other Theoretical Work
A. Arcuri, P. K. Lehre, and X. Yao, “Theoretical runtime analyses of search algorithms on the test data generation for the triangle classification problem,” SBST workshop 2008, Proc. ICST 2008. IEEE, 2008, pp. 161–169.
A. Arcuri, “Longer is better: On the role of test sequence length in software testing,” Proc. ICST 2010. IEEE, 2010.
![Page 283: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/283.jpg)
Future directions...
![Page 284: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/284.jpg)
The Oracle ProblemDetermining the correct output for a given input is called the oracle problem
Software engineering research has devoted much attention to automated oracles
... but many systems do not have automated oracles
![Page 285: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/285.jpg)
Human Oracle Cost Typically the responsibility falls on the human
Reducing human effort, the human oracle cost remains an important problem
![Page 286: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/286.jpg)
Test data generation and human oracle cost
generate test data that maximises coverage but minimises the number of test cases
reduce size of test cases
Quantitative
![Page 287: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/287.jpg)
Test data generation and human oracle cost
Quantitative generate test data that maximises coverage but minimises the number of test cases
reduce size of test casesA. Leitner, M. Oriol, A. Zeller, I. Ciupa, and B. Meyer.
Efficient unit test case minimization. ASE 2007, pp. 417–420. ACM.
M. Harman, S. G. Kim, K. Lakhotia, P. McMinn, and S. Yoo. Optimizing for the number of tests generated in search based test
data generation with an application to the oracle cost problem. 3rd Search-Based Software Testing workshop (SBST 2010), 2010. IEEE
digital library.
![Page 288: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/288.jpg)
Test data generation and human oracle cost
Qualitative e.g. how easily can the scenario comprising a test case be understood so that the output can be evaluated ?
![Page 289: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/289.jpg)
Test data generation and human oracle cost
Qualitative e.g. how easily can the scenario comprising a test case be understood so that the output can be evaluated ?
![Page 290: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/290.jpg)
Test data generation and human oracle cost
Qualitative e.g. how easily can the scenario comprising generated test databe understood so that the output can be evaluated ?
Phil McMinn, Mark Stevenson and Mark Harman.Reducing Qualitative Human Oracle Costs associated with Automatically Generated Test Data.
Proceedings of the 1st International Workshop on Software Test Output Validation (STOV 2010), to appear.
![Page 291: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/291.jpg)
Calendar program
Takes two dates (represented by 3 integers each)
Finds the number of days between the dates
![Page 292: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/292.jpg)
Calendar program
-4048/-10854/-29141 10430/3140/6733 3063/31358/8201
Some example dates generated:
Machine-generated test data tends to not fit the operational profile of a program particularly well
![Page 293: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/293.jpg)
Seeding knowledgeProgrammer test case used as the starting point of the search process
16/1/20102/1/20092/32/2010
![Page 294: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/294.jpg)
In what ways can operational profile knowledge be obtained?
Programmers
The Program
![Page 295: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/295.jpg)
In what ways can operational profile knowledge be obtained?
Likely to have run the program with a sanity check
These can be seeded to bias the test data generation process
Programmers
The Program
![Page 296: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/296.jpg)
In what ways can operational profile knowledge be obtained?
Programmers
The Program
Identifier names
Code comments
Sanitisation routines
![Page 297: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/297.jpg)
Identifier namesGive clues to the sort of inputs that
might be expected
dayOfTheWeek
country_of_origin
url
Can be analysed in conjunction with large-scale natural language lexicons such as WordNet
![Page 298: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/298.jpg)
Sanitisation Routines
Defensive programming routines might be used to ‘correct’ a program’s own test data
![Page 299: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/299.jpg)
Test Data Re-use
![Page 300: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/300.jpg)
Test Data Re-use
![Page 301: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/301.jpg)
Program similarity and test data re-use
Code structure
Identifier names
Code comments
clone detection
plagiarism detection
![Page 302: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/302.jpg)
Will these techniques work?
Will they compromise fault-finding capability?
![Page 304: Search-Based Testing · Applications in Mutation Testing Test suite prioritisation. Acknowledgements ... Search-Based Testing Write a fitness function to determine how good a test](https://reader033.fdocuments.us/reader033/viewer/2022050201/5f54cd35f7db72109523bd36/html5/thumbnails/304.jpg)
Questions &Discussion