Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos.
-
Upload
abner-mcdowell -
Category
Documents
-
view
214 -
download
0
Transcript of Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos.
Using Dynamic Using Dynamic Compilers for Software Compilers for Software
TestingTesting
Ben BreechBen BreechLori PollockLori Pollock
John CavazosJohn Cavazos
Background: Static CompilersBackground: Static Compilers
sourcecode
Lexical, syntactic, Semantic analysis
Inter. Rep.(basic blocks)
Static AnalysisOptimization
Code generationExecutable
Code
Static Compiler
Background: Background: Software TestingSoftware Testing
Assesses Assesses qualityquality of software of software• Correctness, performance, securityCorrectness, performance, security
Most testing performed by executing codeMost testing performed by executing code
ExecutableCode
TestInput
ExecuteProgram
ProgramResults
ExpectedResults
Check Results(oracle)
Motivating ExampleMotivating Exampleif ((sptr = malloc (size + 1)) == NULL) {if ((sptr = malloc (size + 1)) == NULL) {
findmem ();findmem ();if ((sptr = malloc (size + 1)) == NULL)if ((sptr = malloc (size + 1)) == NULL)
xlfail (“insufficient string space”); }xlfail (“insufficient string space”); }
How do I test this callsite?How do I test this callsite?
Make the machine run out of memory?Make the machine run out of memory?Flip the conditional, recompile, flip back?Flip the conditional, recompile, flip back?
Pretend it doesn’t exist during testing?Pretend it doesn’t exist during testing?
Generalizing the ProblemGeneralizing the Problem
Code to handle Code to handle uncommon situationsuncommon situations• Difficult to testDifficult to test• May need external environment event to triggerMay need external environment event to trigger
Examples:Examples:• Error handling codeError handling code• Testing program security mechanismsTesting program security mechanisms
ObservationObservation
Hard to reach code executes when program Hard to reach code executes when program thinksthinks something uncommon has occurred something uncommon has occurred
if ((sptr = malloc (size + 1)) == NULL) {if ((sptr = malloc (size + 1)) == NULL) { findmem ();findmem ();
xlfail (“insufficient string space”); }xlfail (“insufficient string space”); }if ((sptr = malloc (size + 1)) == NULL)if ((sptr = malloc (size + 1)) == NULL)
Could test Could test findmemfindmem() by simulating error() by simulating error• E.g., could add instructions to program so program E.g., could add instructions to program so program
believes believes mallocmalloc failed failed
RUGRAT ApproachRUGRAT Approach
Use Use Dynamic CompilersDynamic Compilers to generate to generate test cases for hard to reach code.test cases for hard to reach code.
AutomaticallyAutomatically add instructions to add instructions to program program during executionduring execution to simulate to simulate uncommon situation.uncommon situation.
Dynamic CompilersDynamic Compilers
Dynamic compilers perform compilation Dynamic compilers perform compilation tasks tasks duringduring program execution program execution
code
Analysis&
optimization
Create basic block
translate
Basicblock
Mod. Basicblock
Executeon CPU
Dynamic Compiler
RUGRAT ArchitectureRUGRAT Architecture
code
Analysis&
optimization
Create basic block
translate
Basicblock
Mod. Basicblock
Executeon CPU
Dynamic Compiler
Create basic block
Dynatest Generator
Testspec
Test OracleTest
Report
Dynatest GeneratorDynatest Generator
Decides when/where/how to add testDecides when/where/how to add test• Decision based on test specificationDecision based on test specification
Test spec examplesTest spec examples• ““test all test all mallocsmallocs in program” in program”• ““test 2nd test 2nd mallocmalloc call in function call in function foofoo””
ExampleExampleif ((sptr = malloc (size + 1)) == NULL) {if ((sptr = malloc (size + 1)) == NULL) {
findmem ();findmem ();
xlfail (“insufficient string space”); }xlfail (“insufficient string space”); }
if ((sptr = malloc (size + 1)) == NULL)if ((sptr = malloc (size + 1)) == NULL)
call malloc (code for malloc)movl <return val> sptrcmpl sptr, 0jnz L1call findmem….L1: …
Dynatest Generator
call malloc (code for malloc)movl 0, <return val>movl ENOMEM, errnomovl <return val> sptrcmpl sptr, 0jnz L1call findmem….L1: …
Good Times, Bad TimesGood Times, Bad Times
The Bad: Not a perfect simulationNot a perfect simulation
The Good: Adequate simulationAdequate simulation Can target system or appl callsCan target system or appl calls Saves quite a lot of tester effortSaves quite a lot of tester effort
Security Mechanism Testing:Security Mechanism Testing:Encrypting Function PointersEncrypting Function Pointers
Protects progs against func pointer attacksProtects progs against func pointer attacks Difficult to test (need vulnerable program and Difficult to test (need vulnerable program and
attack)attack) RUGRAT can simulate attack by adding RUGRAT can simulate attack by adding
instructionsinstructions• Very different from error handling code caseVery different from error handling code case
RUGRAT can be used for variety of RUGRAT can be used for variety of testing tasks.testing tasks.
Experiments SummaryExperiments Summary
Tested variety programs with RUGRATTested variety programs with RUGRAT 120+ error code handling callsites covered120+ error code handling callsites covered
• Both application and system callsBoth application and system calls Increased error code coverage ~ 50% over Increased error code coverage ~ 50% over
regular test casesregular test cases• Not all error code statements could be coveredNot all error code statements could be covered
• Different options, etcDifferent options, etc
Reasonable time overheadReasonable time overhead
Some related workSome related work
Holodeck [1], FIG [2] Holodeck [1], FIG [2] • Require tester provide alternative “stub” Require tester provide alternative “stub”
functions to do testingfunctions to do testing• Miss application callsMiss application calls
Dynamic branch switching [3]Dynamic branch switching [3]• Not originally intended for testing error codeNot originally intended for testing error code• Need to know which branch to changeNeed to know which branch to change• Far less accurate simulationFar less accurate simulation
[1] Thompson et al., SAC 2002[1] Thompson et al., SAC 2002[2] Broadwell et al., SHAMAN 2002[2] Broadwell et al., SHAMAN 2002[3] Zhang et al., ICSE 2006[3] Zhang et al., ICSE 2006
Conclusions and SummaryConclusions and Summary
Presented RUGRAT architecturePresented RUGRAT architecture• Can test hard to reach (and seldom tested) Can test hard to reach (and seldom tested)
code by using dynamic compilerscode by using dynamic compilers• Saves tester effortSaves tester effort
RUGRAT is a general toolRUGRAT is a general tool
RUGRAT ArchitectureRUGRAT Architecture
code
Basicblock
Mod. Basicblock
Executeon CPU
Dynamic Compiler
Create basic block
Dynatest Generator
Testspec
Test OracleTest
Report