Faults and Regression Testing - Fault interaction and its repercussions

download Faults and Regression Testing - Fault interaction and its repercussions

of 17

  • date post

    13-Jan-2015
  • Category

    Technology

  • view

    415
  • download

    3

Embed Size (px)

description

Paper: Fault Interaction and its RepercussionsAuthors: Nicholas DiGiuseppe and James A. JonesSeesion: Research Track 1: Faults and Regression Testing

Transcript of Faults and Regression Testing - Fault interaction and its repercussions

  • 1. Fault Interaction and its RepercussionsNicholas DiGiuseppeJames A. Jones UC IrvineSpider Lab

2. Fault Interaction Defined4 Types of InteractionPass/Fail IndependenceObfuscationSynergyMulti-Type 3. Fault Interaction Defined4 Types of InteractionPass/Fail IndependenceObfuscationSynergyMulti-Type 4. Fault Interaction Defined4 Types of InteractionPass/Fail IndependenceObfuscationSynergyMulti-Type 5. Fault Interaction Defined4 Types of InteractionPass/Fail IndependenceObfuscationSynergyMulti-Type 6. We Need More Fault Understanding Conferences:ASE, FSE, ICSE, ICSM, ICST, ISSTA Years: 2008200927 Fault Localization = 2.25 per conference per year4 Fault Behavior = 0.333 per conference per year 7. Motivating Example X = input [0] - 2 Y = input [0] - 2 If (X == Y) print 1 else print 0 ExpectedActualPass/Fail Results Results StatusTest Case 1: input [0,0] 1 1 PassTest Case 2: input [2,0] 0 1 Fail 8. Motivating Example X = input [0] - 2 Y = input [1] / 2 If (X == Y) print 1 else print 0 ExpectedActualPass/Fail Results Results StatusTest Case 1: input [0,0] 1 0 FailTest Case 2: input [2,0] 0 1 Fail 9. Motivating Example X = input [0] * 2 Y = input [1] / 2 If (X == Y) print 1 else print 0 ExpectedActualPass/Fail Results Results StatusTest Case 1: input [0,0] 1 1 PassTest Case 2: input [2,0] 0 0 Pass 10. Our Experiment FlexGzipReplaceSchedule Sed SpaceTotal1 Fault212025 20 2033 1392 Faults 189 182 185165188 20011093 Faults 502 505 598500660 20029654 Faults 10001000700500950 20043505 Faults 10001000700500950 200435017 Faults10001000700500950 2004350Total Versions 14,71214,70710,608 7,68514,1683,23365,113Size of Test Suite 527 214 5,5422,650363 13,527 22,733Number of7,753,224 3,147,298 58,798,536 17,673,600 6,253,984 43,732,791 138,239,433Executions 11. Our Experiment FlexGzipReplaceSchedule Sed SpaceTotal1 Fault212025 20 2033 1392 Faults 189 182 185165188 20011093 Faults 502 505 598500660 20029654 Faults 10001000700500950 20043505 Faults 10001000700500950 200435017 Faults10001000700500950 2004350 Total VersionsTotal Versions 14,712 14,707 10,608 65,1137,68514,1683,23365,113Size of Test Suite 527 214 5,5422,650363 13,527 22,733Number of Executions 138,239,433Number ofExecutions 7,753,224 3,147,298 58,798,536 17,673,600 6,253,984 43,732,791 138,239,433 12. aResults1.0 1.0ObfuscationMulti Type 1.00.8 0.80.8Percentage of InteractionPercentage of InteractionPercentage of Interaction0.6 0.6Synergy Independence0.60.4 0.40.40.2 0.20.20.0 0.00.0 00 11 22 33 44 55 66 77 88 99 10 11 12 13 14 15 16 1710 11 12 13 14 15 16 170 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17The Quantity of Faults The Quantity of Faults The Quantity of Faults1.0 1.00.8 0.8Percentage of InteractionPercentage of Interaction0.6 0.60.4 0.40.2 0.20.0 0.0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17The Quantity of FaultsThe Quantity of Faults1.0 1.00.8 0.8Percentage of InteractionPercentage of Interaction0.6 0.60.4 0.40.2 0.20.0 0.0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17The Quantity of FaultsThe Quantity of Faults 13. Impact of Fault InteractionRegression TestingFault LocalizationTwo Faults Fix a Fault One Fault Test 1 Pass Fail Test 2 Pass Pass Test 3 Fail Pass Time Failure ClusteringRun Test Fix Faults Suite in ClustersCluster Assign ClustersFailures to Developers 14. Results ContinuedGzip FlexReplace Schedule Sed SpaceLoEC2013 4020242 15022533651Branches1727 2680182 66 25541190Methods 51412616341 416 717Data Dep45,291 143,075 20051327 94,61429,883Control Dep 26,766 61,27914441074 21,99125,047Total Dep 72,057 204,354 34492401 116,605 54,930LoEC: 1.1651.5 3.818 2.2731.893 3.068BranchesLoEC: Methods 3.9163.188 3.841 3.6595.416 5.092LoEC: Data0.0440.028 0.121 0.1130.024 0.122DepLoEC: Control 0.0750.066 0.168 0.1400.103 0.146DepLoEC: Total 0.0280.020 0.070 0.0620.019 0.067Dep 15. ConclusionsDefined Fault InteractionDemonstrated that Fault Interaction is prominent (primarilyfault obfuscation)Demonstrated that Fault Interaction can have a realimpact on practice and researchDemonstrated that the size of a program affects FaultInteraction 16. Conclusions Take Home MessageFault Interaction is prevalent Fault Obfuscation is prevalentWe need to adapt our techniques 17. Thank You!This work was partially funded by:University of California, Irvine