Fault Interaction and its Repercussions
Nicholas DiGiuseppe James A. Jones
UC Irvine Spider Lab
Fault Interaction Defined
Multi-Type Synergy Obfuscation Pass/Fail Independence Pass/Fail Independence
4 Types of Interaction
Fault Interaction Defined
4 Types of Interaction
Multi-Type Synergy Obfuscation Pass/Fail Independence Obfuscation
Fault Interaction Defined
4 Types of Interaction
Multi-Type Synergy Obfuscation Pass/Fail Independence
Synergy
Fault Interaction Defined
4 Types of Interaction
Multi-Type Synergy Obfuscation Pass/Fail Independence
Multi-Type
We Need More Fault Understanding
Conferences: ASE, FSE, ICSE, ICSM, ICST, ISSTA
Years: 2008–2009
27 Fault Localization = 2.25 per conference per year 4 Fault Behavior = 0.333 per conference per year
Motivating Example
Expected Results
Test Case 1: input [0,0] 1 Test Case 2: input [2,0] 0
X = input [0] - 2 Y = input [0] - 2 If (X == Y) print 1 else print 0
Actual Results
Pass/Fail Status
1 Pass 1 Fail
Motivating Example
Expected Results
Test Case 1: input [0,0] 1 Test Case 2: input [2,0] 0
X = input [0] - 2 Y = input [1] / 2 If (X == Y) print 1 else print 0
Actual Results
Pass/Fail Status
0 Fail 1 Fail
Motivating Example
Expected Results
Test Case 1: input [0,0] 1 Test Case 2: input [2,0] 0
X = input [0] * 2 Y = input [1] / 2 If (X == Y) print 1 else print 0
Actual Results
Pass/Fail Status
1 Pass 0 Pass
Our Experiment Flex Gzip Replace Schedule Sed Space Total
1 Fault 21 20 25 20 20 33 139
2 Faults 189 182 185 165 188 200 1109
3 Faults 502 505 598 500 660 200 2965
4 Faults 1000 1000 700 500 950 200 4350
5 Faults 1000 1000 700 500 950 200 4350
17 Faults 1000 1000 700 500 950 200 4350
Total Versions 14,712 14,707 10,608 7,685 14,168 3,233 65,113
Size of Test Suite 527 214 5,542 2,650 363 13,527 22,733
Number of Executions
7,753,224 3,147,298 58,798,536 17,673,600 6,253,984 43,732,791 138,239,433
Flex Gzip Replace Schedule Sed Space Total
1 Fault 21 20 25 20 20 33 139
2 Faults 189 182 185 165 188 200 1109
3 Faults 502 505 598 500 660 200 2965
4 Faults 1000 1000 700 500 950 200 4350
5 Faults 1000 1000 700 500 950 200 4350
17 Faults 1000 1000 700 500 950 200 4350
Total Versions 14,712 14,707 10,608 7,685 14,168 3,233 65,113
Size of Test Suite 527 214 5,542 2,650 363 13,527 22,733
Number of Executions
7,753,224 3,147,298 58,798,536 17,673,600 6,253,984 43,732,791 138,239,433
Total Versions 65,113
Number of Executions 138,239,433
Our Experiment
Results
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17The Quantity of Faults
0.0
0.2
0.4
0.6
0.8
1.0
Perc
enta
geof
Inte
ract
ion
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17The Quantity of Faults
0.0
0.2
0.4
0.6
0.8
1.0
Perc
enta
geof
Inte
ract
ion
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17The Quantity of Faults
0.0
0.2
0.4
0.6
0.8
1.0
Perc
enta
geof
Inte
ract
ion
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17The Quantity of Faults
0.0
0.2
0.4
0.6
0.8
1.0
Perc
enta
geof
Inte
ract
ion
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17The Quantity of Faults
0.0
0.2
0.4
0.6
0.8
1.0
Perc
enta
geof
Inte
ract
ion
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17The Quantity of Faults
0.0
0.2
0.4
0.6
0.8
1.0
Perc
enta
geof
Inte
ract
ion
a
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17The Quantity of Faults
0.0
0.2
0.4
0.6
0.8
1.0
Perc
enta
geof
Inte
ract
ion
Obfuscation
Synergy Independence
Multi Type
Impact of Fault Interaction
Fault Localization
Failure Clustering
Regression Testing
Run Test
Suite
Cluster
FailuresAssign Clusters
to Developers
Fix Faults
in Clusters
Pass
Two Faults Fix a Fault One Fault
Test 1
Test 2
Test 3
Pass
Fail
Pass
Pass
Fail
Time
Results Continued Gzip Flex Replace Schedule Sed Space
LoEC 2013 4020 242 150 2253 3651
Branches 1727 2680 182 66 2554 1190
Methods 514 1261 63 41 416 717
Data Dep 45,291 143,075 2005 1327 94,614 29,883
Control Dep 26,766 61,279 1444 1074 21,991 25,047
Total Dep 72,057 204,354 3449 2401 116,605 54,930
LoEC: Branches
1.165 1.5 3.818 2.273 1.893 3.068
LoEC: Methods 3.916 3.188 3.841 3.659 5.416 5.092
LoEC: Data Dep
0.044 0.028 0.121 0.113 0.024 0.122
LoEC: Control Dep
0.075 0.066 0.168 0.140 0.103 0.146
LoEC: Total Dep
0.028 0.020 0.070 0.062 0.019 0.067
Conclusions
Defined Fault Interaction Demonstrated that Fault Interaction is prominent (primarily fault obfuscation) Demonstrated that Fault Interaction can have a real impact on practice and research Demonstrated that the size of a program affects Fault Interaction
Take Home Message
Fault Interaction is prevalent Fault Obfuscation is prevalent
We need to adapt our techniques
Conclusions
Thank You!
This work was partially funded by: University of California, Irvine
Top Related