A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le...
-
Upload
giselle-frisbie -
Category
Documents
-
view
217 -
download
3
Transcript of A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le...
![Page 1: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/1.jpg)
A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH
Claire Le Goues
Michael Dewey-Vogt
Stephanie Forrest
Westley Weimer
http://genprog.cs.virginia.edu 1
![Page 2: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/2.jpg)
Claire Le Goues, ICSE 2012 2
PROBLEM: BUGGY SOFTWARE
http://genprog.cs.virginia.edu
“Everyday, almost 300 bugs appear […] far too many for only the Mozilla programmers to handle.”
– Mozilla Developer, 2005
Annual cost of software errors in the
US: $59.5 billion (0.6% of GDP).
Average time to fix a security-critical error:
28 days.
90%: Maintenance
10%: Everything Else
![Page 3: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/3.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 3
HOW BAD IS IT?
![Page 4: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/4.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 4
![Page 5: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/5.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 5
![Page 6: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/6.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 6
Tarsnap:
125 spelling/style
63 harmless
11 minor+ 1 major
75/200 = 38% TP rate
$17 + 40 hours per TP
…REALLY?
![Page 7: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/7.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 7
Tarsnap:
125 spelling/style
63 harmless
11 minor+ 1 major
75/200 = 38% TP rate
$17 + 40 hours per TP
…REALLY?
![Page 8: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/8.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 8
…REALLY?
![Page 9: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/9.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 9
SOLUTION: PAY STRANGERS
![Page 10: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/10.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 10
SOLUTION: PAY STRANGERS
![Page 11: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/11.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 11
SOLUTION: AUTOMATE
![Page 12: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/12.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 12
GENPROG: AUTOMATIC1, SCALABLE, COMPETITIVE BUG REPAIR.
AUTOMATED PROGRAM REPAIR
1 C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer, “GenProg: A generic method for automated software repair,” Transactions on Software Engineering, vol. 38, no. 1, pp. 54– 72, 2012.
W. Weimer, T. Nguyen, C. Le Goues, and S. Forrest, “Automatically finding patches using genetic programming,” in International Conference on Software Engineering, 2009, pp. 364–367.
![Page 13: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/13.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 13
GENPROG: AUTOMATIC1, SCALABLE, COMPETITIVE BUG REPAIR.
AUTOMATED PROGRAM REPAIR
1 C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer, “GenProg: A generic method for automated software repair,” Transactions on Software Engineering, vol. 38, no. 1, pp. 54– 72, 2012.
W. Weimer, T. Nguyen, C. Le Goues, and S. Forrest, “Automatically finding patches using genetic programming,” in International Conference on Software Engineering, 2009, pp. 364–367.
![Page 14: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/14.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 14
GENPROG: AUTOMATIC, SCALABLE, COMPETITIVE BUG REPAIR.
AUTOMATED PROGRAM REPAIR
![Page 15: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/15.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 15
GENPROG: AUTOMATIC, SCALABLE, COMPETITIVE BUG REPAIR.
AUTOMATED PROGRAM REPAIR
![Page 16: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/16.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 16
GENPROG: AUTOMATIC, SCALABLE, COMPETITIVE BUG REPAIR.
AUTOMATED PROGRAM REPAIR
![Page 17: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/17.jpg)
Claire Le Goues, ICSE 2012
INPUT
OUTPUT
EVALUATE FITNESS
DISCARD
ACCEPT
MUTATE
![Page 18: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/18.jpg)
Claire Le Goues, ICSE 2012
DISCARD
INPUT EVALUATE FITNESS
MUTATE
ACCEPT
OUTPUT
![Page 19: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/19.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 19
Search: random (GP) search through nearby patches.
Approach: compose small random edits.
•Where to change?•How to change it?
BIRD’S EYE VIEW
![Page 20: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/20.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 20
Input:
2
5 6
1
3 4
8
7
9
11
10
12
![Page 21: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/21.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 21
Input:
2
5 6
1
3 4
8
7
9
11
10
12
Legend:
High change probability.
Low change probability.
Not changed.
![Page 22: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/22.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 22
2
5 6
1
3 4
8
7
9
11
10
12
An edit is:
• Replace statement X with statement Y
• Insert statement X after statement Y
• Delete statement X
![Page 23: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/23.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 23
2
5 6
1
3 4
8
7
9
11
10
12
An edit is:
• Replace statement X with statement Y
• Insert statement X after statement Y
• Delete statement X
![Page 24: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/24.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 24
2
5 6
1
3 4
8
7
9
11
10
12
An edit is:
• Replace statement X with statement Y
• Insert statement X after statement Y
• Delete statement X
![Page 25: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/25.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 25
2
5 6
1
3 4
8
7
9
11
10
12
An edit is:
• Replace statement X with statement Y
• Insert statement X after statement Y
• Delete statement X
![Page 26: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/26.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 26
2
5 6
1
3 4
8
7
9
11
10
12
An edit is:
• Replace statement X with statement Y
• Insert statement X after statement Y
• Delete statement X
4
![Page 27: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/27.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 27
2
5 6
1
3 4
8
7
9
11
10
12
An edit is:
• Replace statement X with statement Y
• Insert statement X after statement Y
• Delete statement X
4
![Page 28: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/28.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 28
2
5 6
1
3 4
7
9
11
10
12
An edit is:
• Replace statement X with statement Y
• Insert statement X after statement Y
• Delete statement X
4
4’
![Page 29: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/29.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 29
2
5 6
1
3 4
7
9
11
10
12
An edit is:
• Replace statement X with statement Y
• Insert statement X after statement Y
• Delete statement X
4
4’
![Page 30: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/30.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 30
GENPROG: AUTOMATIC, SCALABLE, COMPETITIVE BUG REPAIR.
AUTOMATED PROGRAM REPAIR
![Page 31: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/31.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 31
GENPROG: AUTOMATIC, SCALABLE, COMPETITIVE BUG REPAIR.
AUTOMATED PROGRAM REPAIR
![Page 32: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/32.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 32
SCALABLE: SEARCH SPACE
http://genprog.cs.virginia.edu 32
2
5 6
1
3 4
8
7
9
11
10
12
![Page 33: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/33.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 33
SCALABLE: SEARCH SPACE
http://genprog.cs.virginia.edu 33
2
5 6
1
3 4
8
7
9
11
10
12
![Page 34: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/34.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 34
SCALABLE: SEARCH SPACE
http://genprog.cs.virginia.edu 34
2
5 6
1
3
8
7
9
11
10
12
4
![Page 35: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/35.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 35
SCALABLE: SEARCH SPACE
http://genprog.cs.virginia.edu 35
2
5 6
1
3
8
7
9
11
10
12
4
Fix localization: intelligently choose code to move.
![Page 36: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/36.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 36
SCALABLE: REPRESENTATION
1
2
54
Naïve:
1
2
4 5
5’
1
32
54
Input:
New:
Delete(3)
Replace(3,5)
![Page 37: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/37.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 37
SCALABLE: REPRESENTATION
1
2
54
Naïve:
1
2
4 5
5’
1
32
54
Input:
New:
Delete(3)
Replace(3,5)
New fitness, crossover, and mutation operators to work with
a variable-length genome.
![Page 38: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/38.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 38
SCALABLE: PARALLELISM
Fitness:
• Subsample test cases.
• Evaluate in parallel.
Random runs:
• Multiple simultaneous runs on different seeds.
![Page 39: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/39.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 39
GENPROG: AUTOMATIC, SCALABLE, COMPETITIVE BUG REPAIR.
AUTOMATED PROGRAM REPAIR
![Page 40: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/40.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 40
GENPROG: AUTOMATIC, SCALABLE, COMPETITIVE BUG REPAIR.
AUTOMATED PROGRAM REPAIR
![Page 41: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/41.jpg)
Claire Le Goues, ICSE 2012 41
COMPETITIVE
http://genprog.cs.virginia.edu
How many bugs can GenProg fix?
How much does it cost?
![Page 42: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/42.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 42
Goal: systematically test GenProg on a general, indicative bug set.
General approach:
• Avoid overfitting: fix the algorithm. • Systematically create a generalizable benchmark set.
• Try to repair every bug in the benchmark set, establish grounded cost measurements.
SETUP
![Page 43: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/43.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 43
Goal: systematically evaluate GenProg on a general, indicative bug set.
General approach:
• Avoid overfitting: fix the algorithm. • Systematically create a generalizable benchmark set.
• Try to repair every bug in the benchmark set, establish grounded cost measurements.
SETUP
![Page 44: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/44.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 44
CHALLENGE: INDICATIVE BUG SET
![Page 45: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/45.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 45
Goal: a large set of important, reproducible bugs in non-trivial programs.
Approach: use historical data to approximate discovery and repair of bugs in the wild.
SYSTEMATIC BENCHMARK SELECTION
![Page 46: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/46.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 46
Consider top programs from SourceForge, Google Code, Fedora SRPM, etc:
• Find pairs of viable versions where test case behavior changes.
• Take all tests from most recent version.• Go back in time through the source control.
Corresponds to a human-written repair for the bug tested by the failing test case(s).
SYSTEMATIC BENCHMARK SELECTION
![Page 47: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/47.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 47
BENCHMARKS
Program LOC Tests Bugs Description
fbc 97,000 773 3 Language (legacy)
gmp 145,000 146 2 Multiple precision math
gzip 491,000 12 5 Data compression
libtiff 77,000 78 24 Image manipulation
lighttpd 62,000 295 9 Web server
php 1,046,000 8,471 44 Language (web)
python 407,000 355 11 Language (general)
wireshark 2,814,000 63 7 Network packet analyzer
Total 5,139,000 10,193 105
![Page 48: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/48.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 48
CHALLENGE: GROUNDED COST MEASUREMENTS
![Page 49: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/49.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 49
![Page 50: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/50.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 50
![Page 51: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/51.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 51
READY
![Page 52: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/52.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 52
GO
![Page 53: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/53.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 53
13 HOURS LATER
![Page 54: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/54.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 54
SUCCESS/COST
Program
Defects Repaired
Cost per non-repair Cost per repair
Hours US$ Hours US$
fbc 1/3 8.52 5.56 6.52 4.08
gmp 1/2 9.93 6.61 1.60 0.44
gzip 1/5 5.11 3.04 1.41 0.30
libtiff 17/24 7.81 5.04 1.05 0.04
lighttpd 5/9 10.79 7.25 1.34 0.25
php 28/44 13.00 8.80 1.84 0.62
python 1/11 13.00 8.80 1.22 0.16
wireshark 1/7 13.00 8.80 1.23 0.17
Total 55/105 11.22h 1.60h
$403 for all 105 trials, leading to 55 repairs; $7.32 per bug repaired.
![Page 55: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/55.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 55
JBoss issue tracking: median 5.0, mean 15.3 hours.1
IBM: $25 per defect during coding, rising at build, Q&A, post-release, etc.2
Tarsnap.com: $17, 40 hours per non-trivial repair.3
Bug bounty programs in general:
• At least $500 for security-critical bugs.• One of our php bugs has an associated security CVE.
1C. Weiß, R. Premraj, T. Zimmermann, and A. Zeller, “How long will it take to fix this bug?” in Workshop on Mining Software Repositories, May 2007.
2 L. Williamson, “IBM Rational software analyzer: Beyond source code,” in Rational Software Developer Conference, Jun. 2008.
3http://www.tarsnap.com/bugbounty.html
PUBLIC COMPARISON
![Page 56: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/56.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 56
GenProg: scalable, automatic bug repair.
• Algorithmic improvements for scalability: fix localization, internal representation, parallelism.
Systematic study:
• Indicative, systematically-generated set of bugs that humans care about.
• Repaired 52% of 105 bugs in 96 minutes, on average, for $7.32 each.
Benchmarks/results/source code/VM images available:
• http://genprog.cs.virginia.edu
CONCLUSIONS/CONTRIBUTIONS
![Page 57: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/57.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 57
I LOVE QUESTIONS.
(Examples: “Which bugs can GenProg fix?” “What happens if you run for more than 13 hours/change the probability distributions/pick a different crossover/etc?” “How do you know the patches are any good?” “How do your patches compare to human patches?” …)
![Page 58: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/58.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 58
WHICH BUGS…?
Slightly more likely to fix bugs where the human:
• restricts the repair to statements.• touched fewer files.
As fault space decreases, success increases, repair time decreases.
As fix space increases, repair time decreases.
![Page 59: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/59.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu
FINDING BUGS IS HARD
Opaque or non-automated GUI testing.
• Firefox, Eclipse, OpenOffice
Inaccessible or small version control histories.
• bash, cvs, openssh
Few viable versions for recent tests.
• valgrind
Require incompatible automake, libtool
• Earlier versions of gmp
No bugs
• GnuCash, openssl
Non-deterministic tests ...
![Page 60: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/60.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 60
1. class test_class { 2. public function __get($n) 3. { return $this; %$ }4. public function b()5. { return; }6. }7. global $test3; 8. $test3 = new test_class(); 9. $test3->a->b();
EXAMPLE: PHP BUG #54372
Relevant code: function zend_std_read_property in zend_object_handlers.c
Note: memory management uses reference counting.
Problem: this line:
449.zval_ptr_dtor(object)
If object points to $this and $this is global, its memory is completely freed, even though we could access $this later.
Expected output: nothing
Buggy output: crash on line 9.
![Page 61: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/61.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 61
GenProg :
% 448c448,451
> Z_ADDROF_P(object);
> if
(PZVAL_IS_REF(object))
> {
>
SEPARATE_ZVAL(&object);
> }
zval_ptr_dtor(&object)
EXAMPLE: PHP BUG #54372
Human : % 449c449,453
< zval_ptr_dtor(&object);
> if (*retval != object)
> { // expected
> zval_ptr_dtor(&object);
> } else {
> Z_DELREF_P(object);
> }
![Page 62: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/62.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 62
Is automatically-patched code more or less maintainable?
Approach: Ask 102 humans maintainability questions about patched code (human vs. GenProg).
Results:
• No difference in accuracy/time between human accepted and GenProg patches.
• Automatically-documented GenProg patches result in higher accuracy and lower effort than human patches.
Zachary P. Fry, Bryan Landau, Westley Weimer: A Human Study of Patch Maintainability. International Symposium on Software Testing and Analysis (ISSTA) 2012: to appear
PATCH QUALITY
![Page 63: A SYSTEMATIC STUDY OF AUTOMATED PROGRAM REPAIR: FIXING 55 OUT OF 105 BUGS FOR $8 EACH Claire Le Goues Michael Dewey-Vogt Stephanie Forrest Westley Weimer.](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649cad5503460f9496f678/html5/thumbnails/63.jpg)
Claire Le Goues, ICSE 2012 http://genprog.cs.virginia.edu 63
PATCH REPRESENTATION
Program Fault LOC Repair Ratio
gcd infinite loop 22 1.07
uniq-utx segfault 1146 1.01
look-utx segfault 1169 1.00
look-svr infinite loop 1363 1.00
units-svr segfault 1504 3.13
deroff-utx segfault 2236 1.22
nullhttpd buffer exploit 5575 1.95
indent infinite loop 9906 1.70
flex segfault 18775 3.75
atris buffer exploit 21553 0.97
Average 6325 1.68