ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert:...
Transcript of ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert:...
![Page 1: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/1.jpg)
ReAssert: Suggesting Repairs for Broken Unit Tests
Brett DanielVilas Jagannath
Danny DigDarko Marinov
![Page 2: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/2.jpg)
http://xkcd.com/662/Creative Commons Attribution-Noncommercial
This is Alice
![Page 3: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/3.jpg)
Her unit tests pass
public class Cart { ... public double getTotalPrice() {...} public String getPrintedBill() {...} ...}
public void testAddTwoDifferentProducts() { Cart cart = ... assertEquals(3.0, cart.getTotalPrice()); assertEquals( "Discount: -$1.00, Total: $3.00", cart.getPrintedBill()); }
![Page 4: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/4.jpg)
But requirements change
public void testAddTwoDifferentProducts() { Cart cart = ... assertEquals(3.0, cart.getTotalPrice()); assertEquals( "Discount: -$1.00, Total: $3.00", cart.getPrintedBill()); }
public class Cart { ... public double getTotalPrice() {...} public String getPrintedBill() {...} ...}
![Page 5: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/5.jpg)
She can delete broken tests
But that reducesthe quality of the
test suite.
![Page 6: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/6.jpg)
Repairing tests is preferable
But that requiresa lot of timeand effort
![Page 7: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/7.jpg)
ReAssert suggests repairs
![Page 8: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/8.jpg)
Alice decides whether to apply
![Page 9: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/9.jpg)
ReAssert reduces effort
![Page 10: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/10.jpg)
assertTrue(true);
assertEquals(3.0, cart.getTotalPrice());
Bad Repair!
What is a Good Repair?
![Page 11: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/11.jpg)
Good Repair
Make tests pass
Make minimal changes to test code (not SUT)
Require developer approval
Produce understandable test code
Repair Criteria
![Page 12: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/12.jpg)
Repair Strategies
● Strategies specific to:● Static structure of the code● The type of failure● The runtime values that caused the failure
● Seven general strategies + custom strategies
![Page 13: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/13.jpg)
Simple Assertion Failure
assertEquals(3.0, cart.getTotalPrice());
![Page 14: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/14.jpg)
Strategy: Replace Literal
assertEquals(6.0, cart.getTotalPrice());
Record actual valueReplace in code
![Page 15: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/15.jpg)
Failure in Helper Method
void testAddTwoDifferentProducts() { Cart cart = ... ... checkCart(cart, 3.0, ...);}
void checkCart( Cart cart, double total, ...) { ... assertEquals(total, cart.getTotalPrice()); ...}
![Page 16: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/16.jpg)
Strategy: TraceDeclaration-Use Path
void testAddTwoDifferentProducts() { Cart cart = ... ... checkCart(cart, 6.0, ...);}
void checkCart( Cart cart, double total, ...) { ... assertEquals(total, cart.getTotalPrice()); ...}
Trace declaration-use path
Record actual value
Replace in code
![Page 17: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/17.jpg)
Object (In)Equality Failure
Product expected = ...Product actual = ...assertEquals(expected, actual);
![Page 18: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/18.jpg)
Strategy: Expand Accessors
Product expected = ...Product actual = ...{ assertEquals( , actual.getPrice()); assertEquals( , actual.getDescription());}
Expand accessors
![Page 19: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/19.jpg)
Strategy: Expand Accessors
Product expected = ...Product actual = ...{ assertEquals(expected.getPrice(), actual.getPrice()); assertEquals("Red pen", actual.getDescription());}
Expected and actualaccessors equal
Actual accessor differs
![Page 20: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/20.jpg)
![Page 21: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/21.jpg)
Instrument
public static void assertEquals ( Object expected, Object actual) { try { // ...assert expected.equals(actual) } catch (Error e) { throw new RecordedAssertFailure( e, expected, actual); }}
...then record valuesthat caused failure
If assertion fails...
![Page 22: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/22.jpg)
Execute
assertEquals(3.0, cart.getTotalPrice());
throw RecordedAssertFailure(e, 3.0, 6.0); edu.illinois.reassert.RecordedAssertFailure: org.junit.AssertionFailedError: expected:<3.0> but was:<6.0> at org.junit.Assert.assertEquals(Assert.java:116) at CartTest.testRedPenCoupon(CartTest.java:6) ...
![Page 23: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/23.jpg)
Find Repair Location
edu.illinois.reassert.RecordedAssertFailure:org.junit.AssertionFailedError:expected:<3.0> but was:<6.0> at org.junit.Assert.assertEquals(Assert.java:116)
at CartTest.testRedPenCoupon(CartTest.java:6) ...
![Page 24: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/24.jpg)
Choose Strategy and Apply
assertEquals(3.0, cart.getTotalPrice());
Replace Literal in Assertion strategy. ..
assertEquals(6.0, cart.getTotalPrice());
Recorded values: literalsFailure type: assertion failure
Structure: assertEquals with literal
![Page 25: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/25.jpg)
Recompile and Repeat
assertEquals(6.0, cart.getTotalPrice());
assertEquals( "Discount: -$1.00, Total: $3.00", cart.getPrintedBill());
![Page 26: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/26.jpg)
Evaluation
Q1: How many failures can ReAssert repair?
Q2: Are ReAssert's suggested repairs useful?
Q3: Does ReAssert reveal or hide regressions?
![Page 27: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/27.jpg)
Evaluation
Repairs? Useful? Regressions?
Case Studies
ControlledUser Study
Failures inOpen-Source
Software
![Page 28: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/28.jpg)
Case Studies
![Page 29: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/29.jpg)
78%(29 of 37)
22%(8 of 37)
Case Studies
Unconfirmed
Confirmed by user
100%(37 of 37)
Repairs? Regressions?Useful?
![Page 30: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/30.jpg)
Controlled User Study
![Page 31: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/31.jpg)
9%(12 of 131)
86%(113 of 131)
Controlled User Study
vs. 8 introduced bythe control groupMatching repairs
Repairs? Regressions?Useful?
97%(131 of 135)
![Page 32: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/32.jpg)
Failures in Open-Source Software
Test Suite n
SUT n
Version n
Test Suite n + 1
SUT n + 1
Version n + 1
execute on
![Page 33: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/33.jpg)
Failures in Open-Source Software
45%(76 of 170)
![Page 34: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/34.jpg)
Unrepairable Failures
● Nondeterminism
● Multiple contexts
● No applicable strategies
for (Product product : cart.getProducts()) { assertEquals(3.0, product.getPrice());}
if (...) { expected = 3.0;}assertEquals(expected, cart.getTotalPrice());
assertEquals(..., cart.getPurchaseDate());
![Page 35: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/35.jpg)
http://mir.cs.illinois.edu/reassert
![Page 36: ReAssert: Suggesting Repairs for Broken Unit Testsosl.cs.illinois.edu/docs/ReAssert: Suggesting...ReAssert: Suggesting Repairs for Broken Unit Tests Brett Daniel Vilas Jagannath Danny](https://reader033.fdocuments.us/reader033/viewer/2022051811/601d696c0bf5025f2303fcd9/html5/thumbnails/36.jpg)
Make tests fail
Make tests pass
Test-Driven Development
...by changing tests
...by changing SUT
Test Repair
...by changing SUT
...by changing tests