Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with...
Transcript of Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with...
![Page 1: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/1.jpg)
Mark HarmanFACEBOOK SOFTWARE ENGINEERING MANAGER
SAPIENZ TEAM; THIS TALK IS BASED ON WORK OF THE WHOLE TEAM
![Page 2: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/2.jpg)
Mark HarmanFACEBOOK SOFTWARE ENGINEERING MANAGER
SAPIENZ TEAM; THIS TALK IS BASED ON WORK OF THE WHOLE TEAM
![Page 3: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/3.jpg)
UNFORTUNATELY, YOUR APP HAS STOPPED.
OKREPORT
WORK IN PROGRESS
![Page 4: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/4.jpg)
SLOW BORING UNIMPORTANTPAINFUL
![Page 5: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/5.jpg)
URGENT
![Page 6: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/6.jpg)
IN PROC. OF SSBSE, 2018.
Deploying Search Based Software Engineering with Sapienz at FacebookNADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO, ALEXANDER MOLS, TAIJIN TEI AND ILYA ZORIN
![Page 7: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/7.jpg)
![Page 8: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/8.jpg)
![Page 9: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/9.jpg)
Search Based Optimization
Software Engineering
S B S E
We use Search Based Software Engineering
widely studied in academia; now starting to reach into industry
![Page 10: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/10.jpg)
... covers history of SBST up to 2015
![Page 11: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/11.jpg)
Test live in a search space
![Page 12: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/12.jpg)
Engineers Design
Machines Execute
![Page 13: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/13.jpg)
![Page 14: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/14.jpg)
Unfortunately, your app has stopped.
![Page 15: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/15.jpg)
System Level Testing
RANDOM FUZZER HUMAN TESTERS
![Page 16: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/16.jpg)
Fault Triage Process
OPERATIONS IN THE EVOLUTION WORKFLOW
TRIAGEBUILD SEARCH CRASHREOCCURRING
OPERATOR
FILE A FIX STACK TRACE
A
B
C
DDIFFOWNER REVIEW LOCALIZER CRASH DATA
![Page 17: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/17.jpg)
>1Msource control
commands run per day
>100Kcommits per week
![Page 18: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/18.jpg)
![Page 19: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/19.jpg)
Workplace for
Android
Messenger for
Android
Instagram for
Android
Facebook foriOS
Facebook for
Android
![Page 20: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/20.jpg)
~75%FIX RATE
![Page 21: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/21.jpg)
Distribution (FB)
Null Pointer
Illegal State
Illegal Argument
RunTime
Class Cast
No Such Method
Assertion
Array Index Out of Bounds
Bad Token
Index Out of Bounds
TOP CRASHES TYPES ON FACEBOOK FOR ANDROID (BY SAPIENZ)
![Page 22: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/22.jpg)
Distribution (Research)
TOP CRASHES TYPES ON 1000 ANDROID APPS (BY SAPIENZ[1])
Null Pointer
Illegal State
Illegal Argument
RunTime
Activity Not Found
Out of Memory
Concurrent
Array Index Out of Bounds
Bad Token
Index Out of Bounds
[1] K. MAO, M. HARMAN, AND Y. JIA, “SAPIENZ: MULTI-OBJECTIVE AUTOMATED TESTING FOR ANDROID APPLICATIONS,” IN PROC. OF ISSTA’16, 2016
![Page 23: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/23.jpg)
Distribution (Research)
TOP CRASHES TYPES ON 1000 ANDROID APPS (BY SAPIENZ[1])
Null Pointer
Illegal State
Illegal Argument
RunTime
Activity Not Found
Out of Memory
Concurrent
Array Index Out of Bounds
Bad Token
Index Out of Bounds
[1] K. MAO, M. HARMAN, AND Y. JIA, “SAPIENZ: MULTI-OBJECTIVE AUTOMATED TESTING FOR ANDROID APPLICATIONS,” IN PROC. OF ISSTA’16, 2016
![Page 24: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/24.jpg)
Auto Fix
Auto BoostAuto Test Design Auto Fix
![Page 25: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/25.jpg)
Auto Fix Workflow (Generation)
CRASH TRIGGERS
REVERT FULL DIFF
REVERTPARTIAL DIFF TEMPLATE MUTATION
SAPIENZ AUTOTRIAGE
FIX PATCH GENERATOR
TRIGGER PATCH GENERATOR
CREATE REVISION IN SAPFIX
![Page 26: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/26.jpg)
Auto Fix Workflow (Validation)
CREATE REVISION IN SAPFIX
PATCH 1
PATCH 2
PATCH N
MANUALLY WRITTEN
TESTS
COMPILATION ERROR?BUILD
CRASH FIXED? NEW CRASH?SAPIENZ E2E
![Page 27: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/27.jpg)
Auto Fix Workflow (Signal)
CREATE REVISION IN SAPFIX
REJECTED/ EXPRED
DEVELOPER REVIEWS
AUTO FIX PROCESSOR
DEVELOPER INTERACTION
TRACKER
ACCEPTED
![Page 28: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/28.jpg)
SAPIENZ
INFER
![Page 29: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/29.jpg)
Localization: Sapienz + Infer
SAPIENZ 1559
INFER
// Sapienz Revealed NPE Crash contextDateMap.put("name", displayedUser.getName());
1559// INFER WarningThe value of 'UserDetailFragment.displayedUser in the call to 'getName()'could be null.
![Page 30: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/30.jpg)
Summary
AUTO TEST DESIGN
AUTO FIX AUTO BOOST
![Page 31: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/31.jpg)
Flaky testsFix detectionAutomated OraclesWider search spacesSmarter white box coverageCombining static and dynamicUnit test from system testsHuman machine test hybridsFully parallel search algorithmsAuto Fix and perf improvement
![Page 32: Mark Harman - PuzzleIN PROC. OF SSBSE, 2018. Deploying Search Based Software Engineering with Sapienz at Facebook NADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO,Search Based](https://reader033.fdocuments.us/reader033/viewer/2022043006/5f8dc6e31915904661554f7e/html5/thumbnails/32.jpg)
IN PROC. OF SSBSE, 2018.
Deploying Search Based Software Engineering with Sapienz at FacebookNADIA ALSHAHWAN, XINBO GAO, MARK HARMAN, YUE JIA, KE MAO, ALEXANDER MOLS, TAIJIN TEI AND ILYA ZORIN
Resources