Post on 15-Apr-2017
Performance Regression Analysis
Cor-Paul Bezemer
Accomplishments and Challenges
A burnt child dreads fire
Regression testing makes sure we do not make the same mistake twice
Write code
Execute regressiontest
Deploy or fix changes
Regression testing also helps us to make sure we are not re-introducing more bugs
Performance regression testing helps ensure we are not breaking performance
Write code
Execute performance test
Deploy if performance is OK or fix changes
Performance regression testing can/should be done on several metrics
Performance regression testing can/should be done on several metrics
Performance regression testing can/should be done on several metrics
Performance regression testing can/should be done on several metrics
Directions in research on performance regression analysis
Directions in performance regression analysis research
1. Execution profile comparison
2. Source code analysis
3. Visualization
Execution profile comparison
Execution profile comparison
Disadvantages of execution profile comparison
- Actually needs performance tests
- Need to run performance tests several times because of variation
- Small changes may be difficult to interpret
Source code analysis
Identify risky commits and recommend those for testing (Huang et al. @ ICSE 2014)
Disadvantages of source code analysis
- Can not replace 'real' performance measurements (yet)
Visualization of performance regression analysis results
Control charts
(Nguyen et al. @ MSR 2014)
Differential flame graphs
(Bezemer et al. @ SANER 2015)
Deciding on the baseline performance
Deciding on the acceptable performance deviation
Reporting results to the developer
Availability of performance tests
Execution time of tests
Performance regression testing helps ensure we are not breaking performance
Write code
Execute performance test
Deploy if performance is OK or fix changes
Directions in performance regression analysis research
1. Execution profile comparison
2. Source code analysis
3. Visualization
Availability of performance tests
Execution time of tests