Post on 23-May-2020
A Framework for Regression Test Prioritization and SelectionMaster Thesis Final Presentation
December 9th, 2015
Christian Plewnia christian.plewnia@rwth-aachen.de
Regressions, Test Automation, and Regression Testing
Master Thesis Final Presentation | Christian Plewnia2
Source: http://www.seguetech.com
Challenge
Master Thesis Final Presentation | Christian Plewnia3
Development Regression testing
Optimized
regression testing
Code
changes
Testing time
exceeded
Regression Test Optimization
Master Thesis Final Presentation | Christian Plewnia4
All test cases
Regression Test Optimization (RTO)
Regression TestSelection (RTS)
Regression TestPrioritization (RTP)
Many techniques
described in
research
RTO in Practice
• RTO is widely practiced
• techniques do not scale or
• the integration is difficult
Challenges of integration
• Integration into testing process
• Optimization requires data gathering
• Execution of optimized test suite
Master Thesis Final Presentation | Christian Plewnia5
manually by developers
RTS & RTP Implementations
Master Thesis Final Presentation | Christian Plewnia6
Ekstazi RTS
InfiniTest RTS (live testing)
Test Load Balancer RTP
Echelon RTP; internal tool by Microsoft
Dependency Graph Tool* RTS; internal tool by Google
Cleanscape Testwise (ATAC) RTS
DejaVOO RTS
SPYDER RTS
Aristotle Analysis System Library used by some RTO techniques
discontinuedavailable* Real name unknown internal
Master Thesis Final Presentation | Christian Plewnia7
Demo
Lazzer Framework Architecture
Master Thesis Final Presentation | Christian Plewnia8
Lazzer Framework Architecture
Master Thesis Final Presentation | Christian Plewnia9
Lazzer Pipeline: Test Discovery
Master Thesis Final Presentation | Christian Plewnia10
• Find all test cases in the giventest suite
• Delegated to the test framework adapter
TestFramework
Lazzer Pipeline: Pre-Optimization Data Collection
Master Thesis Final Presentation | Christian Plewnia11
• Enable data stores to collectinformation
• Example: determine source code changes
Data Store
Lazzer Pipeline: Optimization
Master Thesis Final Presentation | Christian Plewnia12
• Perform regression testoptimization
• Delegated to optimization strategies
• The input and output is a test suite(a collection of test classes containing tests)
OptimizationStrategy
Lazzer Pipeline: Test Run
Master Thesis Final Presentation | Christian Plewnia13
• Run the optimized test suite
• Delegated to the test framework
TestFramework
Lazzer Pipeline: Post-Test-Run Data Collection
Master Thesis Final Presentation | Christian Plewnia14
• Enable data stores to collectinformation
• Example: Save test results in database
Data Store
Lazzer Pipeline: Reporting
Master Thesis Final Presentation | Christian Plewnia15
• Report test results
• Example: write to the log or an XML file
Lazzer Framework Architecture
Master Thesis Final Presentation | Christian Plewnia16
Evaluation: Running Lazzer via its API
// Configuration
LazzerSettings settings =
new LazzerSettings.Builder(
// Test Framework Adapter
new JUnit4Adapter(),
// Optimization Strategy
Arrays.asList(new FailedFirstPrioritization()),
// Data Stores
Arrays.asList(new TestHistory(…))
.build();
// Instantiate lazzer
Lazzer lazzer = LazzerFactory.createLazzer(settings);
// Execute test optimization & test run
lazzer.run();
Master Thesis Final Presentation | Christian Plewnia17
ClientAdapter
Lazzer Framework Architecture
Master Thesis Final Presentation | Christian Plewnia18
Developing a Lazzer Optimization Strategy
public interface OptimizationStrategy {
TestSuite optimize(TestSuite testSuite);
}
Master Thesis Final Presentation | Christian Plewnia19
Called by the framework to
optimize the test suite
OptimizationStrategy
Lazzer Framework Architecture
Master Thesis Final Presentation | Christian Plewnia21
Developing a Lazzer TestRunner
public interface DataStore {
void preOptimizationDataCollection();
void postTestRunDataCollection
(TestRunReport testRunReport);
}
Master Thesis Final Presentation | Christian Plewnia22
Called by the framework before the optimization
Called by the framework after the test run
Data Store
Lazzer Framework Architecture
Master Thesis Final Presentation | Christian Plewnia24
Developing a Lazzer TestFrameworkAdapter
public interface TestFrameworkAdapter {
TestDiscovery provideTestDiscovery(
Classpath classpathForSystemUnderTest,
Classpath classpathForTests
);
Optional<TestRunReport> runTests(
TestSuite testSuite,
Classpath classpathForSystemUnderTest,
Classpath classpathForTests);
}
Master Thesis Final Presentation | Christian Plewnia25
Executes test run
Finds test cases
TestFramework
Lazzer Framework Architecture
Master Thesis Final Presentation | Christian Plewnia26
Evaluation
• Software quality assessed according to ISO/IEC 25010
• Aspects:
Master Thesis Final Presentation | Christian Plewnia27
UsabilityPerformance
EfficiencyFunctionality
Maintainability Reliability Portability
Compatibility Security
Evaluation: Performance Efficiency
Master Thesis Final Presentation | Christian Plewnia28
0
10
20
30
40
50
60
70
80
90
100
110
120
130
0 100 200 300 400 500 600 700 800 900 1000
Run T
ime [m
s]
Runs [1]
Lazzer Run Time Evaluation 1,000 runs; 1 outlier (989ms) not considered
0
10
20
30
40
50
60
70
80
90
100
110
120
130
Run T
ime [m
s]
Maximum: 121 msAverage: 6 msMedian: 4 msMinimum: 1 ms
Evaluation: Usability
• Setup
• Command Line: deploy framework JAR + extensions
• Maven: add some lines to project configuration
• Invocation
• Command Line: execution of a Java application
• Maven: run “test” phase
• Inspection of test results
• View log output
Master Thesis Final Presentation | Christian Plewnia29
Evaluation: Functionality
• Working prototype
• Limitations
Configuration of data stores difficult
Tests of JUnit version (<4) not supported
Parameterized tests not recognized
Test results are only logged ⇒ extend API for custom reporting
No prioritization on method level
Not supported order: TestA.x(), TestB.y(), TestA.y()
Master Thesis Final Presentation | Christian Plewnia30
Future Work on the Prototype
Tasks
• Overcoming current limitations
• Support for parallel testing
• More framework configuration options
• Lazzer needs more tests
Master Thesis Final Presentation | Christian Plewnia31
Future Research
Some topics for future research:
• Implementation and evaluation of RTO techniques
• Support for other programming languages
• How to choose the best RTO technique?
• Recommendation for test suite reduction
• Test optimization as a service
Master Thesis Final Presentation | Christian Plewnia32
Conclusion
Contributions
• Overview of state of research on RTO
• Framework for RTO
• Working prototype
Master Thesis Final Presentation | Christian Plewnia33
First step towards an RTO platform
• Architecture
• Unit Testing
SonarQube Static Code Analysis of Lazzer Framework
Master Thesis Final Presentation | Christian Plewnia35
• Complexity
• Rule Compliance
• Documentation
Version 51 / 7th December 2015 14:25 Uhr