USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in...
Transcript of USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in...
![Page 1: USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in efficient test‐suite: • Full coverage • Optimized test effort • Integration of](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f058b3f7e708231d4137cbd/html5/thumbnails/1.jpg)
Sophia Antipolis, French Riviera20-22 October 2015
USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATIONPresented by Rupert Schlick
© All rights reserved
![Page 2: USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in efficient test‐suite: • Full coverage • Optimized test effort • Integration of](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f058b3f7e708231d4137cbd/html5/thumbnails/2.jpg)
Railway Interlocking ‐The Application
• Application Purpose: • Ensure safe train movement• Prevent collisions and derailing of rolling stock
• Experimental Evaluation• used a functional subset of interlocking logic following Austrian railway operation rules
• THALES product LockTrac 6131 Elektra,approx. 250 installations, 4 countries
2 © All rights reserved
![Page 3: USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in efficient test‐suite: • Full coverage • Optimized test effort • Integration of](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f058b3f7e708231d4137cbd/html5/thumbnails/3.jpg)
Railway Interlocking ‐Process Challenges for Testing
• Complex Application Domain• 30 years in service• country specific requirement variants• multiple HW and OS platforms
• Highly regulated domain • CENELEC standards, e.g EN50128 (software safety)• require controllable, documented test and verification process
• traceability, certification of SW increments
3 © All rights reserved
![Page 4: USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in efficient test‐suite: • Full coverage • Optimized test effort • Integration of](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f058b3f7e708231d4137cbd/html5/thumbnails/4.jpg)
Railway Interlocking ‐Technical Challenges for Testing
Configuration• train route logic isgeneric
• test needs instantiationto a station
4 © All rights reserved
Complexity• 71 rule requirements in simplified eval. example
• example test station has:• 34 points, 56 track relais, 22 signals, 145 train routes
Example Rule Requirement:IL:RULE:121: A switch shall reject any kind of moving command (both if it is a manual command and if it is an automatically generated command), if the switch holds a lock or any interlock or an interlock request.
![Page 5: USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in efficient test‐suite: • Full coverage • Optimized test effort • Integration of](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f058b3f7e708231d4137cbd/html5/thumbnails/5.jpg)
Test Environ‐ment
Principle of Model Based Testing (MBT)
• Test Model:• sequences/scenarios• state machines• formal requirements• usage probabilities …
• Test Goal:• target state (condition)• number of tests (random walks)• coverage
• requirement• model structure• user inputs ..
• Test oracle:• no crash, no deadlock• correct behaviour (subset)• invariants…
5 © All rights reserved
Requirements
Modelling
Test Model
Test Case Generation
Tests
System under Test
Test Verdict
Test Oracle
Test Goal
![Page 6: USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in efficient test‐suite: • Full coverage • Optimized test effort • Integration of](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f058b3f7e708231d4137cbd/html5/thumbnails/6.jpg)
Principle of fault based MBT
• behaviour model• mutant: model with a small, syntactically correct change
• used for both:• test quality analysis • as a test goal (fault coverage)
6 © All rights reserved
Test Environ‐ment
Requirements
Modelling
Test Model
Test Case Generation
Tests
System under Test
Test Verdict
Test Oracle
Mutant(s)
Mutation Engine
Fault Model
Kill Check
Coverage Measure
![Page 7: USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in efficient test‐suite: • Full coverage • Optimized test effort • Integration of](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f058b3f7e708231d4137cbd/html5/thumbnails/7.jpg)
Comparison with other coverage driven approaches
• structural coverage alone in state machines (e.g. transitions) is not enough ‐> decision, data flow
• data flow coverage not easily done in concurrent models with instances
• observability not inherent in classic coverage• safety standards request certain coverage criteria for code
7 © All rights reserved
![Page 8: USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in efficient test‐suite: • Full coverage • Optimized test effort • Integration of](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f058b3f7e708231d4137cbd/html5/thumbnails/8.jpg)
Combining Strategies
Combine cheap and expensive approachesResults in efficient test‐suite:
• Full coverage• Optimized test effort• Integration of legacy‐tests
8
Test Model
Test‐Casesfrom:different tools, earlier versions
Coverage Analysis
Test‐Case generation Test‐Cases
?
![Page 9: USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in efficient test‐suite: • Full coverage • Optimized test effort • Integration of](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f058b3f7e708231d4137cbd/html5/thumbnails/9.jpg)
The Test Case Generator: MoMuT
• TCG engine • Input from different modelling tools
• Papyrus UML language front‐end used in evaluation• Generation from UML state machines
• Other modelling languages planned:• DSLs from industrial users• Timed Automata• Event‐B
9 © All rights reserved
![Page 10: USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in efficient test‐suite: • Full coverage • Optimized test effort • Integration of](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f058b3f7e708231d4137cbd/html5/thumbnails/10.jpg)
Behaviour Model
• 32 classes ( 4 environment, 18 field element, 10 trainroute logic)
• 18 active classes (state machines)
10 © All rights reserved
![Page 11: USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in efficient test‐suite: • Full coverage • Optimized test effort • Integration of](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f058b3f7e708231d4137cbd/html5/thumbnails/11.jpg)
Example Stations + Model Size
11 © All rights reserved
Station MMS LBT
Characterisation A small meeting station
Layout used for train route tests
# track relays 4 56
# signals 8 22
# points 2 34
# train routes 10 145
# instances 125 2847
# controllable inputs 172 1652
State size / kB 22,3 > 184,9
![Page 12: USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in efficient test‐suite: • Full coverage • Optimized test effort • Integration of](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f058b3f7e708231d4137cbd/html5/thumbnails/12.jpg)
Evaluation of generated tests
• UML mutation coverage of:• original tests from production use• random tests generated from model
• evaluation of test coverage• option to prioritize tests• derive traces from test to requirement
12 © All rights reserved
Mutants
Tests
![Page 13: USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in efficient test‐suite: • Full coverage • Optimized test effort • Integration of](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f058b3f7e708231d4137cbd/html5/thumbnails/13.jpg)
Performance/Applicability
Generates tests with overall 450 steps for MMS in 23 minutes, covering 680 of 2044 mutants
• Abstract tests including oracle and coverage information• Not cleaned up for unreachable mutants
Use of enumerative exploration• Just‐In‐Time Compilation based on LLVM 3.6• Partial Order Reduction• Partial Orders Encoded in Test Cases• Exploring mutants only for needed steps (<5 steps for 99 %)• Search based exploration driven by mutants (LBT + 10 % cov.)
13 © All rights reserved
![Page 14: USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in efficient test‐suite: • Full coverage • Optimized test effort • Integration of](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f058b3f7e708231d4137cbd/html5/thumbnails/14.jpg)
Conclusion –How are the challenges addressed?
• reduced effort• automated test development• efficient tests ‐> affordable test run time• less maintenance effort
• sufficient test quality• better suited coverage criterion (for generation)
• certification of increments• only needed changes to test suite (improvement support)
• complexity can be handled• automated generation of tests in reasonable time
14 © All rights reserved
![Page 15: USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in efficient test‐suite: • Full coverage • Optimized test effort • Integration of](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f058b3f7e708231d4137cbd/html5/thumbnails/15.jpg)
Acknowledgements
• Partners• Thales Austria GmbH
• Graz University of TechnologyInstitute for Software Technology
• Funding Agencies:
• Projects:
15 © All rights reserved
![Page 16: USING FAULTS FOR EFFICIENT MBT FOR A COMPLEX RAILWAY APPLICATION · 2015-10-21 · Results in efficient test‐suite: • Full coverage • Optimized test effort • Integration of](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f058b3f7e708231d4137cbd/html5/thumbnails/16.jpg)
Contacts
Rupert Schlick, Willibald KrennDepartment Digital Safety and SecurityBusiness Unit Safe and Autonomous SystemsAIT Austrian Institute of Technology GmbHDonau‐City‐Straße 1 | 1220 Vienna | Austriahttp://www.ait.ac.at | F +43(0) 50550‐4150
[email protected] | T +43(0) 50550‐[email protected] | T +43(0) 50550‐4109
16 © All rights reserved
www.momut .org
Werner SchützHead Methods and ToolsThales Austria GmbHHandelskai 92 | 1200 Vienna | Austriahttp://www.thalesgroup.com/[email protected] +43(0)1‐27711‐3115