Post on 22-Jan-2018
Testing the Untestable: Model Testing
of Complex Software-Intensive Systems"
Lionel Briand joint work with Shiva Nejati, Mehrdad Sabetzadeh, Domenico Bianculli
Untestable Systems
2
“systems for which testing is highly expensive "or impossible to fully automate”
Untestable Systems"Test Generation
3
large, complex input spaces
Untestable Systems"Test Execution
4
hardware-in-the-loop
Untestable Systems"Identification of failures
5
dynamic behavior
• Inputs are signals over time
• Properties to be checked consider change over time, for independent or related output signals
• Common when dealing with physical entities
Initi
al D
esire
d(ID
)
Desired ValueI (input)Actual Value (output)
Fina
l Des
ired
(FD
)
timeT/2 T
Smoothness
Responsiveness
Stability
Untestable Systems
6
limited testing
only a small number of test scenarios "can be executed and analyzed "
within time and budget constraints
Cyber-physical Systems as Untestable Systems
7
Testing Cyber-physical Systems: Challenges
8
complex systems
hybrid behavior
complex"environment
combinatorial"state
explosion complex"
requirements uncertainty
Example: Night Vision System
9
camera-based assistance system"providing improved vision at night
Large, complex input space: • traffic situations • shape, speed, "
and distance • lightning, weather
Example: Controllers
10
• how to select test cases from the vast space of possible input signals? • how to automatically distinguish faulty from non-faulty output signals?
Software V&V: Common Model-based Approaches
11
Model-based"Testing
Model"Checking
Model-based Testing
• Relies on models to generate test scenarios and oracles for the implemented system
• Has challenges with:
• hardware-in-the-loop
• uncertainty
• hybrid, discrete-dynamic behavior 12
Model Checking
• Verification of temporal properties over concurrent, finite state models
• Has challenges with:
• modeling complex properties of physical devices with continuous dynamics
• modeling complex, concurrent interactions with the environment
• Not easy to parallelize 13
Software V&V: Common Model-based Approaches
14
Model-based"Testing
Model"Checking
Scalability and practicality challenges in the context of untestable systems!!!
Model Testing
Model Testing
16
Raise the level of abstraction"at which (most of the) testing is performed
Model Testing
17
Shift the bulk of testing"from implemented systems "
to models of such systems and their environments
Model Testing: Strategy
• Make the best out of available time and resources for testing the implemented system
• Focus on high-risk test scenarios within budget and time constraints
18
• Capture relevant aspects of system behavior, structure, environment, and properties, as well as their uncertainty
• Are used as a basis for execution and risk analysis
• Use the results to focus the testing of the implemented system
The Role of Models
19
Model Testing vs. "Model-based Testing
20
The focus of model testing is on"testing the model,
not the implementation
Model Testing vs."Model Checking
• Not exhaustive, but risk-driven
• Oracles can be dynamic properties over time intervals and handle uncertainty
• Supports modeling the physical world, heterogeneous formalisms and integration languages
21
Model Testing: Objectives and Challenges (1/5)
22
Executable and testable models of the system and
its environment
Objective Definition and execution of testable models, including
dynamic and uncertain behavior
Challenge
Model Testing: Objectives and Challenges (2/5)
23
Finding, executing, "and analyzing "
high-risk test scenarios
Objective Automated, effective test selection in large input
spaces, partly defined by input signals
Challenge
Model Testing: Objectives and Challenges (3/5)
24
Modeling"uncertainty and risk
Objective
Automated detection of failures under uncertainty;
risk assessment
Challenge
Model Testing: Objectives and Challenges (4/5)
25
Change impact analysis "and regression testing
Objective
Regression tests prioritization or selection
when models change
Challenge
Model Testing: Objectives and Challenges (5/5)
26
Guiding testing on the implemented system
(with HiL)
Objective
Selecting test cases to run so as to reduce overall risk
Challenge
Other Questions
27
• Cost of modeling
• Fidelity of modeling and its impact on cost-effectiveness
• Impact of uncertainty on finding high-risk scenarios and fault detection
Model Testing
28
Shift the bulk of testing"from implemented systems "
to models of such systems and their environments
Accounts for uncertainty, dynamic behavior, a diversity of modeling languages from different
domains, risk modeling Better focus testing of implemented systems"
on high-risk test scenarios within limited resources
Testing the Untestable: Model Testing
of Complex Software-Intensive Systems"
Lionel Briand joint work with Shiva Nejati, Mehrdad Sabetzadeh, Domenico Bianculli
We’re hiring!!!