Post on 03-Feb-2016
description
Tom Gullion
SRMV Toolsmithtgullion@msisinc.com
(608) 315-2780
Executing UML Models as Verification Oracles
Overview
• Charter– Execute UML models as oracles of behavior exhibited by software
under test• Challenge
– Testing facilities for flight control software and related domains are expensive
• in terms of cost and resources and access– Verification teams often find it difficult to get adequate access to
those testing facilities• Goal
– Reanimating test logs from the test labs against System Reference Models
– That is, verify the SRM against test logs. And vice versa!
The Challenge• Testing facilities for flight control software and related
domains are expensive– in terms of cost and resources and access
• Verification teams often find it difficult to get adequate access to those testing facilities
• Access to logs from test runs is common• Access to software simulations may happen• Access to live test rigs could possibly happen…
• How can we leverage this test data in Verification?
Test Data Reanimation
• Data from a test log is used as a stream of input events
• SRM is implemented/extended as a test oracle
• Behavior of the model, under that stimulus stream, is compared with the behavior logged for the software under test
Basic Reanimation Approach
test definition
test definition
comand/eventstack
comand/eventstack
ReanimateSRM
ReanimateSRM
data convertedto stream of events
data convertedto stream of events
Behavior of the model, under that stimulus stream, is compared with the behavior logged for the software under test
Behavior of the model, under that stimulus stream, is compared with the behavior logged for the software under test
SRM astest oracle
SRM astest oracle
execexec
Test log expectations
• Logs from testing / simulation facilities will have timeframes, command names, parameter data and other metadata
• We won’t be interested in all of it (at least, at first)
• We may even prefer to extract relevant data from multiple sources for our own scenarios (e.g., testing across component boundaries)
Case study: Ejection Seat Behavior
Test scenario
• Example: the testing facility ran a simulation of a flight where the crew ejected
Reanimate Ejection Seat Events
• Focus only on ejection seat commands in test logs
• Configure model runner environment
• Execute commands against SRM to reanimate the events
Reanimation details
COMMENT START SCRIPT Flight with Eject 3330 EJECTION_SEAT arm6322 EJECTION_SEAT enable6323 EJECTION_SEAT eject6326 EJECTION_SEAT separate6330 EJECTION_SEAT releaseMainCOMMENT END SCRIPT
FRAME: 3330 EJECTION SEAT PHASE CHANGE TO Armed
FRAME: 6322 EJECTION SEAT PHASE CHANGE TO Ready
FRAME: 6323 EJECTION SEAT PHASE CHANGE TO Ejected
FRAME: 6326 EJECTION SEAT PHASE CHANGE TO Separated
FRAME: 6330 EJECTION SEAT PHASE CHANGE TO MainChute
Scenario complete: SUCCESS
Input events(Frame, Command, Parameter(s))
Input events(Frame, Command, Parameter(s))
ResultsResultsplay
back
again
st
SRM
mod
el ele
men
t(s)
play
back
again
st
SRM
mod
el ele
men
t(s)
Ejection seat scenario results
• Use reanimation results to verify logged events match SRM modeled behavior
• But this scenario only verifies a single slice of the SRM. Let’s expand the reanimation to include flight modes…
Reanimating multiple statemachines
3320 FLIGHT_MODE preflight3330 EJECTION_SEAT arm4532 FLIGHT_MODE taxi5832 FLIGHT_MODE takeoff6322 EJECTION_SEAT enable6323 EJECTION_SEAT eject6326 EJECTION_SEAT separate6330 EJECTION_SEAT releaseMainCOMMENT END SCRIPT
3320 FLIGHT_MODE preflight3330 EJECTION_SEAT arm4532 FLIGHT_MODE taxi5832 FLIGHT_MODE takeoff6322 EJECTION_SEAT enable6323 EJECTION_SEAT eject6326 EJECTION_SEAT separate6330 EJECTION_SEAT releaseMainCOMMENT END SCRIPT
Input events(Frame, Command, Parameter(s))
Input events(Frame, Command, Parameter(s)) ResultsResults
play
back
again
st
SRM
mod
el ele
men
t(s)
play
back
again
st
SRM
mod
el ele
men
t(s)
FRAME: 3320 FLIGHT MODE CHANGE TO PreFlightFRAME: 3330 EJECTION SEAT PHASE CHANGE TO ArmedFRAME: 4532 FLIGHT MODE CHANGE TO TaxiFRAME: 5832 FLIGHT MODE CHANGE TO FlightFRAME: 6322 EJECTION SEAT PHASE CHANGE TO ReadyFRAME: 6323 EJECTION SEAT PHASE CHANGE TO EjectedFRAME: 6326 EJECTION SEAT PHASE CHANGE TO SeparatedFRAME: 6330 EJECTION SEAT PHASE CHANGE TO MainChuteScenario complete: SUCCESS
FRAME: 3320 FLIGHT MODE CHANGE TO PreFlightFRAME: 3330 EJECTION SEAT PHASE CHANGE TO ArmedFRAME: 4532 FLIGHT MODE CHANGE TO TaxiFRAME: 5832 FLIGHT MODE CHANGE TO FlightFRAME: 6322 EJECTION SEAT PHASE CHANGE TO ReadyFRAME: 6323 EJECTION SEAT PHASE CHANGE TO EjectedFRAME: 6326 EJECTION SEAT PHASE CHANGE TO SeparatedFRAME: 6330 EJECTION SEAT PHASE CHANGE TO MainChuteScenario complete: SUCCESS
Flight with Ejection seat scenario results
• Now we’ve verified logged events match SRM modeled behavior in two statemachines
• This scenario only verifies proper event sequencing within its scope. But we’re not checking preconditions and other rules.
• Extend interfaces defined in SRM to implement rules…
Reanimating the SRM
3320 FLIGHT_MODE preflight3330 EJECTION_SEAT arm4532 FLIGHT_MODE taxi5832 FLIGHT_MODE takeoff6322 EJECTION_SEAT enable6323 EJECTION_SEAT eject6326 EJECTION_SEAT separate6330 EJECTION_SEAT releaseMainCOMMENT END SCRIPT
3320 FLIGHT_MODE preflight3330 EJECTION_SEAT arm4532 FLIGHT_MODE taxi5832 FLIGHT_MODE takeoff6322 EJECTION_SEAT enable6323 EJECTION_SEAT eject6326 EJECTION_SEAT separate6330 EJECTION_SEAT releaseMainCOMMENT END SCRIPT
Input events(Frame, Command, Parameter(s))
Input events(Frame, Command, Parameter(s))
ResultsResults
play
back
again
st
SRM
mod
el ele
men
t(s)
play
back
again
st
SRM
mod
el ele
men
t(s)
FRAME: 3320 FLIGHT MODE CHANGE TO PreFlightFRAME: 3330 FlightController: Ejection seat armedEJECTION SEAT PHASE CHANGE TO ArmedFRAME: 4532 FLIGHT MODE CHANGE TO TaxiFRAME: 5832 FLIGHT MODE CHANGE TO FlightFRAME: 6322 EJECTION SEAT PHASE CHANGE TO ReadyFRAME: 6323 FlightController: Canopy releasedFlightController: Ejection seat positionedFlightController: Ejection seat ejectedFlightController: main chute deployedEJECTION SEAT PHASE CHANGE TO EjectedFRAME: 6326 EJECTION SEAT PHASE CHANGE TO SeparatedFRAME: 6330 EJECTION SEAT PHASE CHANGE TO MainChuteScenario complete: SUCCESS
FRAME: 3320 FLIGHT MODE CHANGE TO PreFlightFRAME: 3330 FlightController: Ejection seat armedEJECTION SEAT PHASE CHANGE TO ArmedFRAME: 4532 FLIGHT MODE CHANGE TO TaxiFRAME: 5832 FLIGHT MODE CHANGE TO FlightFRAME: 6322 EJECTION SEAT PHASE CHANGE TO ReadyFRAME: 6323 FlightController: Canopy releasedFlightController: Ejection seat positionedFlightController: Ejection seat ejectedFlightController: main chute deployedEJECTION SEAT PHASE CHANGE TO EjectedFRAME: 6326 EJECTION SEAT PHASE CHANGE TO SeparatedFRAME: 6330 EJECTION SEAT PHASE CHANGE TO MainChuteScenario complete: SUCCESS
Complete scenario results
• Now we’ve verified logged events match SRM modeled behavior in two statemachines as well as a flight controller class
• From this framework, it is now possible to add in telemetry data and other command data.
• This structure is setup for reuse across other reanimations.
live data feedlive data feed
Live Test Rig
Live Test Rig
Extensible Reanimation Approach
comand/eventstack
comand/eventstack
Pluggable input data types;Pluggable exec environments
Pluggable input data types;Pluggable exec environments
SoftwareSimulator
SoftwareSimulator
execexec
ReanimateSRM
ReanimateSRM
adversecondition data
adversecondition data
test definition(s)
test definition(s)
Extensible Reanimation Approach
Pluggable input streams allow for future use of• Other test logs (related tests, adverse conditions, etc.)
• "live" data feeds
• the use of accredited simulations as live or stale data sources
Some or all these data sources can be used without changing the overall approach
Conclusions• This approach is appropriate for complex collections of analysis
cases– Simple analysis cases probably do not require reanimation
• We are building part of a larger construct– Analyze adverse conditions against specific tests logs or accredited
simulations– Enables use/reuse of SRM without extensive modification– Enables traceability from reanimation result back through SRM to
originating requirements– Allow distributed machines to play different roles (data source vs data
analysis)– Flexibility for machines (real and virtual) to interface with existing
infrastructure (IV&V not required to manage all machines)• This approach embraces the existing overall framework and is
designed to scale
Applying the 3 Questions
• What is the system supposed to do?– Verified by success and failure scenarios
• What is the system not supposed to do?– Verified by success and failure scenarios
• What does the system do under adverse conditions?– Verified by adverse conditions test data
– Can predict possible failure scenarios that don’t seem adequately handled, propose test scenarios for dev team
Acknowledgements
• Thanks to NASA IV&V for the opportunity to present these concepts
• Special thanks to Frank Huy, Bill Stanton, Todd Gauer, Karl Frank, Steve Driskell, Dan Sivertson, Tom Hempler and Jeff Micke for technical contributions and assistance
Questions? / Comments?
Thank you!
Tom Gullion
tgullion@msisinc.com
(608) 315-2780