ARTEMIS Project MBAT: Advanced Validation & Verification of Embedded Systems af Brian Nielsen,...
-
Upload
infinit-innovationsnetvaerket-for-it -
Category
Technology
-
view
631 -
download
3
description
Transcript of ARTEMIS Project MBAT: Advanced Validation & Verification of Embedded Systems af Brian Nielsen,...
© MBAT, ARTEMIS project 269335ARTEMIS Joint UndertakingARTEMIS Joint UndertakingThe public private partnership for R&D actors in embedded systems
http://www.mbat-artemis.eu/
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking 2
© MBAT, ARTEMIS project 269335ARTEMIS Joint UndertakingARTEMIS Joint Undertaking
© Daimler
Embedded Systems in a Car (ECUs) as example Targets of MBAT
3
3‐4 networks100+ ECUs2017 autonomous2020 driverless
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
© Daimler © Daimler
Example application domain for MBAT:automotive HIL integration test environment for model-based testing of embedded systems (interactions)
Automotive Test Environment
4
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
MBAT’s overall Challenges
V&V technologies are still not effective and efficient enough
V&V costs for Embedded Systems are too high (still up to 50% of Embedded System’s total development costs)
V&V technologies should improve the error detection rate
HIL Test Environment © Daimler
5
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
MBAT‘s European impact
MBAT outcomes will contribute to increase the competitiveness of European transportation products industry
MBAT will support higher quality European transportation products at reduced development costs
6
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
MBAT’s Project Character
MBAT is an ARTEMIS project, thus focussing on embedded
systems an industrial-oriented R&D project to transfer tool
innovations and academic research into industrial application
strongly driven and evaluated by industrial use cases
7
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
MBAT’s Market Impact
MBAT will increase the competitiveness of European key players in transportation domain by reducing V&V costs for embedded systems by at least 20 % (keeping the
planned level of quality) shortening time‐to‐market by at least 20 % increasing the coverage of the embedded system under V&V by at least 30
% significantly increasing the probablitity to uncover errors enabling higher quality and safer embedded systems & embedded
systems based products
8
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
MBAT Technological Innovation
MBAT = Combined Model‐based Static Analysis and Dynamic Testing of Embedded Systems
Test & Analysis Models
Test Cases
Analysis Results
Test Results
Analysis Cases
Embedded Systems Descriptions
Dynamic Tests
Static Analysis
Test
Analysis
10
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
MBAT Outcomes
Industrial‐approved MBAT Reference Technology Platform (MBAT RTP) supporting Validation & Verification of Embedded Systems
Experience Packages describing the usage of the RTP in industrial domains (automotive, aerospace, rail)
11
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
UC No. Use case name Use case driverAutomotive Use Cases
UC A1 Brake-by-Wire VOLVOUC A2 Common Powertrain Control (CPC) DAIUC A3 Adaptive Brake Light (ABL) DAIUC A4 Turn Indicator Control (TIC) DAIUC A5 Transmission Controller Product Line RICUC A6 Passive Balancing AVLUC A7 Hybrid Power Train Control Unit AVLUC A8 Virtual Prototype Airbag ECU IFAT
Aerospace Use CasesUC AE1 Flight Control Program AIRUC AE2 ACSL Component for Flight Control Computer AIRUC AE3 Flight Warning Program AIRUC AE4 Flight Management System/UAV ASIA
UC AE5 Degraded Vision Landing Aid System DeViLASystem EADS DE
UC AE6 TALARION - Unmanned Aerial Vehicle (UAV) EADS IWUC AE7 Flight Guidance System (FGS) RCFUC AE8 Attitude and Altitude (A&A) for Helicopters RCFUC AE9 Spacecraft Central Software-Sentinel 3 TAS
Rail Use CasesUC T1 Automatic Train Control ALSTOMUC T2 Rapid Transit Metro System (Ansaldo STS) ANSALDOUC T3 Validator of the ZLB ATOP System SIEMENS
MBAT Use Cases
12
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
(Automated) V&V
Techniques
Dynamic Techniques
Static Techniques
Testing Monitor‐ing Simulation Theorem
Proving
Symbolic Exec
Model‐checking
Abstract Interpreta‐
tionRefinement‐checking
MiL TestingStatistical Model
Checking
Runtime Verifi‐cation
Software Model‐checking
(hybrids)
Classification of Techniques
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
Req. VerificationPlan/Status T&A Model(s)
Analysis Cases
Test Cases
Analysis
Test
CoverageResults,
Main MBAT Method
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
Correct code, or verify weaker invariant, and analyse implicationon models level
Verify req using sim/test as approximation,or re-verify using strengthenassumption (or refine req.)
V&
V P
lanning
Success(verified)
Failed Reqs
Inconclusive
Suspectedor new case
Correct model, design, code, and repeat V&V of all impactedsys. Add cases for regression check
Success(pass)
Failed Reqs
Inconclusive
Suspectedor new case
Correct system, and analyze model in context of trace and test case to ruleout similar errors
V&V Objectivesto be analysed
V&V Objectivesto be codechecked
Success(verified
Failed Reqs
Inconclusive
Suspectedor new case
”Maybe satisfied” property: derivetest High warning densityHigh complexity
Define analysis cases for model-analysisDefine invariants for static codeanalysisUncovered items: try to target theseusing model analysis
1) Make initial V&V Plan that maprequirements/V&V objectives to most suitable techique
5) Update V&V plan and status based on results
Define new analysis or test cases for model-analysis (or invariants for static codeanalysis)
1 2
3
3
3
2) Construct analysisand/or test model(s)
3) Execute4) evaluate results
Model-Analysis
(MB) Testing
Code-Analysis
use testing
Engineering Artifacts
V&V Objectivesto be tested
4 Feedback
T&A
mod
el(s
)4 Feedback
55
Refine req and test case
ok
ok
ok
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
Hybrid powertrain control unit (HCU) that is responsible for coordinating the energy flows between engine, electrical motor, and the battery.
AVL’s HCU Initial Combination
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
Refinement ProcessOverall Methodology
Workflow / Combination Patterns
(sub) MethodInstance
RTP Instance
• Framework describing general workflow and most A&T combination strategies
• Holistic view• Domain‐ and tool‐independent
• Pattern=reusable solution to a commonly occurring problem
• Pattern for common A&T Combination strategies• Typically focuses on only a part of the V&V flow
• A specific chosen set of notations (reqs, models, traces, etc.)
• Specific type of results and data to be exchanged (syntax and semantics)
• Specific set of tools
• Workflow, and data exchange supported by the RTP• Tools integrated/interoperable
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
Reduce warnings from Static Code Analysis
Static Code Analyzer
Model-Analyzer
Code +config
Model generator
report
• Program slice
• Path precondition
warnings
Report merger
Confirmed defects Remaining warnings
Confirmed defects
UC T3 SIE “ZLB ATOP System”: SAT‐solving using RTT+UC A2 DAI “CPC”: model‐checking
• Semantic preserving Model
• Property
1
32
4
Purpose: Reduce number of warnings from static code analysis by more exact analysis
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
Instance (Work in progress)
Code +config
report
Report merger
Confirmed defects Remaining warnings
Confirmed defects
Purpose: Reduce number of warnings from static code analysis by more exact analysisPre-condition: first step conducted by abstract interpretation (over-approximation)Maturity: researchVariants:Notes:
Astrée+slicer+exchange format for invariants
Uppaal
Static Code Analyzer Model generator Model Analyzer
WarningsPreconditionSlice
Significant effort!!UC A2 DAI “CPC”: model‐checking
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
Reduce warnings from Static Code Analysis
Static Code Analyzer
Test Execution
Code +config
Test input generator
report
• Program slice
• Path precondition
warnings
Report merger
Confirmed defects Remaining warnings
Confirmed defects
• Instrumented Program (Oracle)
• Test case
UC AE8 RWC “Attitude and Altitude for Helicopters”
1
32
4
UC T3 / SIE “ZLB ATOP System”
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
Increase Coverage by Analysis
Simulation based test generator
Path synthesizer
Test suite
Executor + Coverage Evaluator
Test Model
Coverage Analyzer
SUT
Cover-age report
Test Case
Test Input
Model-checker
Cover-age report
(Counter examplebased) Test Case
Two Patterns? • Model‐coverage• White‐box SUT/Code
Coverage
Same test suite / test format?Is it possible to transfer a path synthesized test case to model level?
Can model serve as Oracle?
Alternative: Coverage based test generation+ Coverage completion by simulation
oracle
UC_AE6 EADS TALARION UC_AE7 RWC “Flight Guidance System”: MC/DC CoverageUC T2 ? ASTS 3.1.9 ANSALDO “Rapid Transit Metro System”
1
3
2
4 5
6
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
MBT with analysis 1
Model-based test generator
Test suite
Executor
Test Model
SUT
Model-checker
Test objectives
(Formalized) Requirements
Fail: Hypothesis: most likely impl is wrong because model was checked wrt req’s Pass: Hypothesis: Impl satisfies requirements because model satisfy reqs and impl refines model
Analysis Objectives
Report
e.g. UC A1 Volvo BBW
1
3
2
4 5
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
MBT with analysis 2
Model-based test generator
Test suite
Executor
Analysis Model(Env + SUT)
SUT
Model-checker
Test objectives
(Formalized) Requirements
1. Model‐check could not verify all requirements on analysis model (spate space too large)
2. Model‐check in context (as environment model/input) of failing test case (to reduce state space)
3. Higher confidence, targeted analysis: confirm/exclude “similar” errors
Analysis Objectives
Report
Abstracted failed test trace (or observed suspect behavior)
13
2
Purpose: Rule out further defects along known failing test
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
Target MBT to failing test case
Model-based test generator
Test suite
Executor
Test Model
SUT
Model-checker
Test objectives
(Formalized) Requirements
neighborhood• A related test path found by
choosing alternative outcome at branching point in the original path[Peled, in FME 2001]
• Small “trace distance”
Analysis Objectives
Report
failed testcase
1
3
2
Purpose: Generate additional related test cases in the same model neighborhood due to bug cluster assumptionPre-condition: failed test case, notion of neighborhood
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
Target MBT to suspect areas
Model-based test generator
Test Cases
Static analysisSU V&V
Model Mapping: links code level element (“defect area” e.g., function/statement) to Model‐level element (eg. component or transition)• Traceability info? • Auto generated code (e.g. Daimler impact
analysis for Simulink)• Manual inspection
Report
Test Objectives
Analysis Objectives
Model
High warning densityHigh (cyclomatic) complexity
Model Mapping
Model-check
1
3b
3a
2
Purpose: Target suspected parts of SUV&V with additional analysis and test cases Pre-condition: notion of neighborhood, mapping
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking
MBAT RTP in more Detail
A Reference Technology Platform (RTP), like the ARTEMIS MBAT RTP, provides a set of management or engineering methods and processes, as well as engineering tools, which will be used to compose/build a complete engineering environment
Integrated subset of RTP components. The interoperability
approach is based on the IOS
(RTP – Tailoring)
An Interoperability Specification (IOS) will guarantee these needs for interoperability and collaboration between tools across the entire engineering lifecycle
30
© MBAT, ARTEMIS project 269335ARTEMIS Joint Undertaking 31
MBAT 2nd Full Plenary Meeting in Copenhagen May 2012