AdaCore Paris Tech Day 2016: Elie Richa - Integrated Unit Testing for a Trusted Code Generator
Transcript of AdaCore Paris Tech Day 2016: Elie Richa - Integrated Unit Testing for a Trusted Code Generator
Elie Richa
Gem: Integrated Unit Testing for a Trusted Code Generator
QGen Developer and QA Coordinator
Model-driven Development of Critical Software • An Automatic Code Generator (ACG) produces the source code from a
model-based specification
• To avoid re-verifying the generated code each time the ACG must be "Qualified" (i.e. Trusted) (or “Qualifiable”)
• Qualification requires a rigorous and thorough verification of the ACG § Extensive testing of the ACG with high exhaustiveness
• Testing is a major cost in ACG qualification
ModelAutomatic Code
Generator !(ACG)
Source Code
2
DO-178C Tool Qualification Levels
Criteria1:Atoolwhoseoutputispartoftheairborneso5wareandthuscouldinsertanerror.Criteria2:Atoolthatautomatesverifica=onprocess(es)andthuscouldfailtodetectanerror,and
whoseoutputisusedtojus0fytheelimina0onorreduc0onof:• Verifica=onprocess(es)otherthanthatautomatedbythetool,or• Developmentprocess(es)thatcouldhaveanimpactontheairborneso5ware.
Criteria3:Atoolthat,withinthescopeofitsintendeduse,couldfailtodetectanerror.
So$wareLevels,failureis…
LevelA:Catastrophic
LevelB:Hazardous
LevelC:Major
LevelD:Minor
3
TQL-1 Qualification: Requirements-Based Testing Model for Tools
ToolOpera=onalRequirements(TORs)
ToolRequirements(TRs)
TestCases
TestProcedures
SourceCode
implementedby
allocatedonToolArchitecture
implementedby
implementedby
verifiedby
implementedbyimplementedby
4
QGen
Architecture of QGen
5
Importer Preprocessor Sequencer
CodeModel
Generator(CMG)
Op=mizer Expander Printer
Simulink®
Model
Source Code
internal model representation internal code representation
Unit Testing Input Output
Integration Testing
Input Output
User Language Ti−1 TiMi
Source Code Ti+1
Unit testing vs. Integration Testing
Integration testing is preferred to unit testing
Objective Achieve unit testing exhaustiveness using only integration tests
Unit Testing Integration Testing
Test Data Editor/Viewer Internal languages have no editors User language has a good editor
Test Data Complexity Intermediate languages are complex User language is simpler, has higher abstraction
Test Exhaustiveness Achievable thanks to isolation of units Hard to achieve with no visibility on the internals of the tool
Mi+1
6
Executable Unit Test Cases
• Write unit test cases as executable queries § A query specifying the required input : Test Requirement § A query specifying the expected output : Test Oracle
• Example: functionSqrt(Arg:Float)returnFloat;
7
User Language Ti−1 TiMi
Source Code Ti+1Mi+1
Unit Test Requirements
tri,jtri,jtri,j tri,jtri,jtoi,j Unit Test Oracles
… …
Test Case ID Test Requirement Test Oracle
1 Arg=0.0 Sqrt'Result=0.0
2 Arg>0.0andthenArg<=1.0 Sqrt'Result>0.0andthenSqrt'Result<=1.0
3 Arg>1.0andthenArg<=100.0 Sqrt'Result>1.0andthenSqrt'Result<=10.0
Instrumentation for Test Case Evaluation and Logging
• All instrumentation is marked as ghost code
• Compiler produces ghost code only when assertions are enabled
Log which test requirements are satisfied by the input data
Evaluate the test oracles of the covered test cases
Subprogram Under Test
control flow
instrumentation
instrumentation
8
Unit Test Cases
Integration Tests tr0,0 tr0,1 tr0,2 to0,0 to0,1 to0,2 tr1,0 tr1,1 to1,0 to1,1 tr2,0 tr2,1 to2,0 to2,1
Test0 SAT – – PASS – – – SAT – PASS – SAT – PASS
Test1 – – SAT – – PASS – SAT – PASS – SAT – PASS
Test2 – – SAT – – FAIL SAT – PASS – – SAT – PASS
Test3 – – – – – – SAT SAT PASS PASS – SAT – FAIL
Test4 SAT – – PASS – – – SAT – PASS – SAT – PASS
Non-covered Unit Test Case
Non-covered Unit Test Case Unit Test Failure Unit Test Failure
Integrated Unit Testing (IUT) Integration Test Input Ti−1 TiMi
Source Code Ti+1Mi+1
Unit Test Requirements
tri,jtri,jtri,j tri,jtri,jtoi,j Unit Test Oracles
… … Test Execution
9
Achieving coverage of Unit Test Cases
• To achieve coverage, Integration Testing must be thorough
• Example: A Simulink® block type with 4 parameters § 3 possible types each (i.e. Scalar, Vector, Matrix) ⇒ 34 = 81 configurations § Taking into account primitive types (int8, int16 …) ⇒ ~600000 configurations!
§ But wait, only 14 out of 81 are valid configurations (~7000 out of ~600000) § The behavior is generally similar for all configurations, with subtle differences
§ How can we do this accurately for 120+ Simulink® block types?
• Use the BlockLibrary specification language § Specify variability parameters: input/output ports, data parameters
§ Specify constraints defining valid combinations of parameter values and types § Factorise common specification elements § Auto-generate a large test set with a meaningful/reasonable coverage criteria of the
possible configurations
• Achieve Unit Testing using only integration test data (Simulink® models)
10
Integrated Unit Testing Framework
• Other features of the Integrated Unit Testing framework include § Checking the completeness and disjointness of test cases § Storing temporary data specific to a test case to improve expressiveness § Selecting/excluding specific groups of test cases in the instrumentation § Optimising the instrumentation by only evaluating previously non-covered test cases § Specifying test cases in separate files than the tested source code
• Let us know if you’d be interested in Integrated Unit Testing for your code § We are considering making it a GNATtest feature
11
Unit Test Requirements
Unit Test Oracles
Integration Test Input Ti−1 TiMi
Source Code Ti+1Mi+1
Unit Test Requirements
tri,jtri,jtri,j tri,jtri,jtoi,j Unit Test Oracles
Test Execution
… …
Conclusion: Integrated Unit Testing • Conventional Unit Testing of multi-pass code generators is painful
§ Difficult to create inputs and check outputs of individual passes
• Integration Testing is generally easier § Can use normal Models as input; generated Source Code as output
• Integrated Unit Testing: accomplish Unit Testing using Integration Testing § Instrument the tested tool with ghost code
§ Monitor internal data during Integration Testing
§ Log evidence of the coverage and passing of Unit Test Cases
• The approach is applicable to other software § Let us know if Integrated Unit Testing is interesting in your context
12