1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code...
Transcript of 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code...
![Page 1: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/1.jpg)
Code Execution and
Runtime VerificationRuntime Verification
Jeff Zemerick
1
![Page 2: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/2.jpg)
Outline
• The source code.
• The profiler.
• Executing unit tests.
• Runtime Verification.
Code Execution and Runtime Verification
• Runtime Verification.
2
![Page 3: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/3.jpg)
Overview of the Code
• ~1.2 million SLOC
• Organized by functional module (~150
modules)
• The code in each directory is independent of
Code Execution and Runtime Verification
• The code in each directory is independent of
other code (can be built separately).
• Compiles and executes on x86 Linux.
• Built as shared libraries but must be built
statically.
3
![Page 4: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/4.jpg)
Build Changes
• Components are compiled as shared libraries.
• Shared libraries cannot be easily
instrumented.
• Modified the build process to do the build so
Code Execution and Runtime Verification
• Modified the build process to do the build so
that the executable is linked statically.
– Determined what source files are needed (cross-
module), build and link them with the unit tests.
4
![Page 5: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/5.jpg)
Outline
• The source code.
• The profiler.
• Executing unit tests.
• Runtime Verification.
Code Execution and Runtime Verification
• Runtime Verification.
5
![Page 6: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/6.jpg)
The Profiler
• Created by my 4 NEAP interns this summer.
• C profiler.
• Event-based (function entrances/exits).
• Captures execution trace and can export the
Code Execution and Runtime Verification
• Captures execution trace and can export the
trace as: plain text, CSV, XML
• Translates function addresses to function
names.
• The interns did a fantastic job.
6
![Page 7: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/7.jpg)
Instrumenting the Code
• Modified the makefile to include support for:
– Profiling (add my interns’ profiler object file when
linking)
– Debugging – allows for translation of (useless)
Code Execution and Runtime Verification
– Debugging – allows for translation of (useless)
function addresses to (useful) function names.
7
![Page 8: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/8.jpg)
Example Execution Trace
main (1) (??)
|=-function1 (1) (main)
|=-|=-function2 (1) (function1)
Indentation
shows depth.
Code Execution and Runtime Verification
|=-|=-function2 (1) (function1)
|=-|=-function3 (1) (function1)
|=-|=-|=-function4 (1) (function3 )
|=-|=-|=-|=-function5(1) (function4 )
8
![Page 9: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/9.jpg)
Outline
• The source code.
• The profiler.
• Executing unit tests.
• Runtime Verification.
Code Execution and Runtime Verification
• Runtime Verification.
9
![Page 10: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/10.jpg)
Executing the Unit Tests
• A wrapper facilitates the execution of the unit
tests.
• The wrapper provides stubs for hardware-
specific functionality.
Code Execution and Runtime Verification
specific functionality.
– Allows for testing the code on X86 Linux by
providing stub functions for the hardware-specific
functionality.
10
![Page 11: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/11.jpg)
UTH Example
Event Report
Module
(.h file)
Event report generation for FSW build:
FSW Event Report
Module
(.c file)
Any FSW ModuleEvent Report It
Code Execution and Runtime Verification
Event Report
Module
(.h file)
Event report generation for Test build:
UTH Event Report
Module
(.c file)
Any FSW ModuleEvent Report It
11
![Page 12: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/12.jpg)
Outline
• The source code.
• The profiler.
• Executing unit tests.
• Runtime Verification.
Code Execution and Runtime Verification
• Runtime Verification.
12
![Page 13: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/13.jpg)
Runtime Verification
• Requirements for runtime verification:
– Code that will compile and execute.
– Ability to instrument the code to monitor the
execution.
Code Execution and Runtime Verification
execution.
– Ability to compare the execution with a model of
the desired behavior.
• None of the FSW or unit tests were modified
for this work.
13
![Page 14: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/14.jpg)
Purpose
• Using the execution trace of the code, can we
identify the presence of implemented
requirements?
Code Execution and Runtime Verification 14
![Page 15: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/15.jpg)
Why We Can Attempt to Answer This
• Unit tests achieve 100% coverage of module
testing, per developer rule.
– If a requirement has been implemented, it should
be in the execution trace.
Code Execution and Runtime Verification
be in the execution trace.
15
Untested lines include
default statements in
switch statements and
code which is tested by
other modules.
![Page 16: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/16.jpg)
Modeling the Behavior
• Model can be created in two forms:
– Plain text
– UML activity diagram (work in progress).
• Only one model per requirement is necessary.
Code Execution and Runtime Verification
• Only one model per requirement is necessary.
• Which model type to create and use is up to
the analyst.
• The behavior can be desired behavior or
undesired behavior.
16
![Page 17: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/17.jpg)
Plain Text Model for Event Reporting
Model Rules:
command: <command>
success: <result>
failure: < result >
At least one command.
Either Success, Failure, or both.
Code Execution and Runtime Verification
failure: < result >
Example Model:
command: disable_bus_cmd
success: OK
failure: ERROR
17
![Page 18: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/18.jpg)
Corresponding UML Model
Code Execution and Runtime Verification 18
![Page 19: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/19.jpg)
Eclipse Integration Overview
• Provides a new Eclipse project type called
“NASA IV&V Runtime Verification.”
• Two new file types:
– Text Model Requirement
Code Execution and Runtime Verification
– Text Model Requirement
– Execution Trace
• Custom editors for both file types that
includes syntax highlighting and error
checking.
19
![Page 20: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/20.jpg)
Eclipse Integration (1)
• Plug-ins allow
for creating a
“Runtime
Verification”
Code Execution and Runtime Verification
Verification”
project.
• Project contains
text models and
execution traces.
20
![Page 21: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/21.jpg)
Eclipse Integration (2)
Code Execution and Runtime Verification 21
![Page 22: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/22.jpg)
Eclipse Integration (3)
Code Execution and Runtime Verification 22
![Page 23: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/23.jpg)
Eclipse Integration (4)
Code Execution and Runtime Verification 23
![Page 24: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/24.jpg)
Model Checking
• Checks the execution trace for the model.
• Takes into consideration the:
– Order of commands.
– The depth of the call tree.
Code Execution and Runtime Verification
– The depth of the call tree.
– The distance between located commands.
• Will likely consider other factors as the algorithm development progresses.
• Will accommodate UML models once algorithm is sufficient.
24
![Page 25: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/25.jpg)
Limitations
• Cannot test requirements that specify timing
or latency constraints.
• Cannot test hardware-specific requirements
without the flight hardware.
Code Execution and Runtime Verification
without the flight hardware.
25
![Page 26: 1020 - Code Execution and Runtime Verification · Code Execution and Runtime Verification – Code coverage metrics: • Locate untested code. • Focus V&V efforts on code executed](https://reader030.fdocuments.us/reader030/viewer/2022040323/5e68aed7f1e80e551a4389cc/html5/thumbnails/26.jpg)
Summary
• Runtime Verification can provide:
– Assurance that a requirement is implemented.
– Confirmation of a non-implemented requirement.
– Assertion checking to monitor states.
• Execution and profiling can provide:
Code Execution and Runtime Verification
• Execution and profiling can provide:
– Code coverage metrics:
• Locate untested code.
• Focus V&V efforts on code executed the most (80/20 rule).
– Isolating requirements in unit tests provides the source code which implements that requirement.
26