Presentation slides: "How to get 100% code coverage"
-
Upload
rapitasystems1 -
Category
Software
-
view
114 -
download
1
Transcript of Presentation slides: "How to get 100% code coverage"
Intelligent Testing 2015
Bristol, 18th June 2015
Dr Guillem Bernat
How to get to 100% code
coverage
Introduction
Code coverage is important
Sometimes 100% is difficult
What do we do if we can't get 100%
Code coverage defined
What does it mean to get 100% code coverage?
“Testing shows the presence of bugs, not their absence”
Edsger Dijkstra (1969)
100% code coverage does not mean:
your code is correct
100% code coverage does not mean:
your software requirements are correct
100% code coverage does not mean:
the compiler translated your code correctly
const volatile int x;
volatile int y;
void foo(void){
for(y=0; y>10; y++)
{
int z = x;
}
}
foo:
mov1 $0, y
mov1 x, %eax
jmp .L2
.L3:
mov1 y, %eax
incl %eax
mov1 %eax, y
.L2:
mov1 y, %eax
cmp1 $10, %eax
jg .L3
ret
At the -0s optimization level, GCC 4.3.0 for IA32
compiles the C code on the left to the assembly on
the right.
The compiler output is wrong: the access to volatile-
qualified variable x should not be hoisted out of the
loop.
100% code coverage does not mean:
you have covered 100% of your object code
Source code Object code Compiled to
Does it matter which code coverage criteria I'm
using?
Coverage type DO-178B/C ISO 26262
(Software architecture)
ISO 26262
(unit test)
Function Used with MC/DC
ASIL C, D Highly
Recommended
ASIL A, B Recommended
-
Call Coverage Not required
ASIL C, D Highly
Recommended
ASIL A, B Recommended
-
Statement Level A, B, C
Required -
ASIL A, B Highly
Recommended
ASIL C, D Recommended
Decision Level A, B
Required - -
Branch Not required -
ASIL B, C, D Highly
Recommended
ASIL A Recommended
MC/DC Level A Required - ASIL D Highly Recommended
ASIL A, B, C Recommended
What are the barriers to 100% code coverage?
Missing requirements.
Missing or incorrect tests.
Dead code.
Deactivated code.
Defensive programming.
Impossible combinations of events.
Compiler-introduced errors.
Barrier #1: Missing Requirements
Add new requirements
Develop tests
Run the tests
What to do about it
Barrier #2: Missing/incorrect tests
What to do about it Implement additional tests
Ensure that they trace to
the correct requirement(s)
Run the new tests
Implement additional tests
Ensure that they trace to
the correct requirement(s)
Run the new tests
What to do about it
Barrier #3: Dead code
foo(a,c){
int b;
mode = MOVING;
if(mode == STATIONARY){
b = 1 + a;
}
c = a;
return c;
}
Remove the dead code if
possible or
Justify why the code cannot
be run
What to do about it
DO-178C recommends "The [dead] code should be removed and
an analysis performed to assess the effect and the need for
reverification." [6.4.4.3c]
For ISO 26262, the general guidance for "insufficient" coverage
applies, namely to provide a rationale for why the dead code is
acceptable [6:9.4.5].
What to do about it…
Barrier #4: Deactivated code
settings = read_input(CONFIG_PINS);
if ((settings & CONFIG_MASK_A) == CONFIG_A) {
/* code to execute in configuration A */
...
} else if ((settings & CONFIG_MASK_B) == CONFIG_B) {
/* code to execute in configuration B */
...
}
Justify why code won’t be
executed
Never executed during normal operation
Will not affect the execution of the system
What to do about it
Barrier #5: Defensive Programming
switch(a % 4){
case 0:
..........
break;
case 1:
..........
break;
case 2:
..........
break;
case 3:
..........
break;
default:
..........
break;
}
Justify why defensive
programming cannot be
executed
Provide evidence that if
triggered, the defensive
programming works
correctly
What to do about it
Barrier #6: Impossible Combinations of Events
-- “enabled” might already be set at this point
if speed < 1700 then
enabled := true;
end if;
if speed < 1500 and enabled then
...
end if;
Simplify the condition?
or
Justify why 100% could not
be achieved
What to do about it
Barrier #7: Compiler Introduced Errors
int a,x;
if a < 10{ /*Compiler generates “a <=10”*/
result = x / 5;
} else {
result = x / 10;
}
a x
Test case 1 3 4
Test case 2 10 4
Review existing code base
Add to coding guidelines /
code review guidelines
What to do about it
Justifying code: in-line justifications
void fatal_error (void)
{
#pragma RVS justification( "COV_FUNCTIONS", "The fatal_error
function cannot be executed during normal execution")
/* some code */
}
...
void check_status (int status)
{
if ( (check_enabled) && (FATAL_ERROR == status) ) {
#pragma RVS justification( "COV_DECISIONS", "Variable 'status'
cannot take value FATAL_ERROR during normal execution")
fatal_error();
}
}
...
Justifying code: justifications in a separate file
#pragma RVS [selftest.c:34] justification("COV_MCDC", "2: (RAM_ERROR ==
status)", "RAM_ERROR cannot be triggered during normal test");
#pragma RVS [selftest.c:439] justification("COV_FUNCTIONS", "Cannot trigger
emergency_shutdown() during normal test");
Justifications editor
Review – establish missing coverage
Identify cause Identify action to take
Dealing with less than 100% code coverage
Review – establish missing coverage
Identify cause Identify action to take
Dealing with less than 100% code coverage
Missing requirements.
Missing or incorrect tests.
Dead code.
Deactivated code.
Defensive programming.
Impossible combinations of events.
Compiler-introduced errors.
Review – establish missing coverage
Identify cause Identify action to take
Dealing with less than 100% code coverage
Remove code Add (or fix)
requirements Justify why code
won't be executed Justify why it isn't
possible to test
reduce testing timescales by running fewer on-target tests
reduce risk through greater tool flexibility
reduced effort for certification activities
Structural coverage analysis to DO-178B/C Level A and ISO 26262
Logos and images can be positioned within this area.
Change layout to blank (without position box) before use
Q A