1 Today Executions, paths, and states Testing via model checking.
-
Upload
leilani-hearst -
Category
Documents
-
view
219 -
download
1
Transcript of 1 Today Executions, paths, and states Testing via model checking.
2
Executions, Paths, and StatesControl flow graph (CFG):structure of a program’s code An execution is simply a
run of the program – the resultof some set of commandsSome executions, with different operations:
A path is a particular waythrough the CFG – ignoringthe commands – where theprogram “goes”
Same path, different executions
A state is the program’s location+ memory contents at a point intime
Same state, different executions,different paths(in the executions, color only shows
the location in the CFG, not thecontents of memory, which may bedifferent even if colors are the same) Same paths, different states
3
A Question
Testing can focus on exploring lots of• Executions
• Random testing• Paths
• Concolic testing• States?
• Model checking• But first: does this gain us anything?
4
A Question
First question: does it make the problem more manageable• That is, are there cases where exploring all
states is more feasible than exploring all paths?
5
Paths vs. Statesint G = 0;
int foo (int x, int y) {if (x < y) G++;if (x == 10) G++;
}
int bar (int x, int y) {if (x < 50) G++;if (y > 8) G++;
}
int main () {foo(x, y);bar(x, y);
if (x + y == 40) G++;
}
Once more, how many paths through thisprogram?
2
4
8
16
32
6
Paths vs. Statesint G = 0;
int foo (int x, int y) {if (x < y) G++;if (x == 10) G++;
}
int bar (int x, int y) {if (x < 50) G++;if (y > 8) G++;
}
int main () {foo(x, y);bar(x, y);
if (x + y == 40) G++;
}
What if we consider the number of states?
If we include x and y’s values in the state,the number is much larger than the numberof paths… so it’s certainly not always animprovement!
7
Paths vs. States
pick(chunk, 0, 10);
pick(nbytes, 0, 10);
write(chunk, wbuf, nbytes);
pick(chunk, 0, 10);
pick(nbytes, 0, 10);
read(chunk, rbuf, nbytes);
pick(chunk, 0, 10);
pick(nbytes, 0, 10);
write(chunk, wbuf, nbytes);
Less clear which is better in thiscase: lots of states, but also lotsof paths
Paths lets us consider inputs thatproduce the same sequence of writesas “the same”
States means we don’t have to worryabout error returns in the past, if thaterror return doesn’t change the systemstate
8
Paths vs. States
So, considering states rather than paths seems to be a trade-off without a clear winner – often at least as un-manageable as complete path exploration
Which is better for error detection?• Hard to say, but for a class of errors
(reachability), we can say that complete state coverage would definitely expose them
9
Model Checking
A model checker is a tool for exploring a state space
Basic idea: generate every reachable state of a transition system• Think of states as nodes in a graph• Directed edges mean “from this state, this is
a possible next state of the program”• Multiple outgoing edges where there is
input/thread scheduling/other nondeterminism
10
Model Checking
Model checking is a huge research field• Clarke, Emerson, and Sifakis received this
year’s Turing Award for coming up with the basic idea
• Some major subfields:• Hardware and software• Symbolic vs. explicit-state
11
Model Checking
Symbolic model checkers “explore” a state space by manipulating logical expressions representing many states
In explicit-state model checking (part of) the graph is actually enumerated• Especially useful for software, where
complex data structures are hard to handle symbolically
• Closer to testingGerard’s SPIN is the most popularexplicit-state model checker
12
Model Checking
We will be looking at one particular kind of explicit-state model checking• Using SPIN• To explore the state spaces of C
programs
Other model checkers used for this approach to testing• Java PathFinder 2 (NASA Ames)• Bogor (U Kansas/Nebraska)
13
Model Checking: State-Based Testing
Model-checking by executing the program• Backtracking search for all states
State already visited!Backtrack and try adifferent operation Done with test!
Backtrack and try adifferent operation
CFG
State already visited!Backtrack and try adifferent operation
Will explore, as a side-effect,many executions and many paths,but the goal is to explore states
15
Two Approaches
CBMC
BLAST
SLAM
JPF2
SPIN
CMC
CRunner
MAGIC
Analysis of derived transition system
Execution of actual code
VeriSoft
CRunner
Our focus in this class
(dynamic: like testing)
(static: like a complete analysis)
Bogor
16
SPIN and Model-Driven Verification
SPIN 4.0 introduced model-driven verification• Can embed C code in a PROMELA model• Code is executed as part of a transition during
depth-first search
SPIN compiles a PROMELA model into a C program: it’s a model checker generator• Embed C code in transitions by executing the
compiled C code – at native speed, with real compiled behavior (including optimizations)
• Take advantage of all SPIN features – hashing, multicore exploration, etc.
17
SPIN and Model-Driven Verification
Used to “model check” significant pieces of implementation code at JPL
Very useful for checking rich properties that automated abstraction engines often fail to handle• Functional correctness (differential checking
with a reference implementation)• Invariants of rich data structures
Using SPIN for testing C programs
18
SPIN and Model-Driven Verification
When SPIN backtracks, it uses information on how to restore the state of the C program:• Tracked memory is
restored on backtrack• Matched memory is also
used to determine if a state has been visited before
• All other memory is not restored on backtrack or stored in state table
Execute C codeuntil control
returns to SPIN
Push tracked & matchedstate on stack
Has state beenvisited before?
Store matchedstate in state
table
Backtrack:pop stack &
restore tracked &matched state
Y N
19
SPIN and Model-Driven Verification
Use state that is tracked but not matched to introduce abstractions• E.g., symmetry reduction: a property that does
not depend on the order of items in a linked list• Track the original version of the list• Copy and sort the list after each transition• Match on the sorted list only• SPIN will backtrack when it visits a state with
same list contents, even in a different order
20
Testing via Model Checking
We do not expect to cover all states• There are far too many (trillions, at least)
Use abstraction (“consider this state and that state to be the same”) to limit the exploration – may miss errors
Use techniques that have a small probability of missing some states (hashing schemes)
21
Testing via Model Checking
Unsound abstractions:• For example, in the flight flash file
systems we’ve checked, the state of the flash device
Used page
Free page
Dirty page
Bad block
live pages? (0-1) x dirty pages (0-1) x block state (bad, free, current)
22
Testing via Model Checking
Does not guarantee that states (or errors) won’t be missed, but provides a better understanding of which states have been explored, based on the program’s purpose
Manages state space size
Used page
Free page
Dirty page
Bad block
live pages? (0-1) x dirty pages (0-1) x block state (bad, free, current)
23
Unsound Abstractions
“Abstraction” can mean a lot of things
What I’ll be talking about in this class:• One operation: from a concrete state, c, we
can compute an abstract state A(c)• Backtrack according to abstract state matching• Goal: we would like to explore every reachable
abstract state of a C program
24
Unsound Abstractions
For some abstractions, the goal is reached if we simply match on abstract state – if
c1, c2 . A(c1) = A(c2) c3 . T(c1, c3) c4 . T(c2, c4) A(c3) = A(c4)
25
Unsound Abstractions
Otherwise:
These states abstract the same
So do theseAnd these
These, however, do not abstract equivalently,and can only be reached through their parents(and are the only concrete representatives oftheir respective abstract states).
Whichever of these concrete states we explorefirst, we will miss one of the children.
26
Unsound Abstractions
We usually don’t have soundness for any abstractions that reduce the state-space to a manageable size
If not, unsound in our context (underapproximating abstract successors by abstracting concrete states)
That’s ok• It’s just testing, we don’t expect to be
complete
27
Testing via Model Checking
Requires the ability to restore a running program to an earlier execution state• Difficult engineering problem, handled by
applying automatic code instrumentation• Rewrite the program to check that the
backtracking works properly• As a side-benefit, can simulate flight
software resets and hardware faults
28
Simple PROMELA Codeint x;
int y;
active proctype main () {
if:: x = 1:: x = 2fi;
assert (x == y);
}
Start simple
This model has 7 states
What are they?
State = (PC, x, y)
1
2
3
5
7
SPIN’s nondeterministic choice construct
Picks any one of the choices that is enabled
How do we guard a choice?if:: (x < 10) -> y = 1:: (x < 5) -> y = 3:: (x > 1) -> y = 4fi;
Not mutuallyexclusive!
29
Simple PROMELA Codeint x;
int y;
active proctype main () {
if:: x = 1:: x = 2fi;
if:: y = 1:: y = 2fi;
if:: x > y -> x = y:: y > x -> y = x:: else -> skipfi;
assert (x == y);
}
This model has 17 states
What are they?
State = (PC, x, y)
123
5
79
13
14
1517
Er…
Don’t worry about state-counting toomuch – SPIN has various automaticreductions and atomicity choices thatcan make that difficult
30
Simple PROMELA Codeint x;
active proctype main () {
do
:: (x < 10) -> x++
:: break
od
Only a couple more PROMELAconstructs to learn for building testharnesses: the do loop
Like if, except it introducesa loop to the top – break choicecan exit the loop
This nondeterministically assigns xa value in the range 0…9
31
Simple PROMELA Codeint x;
inline pick (var, MAX)
var = 0;
do
:: (var < MAX) -> var++
:: break
od
inline gives us a macro facility
As you can imagine, this is auseful macro for building a test harness!
32
Testing via Model Checking
Basic idea:• We’ll write a test harness in PROMELA• Use SPIN to backtrack and explore all
input choices• Call C code we’re testing• Use abstraction to limit the number of
states we consider
• We can even “trick” SPIN into doing pure random testing!
33
Testing via Model Checking
Rest of class:• We’ll look at the idea of “model-driven verification” that
drives all this• I’ll show how to write a simple PROMELA harness to
test C code
• Thursday: we’ll investigate automatic code instrumentation, some novel coverage-based code strategies, and see how model checking and random testing compare
• Download SPIN yourself, and play with it!• http://www.spinroot.com• Examples available on class website