New frontiers in formal software verification Gerard J. Holzmann [email protected] s0s0 s1s1 f.
-
Upload
eric-perry -
Category
Documents
-
view
217 -
download
1
Transcript of New frontiers in formal software verification Gerard J. Holzmann [email protected] s0s0 s1s1 f.
![Page 2: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/2.jpg)
23 spacecraft and 10 science instrumentsExploring the Solar
System
OPPORTUNITY
SPITZER
EPOXI/DEEP
IMPACT
MRO
CLOUDSAT
DAWN
JASON 2KEPLER
WISE
AQUARIUS
JUNO
GRAIL
MSL
GRACE
JASON 1
MARS ODYSSEY
ACRIMSAT
STARDUST
CASSINI
VOYAGER 1
VOYAGER 2
GALEX
INSTRUMENTS
Earth Science
• ASTER
• MISR
• TES
• MLS
• AIRS
Planetary
• MIRO
• Diviner
• MARSIS
Astrophysics
• Herschel
• Planck
model checking
static analysis
2
![Page 3: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/3.jpg)
3
formal software verification
• after some ~30 years of development, is still rarely used on industrial software– primary reasons:
• it is (perceived to be) too difficult • it is takes too long (months to years)
• even in safety critical applications, software verification is often restricted to the verification of models of software, instead of software
• goal:– make software verification as simple as
testing and as fast as compilation
![Page 4: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/4.jpg)
4
verification of the PathStar switch(flashback to 1999)
• a commercial data/phone switch designed in Bell Labs research (for Lucent Technologies)
• newly written code for the core call processing engine
• the first commercial call processing code that was formally verified– 20 versions of the code were verified with
model checking during development 1998-2000 with a fully automated procedure
![Page 5: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/5.jpg)
5
traditional hurdles:
feature interactionfeature breakageconcurrency problemsrace conditionsdeadlock scenariosnon-compliance withlegal requirements, etc.
software structure
basic call processingplus a long list of features(call waiting, call forwarding, three-way calling, etc., etc.)
call processing(~30KLOC)
~10%
call processingcontrol kernel
PathStar Code (C)
![Page 6: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/6.jpg)
6
complex feature precedence relationsdetecting undesired feature interaction is a serious problem
![Page 7: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/7.jpg)
7
the verification was automated in 5 steps
@PathStarcall processing 1
context3
featurerequirements
4
2 abstraction map
bug reportsproperty violations
msc
code
5
Spin
![Page 8: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/8.jpg)
8
1. code conversion...
@dial:switch(op) {default: /* unexpected input */
goto error;case Crdtmf: /* digit collector ready */
x->drv->progress(x, Tdial);time = MSEC(16000); /* set timer */
@: switch(op) {default: /* unexpected input */
goto error;case Crconn:
goto B@1b;case Cronhook: /* caller hangs up
*/x->drv->disconnect(x);
@: if(op!=Crconn && op!=Crdis)
goto Aidle;...etc...
implied state machine:
Crconn
Crdtmf
Cronhook
else
else
dial
dial1
errordial2
a control state
a state change
PathStar
![Page 9: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/9.jpg)
9
2. defining abstractions
• to verify the code we convert it into an automaton: a labeled transition system– the labels (transitions) are the basic statements from C
• each statement can be converted via an abstraction – which is encoded as a lookup table that supports three possible conversions :– relevant: keep (~ 60%)– partially relevant: map/abstract (~ 10%)– irrelevant to the requirements: hide (~ 30%)
• the generated transition system is then checked against the formal requirements (in linear temporal logic) with the model checker– the program and the negated requirements are converted into -
automata, and the model checker computes their intersection
![Page 10: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/10.jpg)
10
3. defining the context
Ccode
Spinmodel
modelextraction
bugreporting
map
database offeature
requirements
environmentmodel
![Page 11: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/11.jpg)
11
4. defining feature requirements
in Linear Temporal Logic this is written:<> (offhook /\ X ( !dialtone U onhook))
a sample property:“always when the subscriber goes offhook, a dialtone is generated”
failure to satisfy this requirement:<> eventually,
the subscriber goes offhook/\ andX thereafter, no dialtone isU generated until the next
onhook.
-automaton
mechanicalconversion
![Page 12: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/12.jpg)
12
5. verification
LTL requirement
logicalnegation
C code
model extractor
abstractionmap
environmentmodel
*
no dialtonegenerated
bug report
![Page 13: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/13.jpg)
13
hardware support (1999)
client/serversockets code
scripts
![Page 14: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/14.jpg)
14
iterative search refinementt=5 min.
t=15 min.t=40 min.
each verification task is run multiple times, with increasing accuracy
![Page 15: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/15.jpg)
15
minutes sincestart of check
performance (1999) “bugs per minute”
percentof bugsreported
25
50
75
100
(15)
(30)
(50)
(60)(numberof bugsreported)
10 20 30 40
15 bug reportsin 3 minutes25% of all bugs
first bug reportin 2 minutes
50% of all bugs reportedin 8 minutes
![Page 16: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/16.jpg)
16
that was 1999, can we do better now?
1999• 16 networked computers
running the plan9 operating system– 500 MHz clockspeed
(~8GHz equiv)– 16x128 Mbyte of RAM
(~2GB equiv)
2012• 32-core off-the-shelf system,
running standard Ubuntu Linux (~ $4K USD)– 2.5 GHz clockspeed
(~80GHz equiv)– 64 Gbyte of shared RAM
difference:approx. 10x faster, and 32x more RAMdoes this change the usefulness of the approach?
![Page 17: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/17.jpg)
17
performance in 2012: “bugs per second”
50% of all bugsin 7 seconds(38 bugs)
11 bug reportsafter 1 second
number of secondssince start of check
32-core PC,64 GB RAM
2.5GHz per coreUbuntu Linux
16 PCs,128 MB per PC
500MHzPlan9 OS
numberof bugsfound
10 min10 seconds
(1999)
(2012)
![Page 18: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/18.jpg)
18
side-by-side comparison
1999• 25% of all bugs
reported in 180 seconds (15 bugs)
• 50% of all bugs reported in 480 seconds (30 bugs)
• 16 CPU networked system
2012• 15% of all bugs
reported in 1 second (11 bugs)
• 50% of all bugs reported in 7 seconds (38 bugs)
• 1 desktop PC
![Page 19: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/19.jpg)
19
generalization: swarm verification
• goal: leverage the availability of large numbers of CPUs and/or cpu-cores– if an application is too large to verify
exhaustively, we can define a swarm of verifiers that each tries to check a randomly different part of the code
• using different hash-polynomials in bitstate hashing and different numbers of polynomials
• using different search algorithms and/or search orders
– use iterative search refinement to dramatically speedup error reporting (“bugs per second”)
![Page 20: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/20.jpg)
20
swarm search
![Page 21: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/21.jpg)
spin front-endhttp://spinroot.com/swarm/
$ swarm –F config.lib –c6 > scriptswarm: 456 runs, avg time per cpu 3599.2
sec $ sh ./script
# rangek 1 4 # min and max nr of
hash functions
# limitsdepth 10000 # max search depthcpus 128 # nr available cpusmemory 64MB # max memory to be used;
recognizes MB,GBtime 1h # max time to be
used; h=hr, m=min, s=secvector 512 # bytes per state,
used for estimatesspeed 250000 # states per second processedfile model.pml # the spin model
# compilation options (each line defines a search mode)-DBITSTATE #
standard dfs-DBITSTATE -DREVERSE # reversed process
ordering-DBITSTATE -DT_REVERSE # reversed transition
ordering-DBITSTATE –DP_RAND # randomized process
ordering-DBITSTATE –DT_RAND # randomized
transition ordering-DBITSTATE –DP_RAND –DT_RAND # both
# runtime options-c1 -x -n
swarm configuration file:
21
the user specifies:1. # cpus to use2. mem / cpu3. maximum time
![Page 22: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/22.jpg)
many small jobs do the work of one large job – but much faster and in less memory
22
100% coverage
#processes (log)
statesreached
linearscaling
swarmsearch
100% coverage witha swarm of 100 using
0.06% of RAM each (8MB) compared to a single
exhaustive run (13GB)
(DEOS O/S model)
![Page 23: New frontiers in formal software verification Gerard J. Holzmann gholzmann@acm.org s0s0 s1s1 f.](https://reader030.fdocuments.us/reader030/viewer/2022032703/56649d055503460f949d91af/html5/thumbnails/23.jpg)
23
tools mentioned:• Spin: http://spinroot.com (1989)
o parallel depth-first search added 2007o parallel breadth-first search added 2012
• Modex: http://spinroot.com/modex (1999)• Swarm: http://spinroot.com/swarm (2008)
thank you!