Verisim: Formal Analysis of Network Simulations

Karthikeyan Bhargavan, Carl A. Gunter, Moonjoo Kim, Insup Lee, Davor Obradovic, Oleg Sokolsky, Mahesh ViswanathanUniversity of Pennsylvania

Internetwork Routing Protocols Internetwork routing protocols

enable interoperability between physical networks.

Routing protocols for the Internet: RIP, OSPF, and BGP.

Routing protocols for packet radio: AODV and DSR.

Routing protocols and software have growing importance and complexity.

Current Analysis Techniques Rigorous mathematical models and

proofs – limited by complexity. Testing.

Testbed – expensive. Operational – risky and inconvenient.

Simulation. Performance attributes: throughput,

latency, reliability, etc.

Limitations of Performance Analysis of Simulations Flaws may not be detected if no

expected performance can be used for comparison.

When flaws are suspected, finer means of analysis are useful.

Some flaws do not manifest themselves as performance problems (e.g. most security gaps).

Verisim Verisim provides support for “logical”

analysis of network simulations. This talk describes its architecture

and logic. We provide a series of experiments

aimed at assessing the approach.

Monitoring and Checking (MaC) Framework

MonitoringScript

ProgramMonitor Checker

Properties

MetatraceTrace

Data

NS MEDL

Meta Event Description Language (MEDL) Expresses properties of traces. Extension of Linear Temporal Logic

(LTL) with auxiliary variables. More expressive than LTL. Properties classified into

(instantaneous) events and (enduring) conditions.

S Kannan, M Kim, I Lee, O Sokolsky, M Viswanathan 98

NS Network SimulatorInstrumentedProtocol Code

P : C++

ConfigurationParameters:

OTcl

TrafficModel:OTcl

Topology:OTcl

Scenario

P

N

src/sink

P

N

src/sink

P

N

src/sink

ProtocolAgents

NetworkModel

TrafficAgents

NS TraceVINT Project

Verisim

Trace Checker

Properties: MEDL

Metatrace

InstrumentedProtocol: C++

Scenario:OTcl

NS

Ad Hoc Networks Routing for a wireless internetwork

without the aid of a central base station.

Connections are low-bandwidth, lossy, and highly transient.

Unique routing assumptions: Most routes are seldom used. Bandwidth must be protected.

Illustration Part 1 of 2

Routing

Movement

Illustration Part 2 of 2

NewRouting

AODV Protocol If a node S needs a route to a destination D

and does not have one, it floods a route-request (RREQ) packet through the network.

Each recipient R of this RREQ keeps a return pointer.

R broadcasts the request to its neighbors if it is not D and does not have a route to D.

If R is D, or has a route to D, it responds with a route-reply (RREP) packet using the return pointers for S.

Perkins and Royer 99

Loop Freedom in AODV Routing loops are undesirable. AODV uses sequence numbers to

indicate freshness of link information. Key Invariant: If next(n) = n’, then

seqno(n) seqno(n’), and if seqno(n) = seqno(n’), then hops(n) >

hops(n’). The invariant ensures that there are

no loops.

Outline of Experiment Run a scenario of modest complexity. Analyze it in Verisim using a list of 9

properties of AODV expressed in MEDL. First pass: Repair First Bug (RFB). Second pass: “tune” the MEDL formulas to

avoid rerunning the simulation discovering bugs in the metatrace.

We instrumented simulation code for AODVv0 supplied by the CMU Monarch Project.

Experiment Scenario1

2 3

4

5

Experiment Scenario1

2 3

4

5

Experiment Scenario1

2 3

4

5

Experiment Scenario1

2 3

4

5

Experiment Scenario1

2 3

4

5

Sample MEDL Alarmalarm LoopInv[at][nxt][dst] = sendrout[at][dst] when ((atnxt) (atdst) (nxtdst) (obs_nexthop[at][dst] == nxt) ((obs_seqno[at][dst] > obs_seqno[nxt][dst]) ((obs_seqno[at][dst] == obs_seqno[nxt][dst]) (obs_hopcont[at][dst] <= obs_hopcnt[at][dst]))))

This is the negation of the fundamental invariant ensuringno loops in AODV:

If the next hop from node at toward destination dst is node nxt then the sequence number (for dst) of at is less than or equal to that of nxt or they are equal and the hop count (for dest) of at is less than or equal to that of nxt.

Repair First Bug Experiment Let be the MEDL property set. Run the simulation to get a trace T. Run the checker to get a metatrace T. Repair the first bug in the metatrace to get

new protocol code. Rerun the simulation with the new protocol

code to get a new trace U. Rerun the checker to get a new metatrace

U. Continue until an empty metatrace is

obtained.

RFB Experiment StatisticsMeta-trace

DestRep

DetectRErr

NodeRep

LoopEnv

TotalAlarms

T 4 54 38 43 220

U 0 54 38 43 216

V 0 48 39 44 206

W 0 0 0 0 1

Tuning Experiment StatisticsMeta-trace

DestRep

DetectRErr

NodeRep

LoopEnv

TotalAlarms

T 4 54 38 43 220

T 0 54 38 43 216

T 0 0 38 50 166

T 0 0 21 0 21

No newsimulation

traces.

Analysis of “Off-The-Shelf” (OTS) Simulations The prior experiment involved fewer

than 10,000 events, and we designed it to exercise key scenarios.

Can we do useful analysis with OTS performance simulations?

Aim: Verisim can be added with small modifications, run alongside OTS simulations, and find logical failures in a practical period of time.

Experiment Took largest available OTS simulation for

AODV and ran it to create a trace. Simulation from Monarch uses 50 mobile nodes on

1500x300m grid moving 20 m/s. 5220 seconds (1.5 hours) to complete simulation. 6,446,316 events.

Naïve effort: ran MEDL with MonSeqNo test on all nodes (2500 relations) using 550Mhz dual processor machine with 1GB of memory.

Aborted the experiment after 4 days; estimate 100 days to complete this analysis.

Optimizations For Network Routing Simulations Population Abstraction: test the

property for a subset of the routers. Packet Type Abstraction: prune the

trace to include only relevant events.

Sample Experiment Results for MonSeqNo Population Abstraction for 5 nodes.

Trace size: 6,446,316 events Property size: 14,543 bytes Time: 51,054 seconds Rate: 0.54 micro seconds per event per property.

Population and Packet Type Abstractions. Trace size: 6812 events Property size: 14,543 bytes Time: 51 seconds (found failures) Rate: 0.51 micro seconds per event per property.

Sample Experiment Results for LoopInv Population Abstraction for 5 nodes.

Trace size: 69,411 events Property size: 75,508 bytes Time: 8064 seconds Rate: 1.54 micro seconds per event per property.

Population and Packet Type Abstractions. Trace size: 48,735 events Property size: 75,508 bytes Time: 5912 seconds (found failures) Rate: 1.61 micro seconds per event per property.

Related Work Communication-based test generation

systems. STRESS and Verisoft. Test oracles based on formulas or

formal operational specifications. GIL and TETRA.

Formal instance verification of routing protocols. SPIN/PITHIA for PNNI.

Simulations based on logical specifications. MTSim and Maude.

Network monitoring systems. Emerald.

Summary Verisim integrates simulation and

logical trace analysis. The combination provides a more

flexible approach to analyzing network simulations for safety properties.

It is able to find bugs in existing simulator studies.

Its flexibility can be exploited to improve debugging turn-around time.