Runtime Refinement Checking of Concurrent Data Structures

(the VYRD project)

Serdar TasiranKoç University, Istanbul, Turkey

Shaz QadeerMicrosoft Research, Redmond, USA

The Problem

Given a specification for a data structure

high-level, executable sequential, with atomic operations

an implementation concurrent not atomic not necessarily linearizable may have race conditions

Verify that (All) executions of the implementation are

consistent with the specification

Why runtime verification?

Static techniques Must reason about entire state space of implementation

Theorem proving techniques: Need representation invariant for implementation

Difficult for multi-threaded implementations Compositional methods

Better-suited for correctness proofs We want to catch concurrency bugs

Proofs difficult in practice Need abstract model for each component’s environment Coordination of proof sub-tasks difficult for large programs

Runtime verification No need for component environment abstractions Gives up exhaustiveness, but can handle practical programs

Outline

Preliminaries Atomicity Refinement

I/O refinement

The multiset example Checking I/O refinement Improving I/O refinement Comparison with other correctness criteria

Semantics: State Transition Systems

A set of program variables The set of program states A transition function

Given current state action performed

specifies next state

Actions Visible: e.g. method calls, returns Invisible: e.g. updates to method local variables

More preliminaries Call action: (thread, “Call”, methodName, args) Return action: (thread, “Return”, methodName, rets)

A run

Corresponding trace: The sequence of visible actions along the run

2 3 … n-1

s0 s1 . . .s2 Sn-1 sn

1 2 3 n-1 n

Well-formed atomic traces

No visible actions by any other thread or by another method between matching “call” and “return” actions

Call0

Retu

rn0

a1

a0

Call1

Retu

rn1

b0

Call2

Retu

rn2

c 1c 0

Matching call and return

Matching call and return

Matching call and return

Well-formed atomic traces

Call0

Retu

rn0

a1

a0

Call1

Retu

rn1

b0

Call2

Retu

rn2

c 1c 0

Atomicfragment

Call0

Retu

rn0

Fragment signature

Determinism

Recall: Return action contains return value Atomic state transition system deterministic iff

same initial state and same fragment signature

imply same final state We require specs to be atomic and deterministic

Call0

Retu

rn0

Fragment signature

Outline

Preliminaries Atomicity Refinement

I/O refinement

The multiset example Checking I/O refinement Improving I/O refinement Comparison with other correctness criteria

Refinement

“Projection of traces (visible actions) to application threads match”

S: State transition system for specification I: State transition system for implementation I refines S iff

for every trace I of I,

there is a trace S of S such that,

for every application thread t

I|t = S|t

Refinement

Call0 b0

a0

Call1

a1

Retu

rn1

Retu

rn0

Call2 c 1c 0

Retu

rn2

Call0

Retu

rn0

a1

a0

Call1

Retu

rn1

b0

Call2

Retu

rn2

c 1c 0

Specification trace

S

Implementation trace

I

I/O Refinement

Refinement: I S I|t = S|t Notion of refinement: Choice of what actions are visible

I/O Refinement: Define only “call” and “return” actions as visible

sequence of calls and returns in the implementation a spec run in with matching calls and return values

Spec atomic and deterministic Spec run gives a linear order of method executions

Called “witness interleaving”

Practical issue: Too many possible interleavings Idea: Infer witness interleaving from runtime analysis

Commit actions and witness interleavings

Call0 b0

a0

Call1 a1

Retu

rn1

Retu

rn0

Call2 c 1c 0

Retu

rn2

Call0

Retu

rn0

a1

a0

Call1

Retu

rn1

b0

Call2

Retu

rn2

c 1c 0

Specification trace

S

Implementation trace

I

Commitaction

Commitaction

Commitaction

Commit action

Intuition: The first line of code that makes visible to other

threads the modified view of the data structure state Ordering of commit actions:

Application’s view of ordering of operations

Can be seen as simplified way to construct abstraction map

User specifies line of code that corresponds to the commit action

Outline

Preliminaries Atomicity Refinement

I/O refinement

The multiset example Checking I/O refinement Improving I/O refinement Comparison with other correctness criteria

Example: Multiset

Multiset data structure Two operations

INSERTPAIR(x,y) If it succeeds, inserts integers x and y into the multiset If it fails, it leaves multiset unmodified

LOOKUP(x) Returns true iff x is in the multiset

Multiset Specification

INSERTPAIR(x,y)status success or

failure

if (status = success) M M U {x,y}return status

LOOKUP(x)

return (x M)

INSERTPAIR allows non-deterministic failure or success

Makes choice visible via return value

Multiset Implementation

Implementation uses array to represent multiset

Only elements with valid bit set to “true” in multiset

content

validA

1 2 3 4

Multiset Implementation

LOOKUP(x)for i 1 to n ACQUIRE(L[i]) if (A[i].valid and A[i].content = x) RELEASE(L[i]) return true

else RELEASE(L[i]) return false

5

7

content

validA

5

7

content

validA

FINDSLOT(x)

for i 1 to n

ACQUIRE(L[i])

if (A[i].content = null)

A[i].content = x

RELEASE(L[i])

return i

else

RELEASE(L[i])

return 0;

Implementation helper method: FindSlot(x)

Finds free slot in array to insert x Returns index if slot found

Returns 0 otherwise

Multiset Implementation

INSERTPAIR(x,y)i FINDSLOT(x)if (i=0) return failure

j FINDSLOT(y)if (j=0) A[i].content null

return failure

ACQUIRE(L[i])ACQUIRE(L[j])A[i].valid true

A[j].valid true

RELEASE(L[i])RELEASE(L[j])return success;

5

content

validA

5

7

content

validA

5

7

content

validA

5

7

content

validA

INSERTPAIR(x,y)

i FINDSLOT(x)

if (i=0)

return failure

j FINDSLOT(y)

if (j=0)

A[i].content null

return failure

ACQUIRE(L[i])

ACQUIRE(L[j])

A[i].valid true

A[j].valid true

RELEASE(L[i])

RELEASE(L[j])

return success;

LOOKUP(x)

LOOKUP(y)

LOOKUP(x)

LOOKUP(y)

Commit

Outline

Preliminaries Atomicity Refinement

I/O refinement

The multiset example Checking I/O refinement Improving I/O refinement Comparison with other correctness criteria

Runtime Checking of I/O Refinement

Spec atomic and deterministic Given a sequence of method calls and return values,

there is at most one run

Checking procedure: Execute implementation Record

order of commit points method calls and return values

Execute spec methods in the order they committed Check fails if spec is not able to execute method with

given return value

Runtime Checking of I/O Refinement

I/O refinement check may fail because Implementation is wrong The selection of commit points is wrong

Can tell which is the case by comparing witness interleaving with implementation trace

Improves testing: In multi-threaded tests, without witness interleaving,

difficult to decide expected return value of method or final state at end of test Must consider all interleavings, or Forced to be too permissive

Off-line checking using a log Avoid overhead and concurrency impact

Write actions of implementation into log Verification: Separate thread

Only reads from the log Log: Sequence of visible actions and commit actions Actions appear in log in the order they happen in implementation

Logged actions (not operations) serialized by log Initial part of log in memory Low impact of contention for log

Must perform action atomically with log entry insertion Optimizations possible

Outline

Preliminaries Atomicity Refinement

I/O refinement

The multiset example Checking I/O refinement Improving I/O refinement Comparison with other correctness criteria

Improving I/O refinement

Why: Test program may not perform “observations”

frequently enough Inserting them may affect concurrency too much Observations may not get interleaved at the most

interesting places

Example: If a multiset test program does no LookUp’s, all I/O refinement tests will pass

Improving I/O refinement: The “view”

Include a state-based condition into the definition of refinement Define auxiliary, hypothetical variable “view”

Both spec and implementation have their copy of view User indicates how view gets updated in both

view represents only the “abstract state” of the data structure Abstract away information not relevant to data structure state

From both spec and implementation Method return values determined uniquely by view

Initialized to same value in spec and implementation In spec: Updated once, atomically between call and return In implementation: Updated once, atomically with commit action

During runtime checking, check that views match for each method

Definition of view for multiset

Spec’s definition of view: Contents of multiset in spec

Implementation’s definition of view:

Computed atomically with commit action

view for i 1 to n lockOK = (L[i] not held by any thread) or (L[i] held by thread currently committing) or (L[i] held by thread executing LOOKUP) if (A[i].valid or lockOK) view view U {A[i].content}

Off-line checking for view

Implementation’s version of view constructed from log, off-line Must log all variable updates that affect view can look forward in log, past commit action,

to compute view for implementation

May have to log certain lock acquisitions and releases in log

Compute, compare view incrementally for efficiency

Overhead, impact on concurrency increases But programs we are working on keep similar logs for recovery

purposes anyway Performance overhead tolerable for these

What does view buy?

Examining program and spec state provides more observability Imagine multiset with REMOVE operation Suppose application executes INSERTPAIR(a,a)

But implementation erroneously inserts a once To expose this error through I/O refinement or testing, error must

show up in return value of LOOKUP(a)

Need execution that inserts a pair of a’s when there are no a’s in the multiset removes a looks “a” up before more a’s get inserted

Checking view catches bug right after INSERTPAIR(a,a)

Non-trivial spec view

Imagine multiset spec given as a binary search tree Executable spec contains detail not part of

abstract data structure state Must abstract structure of search tree view just takes union of tree node contents

Abstracts away parent-child, left-right relationships

Outline

Preliminaries Atomicity Refinement

I/O refinement

The multiset example Checking I/O refinement Improving I/O refinement Comparison with other correctness criteria

Multiset is not atomic

INSERTPAIR(x,y)i FINDSLOT(x)if (i=0) return failure

j FINDSLOT(y)if (j=0) A[i].content null

return failure

ACQUIRE(L[i])ACQUIRE(L[j])A[i].valid true

A[j].valid true

RELEASE(L[i])RELEASE(L[j])return success;

Not possible to re-order different threads’ executions of FINDSLOT

Multiset has a race condition

INSERTPAIR(x,y)i FINDSLOT(x)if (i=0) return failure

j FINDSLOT(y)if (j=0) A[i].content null

return failure

ACQUIRE(L[i])ACQUIRE(L[j])A[i].valid true

A[j].valid true

RELEASE(L[i])RELEASE(L[j])return success;

LOOKUP(x)for i 1 to n ACQUIRE(L[i]) if (A[i].valid and A[i].content = x) RELEASE(L[i]) return true

else RELEASE(L[i]) return false

Multiset is not linearizable thread t1 executing

INSERTPAIR(1,2) concurrently with thread t2 executing

INSERTPAIR(5,6)

content

valid

1

content

valid

FINDSLOT(1)by t1 succeeds

1

5

content

valid

FINDSLOT(5)by t2 succeeds

1

5

content

valid

FINDSLOT(2),INSERTPAIR(1,2) by t1 fail

FINDSLOT(6),INSERTPAIR(5,6) by t2 fail

No linearized execution of implementation fails first call to INSERTPAIR(5,6) Multiset implementation not linearizable

Conclusions, future work

Run-time refinement checking a promising approach Caught artificial bugs in Multiset Caught real bugs in Scan file system for WindowsNT In the process of applying it to Boxwood

A concurrent implementation of a B-link tree

When no formal, executable spec exists, we use “atomic version” of implementation as spec Lowers barrier to application of

refinement-based methods Concentrates on concurrency bugs