Argonne National Laboratory School of Computing and SCI Institute, University of Utah Formal...

Argonne National Laboratory School of Computing and SCI Institute, University of Utah

Formal Verification of Programs That Use MPI One-Sided Communication

Salman Pervez, Ganesh Gopalakrishnan, Robert M. KirbySchool of Computing

University of Utah

Rajeev Thakur, William GroppMathematics and Computer Science Division

Argonne National Laboratory


• The demand for concurrent software is increasing.

• Concurrent algorithms are notoriously hard to design and verify.

• Formal methods, and in particular finite-state model checking, provide a means of reasoning about concurrent algorithms.

• Principle advantages of modeling checking approach:- provides formal framework for reasoning- allows coverage – examination of all possible process interleavings

Thesis of the Talk

Thesis: If finite-state models are created and exhaustively analyzed for desired formal properties, robust

algorithms and implementations will result.


What is Model Checking?

Navier-Stokes Equations are a mathematical model of fluid flow physics

“V&V” – Validation and Verification“Validate Models, Verify Codes”

“Formal models” can be generated eitherautomatically or by a modeler which

translate and abstract algorithms and implementations.


Model Checking: History and Current Practice• History

– Approach invented around 1981 by:• Clarke and Emerson, Queille and Sifakis

– Widely used in Hardware Verification since the 90’s– Uses in Software Verification is the current rage

• Notable Successes– Bell Labs : Telephone Switch Software Verification– NASA : Concurrent Java Program Verification– Microsoft : Device Driver Verification

• Applications in HPC by others:– Siegel and Avrunin: MPI two-sided communication programs– Matlin, Lusk, McCune: Verifying parts of MPD


MPI One-Sided Communication

• MPI One-Sided Constructs Examined:– MPI_Win_lock– MPI_Win_unlock– MPI_Put– MPI_Get

• The desired atomicity is provided by the constructs MPI_Win_Lock / MPI_Win_Unlock

• Once the lock is relinquished, data values can no longer be trusted


Test Case: Byte-Range Algorithm

• Algorithm implemented using MPI one-sided communication (with passive-target lock-unlock synchronization) for coordinating a collection of parallel processes contending for byte-range locks.

Notes Concerning Algorithm:• To acquire a lock, a process must checkpoint the

global state by ‘simultaneously’ indicating its intent and reading others’ status.

• When the lock owner release the lock, he wakes up all conflicting ‘sleeping’ processes.


Lock Acquire

lock_acquire (start, end) {

Stage 11 val[0] = 1; /* flag */ val[1] = start; val[2] = end;2 while(1) {3 lock_win4 place val in win5 get values of other processes from win6 unlock_win7 for all i, if (Pi conflicts with my range)8 conflict = 1;

Stage 29 if(conflict) {10 val[0] = 011 lock_win12 place val in win13 unlock_win14 MPI_Recv(ANY_SOURCE)15 }16 else{17 /* lock is acquired */18 break;19 }20 }//end while

Window:

P0 P1

flag start end 0 -1 -1 0 -1 -1 0 -1 -1


Lock Release

lock_release (start, end) { val[0] = 0; /* flag */ val[1] = -1; val[2] = -1;

lock_win place val in win get values of other processes from win unlock_win

for all i, if (Pi conflicts with my range) MPI_Send(Pi);

}

Window:

P0 P1

flag start end 0 -1 -1 0 -1 -1 0 -1 -1


Window:

P0 P1

1 3 5 0 -1 -1 0 -1 -1 0 -1 -1

Process 0 Process 1

lock_acquire(3,5)lock_release()

lock_acquire(3,5)

Example 1: Demonstration of Lock Acquire/Release Strategy


Window:

P0 P1

1 3 5 1 3 5 0 -1 -1 0 -1 -1

Process 0 Process 1


lock_acquire(3,5)

Deduces Conflict – Stage 1



Window:

P0 P1

1 3 5 0 3 5 0 -1 -1 0 -1 -1

Process 0 Process 1


lock_acquire(3,5)

Deduces Conflict – Stage 2Blocks on Receive



Window:

P0 P1

0 -1 -1 0 3 5 0 -1 -1 0 -1 -1

Process 0 Process 1


lock_acquire(3,5)

Deduces Conflict – Stage 2Blocks on Receive

Send Signal to P1



Window:

P0 P1

0 -1 -1 0 3 5 0 -1 -1 0 -1 -1

Process 0 Process 1


lock_acquire(3,5)

Receives SignalRetry Stage 1

Send Signal to P1



Window:

P0 P1

0 -1 -1 1 3 5 0 -1 -1 0 -1 -1

Process 0 Process 1





Window:

P0 P1

0 -1 -1 0 -1 -1 0 -1 -1 0 -1 -1

Process 0 Process 1





inlineMPI_Win_lock(proc_i){ /* try sending a message

on a channel of size 1,

will block if a message is already in

the queue. */

lock_chan!proc_id; }

Modeling in Promela

Example Promela Code for lock_release• C-like structure• Powerful abstractions like channels


Window:

P0 P1

1 3 5 0 -1 -1 0 -1 -1 0 -1 -1

Process 0 Process 1

lock_acquire(3,5)lock_release()lock_acquire(3,5)

lock_acquire(3,5)

Example 2: Demonstration of Lock Acquire/Release Limitation


Window:

P0 P1

1 3 5 1 3 5 0 -1 -1 0 -1 -1

Process 0 Process 1


lock_acquire(3,5)




Window:

P0 P1

0 -1 -1 1 3 5 0 -1 -1 0 -1 -1

Process 0 Process 1


lock_acquire(3,5)



Send Signal to P1


Window:

P0 P1

1 3 5 1 3 5 0 -1 -1 0 -1 -1

Process 0 Process 1


lock_acquire(3,5)

Deduces Conflict – Stage 1Deduces Conflict – Stage 1



Window:

P0 P1

1 3 5 0 3 5 0 -1 -1 0 -1 -1

Process 0 Process 1


lock_acquire(3,5)



Deduces Conflict – Stage 2Block on Receive


Window:

P0 P1

1 3 5 0 3 5 0 -1 -1 0 -1 -1

Process 0 Process 1


lock_acquire(3,5)

Receive SignalRetry Stage 1




Window:

P0 P1

1 3 5 1 3 5 0 -1 -1 0 -1 -1

Process 0 Process 1


lock_acquire(3,5)





Window:

P0 P1

0 3 5 1 3 5 0 -1 -1 0 -1 -1

Process 0 Process 1


lock_acquire(3,5)

Deduces Conflict – Stage 1Deduces Conflict – Stage 2Block on Receive



Window:

P0 P1

0 3 5 0 3 5 0 -1 -1 0 -1 -1

Process 0 Process 1


lock_acquire(3,5)





Window:

P0 P1

0 3 5 0 3 5 0 -1 -1 0 -1 -1

Process 0 Process 1


lock_acquire(3,5)




DEADLOCK


ObservationsAfter Model Checking

• P0 releases lock before it can see that P1 will be blocked.

• There is no way for P0 to figure out whether P1 merely wants the lock or is actually blocked.

• Multiple unmatched sends can occur (example to follow)


Window:

P0 P1

1 3 5 1 6 8 0 -1 -1 0 -1 -1

Process 0 Process 1 Process 2



lock_acquire(5,6)

P2



Window:

P0 P1

1 3 5 1 6 8 1 5 6 0 -1 -1




lock_acquire(5,6)

P2




Window:

P0 P1

1 3 5 1 6 8 0 5 6 0 -1 -1




lock_acquire(5,6)

P2




Window:

P0 P1

0 -1 -1 0 -1 -1 0 5 6 0 -1 -1




lock_acquire(5,6)

P2


Send Signal to P2 Send Signal to P2



Proposed Solution 1• Main idea: Distinguish between processes that want

the lock and those that are blocked.• Three possible flag values:

– 0 = I do not have the lock– 1 = I have the lock– 2 = I am trying for the lock

• If a process wants the lock, but finds another conflicting process with a flag value of 2, it must wait until this value changes to either 1 or 0.

• We have added more certainty to the algorithm but taken a possible performance hit and possible livelock.


Proposed Solution 2• Main Idea: The process about to be blocked picks who

will wake it up and indicates so by writing to shared memory

• Once processes declare their intentions globally, deadlock can be avoided.

• For there to be deadlock, a dependency cycle must exist.

• The last process to complete this cycle will know about it and must not do so.

Window:

P0 P1

flag start end pick -1 -1 0 -1 -1 0 -1 -1


Window:

P0 P1

1 3 5 -1 0 -1 -1 -1 0 -1 -1 -1

Process 0 Process 1


lock_acquire(3,5)

Example 3: Demonstration of Lock Acquire/Release Proposed Solution 2


Window:

P0 P1

1 3 5 -1 1 3 5 -1 0 -1 -1 -1

Process 0 Process 1


lock_acquire(3,5)




Window:

P0 P1

0 -1 -1 -1 1 3 5 -1 0 -1 -1 -1

Process 0 Process 1


lock_acquire(3,5)




Window:

P0 P1

1 3 5 -1 1 3 5 -1 0 -1 -1 -1

Process 0 Process 1


lock_acquire(3,5)

Deduces Conflict – Stage 1Deduces Conflict – Stage 1



Window:

P0 P1

1 3 5 -1 0 3 5 0 0 -1 -1 -1

Process 0 Process 1


lock_acquire(3,5)

Deduces Conflict – Stage 1 Deduces Conflict – Stage 2Block on Receive



Window:

P0 P1

1 3 5 -1 0 3 5 0 0 -1 -1 -1

Process 0 Process 1


lock_acquire(3,5)


No Conflict – Stage 1



Window:

P0 P1

0 3 5 -1 0 3 5 0 0 -1 -1 -1

Process 0 Process 1


lock_acquire(3,5)


Deduces Deadlock – Stage 2Reset to Stage 1



Discussion and Future Work“Execution Checking”

“Model Checking”

In current practice, concrete executions on a few diverse platforms are often used to verifyalgorithms/codes.

Consequence: Many feasible executions mightnot be manifested.

Model checking forces all executions of a judiciously down-scaled model to be examined.

Current focus of our research: minimize modeling effort and error.


Funding Acknowledgements:

• NSF (CSR–SMA: Toward Reliable and Efficient Message Passing Software Through Formal Analysis)• Microsoft (Formal Analysis and Code Generation Support for MPI)• Office of Science – Department of Energy

Summary• Paradigms such as one-sided MPI and threading creates a plethora of execution possibilities – many of which might be algorithmically fatal yet lay dormant at testing time.

• Model checking provides a formal and practical means of reasoning about all possible executions as part of the design, verification and optimization process.

Closing Question (“Food for Thought”):• Can one come up with safe usages (i.e. easier to verify yet not overly restrictive) of one-sided communication?

Argonne National Laboratory School of Computing and SCI Institute, University of Utah Formal...

Documents

Transcript of Argonne National Laboratory School of Computing and SCI Institute, University of Utah Formal...