http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS...
description
Transcript of http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS...
![Page 1: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/1.jpg)
Practical Formal Verification ofMPI and Thread Programs
Sarvani VakkalankaAnh Vo*
Michael DeLisiSriram Aananthakrishnan
Alan HumphreyChristopher Derrick
Yu YangGanesh Gopalakrishnan*
Robert M. Kirby** = presenters
School of Computing, University of Utah,Salt Lake City, UT 84112, USA
http:// www.cs.utah.edu / formal_verification / europvm09-tutorial-mpi-threading-fvSupported by NSF CNS 0509379, CCF 0811429, CCF 0903408,
SRC tasks TJ 1847.001 and TJ 1993, and Microsoft 1
![Page 2: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/2.jpg)
• Other students involved:Salman Pervez, Robert Palmer, Guodong Li, Geof Sawaya, Subodh Sharma, Grzegorz Szubzda, Jason Williams, Simone Atzeni, Wei-Fan Chiang
• External Collaborators:ANL / UIUC : Rajeev Thakur, Bill Gropp, Rusty Lusk
IBM : Beth Tibbits LLNL : Bronis de Supinski, Martin Schulz, Dan Quinlan Microsoft : Robert Palmer, Dennis Crain, Shahrokh Mortazavi
Additional Acknowledgements for this tutorial
![Page 3: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/3.jpg)
• Overview of Formal Verification, especially Dynamic Verification • Overview of MPI • Demo of our tool ISP• Architecture of ISP• Presentation of Any_src_can_deadlock (from Umpire test suite)• Our algorithm POE (Partial Order avoiding Elusive interleavings)• Presentation of POE-Illustration• Present details of POE-Illustration: ISP’s Eclipse framework and
GUI• Boot into LiveDVD and practice on POE-Illustration
9:00 to 10:30
![Page 4: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/4.jpg)
• Coffee Break
• IMPORTANT : Please give feedback before it is too late
• Too fast ?
• Too slow ?
• Just right !! ?
• Assuming a lot ?
• Other suggestions ?
• We will TRY to take into account these valuable suggestions!
10:30 to 11:00
![Page 5: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/5.jpg)
• Illustration of Resource Dependent Deadlocks, and Detection• Illustration of Resource Leak, and Detection• Iprobe behavior, and illustration using GUI• Assertion Violation in Red/Blue Problem• Audience Participation in Above Exercises• ISP’s Theory : MPI Happens-before
• Also called “matches before, completes before” in the tool
11:00 to 12:00
![Page 6: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/6.jpg)
• Example of Matrix Multiplication: Four Variations• Analysis of these variations using ISP, with Audience
Participation
12:00 to 12:30
![Page 7: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/7.jpg)
• Assisted Problem Solving by Audience
14:00 to 15:00
![Page 8: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/8.jpg)
• Overview of Dynamic Verification of Shared Memory Thread Programs
15:00 to 15:30
![Page 9: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/9.jpg)
• Dynamic Verification of Thread Programs using Inspect• Concluding Remarks
16:00 to 17:30
![Page 10: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/10.jpg)
Overview of Formal Verification methods
for Validating Concurrent Systems
About 30 minutes – by Ganesh
10
![Page 11: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/11.jpg)
11
Problem: Engineering Reliable Concurrent Systems
![Page 12: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/12.jpg)
• Designers require a push-button debugger-like interface– But one that offers coverage guarantees and deeper
insights
For many important reasons, we advocate Dynamic Formal Verification methods
![Page 13: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/13.jpg)
• Testing methods suffer from bug omissions
13X
Bug Omissions
For many important reasons, we advocate Dynamic Formal Verification methods
![Page 14: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/14.jpg)
• Testing methods suffer from bug omissions• Static analysis methods generate many false alarms
14X X
Bug Omissions False Alarms
For many important reasons, we advocate Dynamic Formal Verification methods
![Page 15: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/15.jpg)
• Testing methods suffer from bug omissions• Static analysis methods generate many false alarms• Model based verification requires tedious model
building
15X X X
Bug Omissions False Alarms Tedious Modeling
For many important reasons, we advocate Dynamic Formal Verification methods
![Page 16: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/16.jpg)
• Testing methods suffer from bug omissions• Static analysis methods generate many false alarms• Model based verification requires tedious model
building• Dynamic verification methods are ideal for
designers!
16X X X √
Bug Omissions False Alarms Tedious Modeling • No omissions• No false alarms• No need for modeling
For many important reasons, we advocate Dynamic Formal Verification methods
![Page 17: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/17.jpg)
Code written using mature libraries (MPI, OpenMP, PThreads, …)
API calls made from real programming languages
(C, Fortran, C++)
Runtime semantics determined by realistic compilers and runtimes
Dynamic VerificationMethods are going to be very important for real engineers !
(static analysis and model based verification canplay importantsupportive roles)
17
Growing Importance of Dynamic Verification
![Page 18: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/18.jpg)
• Verisoft Project– Used for telephone switch software verification in Bell Labs– Available
A Brief Survey of Dynamic Verification tools
![Page 19: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/19.jpg)
• Verisoft Project– Used for telephone switch software verification in Bell Labs– Available
• The Java Pathfinder Project– Developed at NASA for Java Control Software– On SourceForge
A Brief Survey of Dynamic Verification tools
![Page 20: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/20.jpg)
• Verisoft Project– Used for telephone switch software verification in Bell Labs– Available
• The Java Pathfinder Project– Developed at NASA for Java Control Software– On SourceForge
• The CHESS Project– Microsoft Research ; available for academic institutions– In use within Microsoft product groups, and used by
academics
A Brief Survey of Dynamic Verification tools
![Page 21: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/21.jpg)
• Verisoft Project– Used for telephone switch software verification in Bell Labs– Available
• The Java Pathfinder Project– Developed at NASA for Java Control Software– On SourceForge
• The CHESS Project– Microsoft Research ; available for academic institutions– In use within Microsoft product groups, and used by
academics• Inspect : Our fairly unique Pthread / C
verifier– Discussed in this tutorial
A Brief Survey of Dynamic Verification tools
![Page 22: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/22.jpg)
• Verisoft Project– Used for telephone switch software verification in Bell Labs– Available
• The Java Pathfinder Project– Developed at NASA for Java Control Software– On SourceForge
• The CHESS Project– Microsoft Research ; available for academic institutions– In use within Microsoft product groups, and used by
academics• Inspect : Our fairly unique Pthread / C
verifier– Discussed in this tutorial
• ISP : Our very unique MPI / C program verifier– Main focus of THIS TUTORIAL !!
A Brief Survey of Dynamic Verification tools
![Page 23: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/23.jpg)
– Through PMPI • Runs the code under a verification scheduler
– ‘Hijacks’ native scheduler• By interposing a profiler
– Exerts its own Interleaving Generation Control• Selective replay, Dynamic Instruction Rewriting
– TRIES HARD to generate only RELEVANT interleavings• Only replays around “non-determinism”
– Does ‘stateless’ (replay) verification• Restarts from MPI_Init for each new interleaving
Example : How ISP Effects Dynamic Verification
![Page 24: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/24.jpg)
• Somehow Instruments the Source / Binary– Through PMPI
• Runs the code under a verification scheduler
– ‘Hijacks’ native scheduler• By interposing a profiler
– Exerts its own Interleaving Generation Control• Selective replay, Dynamic Instruction Rewriting
– TRIES HARD to generate only RELEVANT interleavings• Only replays around “non-determinism”
– Does ‘stateless’ (replay) verification• Restarts from MPI_Init for each new interleaving
Example : How ISP Effects Dynamic Verification
![Page 25: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/25.jpg)
• Somehow Instruments the Source / Binary– Through PMPI at source level
• Runs the code under a verification scheduler
– ‘Hijacks’ native scheduler• By interposing a profiler
– Exerts its own Interleaving Generation Control• Selective replay, Dynamic Instruction Rewriting
– TRIES HARD to generate only RELEVANT interleavings• Only replays around “non-determinism”
– Does ‘stateless’ (replay) verification• Restarts from MPI_Init for each new interleaving
Example : How ISP Effects Dynamic Verification
![Page 26: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/26.jpg)
• Somehow Instruments the Source / Binary– Through PMPI
• Runs the code under a verification scheduler
– ‘Hijacks’ native scheduler• By interposing a profiler
– Exerts its own Interleaving Generation Control• Selective replay, Dynamic Instruction Rewriting
– TRIES HARD to generate only RELEVANT interleavings• Only replays around “non-determinism”
– Does ‘stateless’ (replay) verification• Restarts from MPI_Init for each new interleaving
Example : How ISP Effects Dynamic Verification
![Page 27: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/27.jpg)
• Somehow Instruments the Source / Binary– Through PMPI
• Runs the code under a verification scheduler
– ‘Hijacks’ MPI Function Calls• By interposing a profiler
– Exerts its own Interleaving Generation Control• Selective replay, Dynamic Instruction Rewriting
– TRIES HARD to generate only RELEVANT interleavings• Only replays around “non-determinism”
– Does ‘stateless’ (replay) verification• Restarts from MPI_Init for each new interleaving
Example : How ISP Effects Dynamic Verification
![Page 28: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/28.jpg)
• Somehow Instruments the Source / Binary– Through PMPI
• Runs the code under a verification scheduler
– ‘Hijacks’ MPI Function Calls• By interposing a profiler
– Exerts its own Interleaving Generation Control• Selective replay, Dynamic Instruction Rewriting
– TRIES HARD to generate only RELEVANT interleavings• Only replays around “non-determinism”
– Does ‘stateless’ (replay) verification• Restarts from MPI_Init for each new interleaving
Example : How ISP Effects Dynamic Verification
![Page 29: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/29.jpg)
• Somehow Instruments the Source / Binary– Through PMPI
• Runs the code under a verification scheduler
– ‘Hijacks’ MPI Function Calls• By interposing a profiler
– Exerts its own Interleaving Generation Control• Selective replay, Dynamic Instruction Rewriting
– TRIES HARD to generate only RELEVANT interleavings• Only replays around “non-determinism”
– Does ‘stateless’ (replay) verification• Restarts from MPI_Init for each new interleaving
Example : How ISP Effects Dynamic Verification
![Page 30: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/30.jpg)
• Somehow Instruments the Source / Binary– Through PMPI
• Runs the code under a verification scheduler
– ‘Hijacks’ MPI Function Calls• By interposing a profiler
– Exerts its own Interleaving Generation Control• Selective replay, Dynamic Instruction Rewriting
– TRIES HARD to generate only RELEVANT interleavings• Only replays around “non-determinism”
– Does ‘stateless’ (replay) verification• Restarts from MPI_Init for each new interleaving
Example : How ISP Effects Dynamic Verification
![Page 31: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/31.jpg)
• Somehow Instruments the Source / Binary– Through PMPI
• Runs the code under a verification scheduler
– ‘Hijacks’ MPI Function Calls• By interposing a profiler
– Exerts its own Interleaving Generation Control• Selective replay, Dynamic Instruction Rewriting
– TRIES HARD to generate only RELEVANT interleavings• Only replays around “non-determinism”
– Does ‘stateless’ (replay) verification• Restarts from MPI_Init for each new interleaving
Example : How ISP Effects Dynamic Verification
![Page 32: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/32.jpg)
• Somehow Instruments the Source / Binary– Through PMPI
• Runs the code under a verification scheduler
– ‘Hijacks’ MPI Function Calls• By interposing a profiler
– Exerts its own Interleaving Generation Control• Selective replay, Dynamic Instruction Rewriting
– TRIES HARD to generate only RELEVANT interleavings• Only replays around “non-determinism”
– Does ‘stateless’ (replay) verification• Restarts from MPI_Init for each new interleaving
Example : How ISP Effects Dynamic Verification
![Page 33: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/33.jpg)
• Somehow Instruments the Source / Binary– Through PMPI
• Runs the code under a verification scheduler
– ‘Hijacks’ MPI Function Calls• By interposing a profiler
– Exerts its own Interleaving Generation Control• Selective replay, Dynamic Instruction Rewriting
– TRIES HARD to generate only RELEVANT interleavings• Only replays around “non-determinism”
– Does ‘stateless’ (replay) verification• Restarts from MPI_Init for each new interleaving
Example : How ISP Effects Dynamic Verification
![Page 34: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/34.jpg)
• Somehow Instruments the Source / Binary– Through PMPI
• Runs the code under a verification scheduler
– ‘Hijacks’ MPI Function Calls• By interposing a profiler
– Exerts its own Interleaving Generation Control• Selective replay, Dynamic Instruction Rewriting
– TRIES HARD to generate only RELEVANT interleavings• Only replays around “non-determinism”
– Does ‘stateless’ (replay) verification• Restarts from MPI_Init for each new interleaving
Example : How ISP Effects Dynamic Verification
![Page 35: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/35.jpg)
L0
U0
L1
L2
U1
U2
L0
U0
L2
U2
L1
U1
35
Sketch of Stateless / Replay Verification
Red, Green, and Blue movesBelong to different processes
Dotted arrow shows someDependency(e.g., runtime non-determinism)
Start systemIn Initial State
![Page 36: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/36.jpg)
Exponential number of TOTAL Interleavings – most are EQUIVALENT – generate only RELEVANT ones !!
36
P0 P1 P2 P3 P4
TOTAL > 10 Billion Interleavings !!
![Page 37: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/37.jpg)
Exponential number of TOTAL Interleavings – most are EQUIVALENT – generate only RELEVANT ones !!
37
P0 P1 P2 P3 P4
A B1
These are the only dependent actionsE.g. One ANY-SOURCE (wildcard) receiveAnd two of its MATCHING SENDS
Point-to-point actions can be issued in ANY order
TOTAL > 10 Billion Interleavings !!
B2
A
B2B1
![Page 38: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/38.jpg)
Exponential number of TOTAL Interleavings – most are EQUIVALENT – generate only RELEVANT ones !!
38
P0 P1 P2 P3 P4
A B1
These are the only dependent actionsE.g. One ANY-SOURCE (wildcard) receiveAnd two of its MATCHING SENDS
Point-to-point actions can be issued in ANY order
Only TWO RELEVANT
Interleavings !
TOTAL > 10 Billion Interleavings !!
B2
A
B2B1
![Page 39: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/39.jpg)
Executable
Proc1
Proc2
……Procn
Scheduler that generates ALL
RELEVANT schedules
(one per partial order)
Run
MPI Runtime
39
MPI Program
Interposition Layer
Workflow of ISP
![Page 40: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/40.jpg)
40
P0 P1 P2
Barrier
Isend(1, req)
Wait(req)
Scheduler
Irecv(*, req)
Barrier
Recv(2)
Wait(req)
Isend(1, req)
Wait(req)
Barrier
Isend(1)
sendNext Barrier
MPI Runtime
Hijack Calls, Generate Relevant Interleavings
![Page 41: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/41.jpg)
P0 P1 P2
Barrier
Isend(1, req)
Wait(req)
Scheduler
Irecv(*, req)
Barrier
Recv(2)
Wait(req)
Isend(1, req)
Wait(req)
Barrier
Isend(1)
sendNextBarrier
Irecv(*)Barrier
41
MPI Runtime
Hijack Calls, Generate Relevant Interleavings
![Page 42: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/42.jpg)
P0 P1 P2
Barrier
Isend(1, req)
Wait(req)
Scheduler
Irecv(*, req)
Barrier
Recv(2)
Wait(req)
Isend(1, req)
Wait(req)
Barrier
Isend(1)
Barrier
Irecv(*)
Barrier
Barrier
Barrier
Barrier
Barrier
42
MPI Runtime
Hijack Calls, Generate Relevant Interleavings
![Page 43: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/43.jpg)
P0 P1 P2
Barrier
Isend(1, req)
Wait(req)
MPI Runtime
Scheduler
Irecv(*, req)
Barrier
Recv(2)
Wait(req)
Isend(1, req)
Wait(req)
Barrier
Isend(1)
Barrier
Irecv(*)
Barrier
Barrier
Wait (req)
Recv(2)
Isend(1)
SendNext
Wait (req)
Irecv(2)Isend
Wait
No Match-Set
43
Deadlock!
Hijack Calls, Generate Relevant Interleavings
![Page 44: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/44.jpg)
• lucky.c has a deadlock that shows upon testing
• unlucky.c does not reveal a deadlock upon testing
• Testing is done using mpicc ; mpirun
• Verification is done using ispcc ; isp
Let us see ISP in action on ‘lucky.c’ and ‘unlucky.c’
![Page 45: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/45.jpg)
Process P0
R(from:*, r1) ;
R(from:2, r2);
S(to:2, r3);
R(from:*, r4);
All the Ws…
Process P1
Sleep(3);
S(to:0, r1);
All the Ws…
Process P2
//Sleep(3);
S(to:0, r1);
R(from:0, r2);
S(to:0, r3);
All the Ws…
45
Example MPI program ‘lucky.c’ (lucky for tester)
![Page 46: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/46.jpg)
Process P0
R(from:*, r1) ;
R(from:2, r2);
S(to:2, r3);
R(from:*, r4);
All the Ws…
Process P1
// Sleep(3);
S(to:0, r1);
All the Ws…
Process P2
Sleep(3);
S(to:0, r1);
R(from:0, r2);
S(to:0, r3);
All the Ws…
46
MPI program ‘unlucky.c’
![Page 47: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/47.jpg)
Runs of lucky.c and unlucky.c on mpich using “standard testing” (“lucky” for tester)
47
mpicc lucky.c -o lucky.outmpirun -np 3 ./lucky.out(0) is alive on ganesh-desktop(1) is alive on ganesh-desktop(2) is alive on ganesh-desktopRank 0 did IrecvRank 2 did SendSleep overRank 1 did Send
[.. hang ..]
mpicc unlucky.c -o unlucky.outmpirun -np 3 ./unlucky.out(0) is alive on ganesh-desktop(2) is alive on ganesh-desktop(1) is alive on ganesh-desktopRank 0 did IrecvRank 1 did SendRank 0 got 11Sleep overRank 2 did Send(2) Finished normally(1) Finished normally(0) Finished normally
[.. OK ..]
![Page 48: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/48.jpg)
Runs of lucky.c and unlucky.c on mpich using “standard testing” (“lucky” for tester)
48
mpicc lucky.c -o lucky.outmpirun -np 3 ./lucky.out(0) is alive on ganesh-desktop(1) is alive on ganesh-desktop(2) is alive on ganesh-desktopRank 0 did IrecvRank 2 did SendSleep overRank 1 did Send
[.. hang ..]
mpicc unlucky.c -o unlucky.outmpirun -np 3 ./unlucky.out(0) is alive on ganesh-desktop(2) is alive on ganesh-desktop(1) is alive on ganesh-desktopRank 0 did IrecvRank 1 did SendRank 0 got 11Sleep overRank 2 did Send(2) Finished normally(1) Finished normally(0) Finished normally
[.. OK ..]
ispcc ; isp will detect deadlock in both cases !!
![Page 49: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/49.jpg)
• With ISP at hand, WE ARE LUCKY IN BOTH CASES
• Not just ‘feeling lucky’ !!
• COMMANDS RUN :
• Ispcc lucky.c [ later try unlucky.c ]
• Isp -n 3 -log /tmp/log1 ./a.out
• ispUI /tmp/log1
Commands to verify lucky.c or unlucky.c
![Page 50: http:// / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408,](https://reader036.fdocuments.us/reader036/viewer/2022062521/5681692b550346895de06cd6/html5/thumbnails/50.jpg)
End of A
50