Distributed Computing with MPI - McGill...
Transcript of Distributed Computing with MPI - McGill...
![Page 1: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/1.jpg)
Distributed Computing with MPI
Sean Lawlor
Computer Networks GroupDepartment of Electrical and Computer Engineering
McGill University Bellairs Institute, Barbados
March 14, 2012
Sean Lawlor MPI-Cluster Computing 1/ 23
![Page 2: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/2.jpg)
Outline
Introduction
MotivationMPI-1 and MPI-2 statuses
Machine Learning with MPI
Deadlock and Knot detection/removal
Determining Computation End
Conclusion
Sean Lawlor MPI-Cluster Computing 2/ 23
![Page 3: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/3.jpg)
Motivation
Why would we want to parallelize applications?
What benefit would pushing to a cluster give?
What options exist for this kind of work?
How do we determine computation end?
Sean Lawlor MPI-Cluster Computing 3/ 23
![Page 4: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/4.jpg)
Motivation
Why would we want to parallelize applications?
What benefit would pushing to a cluster give?
What options exist for this kind of work?
How do we determine computation end?
Sean Lawlor MPI-Cluster Computing 3/ 23
![Page 5: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/5.jpg)
Motivation
Why would we want to parallelize applications?
What benefit would pushing to a cluster give?
What options exist for this kind of work?
How do we determine computation end?
Sean Lawlor MPI-Cluster Computing 3/ 23
![Page 6: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/6.jpg)
Motivation
Why would we want to parallelize applications?
What benefit would pushing to a cluster give?
What options exist for this kind of work?
How do we determine computation end?
Sean Lawlor MPI-Cluster Computing 3/ 23
![Page 7: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/7.jpg)
Cluster Computing
Figure: Example of computer cluster [Source: Boise State CS Dept,Beowulf Cluster Lab(http://cs.boisestate.edu/∼amit/research/beowulf/)]
Sean Lawlor MPI-Cluster Computing 4/ 23
![Page 8: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/8.jpg)
MPI-1
MPI is a message passing interface library standard
Is a specification, not an implementationLibrary, not a languageBased on the classical message passing programming modelBarriersMulti-Party messages
1:n → broadcastn:1 → reduce (Hadoop)n:n → all-to-all (1-step consensus)
Sean Lawlor MPI-Cluster Computing 5/ 23
![Page 9: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/9.jpg)
MPI-1
MPI was defined in 1994 1 by a broadly-based group ofparallel computer vendors, computer scientists, andapplication developers.
Implementations and support grew quickly
Basis for cluster environments with free and openimplementations (OpenMPI, MPICH2, ...)
1I. Foster, Designing and Building Parallel Programs. Addison-Wesley,1994, online Ch. 8 - Message Passing Interface
Sean Lawlor MPI-Cluster Computing 6/ 23
![Page 10: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/10.jpg)
MPI-2
Same as MPI-1 with extended functionality
Extends message passing model
Parallel I/ORemote memory operations (not covered here)Dynamic process management
Adds bindings to include C++ and Fortran-90
Interaction with Threads
Sean Lawlor MPI-Cluster Computing 7/ 23
![Page 11: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/11.jpg)
MPI-2 Status
Most parallel systems implement some implementation ofMPI-2
Cluster MPIs, such as MPICH2 and LAM, support most ofMPI-2 including dynamic process management
We’ll refer mostly to the OpenMPI implementation
Sean Lawlor MPI-Cluster Computing 8/ 23
![Page 12: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/12.jpg)
Problem Definition
Problem Types solved by MPI
Large Dataset ProcessingDecreased Computation Time (if possible)Redundancy for Distributed Systems
Example System Definition
Deepthought: (Beowulf Cluster)15 NodesEach node has a dual-core Pentium IV processor1 GB of RAM1 (non-included node) as coordinator
Sean Lawlor MPI-Cluster Computing 9/ 23
![Page 13: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/13.jpg)
Asynchronous vs. Synchronous
What’s the difference?
Can asynchronous send and receives be done?
What’s the best option?
Sean Lawlor MPI-Cluster Computing 10/ 23
![Page 14: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/14.jpg)
Asynchronous vs. Synchronous
What’s the difference?
Can asynchronous send and receives be done?
What’s the best option?
Sean Lawlor MPI-Cluster Computing 10/ 23
![Page 15: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/15.jpg)
Asynchronous vs. Synchronous
What’s the difference?
Can asynchronous send and receives be done?
What’s the best option?
Sean Lawlor MPI-Cluster Computing 10/ 23
![Page 16: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/16.jpg)
Machine Learning
MPI Uses for machine learning applications
Used to handle massive datasets
Many programs are parallelizable
Tailored applications for problems
Example:
MNIST Dataset15 Node plot of error vs 2 Node calculationSynchronous
Sean Lawlor MPI-Cluster Computing 11/ 23
![Page 17: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/17.jpg)
Machine Learning
MPI Uses for machine learning applications
Used to handle massive datasets
Many programs are parallelizable
Tailored applications for problems
Example:
MNIST Dataset15 Node plot of error vs 2 Node calculationSynchronous
Sean Lawlor MPI-Cluster Computing 11/ 23
![Page 18: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/18.jpg)
Machine Learning
MPI Uses for machine learning applications
Used to handle massive datasets
Many programs are parallelizable
Tailored applications for problems
Example:
MNIST Dataset15 Node plot of error vs 2 Node calculationSynchronous
Sean Lawlor MPI-Cluster Computing 11/ 23
![Page 19: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/19.jpg)
Machine Learning
MPI Uses for machine learning applications
Used to handle massive datasets
Many programs are parallelizable
Tailored applications for problems
Example:
MNIST Dataset
15 Node plot of error vs 2 Node calculationSynchronous
Sean Lawlor MPI-Cluster Computing 11/ 23
![Page 20: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/20.jpg)
Machine Learning
MPI Uses for machine learning applications
Used to handle massive datasets
Many programs are parallelizable
Tailored applications for problems
Example:
MNIST Dataset15 Node plot of error vs 2 Node calculation
Synchronous
Sean Lawlor MPI-Cluster Computing 11/ 23
![Page 21: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/21.jpg)
Machine Learning
MPI Uses for machine learning applications
Used to handle massive datasets
Many programs are parallelizable
Tailored applications for problems
Example:
MNIST Dataset15 Node plot of error vs 2 Node calculationSynchronous
Sean Lawlor MPI-Cluster Computing 11/ 23
![Page 22: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/22.jpg)
Machine Learning
MPI Uses for machine learning applications
Used to handle massive datasets
Many programs are parallelizable
Tailored applications for problems
Example:
MNIST Dataset15 Node plot of error vs 2 Node calculationSynchronous
Sean Lawlor MPI-Cluster Computing 11/ 23
![Page 23: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/23.jpg)
Logistic Regression Plot
Figure: Results of parallel Logistic Regression(x-axis denotes time andy-axis is error on test set)
Sean Lawlor MPI-Cluster Computing 12/ 23
![Page 24: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/24.jpg)
Deadlock Detection
Many works into this (Example2)
Determine if directed cyclic graph exists in network
START
Figure: Example of cyclic graph in network
2K. M. Chandy, J. Misra, and L. M. Haas, “Distributed deadlock detection,”ACM Trans. Comput. Syst., vol. 1, no. 2, pp. 144–156, May 1983
Sean Lawlor MPI-Cluster Computing 13/ 23
![Page 25: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/25.jpg)
Deadlock Removal
A deadlock has been detected, now what?
Need to determine which node should give up computation
Problem Specific (Eg. Will you destroy everything or justsome things by killing a specific node?)
Sean Lawlor MPI-Cluster Computing 14/ 23
![Page 26: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/26.jpg)
Deadlock Removal
A deadlock has been detected, now what?
Need to determine which node should give up computation
Problem Specific (Eg. Will you destroy everything or justsome things by killing a specific node?)
Sean Lawlor MPI-Cluster Computing 14/ 23
![Page 27: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/27.jpg)
Deadlock Removal
A deadlock has been detected, now what?
Need to determine which node should give up computation
Problem Specific (Eg. Will you destroy everything or justsome things by killing a specific node?)
Sean Lawlor MPI-Cluster Computing 14/ 23
![Page 28: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/28.jpg)
Knot Detection
Knots are a special case of deadlocks3
Cause larger problemsKilling any single cycle may not kill all cyclesWhat should we do here?
Figure: Example of network graph with knot
3J. Misra and K. M. Chandy, “A distributed graph algorithm: Knotdetection,” ACM Trans. Program. Lang. Syst., vol. 4, no. 4, pp. 678–686, Oct.1982
Sean Lawlor MPI-Cluster Computing 15/ 23
![Page 29: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/29.jpg)
Multiple Knots
Figure: Graph with multiple (> 2) distinct cycles, forming multiple knots
Sean Lawlor MPI-Cluster Computing 16/ 23
![Page 30: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/30.jpg)
Determining Computation End
Again problem specific
However there are many standard ways depending on theproblem
Variations:
Timelimit reachedGoal achievedNo more messages to send
Focus on no more messages situation
Upon end, global solution must be attained
Sean Lawlor MPI-Cluster Computing 17/ 23
![Page 31: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/31.jpg)
Determining Computation End
Again problem specific
However there are many standard ways depending on theproblem
Variations:
Timelimit reachedGoal achievedNo more messages to send
Focus on no more messages situation
Upon end, global solution must be attained
Sean Lawlor MPI-Cluster Computing 17/ 23
![Page 32: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/32.jpg)
Determining Computation End
Again problem specific
However there are many standard ways depending on theproblem
Variations:
Timelimit reached
Goal achievedNo more messages to send
Focus on no more messages situation
Upon end, global solution must be attained
Sean Lawlor MPI-Cluster Computing 17/ 23
![Page 33: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/33.jpg)
Determining Computation End
Again problem specific
However there are many standard ways depending on theproblem
Variations:
Timelimit reachedGoal achieved
No more messages to send
Focus on no more messages situation
Upon end, global solution must be attained
Sean Lawlor MPI-Cluster Computing 17/ 23
![Page 34: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/34.jpg)
Determining Computation End
Again problem specific
However there are many standard ways depending on theproblem
Variations:
Timelimit reachedGoal achievedNo more messages to send
Focus on no more messages situation
Upon end, global solution must be attained
Sean Lawlor MPI-Cluster Computing 17/ 23
![Page 35: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/35.jpg)
Determining Computation End
Again problem specific
However there are many standard ways depending on theproblem
Variations:
Timelimit reachedGoal achievedNo more messages to send
Focus on no more messages situation
Upon end, global solution must be attained
Sean Lawlor MPI-Cluster Computing 17/ 23
![Page 36: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/36.jpg)
Determining Computation End
Again problem specific
However there are many standard ways depending on theproblem
Variations:
Timelimit reachedGoal achievedNo more messages to send
Focus on no more messages situation
Upon end, global solution must be attained
Sean Lawlor MPI-Cluster Computing 17/ 23
![Page 37: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/37.jpg)
Determining Computation End
Again problem specific
However there are many standard ways depending on theproblem
Variations:
Timelimit reachedGoal achievedNo more messages to send
Focus on no more messages situation
Upon end, global solution must be attained
Sean Lawlor MPI-Cluster Computing 17/ 23
![Page 38: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/38.jpg)
Naive Computation End Algorithm
Setup:
Ring network
Single marker used at a time4
White determines node is idle
Black means node is currently responding to a message (i.e.computing)
Marker colors node white when it leaves that node if idle
If marker sees a white node, assumed white since last iteration
4J. Misra, “Detecting termination of distributed computations usingmarkers,” in Proceedings of the second annual ACM symposium on Principlesof distributed computing, ser. PODC ’83. New York, NY, USA: ACM, 1983,pp. 290–294
Sean Lawlor MPI-Cluster Computing 18/ 23
![Page 39: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/39.jpg)
Example Network Graph
Figure: Example ring network graph
Sean Lawlor MPI-Cluster Computing 19/ 23
![Page 40: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/40.jpg)
Naive Computation End Algorithm
Algorithm:
INIT: ∀nodes (n) ∈ N : color black
ni sends initial marker m along outgoing edge
Node nj , where nj is the child of ni forwards message withfollowing properties:
0, if nj is blackm + 1, if nj is whiteMarker paints node whiteIf node receives message, paints itself black
Termination is achieved when m = N
Collect data at gate, and return
Sean Lawlor MPI-Cluster Computing 20/ 23
![Page 41: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/41.jpg)
Algorithm by example
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 42: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/42.jpg)
Algorithm by example
0
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 43: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/43.jpg)
Algorithm by example
0
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 44: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/44.jpg)
Algorithm by example
0
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 45: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/45.jpg)
Algorithm by example
0
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 46: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/46.jpg)
Algorithm by example
0
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 47: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/47.jpg)
Algorithm by example
0
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 48: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/48.jpg)
Algorithm by example
0
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 49: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/49.jpg)
Algorithm by example
0
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 50: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/50.jpg)
Algorithm by example
1
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 51: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/51.jpg)
Algorithm by example
2
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 52: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/52.jpg)
Algorithm by example
3
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 53: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/53.jpg)
Algorithm by example
4
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 54: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/54.jpg)
Algorithm by example
0
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 55: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/55.jpg)
Algorithm by example
1
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 56: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/56.jpg)
Algorithm by example
2
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 57: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/57.jpg)
Algorithm by example
3
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 58: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/58.jpg)
Algorithm by example
1
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 59: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/59.jpg)
Algorithm by example
2
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 60: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/60.jpg)
Algorithm by example
3
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 61: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/61.jpg)
Algorithm by example
4
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 62: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/62.jpg)
Algorithm by example
5
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 63: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/63.jpg)
Algorithm by example
6
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 64: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/64.jpg)
Algorithm by example
7
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 65: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/65.jpg)
Algorithm by example
8
Sean Lawlor MPI-Cluster Computing 21/ 23
![Page 66: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/66.jpg)
Conclusion
Definition of MPI 1 and 2
How MPI can be leveraged for distributed computation
Determining cyclic graphs and avoiding them
Determining computation end on the fly
Sean Lawlor MPI-Cluster Computing 22/ 23
![Page 67: Distributed Computing with MPI - McGill Universitynetworks.ece.mcgill.ca/sites/default/files/Lawlor_Mpi_Cluter_Bellairs_2012.pdf · Distributed Computing with MPI Sean Lawlor Computer](https://reader030.fdocuments.us/reader030/viewer/2022040820/5e68ba88d892fb20d10e7fbe/html5/thumbnails/67.jpg)
Thank you
Thank you!Questions?
Sean Lawlor MPI-Cluster Computing 23/ 23