Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns -...
Transcript of Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns -...
![Page 1: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/1.jpg)
Parallel Programming Patterns
Eun-Gyu Kim June 10, 2004
![Page 2: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/2.jpg)
Overview
• Definition • Algorithm Patterns Example • Problems / Weakness • More Examples • Directions
![Page 3: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/3.jpg)
Defining Patterns
• Design Patterns - Quality description of problem and solution to a frequently occurring problem in some domain. – (object oriented programming, network apps…) – Description usually involves problem definition, dr
iving forces, solution, benefits, difficulties, related patterns.
• Pattern Language - a collection of design patterns, guiding the users through the decision process in building a system.
![Page 4: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/4.jpg)
Definition - we get hungry every lunch hour. - usually around 11:30am – 1:00pm. - must resolve hunger. Driving Forces - appetite, intensity of hunger, nearby restaurants Solution - eat at nearest restaurant that satisfies minimum appetite requirement. Benefits - hunger is resolved Difficulties - must not fall asleep afterwards. Related Patterns - dinner, breakfast, brunch, and snack
Example Design Space “Lunch” Pattern “Eat” Pattern Language
Wake-Up
Breakfast
Lunch
Dinner
Brunch
![Page 5: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/5.jpg)
Parallel Programming Pattern : WHY?
• Parallel software does not fully exploit parallel hardware. – Too difficult for programmers – Parallel Programming Environments do not focus on design
issues. • Need a “cookbook” that will guide the programmers
systematically to achieve peak parallel performance. – (decomposition, algorithm, program structure, programmin
g environment, optimizations) • Provide common vocabulary to the programming co
mmunity. • Software reusability & modularity.
![Page 6: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/6.jpg)
Some approaches
• Foster Methodology – Decomposition – Communication – Agglomeration – Mapping
• Massingil – FindingConcurrency
• data vs. control
– AlgorithmStructure • pipeline, replicate…
– SupportingStructure • SPMD, fork/join…
– Implementation • barriers, locks…
![Page 7: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/7.jpg)
Common Parallel Patterns
• Embarassingly Parallel • Replicable • Repository • Divide&Conquer • Pipeline • Recursive Data • Geometric • IrregularMesh • Inseparable
![Page 8: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/8.jpg)
Embarassingly Parallel
A B D E
Independent Tasks
Problem: Need to perform same operations to tasks that are independent
C
A B
C
E D
manager
worker worker worker worker
![Page 9: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/9.jpg)
Replicable
Global Data
Sets of operations need to be performed using global data structure, causing dependency.
local local local
replicate
Operation I Operation II Operation III
Solution
reduce
![Page 10: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/10.jpg)
Repository Independent computations needs to applied to centralized data structure in non-deterministic way.
repository
Compute A
Compute B
Compute E
Compute D
Compute C
Asynchronous Function calls
Compute object cannot access Same element simultaneously. (Repository controls access)
![Page 11: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/11.jpg)
Divide & Conquer
problem
A problem is structured to be solved in sub-problems independently, and merging them later.
subproblem subproblem
Compute subproblem
Compute subproblem
Compute subproblem
Compute subproblem
subproblem subproblem
solution
merge merge
merge
split split
split
* Split level needs to be adjusted appropriately.
![Page 12: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/12.jpg)
Pipeline
Stage 1
Stage 2
Stage 3
Stage 4
Time
C1 C2 C3 C4 C5 C6
C1 C2 C3 C4 C5 C6
C1 C2 C3 C4 C5 C6
C1 C2 C3 C4 C5 C6
A series of ordered but independent computation stages need to be applied on data, where each output of a computation becomes input of subsequent computation.
![Page 13: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/13.jpg)
Recursive Data Recursive data structures seem to have little exploitable concurrency. But in some cases, the structure can be transformed.
4
3
2
1 6
5 7
4
3
2
1 6
5 7
4
3
2
1 6
5 7
Step 1
Step 2
Step 3
Find Root Problem
![Page 14: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/14.jpg)
Geometric Dependencies exist, but communicate in predictable (geometric) neighbor-to-neighbor paths.
Neighbor-To-Neighbor communication
![Page 15: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/15.jpg)
Irregular Mesh Communication in non-predictable paths in mesh topology.
Hard to define due to varying communication patterns. Start point : Pattern that constructed this mesh.
![Page 16: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/16.jpg)
Inseparable
• All other parallel patterns fails, and need explicit protection when dependent data elements are accessed. – i.e. mutual exclusion, producer-consumer.
• Very vague definition.
![Page 17: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/17.jpg)
They all starts from
• Decomposing a sequential problem to expose concurrency.
• Data decomposition – Concurrency comes from working on different
data elements simultaneously. • Functional decomposition
– Concurrency comes from working on independent functional tasks simultaneously.
![Page 18: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/18.jpg)
It’s Parallel Programming 101
• Why can’t we advance from here? • Almost all of the patterns discussed are either
intuitive, or covered in introductory courses. • Do these patterns capture most of parallel
programs today? • Can these patterns be used to exploit full
parallelism?
![Page 19: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/19.jpg)
Why so trivial?
• Only two ways to decompose: – Data and Functional.
• Over-simplified just for – Intuitiveness & “natural representation” – Fitting the problem to “imperative language”
• By defining functional task: – We have chosen to reuse a variable. – It’s a conscious choice, and brings new constraint. – Removing this does not change the problem definition.
![Page 20: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/20.jpg)
Same Effect In many cases, different decomposition leads to same computation.
= x Owner-Compute
= x Row-major Decomposition
Implies / Equivalent to
![Page 21: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/21.jpg)
Exposing Better Parallelism
• Loosen the “natural representation”. • Free the assumptions
– Variable reuse – Thinking in terms of imperative language
![Page 22: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/22.jpg)
Non-Trivial Parallelism I :Molecular Dynamics
Spatial Domain (of molecules)
Fast Fourier Transform K-Space Periodic Domain
Repeated data of molecules can be processed better in periodic domain. (but little bit harder to represent)
*Decomposition is done in two domains. Spatial + Periodic
![Page 23: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/23.jpg)
Non-Trivial Parallelism II : Climate Modeling
Spatial Domain (2D/3D representation of sphere)
Fast Fourier Transform
Spectral Domain
Physics equation called “Spectral Transform Method” needs spectral domain to work on.
![Page 24: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/24.jpg)
Direction
• Capture fundamental decomposition in – Systematic – Constraint-Free Ways.
• Document more non-trivial patterns • Map patterns to programming environments • Structural Patterns
![Page 25: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/25.jpg)
Any Questions?
![Page 26: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/26.jpg)
![Page 27: Parallel Programming Patternssnir.cs.illinois.edu/patterns/patterns.pdf · • Design Patterns - Quality description of proble m and solution to a frequently occurring proble m in](https://reader035.fdocuments.us/reader035/viewer/2022070708/5eae81e2652276674505f8ca/html5/thumbnails/27.jpg)