Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance...
Transcript of Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance...
![Page 1: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/1.jpg)
Parallel System Performance
Jan 6, 2020
![Page 2: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/2.jpg)
Course outline (Pacheco; GGKK; Quinn)
● Motivation (1;1;1)● How to quantify performance improvement (2.6; 5; 7)● Parallel hardware architecture (2.2-2.3; 2,4; 2)● Parallel programming frameworks
○ Pthreads for shared memory (4; 7; -)○ OpenMP for shared memory (5; 7.10; 17)○ MPI for distributed memory (3; 6; 4)○ CUDA/OpenCL for GPU, ○ Hadoop/Spark/Mapreduce for distributed systems
● Parallel program verification● Parallel algorithm design● Some case studies
![Page 3: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/3.jpg)
Why is performance analysis important?● Being able to accurately predict the performance of a parallel algorithm
○ can help decide whether to actually go to the trouble of coding and debugging it.
● Being able to analyze the execution time exhibited by a parallel program○ Can help understand barriers to higher performance○ Can help predict how much improvement can be realized by increasing number of processors
![Page 4: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/4.jpg)
Well-known performance prediction formulas● Amdahl’s Law
○ Help decide whether a program merits parallelization
● Gustafson-Barsi’s Law○ Way to evaluate performance of a parallel program
● Karp-Flatt metric○ Decide whether the principal barrier to speedup is the amount of inherently sequential code or
parallel overhead
● Iso-efficiency metric○ Way to evaluate the scalability of a parallel algorithm executing on a parallel computer. Help
choose the design that will achieve higher performance when the number of processors increase.
![Page 5: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/5.jpg)
Operations performed by a parallel algorithm● Computations that must be performed sequentially σ(n)● Computations that can be performed in parallel φ(n)● Parallel overhead (communication operations and redundant computations) κ(n,p)
![Page 6: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/6.jpg)
Operations performed by a parallel algorithm● Computations that must be performed sequentially σ(n)● Computations that can be performed in parallel φ(n)● Parallel overhead (communication operations and redundant computations) κ(n,p)
![Page 7: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/7.jpg)
Speedup and efficiencyWe design and implement parallel programs in the hope that they will run faster than their sequential counterparts.
● Speedup = (Sequential execution time)/(Parallel execution time)
● Efficiency = Speedup/(Processors used)
![Page 8: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/8.jpg)
Speedup and efficiencyWe design and implement parallel programs in the hope that they will run faster than their sequential counterparts.
● Speedup = (Sequential execution time)/(Parallel execution time)
● Efficiency = Speedup/(Processors used)
![Page 9: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/9.jpg)
Amdahl’s Law
![Page 10: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/10.jpg)
Amdahl’s Law
![Page 11: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/11.jpg)
Numerical Examples
![Page 12: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/12.jpg)
Numerical Examples
![Page 13: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/13.jpg)
Limitations of Amdahl’s Law
![Page 14: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/14.jpg)
Limitations of Amdahl’s Law
![Page 15: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/15.jpg)
Amdahl Effect
![Page 16: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/16.jpg)
Gustafson-Barsi’s Law
![Page 17: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/17.jpg)
Gustafson-Barsi’s Law
![Page 18: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/18.jpg)
Numerical Examples
![Page 19: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/19.jpg)
The Karp-Flatt Metric
![Page 20: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/20.jpg)
The Karp-Flatt Metric
![Page 21: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/21.jpg)
Numerical Examples
![Page 22: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/22.jpg)
Numerical Examples
![Page 23: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/23.jpg)
Iso-efficiency metric
![Page 24: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/24.jpg)
Iso-efficiency metric
![Page 25: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/25.jpg)
Iso-efficiency metric
![Page 26: Parallel System Performancerijurekha/col380/lec3.pdf · 2020-01-07 · Well-known performance prediction formulas Amdahl’s Law Help decide whether a program merits parallelization](https://reader033.fdocuments.us/reader033/viewer/2022042119/5e992d8807d27903ee23c1a4/html5/thumbnails/26.jpg)
Course outline (Pacheco; GGKK; Quinn)
● Motivation (1;1;1)● How to quantify performance improvement (2.6; 5; 7)● Parallel hardware architecture (2.2-2.3; 2,4; 2)● Parallel programming frameworks
○ Pthreads for shared memory (4; 7; -)○ OpenMP for shared memory (5; 7.10; 17)○ MPI for distributed memory (3; 6; 4)○ CUDA/OpenCL for GPU, ○ Hadoop/Spark/Mapreduce for distributed systems
● Parallel program verification● Parallel algorithm design● Some case studies