High Performance Computing: Concepts, Methods & Means Performance 3 : Measurement

High Performance Computing: Concepts, Methods & Means

Performance 3 : Measurement

Prof. Thomas SterlingDepartment of Computer Science

Louisiana State University

February 27th, 2007

Term Projects

• Graduate students only• Due date: April 19th, 2007• Total time: approx. 40 hours• 20% of final grade• 4 categories of projects:

– A) technology evolution report (20 page report)– B) Fixed application code execution scaling (7 page report)– C) Synthetic code parametric studies (7 page report)– D) Parallel application development (7 page report)

• 1 paragraph abstract due March 9th

– Email to Chirag by COB Friday

Term Projects – Technology Evolution

• In depth survey of an enabling technology• Report on capability with respect to time and factors• Two general classes of technology:

– Device technology• Main memory

• Secondary storage

• System network

• Logic

– Architecture• SIMD

• Vector

• Systolic

• Dataflow

Term Project – Fixed Application Scaling• Select an application code

– Need not be one of those in class– Must be a parallel code– You need not write this yourself

• Select two or more system parameters to scale with– # processors– # nodes– Network bandwidth and/or latency– Data block partition size

• Use performance measurement and profiling tools– Describe measured trends– Diagnose reasons for observed results

Term Project – Synthetic Parametric Study

• Write a code expressly to exercise one or more system functions– Parallelism– Network bandwidth– Memory bandwidth

• Allow at least one dimension to be independent and adjust– Message insert rate– Message packet size– Overhead time

• Show system operation with respect to parameter

Term Project – Roll your own

• Write a small parallel application program– Preferably not one we’ve done in class– Can be something you’ve done in another class or research

project modified for MPI or OpenMP– Please! Do this yourself!!– Libraries permitted

• Use profiling tools to determine where most of the work is being done

• Demonstrate scaling wrt # processors

7

Topics

• Introduction• Performance Characteristics & Models• Performance Models : LogP • Performance Models : LogGP• Benchmarks : b_eff• MPI Tracing : PMPI• TAU & MPI• Summary – Materials for Test

8

Topics


9

Please understand when to use the following and what they mean :• API Elements :

– MPI_Init(), MPI_Finalize()– MPI_Comm_size(), MPI_Comm_rank() – MPI_COMM_WORLD– Error checking using MPI_SUCCESS– MPI basic data types (slide 27)– Blocking : MPI_Send(), MPI_Recv()– Non-Blocking : MPI_Isend(), MPI_Irecv(), MPI_Wait()– Collective Calls : MPI_Barrier(), MPI_Bcast(), MPI_Gather(),

MPI_Scatter(), MPI_Reduce()

• Commands : – Running MPI Programs : mpirun– Compile : mpicc – Compile : mpif77

10

Where are we?

• Three classes of parallel computing– Capacity– Cooperative– Capability

• Three execution models– Throughput – Shared memory multithreaded– Communicating sequential processes (message passing)

• Three programming formalisms– Condor– OpenMP– MPI

• More performance modeling and measurement– For cooperative/message passing/MPI

11

Topics

• Introduction

• Performance Characteristics & Models• Performance Models : LogP • Performance Models : LogGP• Benchmarks : b_eff• MPI Tracing : PMPI• TAU & MPI• Summary – Materials for Test

12

What has changed? SMP to MPP

• SMP – symmetric multiprocessor– Shared memory

• UMA – uniform memory access with cache coherence

– Multithreaded parallelism– Communication through main memory– Not scalable– Programming in OpenMP– DSM and PGAS provide alternative shared memory structures

• DSM – distributed shared memory (with cache coherence)• PGAS – Partitioned global address space (without cache coherence)• Both are NUMA

• MPP – massively parallel processor– Distributed memory

• NUMA – non-uniform memory access

– Concurrent sequential processes parallelism– Communication through messages between nodes– Scalable– Programming in MPI– Same for commodity clusters but usually with weaker networks

13

MPI Performance Characteristics

• Latency– Time to send first bits of data across link to remote node– Does not include overhead

• Bandwidth– Rate of data transfer across link to remote node

• Buffers– System or user buffers take up time to manage capacity etc.

• Blocking versus Asynchronous– Forced ordering of computation and communication

14

Granularities of Time Measurements

Timer Usage Wallclock / CPU Time

Resolution Languages Portable?

time shell / script both 1/100th second any yes

timex shell / script both 1/100th second any yes

gettimeofday subroutine wallclock microsecond C/C++ yes

read_real_time subroutine wallclock nanosecond C/C++ no

rtc subroutine wallclock microsecond Fortran no

irtc subroutine wallclock nanosecond Fortran no

dtime_ subroutine CPU 1/100th second Fortran no

etime_ subroutine CPU 1/100th second Fortran no

mclock subroutine CPU 1/100th second Fortran no

timef subroutine wallclock millisecond Fortran no

MPI_Wtime subroutine wallclock microsecond C/C++, Fortran yes

AIX Trace Facility shell / script / subroutine

wallclock microsecond any no

time

15

Performance Factors

• Platform / Architecture Related: – cpu - clock speed, number of cpus – Memory subsystem - memory and cache configuration,

memory-cache-cpu bandwidth, memory copy bandwidth – Network adapters - type, latency and bandwidth

characteristics – Operating system characteristics - many

• Network Related: – Hardware - ethernet, FDDI, switch, intermediate hardware

(routers) – Protocols - TCP/IP, UDP/IP, other – Configuration, routing, etc – Network tuning options ("no" command) – Network contention / saturation

source : http://www.llnl.gov/computing/tutorials/mpi_performance/

16

Performance Factors (2)• Application Related:

– Algorithm efficiency and scalability – Communication to computation ratios – Load balance – Memory usage patterns – I/O – Message size used – Types of MPI routines used - blocking, non-blocking, point-to-point,

collective communications

• MPI Implementation Related: – Message buffering – Message passing protocols - eager, rendezvous, other – Sender-Receiver synchronization - polling, interrupt – Routine internals - efficiency of algorithm used to implement a given

routine

source : http://www.llnl.gov/computing/tutorials/mpi_performance/

17

Performance Impact of Message Sizes

• Message size can be a very significant contributor to MPI application performance. In most cases, increasing the message size will yield better performance.

• For communication intensive applications, algorithm modifications that take advantage of message size "economies of scale" may be worth the effort. Performance can often improve significantly within a relatively small range of message sizes.

• The following three graphs demonstrate how increasing message size can improve bandwidth for different message size ranges

18

MPI Performance Models

• Hockney: Point to Point– Time to send: t=t0+m/rinf

– t0: fixed cost per message, startup cost

– m: message length

– rinf: bandwidth for very large messages

• Xu/Hwang: Collective– Time to send: t=t0 (n)+m/rinf(n)

– same parameters, but now they are functions of n, the number of nodes in the communication

• Source: http://wwwinfo.deis.unical.it/~talia/hpcn98.ps

19

Topics


20


• LogP (fixed message size)– time to send = L+2*o– L: Latency, min send/recv time– o: Overhead, time waiting on processor– g: Gap, min time between successive sends or recvs & does

include message length– P: Number of Processors– L/g: Max number of simultaneous messages

21

Measuring LogP Parameters

• Finding g (implementation dependent)

– Proc 0: MPI_ISend() x N

– Proc 1: MPI_Recv() x N

– g = total time / N• Finding L+2*o

– Proc 0: (MPI_Send() then MPI_Recv()) x N

– Proc 1: (MPI_Recv() then MPI_Send()) x N

– L+2*o = total time/N• Finding o

– Proc 0: (MPI_Send() then MPI_Recv() then some_work) x N

– Proc 1: (MPI_Recv() then some_work then MPI_Send()) x N

– o = (1/2)total time/N – time(some_work)

– requires time(some_work) > 2*L+2*o

22


• Finding L+2*o

– Proc 0: (MPI_Send() then MPI_Recv()) x N

– Proc 1: (MPI_Recv() then MPI_Send()) x N

– L+2*o = total time/N

Figure 1: Time diagram for benchmark 1(a) is Time diagram of processor 0(b) is Time diagram of processor 1

23


• Finding o

– Proc 0: (MPI_Send() then some_work then MPI_Recv() ) x N

– Proc 1: (MPI_Recv() then MPI_Send() then some_work) x N

– o = (1/2)total time/N – time(some_work)

– requires time(some_work) > 2*L+2*o

Figure 2: Time diagram for benchmark 2 with X > 2*L+Or+Os(a) is Time diagram of processor 1(b) is Time diagram of processor 2

24

Demo

• Measure LogP parameters

25

Topics


26


• LogGP (variable message size)– time to send = L+2*o+(m-1)*G– L: Latency, min send/recv time– o: Overhead, time waiting on processor– g: Gap, min time between send/recvs– G: Gap per byte = 1/Bandwidth– P: Number of Processors– L/g: Max number of simultaneous messages– http://citeseer.ist.psu.edu/cache/papers/cs/756/

http:zSzzSzwww.cs.berkeley.eduzSz~cullerzSzpaperszSzsort.pdf/dusseau96fast.pdf

Effective Bandwidth (LogGP)

Toy Calculation– BW = m/(L+2*o+G(m-1))– let: L+2*o-G = 5– let: G = 3– Asymptotically approaches

bandwidth of 1/G for very large messages.

28

Topics


29

HPC Challenge Benchmarks

• HPC Challenge: http://icl.cs.utk.edu/hpcc/– See results tab– b_eff benchmark is a part of this larger database– more info than just HPL!

30

b_eff

• Standard Benchmark – part of HPC Challenge– Provides effective bandwidth and latency

• Averages a variety of message sizes and communication patterns

• Determines an effective latency and bandwidth• b_eff depends on:

– hardware: interconnect, memory– software: MPI implementation– tuneable parameters of the os: buffers– etc.

See : http://www.hlrs.de/organization/par/services/models/mpi/b_eff/

31

Effective Bandwidth Benchmark

32

Example: Send/Recv, ring & random

33

Demo

• running of b_eff

34

Topics


35

Portable MPI Tracing: PMPI

• An API to MPI for tracing, debugging, performance measurements of MPI applications

• MPI_<command>() calls PMPI_<command>()• MPI_Pcontrol(int)

– 0: disabled– 1: enabled – Default Level– 2: flush trace buffers

36

Demo : MPI_Pcontrol…

int sends = 0;

int pcontrol = 1;

int main(int argc,char **argv) {

MPI_Init(&argc,&argv);

int imax = 10000000;

int nmax = 8;

int rank;

int data = 27;

MPI_Status st;

MPI_Comm_rank(MPI_COMM_WORLD,&rank);

time_t start,end;

double fac = (1.0/imax);

double g,lp2o,o;

// Find g

time(&start);

for(int i=0;i<imax;i++) {

if(rank == 0) {

MPI_Send(&data,1,MPI_INT,1,1,MPI_COMM_WORLD);

} else {

MPI_Recv(&data,1,MPI_INT,0,1,MPI_COMM_WORLD,&st);

}

}

time(&end);

if(rank == 0) {

g = fac*(end-start);

printf("gap=%g sec\n",g);

}

// Find L+2*o

time(&start);

const int step = 5;

for(int i=0;i<imax;i+=step) {

if(rank == 0) {



} else {



}

}

time(&end);

if(rank == 0) {

lp2o = 0.5*step*fac*(end-start);

printf("L+2*o=%g sec\n",lp2o);

if(sends > 0) printf("sends = %d\n",sends);

}

MPI_Finalize();

return 0;

}

int MPI_Pcontrol(int n) {

pcontrol = n;

return PMPI_Pcontrol(n);

}

int MPI_Send( void *buf, int count, MPI_Datatype datatype, int dest,

int tag, MPI_Comm comm ) {

if(pcontrol >= 1) sends++;

return PMPI_Send(buf,count,datatype,dest,tag,comm );

}

37

Demo

• MPI tracing, custom implementation

38

Topics


39

TAU and MPI

• Tau uses the PMPI interface to track MPI calls• Jumpshot is used as the viewer

– Shows subroutine calls and mpi calls

40

TAU Performance System Architecture

EPILOG

Paraver

41

TAU Measurement Options

• Parallel profiling– Function-level, block-level, statement-level– Supports user-defined events– TAU parallel profile data stored during execution– Hardware counts values– Support for multiple counters– Support for callpath profiling

• Tracing– All profile-level events– Inter-process communication events– Timestamp synchronization– Trace merging and format conversion

42

How To Use TAU?

• Instrumentation– Application code and libraries– Selective instrumentation

• Install, compile, and link with TAU measurement library– % configure; make clean install– Multiple configurations for different measurements options– Does not require change in instrumentation– Selective measurement control

• Execute “experiments” to produce performance data– Performance data generated at end or during execution

• Use analysis tools to look at performance results

43

Using Tau

• Setup Environment:– source /home/packages/Tau/gcc-papi-mpi-slog2/env.sh– export COUNTER1=GET_TIME_OF_DAY

• Use tau_cc.sh, tau_f90.sh, etc. to compile• Run with mpirun• Post-process:

– tau_treemerge.pl– tau2slog2 tau.trc tau.edf -o tau.slog2

• Run: http://www.cct.lsu.edu/~sbrandt/perf_vis.html

44

Demo

• Tau and Jumpshot

45

Topics


46

Summary – Material for the Test

• Essential MPI - Slide: 9• Performance Models - Slide: 12, 15, 16, 18 (Hockney)• LogP - Slide: 20 – 23• Effective Bandwidth – Slide: 30• Tau/MPI – Slide: 41, 43

47

Sources

• http://www.cs.uoregon.edu/research/tau/docs.php (tau)• http://www.llnl.gov/computing/tutorials/mpi_performance/• http://www.netlib.org/utk/papers/mpi-book/node182.html (mpi profiling interface)• http://www-unix.mcs.anl.gov/mpi/tutorial/perf/index.html (Gropp course)• http://www.ecs.umass.edu/ece/ssa/papers/jpdcmpi.ps (LogP paper with figures)• http://www.netlib.org/utk/people/JackDongarra/PAPERS/coll-perf-analysis-

cluster-2005.pdf (more LogP stuff)• http://www.hlrs.de/organization/par/services/models/mpi/b_eff/ (b_eff bench)• http://icl.cs.utk.edu/hpcc/ (hpc challenge)

High Performance Computing: Concepts, Methods & Means Performance 3 : Measurement

Documents

Transcript of High Performance Computing: Concepts, Methods & Means Performance 3 : Measurement