Art of Multiprocessor Programming1 This work is licensed under a Creative Commons Attribution-...

Art of Multiprocessor Programming 1

This work is licensed under a

Creative Commons Attribution-ShareAlike 2.5 License.

• You are free:– to Share — to copy, distribute and transmit the work – to Remix — to adapt the work

• Under the following conditions:– Attribution. You must attribute the work to “The Art of

Multiprocessor Programming” (but not in any way that suggests that the authors endorse you or your use of the work).

– Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, similar or a compatible license.

• For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to– http://creativecommons.org/licenses/by-sa/3.0/.

• Any of the above conditions can be waived if you get permission from the copyright holder.

• Nothing in this license impairs or restricts the author's moral rights.

http://creativecommons.org/licenses/by-sa/2.5/



Parallel and distributed

programmingAdam Piotrowski

Grzegorz Jabłoński

Lecture I

3

Parallel and distributed programming

lux.dmcs.pl/padp

Grading policy Lecture slides and material

Additional material

Course Overview

4

Introduction to multiprocessor programming Using OpenMP

Distributed programmingMPI, CORBA programming

Theoretical approche to multiprocessor

programmingFundamentals - Models, algorithms

Real-World programming - ArchitecturesTechniques

Books

5

The Art of Multiprocessor Programming

by Maurice Herlihy and Nir Shavit

Using OpenMPPortable Shared Memory Parallel

Programming

by Barbara Chapman, Gabriele Jost and Ruud van der Pas


Still on some of your desktops: The Uniprocesor

memory

cpu


In the Enterprise: The Shared Memory

Multiprocessor(SMP)

cache

BusBus

shared memory

cachecache


Traditional Scaling Process

User code

TraditionalUniprocessor

Speedup1.8x1.8x

7x7x

3.6x3.6x

Time: Moore’s law


Multicore Scaling Process

User code

Multicore

Speedup 1.8x1.8x

7x7x

3.6x3.6x

Unfortunately, not so simple…


Real-World Scaling Process

1.8x1.8x 2x2x 2.9x2.9x

User code

Multicore

Speedup

Parallelization and Synchronization require great care…

Art of Multiprocessor Programming

11

Sequential Computation

memory

object object

thread


12

Concurrent Computation

memory

object object

thre

ads


Asynchrony

• Sudden unpredictable delays– Cache misses (short)– Page faults (long)– Scheduling quantum used up (really

long)


Model Summary

• Multiple threads– Sometimes called processes

• Single shared memory• Objects live in memory• Unpredictable asynchronous

delays


Parallel Primality Testing

• Challenge– Print primes from 1 to 1010

• Given– Ten-processor multiprocessor– One thread per processor

• Goal– Get ten-fold speedup (or close)


Load Balancing

• Split the work evenly• Each thread tests range of 109

…

…109 10102·1091

P0 P1 P9


17

Procedure for Thread i

void primePrint { int i = ThreadID.get(); // IDs in {0..9} for (j = i*109+1, j<(i+1)*109; j++) { if (isPrime(j)) print(j); }}


Issues

• Higher ranges have fewer primes• Yet larger numbers harder to test• Thread workloads

– Uneven– Hard to predict

• Need dynamic load balancing


Issues

• Higher ranges have fewer primes• Yet larger numbers harder to test• Thread workloads

– Uneven– Hard to predict

• Need dynamic load balancingre

jected


20

17

18

19

Shared Counter

each thread takes a number


21


int counter = new Counter(1); void primePrint { long j = 0; while (j < 1010) { j = counter.getAndIncrement(); if (isPrime(j)) print(j); }}


22

Counter counter = new Counter(1); void primePrint { long j = 0; while (j < 1010) { j = counter.getAndIncrement(); if (isPrime(j)) print(j); }}


Shared counterobject


23



Stop when every value taken


24



Increment & return each new

value


25

Counter Implementation

public class Counter { private long value;

public long getAndIncrement() { return value++; }}


26

Counter Implementation


public long getAndIncrement() { return value++; }} OK for single thread,

not for concurrent threads


27

What It Means




28

What It Means



temp = value; value = value + 1; return temp;


29

time

Not so good…

Value… 1

read 1

read 1

write 2

read 2

write 3

write 2

2 3 2


30

Is this problem inherent?

If we could only glue reads and writes…

read

write read

write


31

Challenge


public long getAndIncrement() { temp = value; value = temp + 1; return temp; }}


32

Challenge



Make these steps atomic (indivisible)


33

Hardware Solution


public long getAndIncrement() { temp = value; value = temp + 1; return temp; }} ReadModifyWrite()

instruction


34

An Aside: Java™


public long getAndIncrement() { synchronized { temp = value; value = temp + 1; } return temp; }}


35

An Aside: Java™



Synchronized block


36

An Aside: Java™



Mutual Exclusion


Why do we care?

• We want as much of the code as possible to execute concurrently (in parallel)

• A larger sequential part implies reduced performance

• Amdahl’s law: this relation is not linear…


Amdahl’s Law

OldExecutionTimeNewExecutionTimeSpeedup=

…of computation given n CPUs instead of 1


Amdahl’s Law

p

pn

1

1Speedup=


Amdahl’s Law

p

pn

1

1Speedup=

Parallel fraction


Amdahl’s Law

p

pn

1

1Speedup=

Parallel fraction

Sequential fraction


Amdahl’s Law

p

pn

1

1Speedup=

Parallel fraction

Number of

processors

Sequential fraction


43

Example

• Ten processors• 60% concurrent, 40% sequential• How close to 10-fold speedup?


44

Example


106.0

6.01

1

Speedup=2.17=


45

Example



46

Example


108.0

8.01

1

Speedup=3.57=


47

Example



48

Example


109.0

9.01

1

Speedup=5.26=


49

Example



50

Example


1099.0

99.01

1

Speedup=9.17=


The Moral

• Making good use of our multiple processors (cores) means – Finding ways to effectively parallelize

our code• Minimize sequential parts• Reduce idle time in which threads wait

without


52

Implementing a Counter



Make these steps indivisible using

locks


53

Locks (Mutual Exclusion)

public interface Lock {

public void lock();

public void unlock();}


54



public void lock();


acquire lock


55



public void lock();


release lock

acquire lock


56

Using Locks

public class Counter { private long value; private Lock lock; public long getAndIncrement() { lock.lock(); try { int temp = value; value = value + 1; } finally { lock.unlock(); } return temp; }}


57

Using Locks


acquire Lock


58

Using Locks


Release lock(no matter what)


59

Using Locks


Critical section


Today: Revisit Mutual Exclusion

• Think of performance, not just correctness and progress

• Begin to understand how performance depends on our software properly utilizing the multiprocessor machine’s hardware

• And get to know a collection of locking algorithms…

(1)


What Should you do if you can’t get a lock?

• Keep trying– “spin” or “busy-wait”– Good if delays are short

• Give up the processor– Good if delays are long– Always good on uniprocessor

(1)


What Should you do if you can’t get a lock?

• Keep trying– “spin” or “busy-wait”– Good if delays are short

• Give up the processor– Good if delays are long– Always good on uniprocessor

our focus


Basic Spin-Lock

CS

Resets lock upon exit

spin lock

critical section

...


Basic Spin-Lock

CS

Resets lock upon exit

spin lock

critical section

...

…lock introduces sequential bottleneck


Review: Test-and-Set

• Boolean value• Test-and-set (TAS)

– Swap true with current value– Return value tells if prior value was

true or false

• Can reset just by writing false



public class AtomicBoolean { boolean value; public synchronized boolean getAndSet(boolean newValue) {

boolean prior = value; value = newValue; return prior; }}

(5)



public class AtomicBoolean { boolean value; public synchronized boolean getAndSet(boolean newValue) {

boolean prior = value; value = newValue; return prior; }}

Swap old and new values



AtomicBoolean lock = new AtomicBoolean(false)…boolean prior = lock.getAndSet(true)



AtomicBoolean lock = new AtomicBoolean(false)…boolean prior = lock.getAndSet(true)

(5)

Swapping in true is called “test-and-set” or TAS


Test-and-Set Locks

• Locking– Lock is free: value is false– Lock is taken: value is true

• Acquire lock by calling TAS– If result is false, you win– If result is true, you lose

• Release lock by writing false


Test-and-set Lock

class TASlock { AtomicBoolean state = new AtomicBoolean(false);

void lock() { while (state.getAndSet(true)) {} } void unlock() { state.set(false); }}


Test-and-set Lock



Lock state is AtomicBoolean


Test-and-set Lock



Keep trying until lock acquired


Test-and-set Lock



Release lock by resetting state to false


Performance

• Experiment– n threads– Increment shared counter 1 million

times

• How long should it take?• How long does it take?


Graph

ideal

tim e

threads

no speedup because of sequential bottleneck


Mystery #1ti

m e

threads

TAS lock

Ideal

(1)

What is going on?


Bus-Based Architectures

Bus

cache

memory

cachecache



Bus

cache

memory

cachecache

Random access memory (10s of cycles)



cache

memory

cachecache

Shared Bus•Broadcast medium•One broadcaster at a time•Processors and memory all “snoop”

Bus



Bus

cache

memory

cachecache

Per-Processor Caches•Small•Fast: 1 or 2 cycles•Address & state information


Jargon Watch

• Cache hit– “I found what I wanted in my cache”– Good Thing™


Jargon Watch

• Cache hit– “I found what I wanted in my cache”– Good Thing™

• Cache miss– “I had to shlep all the way to memory

for that data”– Bad Thing™


Cave Canem

• This model is still a simplification– But not in any essential way– Illustrates basic principles

• Will discuss complexities later


Bus

Processor Issues Load Request

cache

memory

cachecache

data


Bus


Bus

cache

memory

cachecache

data

Gimmedata


cache

Bus

Memory Responds

Bus

memory

cachecache

data

Got your data right here data


Bus


memory

cachecachedata

data

Gimmedata


Bus


Bus

memory

cachecachedata

data

Gimmedata


Bus


Bus

memory

cachecachedata

data

I got data


Bus

Other Processor Responds

memory

cachecache

data

I got data

datadata

Bus


Bus

Other Processor Responds

memory

cachecache

data

datadata

Bus


Modify Cached Data

Bus

data

memory

cachedata

data

(1)


Modify Cached Data

Bus

data

memory

cachedata

data

data

(1)


memory

Bus

data

Modify Cached Data

cachedata

data


memory

Bus

data

Modify Cached Data

cache

What’s up with the other copies?

data

data


Cache Coherence

• We have lots of copies of data– Original copy in memory – Cached copies at processors

• Some processor modifies its own copy– What do we do with the others?– How to avoid confusion?


Write-Back Caches

• Accumulate changes in cache• Write back when needed

– Need the cache for something else– Another processor wants it

• On first modification– Invalidate other entries– Requires non-trivial protocol …


Write-Back Caches

• Cache entry has three states– Invalid: contains raw seething bits– Valid: I can read but I can’t write– Dirty: Data has been modified

• Intercept other load requests• Write back to memory before using cache


Bus

Invalidate

memory

cachedatadata

data


Bus

Invalidate

Bus

memory

cachedatadata

data

Mine, all mine!


Bus

Invalidate

Bus

memory

cachedatadata

data

cache

Uh,oh


cache

Bus

Invalidate

memory

cachedata

data

Other caches lose read permission


cache

Bus

Invalidate

memory

cachedata

data

Other caches lose read permission

This cache acquires write permission


cache

Bus

Invalidate

memory

cachedata

data

Memory provides data only if not present in any cache, so no need

to change it now (expensive)

(2)


cache

Bus

Another Processor Asks for Data

memory

cachedata

data

(2)

Bus


cache data

Bus

Owner Responds

memory

cachedata

data

(2)

Bus

Here it is!


Bus

End of the Day …

memory

cachedata

data

(1)

Reading OK, no writing

data data


Simple TASLock

• TAS invalidates cache lines• Spinners

– Miss in cache– Go to bus

• Thread wants to release lock– delayed behind spinners


Test-and-test-and-set

• Wait until lock “looks” free– Spin on local cache– No bus use while lock busy

• Problem: when lock is released– Invalidation storm …


Local Spinning while Lock is Busy

Bus

memory

busybusybusy

busy


Bus

On Release

memory

freeinvalidinvalid

free


On Release

Bus

memory

freeinvalidinvalid

free

miss miss

Everyone misses, rereads

(1)


On Release

Bus

memory

freeinvalidinvalid

free

TAS(…) TAS(…)

Everyone tries TAS

(1)


Mystery Explained

TAS lock

TTAS lock

Ideal

tim e

threads

Better than TAS but still

not as good as ideal

OpenMP• (Shared Memory, Thread Based Parallelism) OpenMP is a

shared-memory application programming interface (API) whose features, are based on prior efforts to facilitate shared-memory parallel programming.

• (Compiler Directive Based) OpenMP provides directives, library functions and environment variables to create and control the execution of parallel programs.

• (Explicit Parallelism) OpenMP’s directives let the user tell the compiler which instructions to execute in parallel and how to distribute them among the threads that will run the code.

116

OpenMP exampleint main(){

for (i = 0; i < n; i++) {a [i] = i * 0.5;b [i] = i * 2.0;

}sum = 0;for (i = 1; i <= n; i++ ) {

sum = sum + a[i]*b[i];}printf ("sum = %f", sum);

}

int main(){

for (i = 0; i < n; i++) {a [i] = i * 0.5;b [i] = i * 2.0

}sum = 0; t = 0;#pragma omp parallel private(t) \

shared(sum, a, b, n){ t = 0;

#pragma omp for for (i = 1; i <= n; i++ ) {

t = t + a[i]*b[i];}#pragma omp critical (update_sum)sum += t;

}printf ("sum = %f \n", sum);

}

117

PARALLEL Region Construct

• A parallel region is a block of code that will be executed by multiple threads. This is the fundamental OpenMP parallel construct.

• Starting from the beginning of this parallel region, the code is duplicated and all threads will execute that code.

• There is an implied barrier at the end of a parallel section. Only the master thread continues execution past this point.

• How Many Threads?– Setting of the NUM_THREADS clause

– Use of the omp_set_num_threads() library function

– Setting of the OMP_NUM_THREADS environment variable

– Implementation default - usually the number of CPUs on a node

• A parallel region must be a structured block that does not span multiple routines or code files

• It is illegal to branch into or out of a parallel region

118


#pragma omp parallel [clause ...] newline

if (scalar_expression)

private (list)

shared (list)

default (shared | none)

firstprivate (list)

num_threads (integer-expression)

structured_block

119


#include <omp.h> void main () {

int nthreads, tid; /* Fork a team of threads with each thread having a private tid variable */ #pragma omp parallel private(tid) {

/* Obtain and print thread id */ tid = omp_get_thread_num(); printf("Hello World from thread = %d\n", tid); /* Only master thread does this */ if (tid == 0) {

nthreads = omp_get_num_threads(); printf("Number of threads = %d\n", nthreads);

} } /* All threads join master thread and terminate */

}

120

Work-Sharing Constructsfor Directive

• The for directive specifies that the iterations of the loop immediately following it must be executed in parallel by the team. This assumes a parallel region has already been initiated, otherwise it executes in serial on a single processor.

121


#pragma omp for [clause ...] newline

schedule (type [,chunk])

ordered

private (list)

firstprivate (list)

lastprivate (list)

shared (list)

nowait

for_loop

122


#include <omp.h> #define CHUNKSIZE 100 #define N 1000 void main () {

int i, chunk; float a[N], b[N], c[N]; /* Some initializations */ for (i=0; i < N; i++) a[i] = b[i] = i * 1.0; chunk = CHUNKSIZE; #pragma omp parallel shared(a,b,c,chunk) private(i) {

#pragma omp for schedule(dynamic,chunk) nowait for (i=0; i < N; i++) c[i] = a[i] + b[i];

} /* end of parallel section */

}

123

Work-Sharing Constructs SECTIONS Directive

• The SECTIONS directive is a non-iterative work-sharing construct. It specifies that the enclosed section(s) of code are to be divided among the threads in the team.

124

Work-Sharing Constructs

SECTIONS Directive#pragma omp sections [clause ...] newline

private (list)

firstprivate (list)

lastprivate (list)

nowait

{

#pragma omp section newline structured_block

#pragma omp section newline structured_block

}

125


SECTIONS Directive#include <omp.h> #define N 1000 void main () {

int i; float a[N], b[N], c[N], d[N]; /* Some initializations */ for (i=0; i < N; i++) a[i] = i * 1.5; b[i] = i + 22.35; #pragma omp parallel shared(a,b,c,d) private(i) {

#pragma omp sections nowait { #pragma omp section

for (i=0; i < N; i++) c[i] = a[i] + b[i]; #pragma omp section for (i=0; i < N; i++) d[i] = a[i] * b[i];

} /* end of sections */


}

126


SINGLE Directive• The SINGLE directive specifies that the enclosed code is to be

executed by only one thread in the team.

• May be useful when dealing with sections of code that are not thread safe (such as I/O)

#pragma omp single [clause ...] newline

private (list)

firstprivate (list)

nowait

structured_block

127

Data Scope Attribute Clauses - PRIVATE

• The PRIVATE clause declares variables in its list to be private to each thread.

• A new object of the same type is declared once for each thread in the team

• All references to the original object are replaced with references to the new object

• Variables declared PRIVATE should be assumed to be uninitialized for each thread

128

Data Scope Attribute Clauses - SHARED

• The SHARED clause declares variables in its list to be shared among all threads in the team.

• A shared variable exists in only one memory location and all threads can read or write to that address

• It is the programmer's responsibility to ensure that multiple threads properly access SHARED variables (such as via CRITICAL sections)

129

Data Scope Attribute Clauses - FIRSTPRIVATE• The FIRSTPRIVATE clause combines the

behavior of the PRIVATE clause with automatic initialization of the variables in its list.

• Listed variables are initialized according to the value of their original objects prior to entry into the parallel or work-sharing construct.

130

Data Scope Attribute Clauses - LASTPRIVATE

• The LASTPRIVATE clause combines the behavior of the PRIVATE clause with a copy from the last loop iteration or section to the original variable object.

• The value copied back into the original variable object is obtained from the last (sequentially) iteration or section of the enclosing construct. For example, the team member which executes the final iteration for a DO section, or the team member which does the last SECTION of a SECTIONS context performs the copy with its own values

131

Synchronization Constructs

CRITICAL Directive• The CRITICAL directive specifies a region

of code that must be executed by only one thread at a time.

• If a thread is currently executing inside a CRITICAL region and another thread reaches that CRITICAL region and attempts to execute it, it will block until the first thread exits that CRITICAL region.

132


CRITICAL Directive#include <omp.h> void main() {

int x; x = 0; #pragma omp parallel shared(x) {

#pragma omp critical x = x + 1;


}

133


• MASTER Directive

– The MASTER directive specifies a region that is to be executed only by the master thread of the team. All other threads on the team skip this section of code

• BARRIER Directive

– The BARRIER directive synchronizes all threads in the team.

– When a BARRIER directive is reached, a thread will wait at that point until all other threads have reached that barrier. All threads then resume executing in parallel the code that follows the barrier.

134


• ATOMIC Directive– The ATOMIC directive specifies that a

specific memory location must be updated atomically, rather than letting multiple threads attempt to write to it. In essence, this directive provides a mini-CRITICAL section.

– The directive applies only to a single, immediately following statement

135

Find an errorint main(){

for (i = 0; i < n; i++) {a [i] = i * 0.5;b [i] = i * 2.0

}sum = 0; t = 0;#pragma omp parallel shared(sum, a, b, n){

#pragma omp for private(t)for (i = 1; i <= n; i++ ) {

t = t + a[i]*b[i];}#pragma omp critical (update_sum)sum += t;

}printf ("sum = %f \n", sum);

}

136

Find an error

for (i=0; i<n-1; i++)a[i] = a[i] + b[i];

for (i=0; i<n-1; i++)a[i] = a[i+1] + b[i];

137

Find an error

#pragma omp parallel{

int Xlocal = omp_get_thread_num();Xshared = omp_get_thread_num(); printf("Xlocal = %d Xshared = %d\n",Xlocal,Xshared);

}int i, j;#pragma omp parallel forfor (i=0; i<n; i++)

for (j=0; j<m; j++) {a[i][j] = compute(i,j);

}

138

Find an error

void compute(int n){

int i;double h, x, sum;h = 1.0/(double) n;sum = 0.0;#pragma omp for reduction(+:sum) shared(h)for (i=1; i <= n; i++) {

x = h * ((double)i - 0.5);sum += (1.0 / (1.0 + x*x));

}pi = h * sum;

}

139

Find an error

void main (){.............

#pragma omp parallel for private(i,a,b)for (i=0; i<n; i++){

b++;a = b+i;

} /*-- End of parallel for --*/c = a + b;.............

}

140


This work is licensed under a

Creative Commons Attribution-ShareAlike 2.5 License.

• You are free:– to Share — to copy, distribute and transmit the work – to Remix — to adapt the work

• Under the following conditions:– Attribution. You must attribute the work to “The Art of

Multiprocessor Programming” (but not in any way that suggests that the authors endorse you or your use of the work).

– Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, similar or a compatible license.

• For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to– http://creativecommons.org/licenses/by-sa/3.0/.

• Any of the above conditions can be waived if you get permission from the copyright holder.

• Nothing in this license impairs or restricts the author's moral rights.




Art of Multiprocessor Programming1 This work is licensed under a Creative Commons Attribution-...

Documents

Transcript of Art of Multiprocessor Programming1 This work is licensed under a Creative Commons Attribution-...