Multiprocessors and Thread Level Parallelism Chapter 4, Appendix H
Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality...
Transcript of Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality...
![Page 1: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/1.jpg)
Parallelism in C++Lecture 1
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
1
![Page 2: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/2.jpg)
Amdahl’s Law (Strong Scaling)
• S: Speedup
• P: Proportion of parallel code
• N: Number of processors
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
2
𝑆 =1
1 − 𝑃 +𝑃𝑁
Figure courtesy of Wikipedia (http://en.wikipedia.org/wiki/Amdahl's_law)
![Page 3: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/3.jpg)
Rule 1
Parallelize Applications as Much as Humanly Possible
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
3
![Page 4: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/4.jpg)
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
4
The 4 Horsemen of the Apocalypse
![Page 5: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/5.jpg)
•Starvation Insufficient concurrent work to maintain high utilization of
resources
•Latencies Time-distance delay of remote resource access and services
•Overheads Work for management of parallel actions and resources on
critical path which are not necessary in sequential variant
•Waiting for Contention resolution
Delays due to lack of availability of oversubscribed shared resources
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
5
The 4 Horsemen of the Apocalypse
![Page 6: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/6.jpg)
Universal Scalability Law
• λ: Scaling efficiency
• δ: Contention
• κ: Latencies (‘Crosstalk’)
• N: Number of processors
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
6
![Page 7: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/7.jpg)
Real-world Problems• Insufficient parallelism imposed by the programming
model OpenMP: enforced barrier at end of parallel loop
MPI: global (communication) barrier after each time step
• Over-synchronization of more things than required by algorithm MPI: Lock-step between nodes (ranks)
• Insufficient coordination between on-node and off-node parallelism MPI+X: insufficient co-design of tools for off-node, on-node, and accelerators
• Distinct programming models for different types of parallelism Off-node: MPI, On-node: OpenMP, Accelerators: CUDA, etc.
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
7
![Page 8: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/8.jpg)
Real-world Problems• Even standard algorithms added to C++17 enforce
fork-join semantics
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
8
![Page 9: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/9.jpg)
Fork/Join Parallelism
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
9
![Page 10: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/10.jpg)
Rule 2
Use a Programming Environment that Embraces SLOW
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
10
![Page 11: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/11.jpg)
Overheads: Thought-Experiment
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
11
0
0.5
1
1.5
2
2.5
3
1 10 100 1000 10000 100000 1000000 10000000
Exec
uti
on
tim
e (r
elat
ive
to s
equ
enti
al t
ime)
Grain Size (amount of work per thread)
Execution Time over Grain Size(for different amounts of overheads per thread, 16 Cores)
Sequential Time
1µs Overhead
100µs Overhead
10ms Overhead
![Page 12: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/12.jpg)
Overheads: The Worst of All?• Even relatively small amounts of work can benefit from being split
into smaller tasks Possibly huge amount of ‘threads’
In the previous thought-experiment we ended up considering up to 10 million threads
Best possible scaling is predicted to be reached when using 10000 threads (for 1 second
worth of work)
• Several problems Impossible to work with that many kernel threads (p-threads)
Impossible to reason about this amount of tasks
Requires abstraction mechanism
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
12
![Page 13: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/13.jpg)
Rule 3
Allow for your Grainsize to be Variable
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
13
![Page 14: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/14.jpg)
Overheads: The Worst of All?
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
14
0
0.5
1
1.5
2
2.5
3
100 1000 10000 100000 1000000 10000000 100000000Exec
uti
on
tim
e (r
elat
ive
to s
equ
enti
al t
ime)
Grainsize (amount of work per thread)
Execution Time over Grainsize(1D Stencil, Solving Heat Diffusion)
Sequential Time
Time (16 Cores)
![Page 15: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/15.jpg)
Rule 4
Oversubscribe and Balance Adaptively
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
15
![Page 16: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/16.jpg)
The Challenges
•We need to find a usable way to fully parallelize our applications
•Goals are:Expose asynchrony to the programmer without exposing additional concurrency
Make data dependencies explicit, hide notion of ‘thread’ and ‘communication’
Provide manageable paradigms for handling parallelism
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
16
![Page 17: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/17.jpg)
The Future of Computation
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
17
![Page 18: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/18.jpg)
What is a (the) Future?• Many ways to get hold of a (the) future, simplest way is to use (std) async:
int universal_answer() { return 42; }
void deep_thought(){
future<int> promised_answer = async(&universal_answer);
// do other things for 7.5 million years
cout << promised_answer.get() << endl; // prints 42}
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
18
![Page 19: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/19.jpg)
What is a (the) future• A future is an object representing a result which has not been calculated yet
Locality 1
Suspend consumerthread
Execute another thread
Resume consumerthread
Locality 2
Execute Future:
Producer thread
Future object
Result is being returned
Enables transparent synchronization with producer
Hides notion of dealing with threads
Represents a data-dependency
Makes asynchrony manageable
Allows for composition of several asynchronous operations
(Turns concurrency into parallelism)
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
19
![Page 20: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/20.jpg)
Ways to Create a future
•Standard defines 3 possible ways to create a future, 3 different ‘asynchronous providers’
std::async
std::packaged_task
std::promise
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
20
![Page 21: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/21.jpg)
Packaging a Future• std::packaged_task is a function object
It gives away a future representing the result of its invocation
• Can be used as a synchronization primitive
Pass to std::thread
• Converting a callback into a future
Observer pattern, allows to wait for a callback to happen
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
21
![Page 22: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/22.jpg)
Packaging a Future
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
22
template <typename F, typename ...Arg>
std::future<typename std::result_of<F(Arg...)>::type>
simple_async(F func, Arg&&... arg)
{
std::packaged_task<F> pt(func);
auto f = pt.get_future();
std::thread t(std::move(pt), std::forward<Arg>(arg)...);
t.detach(); // detach the thread from this object
return f;
}
![Page 23: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/23.jpg)
Promising a Future• std::promise is also an asynchronous provider ("an object
that provides a result to a shared state") The promise is the thing that you set a result on, so that you can
get it from the associated future.
The promise initially creates the shared state
The future created by the promise shares the state with it
The shared state stores the value
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
23
![Page 24: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/24.jpg)
Promising a futuretemplate <typename F> class simple_packaged_task;
template <typename R, typename... Args>
class simple_packaged_task<R(Args...)> // must be move-only
{
std::function<R(Args...)> fn;
std::promise<R> p; // the promise for the result
// ...
public:
template <typename F>
explicit simple_packaged_task(F && f) : fn(std::forward<F>(f)) {}
template <typename ...T>
void operator()(T&&... t) { p.set_value(fn(std::forward<T>(t)...)); }
std::future<R> get_future() { return p.get_future(); }
};
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
24
![Page 25: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/25.jpg)
Extending std::future
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
25
![Page 26: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/26.jpg)
Extending std::future
• Several proposals for next C++ Standard, also HPX Extension for std::future
Compositional facilities
Parallel composition
Sequential composition
Parallel Algorithms
Parallel Task Regions
Extended async semantics: dataflow
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
26
![Page 27: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/27.jpg)
Make a ready Future• Create a future which is ready at construction (N3857)
future<int> compute(int x)
{
if (x < 0) return make_ready_future(-1);
if (x == 0) return make_ready_future(0);
return async(
[](int param) { return do_work(param); },
x);
}
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
27
![Page 28: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/28.jpg)
Compositional facilities• Sequential composition of futures
string make_string()
{
future<int> f1 = async([]() -> int { return 123; });
future<string> f2 = f1.then(
[](future<int> f) -> string {
return to_string(f.get()); // here .get() won’t block
});
}
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
28
![Page 29: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/29.jpg)
Compositional facilities• Parallel composition of futures
void test_when_all()
{
shared_future<int> shared_future1 = async([]() -> int { return 125; });
future<string> future2 = async([]() -> string { return string("hi"); });
future<tuple<shared_future<int>, future<string>>> all_f =
when_all(shared_future1, future2); // also: when_any, when_some, etc.
future<int> result = all_f.then(
[](auto f) -> int {
return do_work(f.get());
});
}
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
29
![Page 30: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/30.jpg)
Parallel Algorithms (C++17)
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
30
![Page 31: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/31.jpg)
Parallel Algorithms (C++17)• Add Execution Policy as first argument
• Execution policies have associated default executor and default executor parameters
execution::parallel_policy, generated with par
parallel executor, static chunk size
execution::sequenced_policy, generated with seq
sequential executor, no chunking
// add execution policystd::fill(
std::execution::par, begin(d), end(d), 0.0);
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
31
![Page 32: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/32.jpg)
Parallel Algorithms (Extensions)
// uses default executor: par
std::vector<double> d = { ... };
fill(execution::par, begin(d), end(d), 0.0);
// rebind par to user-defined executor (where and how to execute)
my_executor my_exec = ...;
fill(execution::par.on(my_exec), begin(d), end(d), 0.0);
// rebind par to user-defined executor and user defined executor
// parameters (affinities, chunking, scheduling, etc.)
my_params my_par = ...
fill(execution::par.on(my_exec).with(my_par), begin(d), end(d), 0.0);
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
32
![Page 33: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/33.jpg)
Execution Policies (Extensions)• Extensions: asynchronous execution policies
parallel_task_execution_policy (asynchronous version of parallel_execution_policy), generated with par(task)
sequenced_task_execution_policy (asynchronous version of sequenced_execution_policy), generated with seq(task)
In all cases the formerly synchronous functions return a future<>
Instruct the parallel construct to be executed asynchronously
Allows integration with asynchronous control flow
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
33
![Page 34: Parallelism in C++pdiehl/teaching/2019/4977/Parallelism in C… · Resume consumer thread Locality 2 Execute Future: Producer thread Future object Result is being returned Enables](https://reader034.fdocuments.us/reader034/viewer/2022050414/5f8a5fa947fb4c7e987dbf1a/html5/thumbnails/34.jpg)
9/2
6/2
019
Para
llel
Pro
gra
mm
ing in
C+
+ (
Lect
ure
1)
Hart
mu
t K
ais
er
34