Don’t Try This at WorkLow Level Threading with C++11

Tony Van Eerd, BlackBerry

May 13, 2013

Copyright Tony Van Eerd (2009-2013) and BlackBerry, Inc. 2012-2013

May 15, 2012 2

•The C++ (11) Memory Model

• C++ (11) Atomics

• What you can but shouldn’t do with them

Things to discuss

The Memory Model

May 15, 2012 3

?

4May 15, 2012

?

6May 15, 2012

?

7May 15, 2012

2009 2013 2 16

Predicted # of Spocks in Star Trek 2013,According to Moore’s Law

(1968: 1 good + 1 evil)(1984: ~1½ Spocks)

← Cache Coherency

9May 15, 2012

David Hilley - http://www.thegibson.org/blog/David Hilley - http://www.thegibson.org/blog/

10May 15, 2012

11May 15, 2012

Want

12May 15, 2012

Got

13May 15, 2012

Got13May 15, 2012

Want

14May 15, 2012

Also Want

14May 15, 2012

Want

15May 15, 2012

Speed15May 15, 2012

SequentialConsistenc

y

...and eat it too

Cake…

May 15, 2012 16

Sequential Consistency:

All memory reads and writes of all threads are executed as if interleaved in some global order, consistent with the program order of each thread.

A Bit of Precision…

ABCabcαβγ

AaαbBβCcγ

αβaγbAcBC

Thread α

αβγ

Thread a

abc

Thread A

ABC

P1: W(x)1 . P2: R(x)1 R(x)2 . P3: R(x)2 R(x)2 . P4: W(x)2 .

May 15, 2012 17

Sequential Consistency:

All memory reads and writes of all threads are executed as if interleaved in some global order, consistent with the program order of each thread.

A Bit of Precision…

P1: W(x)1 . P2: R(x)1 R(x)2 . P3: R(x)1 R(x)2 . P4: W(x)2 .

P1: W(x)1 . P2: R(x)1 R(x)2 . P3: R(x)2 R(x)1 . P4: W(x)2 .

P1: W(x)1 . P2: R(x)? R(x)? . P3: R(x)? R(x)? . P4: W(x)2 .

P1: W(x)1 . P2: R(x)? R(x)? . P3: R(x)? R(x)? . P4: W(x)2 .

Joe Pfeiffer, New Mexico State University - http://www.cs.nmsu.edu/~pfeiffer

P1: W(x)1 . P2: R(x)2 R(x)1 . P3: R(x)2 R(x)1 . P4: W(x)2 .

P1: W(x)1 . P2: R(x)2 R(x)1 . P3: R(x)2 R(x)1 . P4: W(x)2 .

P1: W(x)1 . P2: R(x)1 R(x)2 . P3: R(x)2 R(x)2 . P4: W(x)2 .

P1: W(x)1 . P2: R(x)1 R(x)2 . P3: R(x)1 R(x)2 . P4: W(x)2 .

P1: W(x)1 . P2: R(x)1 R(x)2 . P3: R(x)2 R(x)1 . P4: W(x)2 .

““RelaxRelax””

Yeah, well, you know, Yeah, well, you know, that's just, like, your that's just, like, your opinion, man.opinion, man.

May 15, 2012 18

Sequential Consistency:

All memory reads and writes of all threads are executed as if interleaved in some global order, consistent with the program order of each thread.

A Bit of Precision…

May 15, 2012 19

Sequential Consistency:

All memory reads and writes of all threads are executed as if interleaved in some global order, consistent with the program order of each thread.

A Bit of Precision…

“Relaxed” Memory Model:

16 egotistical and/or layed-back Spocks that each don’t care what the others think. (But are each individually, internally, consistent.)

May 15, 2012 20

Sequential Consistency:

All memory reads and writes of all threads are executed as if interleaved in some global order, consistent with the program order of each thread.

A Bit of Precision…

“Relaxed” Memory Model:

16 egotistical and/or layed-back Spocks that each don’t care what the others think. (But are each individually, internally, consistent.)

(relaxing what we mean by “precision” here)

C++ Atomics

May 15, 2012 21

Have your cake and eat it too.

C++ Atomics

May 15, 2012 22

Have your cake and eat it too.

But be careful, you baked it!

23May 15, 2012

namespace std {// 29.3, order and consistencyenum memory_order;template <class T>T kill_dependency(T y) noexcept;// 29.4, lock-free property#define ATOMIC_BOOL_LOCK_FREE unspecified#define ATOMIC_CHAR_LOCK_FREE unspecified#define ATOMIC_CHAR16_T_LOCK_FREE unspecified#define ATOMIC_CHAR32_T_LOCK_FREE unspecified#define ATOMIC_WCHAR_T_LOCK_FREE unspecified#define ATOMIC_SHORT_LOCK_FREE unspecified#define ATOMIC_INT_LOCK_FREE unspecified#define ATOMIC_LONG_LOCK_FREE unspecified#define ATOMIC_LLONG_LOCK_FREE unspecified#define ATOMIC_POINTER_LOCK_FREE unspecified// 29.5, generic typestemplate<class T> struct atomic;template<> struct atomic<integral >;template<class T> struct atomic<T*>;// 29.6.1, general operations on atomic types// In the following declarations, atomic-type is either// atomic<T> or a named base class for T from// Table 145 or inferred from Table 146 or from bool.// If it is atomic<T>, then the declaration is a template// declaration prefixed with template <class T>.bool atomic_is_lock_free(const volatile atomic-type *) noexcept;bool atomic_is_lock_free(const atomic-type *) noexcept;void atomic_init(volatile atomic-type *, T) noexcept;void atomic_init(atomic-type *, T) noexcept;void atomic_store(volatile atomic-type *, T) noexcept;void atomic_store(atomic-type *, T) noexcept;void atomic_store_explicit(volatile atomic-type *, T, memory_order) noexcept;void atomic_store_explicit(atomic-type *, T, memory_order) noexcept;T atomic_load(const volatile atomic-type *) noexcept;T atomic_load(const atomic-type *) noexcept;T atomic_load_explicit(const volatile atomic-type *, memory_order) noexcept;T atomic_load_explicit(const atomic-type *, memory_order) noexcept;T atomic_exchange(volatile atomic-type *, T) noexcept;T atomic_exchange(atomic-type *, T) noexcept;T atomic_exchange_explicit(volatile atomic-type *, T, memory_order) noexcept;T atomic_exchange_explicit(atomic-type *, T, memory_order) noexcept;bool atomic_compare_exchange_weak(volatile atomic-type *, T*, T) noexcept;bool atomic_compare_exchange_weak(atomic-type *, T*, T) noexcept;bool atomic_compare_exchange_strong(volatile atomic-type *, T*, T) noexcept;bool atomic_compare_exchange_strong(atomic-type *, T*, T) noexcept;bool atomic_compare_exchange_weak_explicit(volatile atomic-type *, T*, T,memory_order, memory_order) noexcept;bool atomic_compare_exchange_weak_explicit(atomic-type *, T*, T.memory_order, memory_order) noexcept;bool atomic_compare)exchange_strong_explicit(volatile atomic-type *, T*, T,memory_order, memory_order) noexcept;bool atomic_compare_exchange_strong_explicit(atomic-type *, T*, T,memory_order, memory_order) noexcept;// 29.6.2, templated operations on atomic typestemplate <class T>T atomic_fetch_add(volatile atomic<T>*, T) noexcept;template <class T>T atomic_fetch_add(atomic<T>*, T) noexcept;template <class T>T atomic_fetch_add_explicit(volatile atomic<T>*, T, memory_order) noexcept;template <class T>T atomic_fetch_add_explicit(atomic<T>*, T, memory_order) noexcept;template <class T>T atomic_fetch_sub(volatile atomic<T>*, T) noexcept;template <class T>T atomic_fetch_sub(atomic<T>*, T) noexcept;template <class T>T atomic_fetch_sub_explicit(volatile atomic<T>*, T, memory_order) noexcept;template <class T>T atomic_fetch_sub_explicit(atomic<T>*, T, memory_order) noexcept;template <class T>T atomic_fetch_and(volatile atomic<T>*, T) noexcept;template <class T>T atomic_fetch_and(atomic<T>*, T) noexcept;template <class T>T atomic_fetch_and_explicit(volatile atomic<T>*, T, memory_order) noexcept;template <class T>T atomic_fetch_and_explicit(atomic<T>*, T, memory_order) noexcept;template <class T>

T atomic_fetch_or(volatile atomic<T>*, T) noexcept;template <class T>T atomic_fetch_or(atomic<T>*, T) noexcept;template <class T>T atomic_fetch_or_explicit(volatile atomic<T>*, T, memory_order) noexcept;template <class T>T atomic_fetch_or_explicit(atomic<T>*, T, memory_order) noexcept;template <class T>T atomic_fetch_xor(volatile atomic<T>*, T) noexcept;template <class T>T atomic_fetch_xor(atomic<T>*, T) noexcept;template <class T>T atomic_fetch_xor_explicit(volatile atomic<T>*, T, memory_order) noexcept;template <class T>T atomic_fetch_xor_explicit(atomic<T>*, T, memory_order) noexcept;// 29.6.3, arithmetic operations on atomic types// In the following declarations, atomic-integral is either// atomic<T> or a named base class for T from// Table 145 or inferred from Table 146.// If it is atomic<T>, then the declaration is a template// specialization declaration prefixed with template <>.integral atomic_fetch_add(volatile atomic-integral *, integral ) noexcept;integral atomic_fetch_add(atomic-integral *, integral ) noexcept;integral atomic_fetch_add_explicit(volatile atomic-integral *, integral , memory_order) noexcept;integral atomic_fetch_add_explicit(atomic-integral *, integral , memory_order) noexcept;integral atomic_fetch_sub(volatile atomic-integral *, integral ) noexcept;integral atomic_fetch_sub(atomic-integral *, integral ) noexcept;integral atomic_fetch_sub_explicit(volatile atomic-integral *, integral , memory_order) noexcept;integral atomic_fetch_sub_explicit(atomic-integral *, integral , memory_order) noexcept;integral atomic_fetch_and(volatile atomic-integral *, integral ) noexcept;integral atomic_fetch_and(atomic-integral *, integral ) noexcept;integral atomic_fetch_and_explicit(volatile atomic-integral *, integral , memory_order) noexcept;integral atomic_fetch_and_explicit(atomic-integral *, integral , memory_order) noexcept;integral atomic_fetch_or(volatile atomic-integral *, integral ) noexcept;integral atomic_fetch_or(atomic-integral *, integral ) noexcept;integral atomic_fetch_or_explicit(volatile atomic-integral *, integral , memory_order) noexcept;integral atomic_fetch_or_explicit(atomic-integral *, integral , memory_order) noexcept;integral atomic_fetch_xor(volatile atomic-integral *, integral ) noexcept;integral atomic_fetch_xor(atomic-integral *, integral ) noexcept;integral atomic_fetch_xor_explicit(volatile atomic-integral *, integral , memory_order) noexcept;integral atomic_fetch_xor_explicit(atomic-integral *, integral , memory_order) noexcept;// 29.6.4, partial specializations for pointerstemplate <class T>T* atomic_fetch_add(volatile atomic<T*>*, ptrdiff_t) noexcept;template <class T>T* atomic_fetch_add(atomic<T*>*, ptrdiff_t) noexcept;template <class T>T* atomic_fetch_add_explicit(volatile atomic<T*>*, ptrdiff_t, memory_order) noexcept;template <class T>T* atomic_fetch_add_explicit(atomic<T*>*, ptrdiff_t, memory_order) noexcept;template <class T>T* atomic_fetch_sub(volatile atomic<T*>*, ptrdiff_t) noexcept;template <class T>T* atomic_fetch_sub(atomic<T*>*, ptrdiff_t) noexcept;template <class T>T* atomic_fetch_sub_explicit(volatile atomic<T*>*, ptrdiff_t, memory_order) noexcept;template <class T>T* atomic_fetch_sub_explicit(atomic<T*>*, ptrdiff_t, memory_order) noexcept;// 29.6.5, initialization#define ATOMIC_VAR_INIT(value) see below// 29.7, flag type and operationsstruct atomic_flag;bool atomic_flag_test_and_set(volatile atomic_flag*) noexcept;bool atomic_flag_test_and_set(atomic_flag*) noexcept;bool atomic_flag_test_and_set_explicit(volatile atomic_flag*, memory_order) noexcept;bool atomic_flag_test_and_set_explicit(atomic_flag*, memory_order) noexcept;void atomic_flag_clear(volatile atomic_flag*) noexcept;void atomic_flag_clear(atomic_flag*) noexcept;void atomic_flag_clear_explicit(volatile atomic_flag*, memory_order) noexcept;void atomic_flag_clear_explicit(atomic_flag*, memory_order) noexcept;#define ATOMIC_FLAG_INIT see below// 29.8, fencesextern "C" void atomic_thread_fence(memory_order) noexcept;extern "C" void atomic_signal_fence(memory_order) noexcept;

typedef enum memory_order {memory_order_relaxed, memory_order_consume, memory_order_acquire,memory_order_release, memory_order_acq_rel, memory_order_seq_cst} memory_order;}

#include <atomic>

24May 15, 2012

template<class T> struct atomic;template<> struct atomic< integral >;template<class T> struct atomic<T*>;

template <class T> struct atomic { void store(T, memory_order = memory_order_seq_cst) volatile noexcept; void store(T, memory_order = memory_order_seq_cst) noexcept; T load(memory_order = memory_order_seq_cst) const volatile noexcept; T load(memory_order = memory_order_seq_cst) const noexcept; ...};

template <class T> struct atomic { ...};

25May 15, 2012

template<class T> struct atomic;template<> struct atomic< integral >;template<class T> struct atomic<T*>;

template <class T> struct atomic { void store(T, memory_order = memory_order_seq_cst) volatile noexcept; void store(T, memory_order = memory_order_seq_cst) noexcept; T load(memory_order = memory_order_seq_cst) const volatile noexcept; T load(memory_order = memory_order_seq_cst) const noexcept; ...};

VOLATILEVOLATILENOTNOTFORFORHEREHERE

26May 15, 2012

VOLATILE

VOLATILE

VOLATILE

VOLATILE

VOLATILE

VOLATILE

27May 15, 2012

template<class T> struct atomic;template<> struct atomic< integral >;template<class T> struct atomic<T*>;

template <class T> struct atomic { void store(T, memory_order = memory_order_seq_cst) volatile noexcept; void store(T, memory_order = memory_order_seq_cst) noexcept; T load(memory_order = memory_order_seq_cst) const volatile noexcept; T load(memory_order = memory_order_seq_cst) const noexcept; ...};

28May 15, 2012

template <class T> struct atomic {

void store(T, memory_order = memory_order_seq_cst); T load(memory_order = memory_order_seq_cst); ...};

template<class T> struct atomic;template<> struct atomic< integral >;template<class T> struct atomic<T*>;

// C-like:T atomic_load(const A * object);T atomic_load(const volatile A * object);T atomic_load_explicit(const A * object, memory_order);T atomic_load_explicit(const volatile A * object, memory_order);

bool atomic_compare_exchange_weak_explicit( volatile A * object, C * expected, C desired, memory_order success, memory_order failure);

29May 15, 2012

template <class T> struct atomic // generic T, integral, pointer, bool{ atomic() = default; constexpr atomic(T); atomic(const atomic&) = delete; atomic& operator=(const atomic&) = delete;

void store(T, memory_order = memory_order_seq_cst); T load(memory_order = memory_order_seq_cst); T operator=(T t) { store(t); } operator T() { return load(); }

T exchange(T, memory_order = memory_order_seq_cst); bool compare_exchange_weak(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_weak(T&, T, memory_order, memory_order); bool compare_exchange_strong(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_strong(T&, T, memory_order, memory_order);

bool is_lock_free();};

template<class T> struct atomic;template<> struct atomic< integral >;template<class T> struct atomic<T*>;

30May 15, 2012

struct atomic_flag{ atomic_flag() = default; atomic_flag(const atomic_flag&) = delete; atomic_flag& operator=(const atomic_flag&) = delete;

bool test_and_set(memory_order = memory_order_seq_cst); void clear(memory_order = memory_order_seq_cst);};

atomic_flag guard = ATOMIC_FLAG_INIT;

struct atomic_flag;

“is_lock_free == true”

31May 15, 2012

template <class T> struct atomic // generic T, integral, pointer, bool{ atomic() = default; constexpr atomic(T); atomic(const atomic&) = delete; atomic& operator=(const atomic&) = delete;

void store(T, memory_order = memory_order_seq_cst); T load(memory_order = memory_order_seq_cst); T operator=(T t) { store(t); } operator T() { return load(); }

T exchange(T, memory_order = memory_order_seq_cst); bool compare_exchange_weak(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_weak(T&, T, memory_order, memory_order); bool compare_exchange_strong(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_strong(T&, T, memory_order, memory_order);

bool is_lock_free();};

template<class T> struct atomic;template<> struct atomic< integral >;template<class T> struct atomic<T*>;

32May 15, 2012

template <class T> struct atomic { // pointers and intergrals ... // as above // both: T fetch_add(T, memory_order = memory_order_seq_cst); T fetch_sub(T, memory_order = memory_order_seq_cst); T operator++(int); T operator--(int); T operator++(); // atomic! not the same as: a = a + 1 T operator--(); T operator+=(T); T operator-=(T);

// integrals only: T fetch_and(T, memory_order = memory_order_seq_cst); T fetch_or(T, memory_order = memory_order_seq_cst); T fetch_xor(T, memory_order = memory_order_seq_cst); T operator&=(T); T operator|=(T); T operator^=(T);};

template<class T> struct atomic;template<> struct atomic< integral >;template<class T> struct atomic<T*>;

33May 15, 2012

template <class T> struct atomic{ atomic() = default; constexpr atomic(T); atomic(const atomic&) = delete; atomic& operator=(const atomic&) = delete;

void store(T, memory_order = memory_order_seq_cst); T load(memory_order = memory_order_seq_cst); T operator=(T t) { store(t); } operator T() { return load(); }

T exchange(T, memory_order = memory_order_seq_cst); bool compare_exchange_weak(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_weak(T&, T, memory_order, memory_order); bool compare_exchange_strong(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_strong(T&, T, memory_order, memory_order);

bool is_lock_free();};

template<class T> struct atomic;template<> struct atomic< integral >;template<class T> struct atomic<T*>;

34May 15, 2012

template <class T> struct atomic{ void store(T, memory_order = memory_order_seq_cst); T load(memory_order = memory_order_seq_cst); T exchange(T, memory_order = memory_order_seq_cst); bool compare_exchange_weak(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_weak(T&, T, memory_order, memory_order); bool compare_exchange_strong(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_strong(T&, T, memory_order, memory_order);};

enum memory_order { memory_order_relaxed, memory_order_consume, memory_order_acquire, memory_order_release, memory_order_acq_rel, memory_order_seq_cst};

template<class T> struct atomic;template<> struct atomic< integral >;template<class T> struct atomic<T*>;

35May 15, 2012

Sequential Consistency vs Acquire/Release vs RelaxedAn operation A synchronizes-with an operation B if A is a store to some atomic variable m, with memory_order_release / memory_order_seq_cst,and B is a load from the same variable m, with memory_order_acquire / memory_order_seq_cst,and B reads the value stored by A.

seq_cst

seq_cst

release

acquire

seq_cst

seq_cst

relaxed

relaxed

relaxed

relaxed

boom

P.S.

Locks useAcquire/Release

(not S.C.)

36May 15, 2012

x 1st

y 2nd

y == 0 implies

y 1st

x 2nd

x == 0 implies

Sequential Consistency vs Acquire/Release vs Relaxed

∴ z != 0

38May 15, 2012

template<class T> struct atomic;template<> struct atomic< integral >;template<class T> struct atomic<T*>;

template <class T> struct atomic{ void store(T, memory_order = memory_order_seq_cst); T load(memory_order = memory_order_seq_cst); T exchange(T, memory_order = memory_order_seq_cst); bool compare_exchange_weak(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_weak(T&, T, memory_order, memory_order); bool compare_exchange_strong(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_strong(T&, T, memory_order, memory_order);};

enum memory_order { memory_order_relaxed, memory_order_consume, memory_order_acquire, memory_order_release, memory_order_acq_rel, memory_order_seq_cst};

template <class T> struct atomic{ T exchange(T, memory_order = memory_order_seq_cst); bool compare_exchange_weak(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_weak(T&, T, memory_order, memory_order); bool compare_exchange_strong(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_strong(T&, T, memory_order, memory_order);};

39May 15, 2012

template<class T> struct atomic;template<> struct atomic< integral >;template<class T> struct atomic<T*>;

static SharedData data;static atomic<bool> locked;

if(!locked.exchange(true, memory_order_acquire)){ do_exclusive(data); locked.store(false, memory_order_release);}

static SharedData data;static atomic_flag locked;

if(!locked.test_and_set()){ do_exclusive(data); locked.clear();}

static SharedData data;static atomic<bool> locked;

if(!locked.exchange(true)){ do_exclusive(data); locked.store(false);}

May 15, 2012 40

template<class T> struct atomic;template<> struct atomic< integral >;template<class T> struct atomic<T*>;

template <class T> struct atomic{ bool compare_exchange_weak(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_weak(T&, T, memory_order, memory_order); bool compare_exchange_strong(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_strong(T&, T, memory_order, memory_order);};

static atomic<int> count;

int next;int was = count.load();

do { next = was + 1;} while (!count.compare_exchange_weak(was, next));

// compare_exchange: if (count == was) count = next; else was = count;

// compare_exchange: if (count == was) count = next; else was = count;

template <class T> struct atomic{ bool compare_exchange_weak(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_weak(T&, T, memory_order, memory_order); bool compare_exchange_strong(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_strong(T&, T, memory_order, memory_order);};

// compare_exchange: if (count untouched) count = next; else was = count;

// compare_exchange: if (count untouched) count = next; else was = count;

template <class T> struct atomic{ bool compare_exchange_weak(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_weak(T&, T, memory_order, memory_order); bool compare_exchange_strong(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_strong(T&, T, memory_order, memory_order);};

static atomic<int> count;

int next;int was = count.load();

do { next = was + 1;} while (!count.compare_exchange_weak(was, next, acq_rel, relaxed));

^ atomically

OR...

May 15, 2012 41

template<class T> struct atomic;template<> struct atomic< integral >;template<class T> struct atomic<T*>;

template <class T> struct atomic{ T fetch_add(T, memory_order = memory_order_seq_cst); T operator++(int);};

static atomic<int> count;

count++;// orcount.fetch_add(memory_order_acq_rel);

do { next = (was + 1) % length;} while (!count.compare_exchange_weak(was, next));

May 15, 2012 42

template<class T> struct atomic;template<> struct atomic< integral >;template<class T> struct atomic<T*>;

template <class T> struct atomic{ bool compare_exchange_weak(T&, T, memory_order = memory_order_seq_cst); bool compare_exchange_weak(T&, T, memory_order, memory_order);};

// Lock Free Stack...

void push(T val){}

T pop(){}

May 15, 2012 43

void push(Val val){ //...?}

May 15, 2012 44

void push(Val val){ Node * newhead = new Node(val);}

May 15, 2012 45

void push(Val val){ Node * newhead = new Node(val); Node * oldhead = stack.head;}

May 15, 2012 46

void push(Val val){ Node * newhead = new Node(val); Node * oldhead = stack.head; newhead->next = oldhead;}

May 15, 2012 47

void push(Val val){ Node * newhead = new Node(val); Node * oldhead = stack.head; newhead->next = oldhead; stack.head = newhead;}

May 15, 2012 48

void push(Val val){ Node * newhead = new Node(val); Node * oldhead = stack.head;

do { next = was + 1; } while (!count.compare_exchange_weak(was, next));}

May 15, 2012 49

void push(Val val){ Node * newhead = new Node(val); Node * oldhead = stack.head;

do { newhead->next = oldhead; } while(!stack.head.compare_exchange_weak(oldhead, newhead));}

May 15, 2012 50

void push(Val val){ Node * newhead = new Node(val); Node * oldhead = stack.head.load(relaxed);

do { newhead->next = oldhead; } while(!stack.head.compare_exchange_weak(oldhead, newhead, release));}

May 15, 2012 51

Val pop(){ //...?}

May 15, 2012 52

Val pop(){ Node * oldhead = stack.head;}

May 15, 2012 53

Val pop(){ Node * oldhead = stack.head; Node * newhead = oldhead->next;}

May 15, 2012 54

Val pop(){ Node * oldhead = stack.head; if (oldhead == NULL) throw StackEmpty(); Node * newhead = oldhead->next;}

May 15, 2012 55

Val pop(){ Node * oldhead = stack.head; if (oldhead == NULL) throw StackEmpty(); Node * newhead = oldhead->next;}

May 15, 2012 56

Val pop(){ Node * oldhead = stack.head; if (oldhead == NULL) throw StackEmpty(); Node * newhead = oldhead->next;}

May 15, 2012 57

Val pop(){ Node * oldhead = stack.head.load(acquire); do { if (!oldhead) throw StackEmpty(); newhead = oldhead->next; } while (!head.compare_exchange_weak(oldhead, newhead, acq_rel));

Val val = oldhead->val; recycle(oldhead); return val;}

May 15, 2012 58

Val pop(){ Node * oldhead = stack.head.load(acquire); do { if (!oldhead) throw StackEmpty(); newhead = oldhead->next; } while (!head.compare_exchange_weak(oldhead, newhead, acq_rel));

Val val = oldhead->val; recycle(oldhead); return val;}

May 15, 2012 59

Val pop(){ Node * oldhead = stack.head.load(acquire); do { if (!oldhead) throw StackEmpty(); newhead = oldhead->next; } while (!head.compare_exchange_weak(oldhead, newhead, acq_rel));

Val val = oldhead->val; recycle(oldhead); return val;}

May 15, 2012 60

Val pop(){ Node * oldhead = stack.head.load(acquire); do { if (!oldhead) throw StackEmpty(); newhead = oldhead->next; } while (!head.compare_exchange_weak(oldhead, newhead, acq_rel));

Val val = oldhead->val; recycle(oldhead); return val;}

May 15, 2012 61

Val pop(){ Node * oldhead = stack.head.load(acquire); do { if (!oldhead) throw StackEmpty(); newhead = oldhead->next; } while (!head.compare_exchange_weak(oldhead, newhead, acq_rel));

Val val = oldhead->val; recycle(oldhead); return val;}

May 15, 2012 62

Val pop(){ Node * oldhead = stack.head.load(acquire); do { if (!oldhead) throw StackEmpty(); newhead = oldhead->next; } while (!head.compare_exchange_weak(oldhead, newhead, acq_rel));

Val val = oldhead->val; recycle(oldhead); return val;}

May 15, 2012 63

Val pop(){ Node * oldhead = stack.head.load(acquire); do { if (!oldhead) throw StackEmpty(); newhead = oldhead->next; } while (!head.compare_exchange_weak(oldhead, newhead, acq_rel));

Val val = oldhead->val; recycle(oldhead); return val;}

May 15, 2012 64

Val pop(){ Node * oldhead = stack.head.load(acquire); do { if (!oldhead) throw StackEmpty(); newhead = oldhead->next; } while (!head.compare_exchange_weak(oldhead, newhead, acq_rel));

Val val = oldhead->val; recycle(oldhead); return val;}

ABAABA

// compare_exchange: if (count == was) count = next; else was = count;

// compare_exchange: if (count == was) count = next; else was = count;

// compare_exchange: if (count untouched) count = next; else was = count;

// compare_exchange: if (count untouched) count = next; else was = count;

More Scary Things 42

memory_order_consume

False Sharing

Bonus Question…

65May 15, 2012

More Scary Things66May 15, 2012

// Thread 1:r1 = y.load(relaxed);x.store(r1, relaxed);

assert(r1 == 42);

// Thread 2:r2 = x.load(relaxed);y.store(42, relaxed);

assert(r2 == 42);

42

More Scary Things67May 15, 2012

foo = 42; // ‘publish’ p = &foo;

memory_order_consume

foo| | |bar

| | p |

int y = bar + 17; if (p != NULL) int x = *p;

assert(x == 42);

More Scary Things68May 15, 2012

prev.compare_exchange(...);

False Sharing

next | prev |

next.load();

struct Node{ atomic<Node*> next; atomic<Node*> prev; //…}

More Scary Things69May 15, 2012

Bonus Question…

• atomic<T> is implemented with locks if/when T is too large to be natively atomic.• locks use acquire/release semantics• atomics offer sequential consistency

How do you implement sequential consistency given only acquire/release?

(Note, that acq + rel != seq_cst, for example, recall…)

70May 15, 2012

x 1st

y 2nd

y == 0 implies

y 1st

x 2nd

x == 0 implies

Sequential Consistency vs Acquire/Release vs Relaxed

∴ z != 0

71May 15, 2012

Thanks to…

Michael Wong, IBM Toronto Lab, michaelw@ca.ibm.com

Hans Boehm, Hewlett-Packard, http://www.hpl.hp.com/personal/Hans_Boehm

Joe Pfeiffer, New Mexico State University, http://www.cs.nmsu.edu/~pfeiffer

Bartosz Milewski, http://bartoszmilewski.com

Anthony Williams, http://www.justsoftwaresolutions.co.uk/

Dmitriy V’jukov, http://www.1024cores.net/

David Hilley, http://www.thegibson.org/blog/

Jeremy Manson, http://jeremymanson.blogspot.com

72May 15, 2012

73May 15, 2012

Use Locks!

74May 15, 2012

from Abstrusegoose.com - licensed under CC BY-NC 3.0 ^atomics