A Brief History Of Time · Consistency One important property of an atomic read/write shared memory...

168
A Brief History Of Time In Riak

Transcript of A Brief History Of Time · Consistency One important property of an atomic read/write shared memory...

Page 1: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

A Brief History Of TimeIn Riak

Page 2: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Time in Riak

✴Logical Time

✴Logical Clocks

✴ Implementation details

Page 3: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Mind the GapHow a venerable, established, simple data structure/algorithm

was botched multiple times.

Page 4: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Order of Events

✴Dynamo And Riak

✴Temporal and Logical Time

✴Logical Clocks of Riak Past

✴Now

Page 5: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Why Riak?

Page 6: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Scale Up

$$$Big Iron (still fails)

Page 7: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Scale Out

Commodity Servers CDNs, App servers

DATABASES!!

Page 8: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Fundamental Trade Off

• Lipton/Sandberg ’88

• Attiya/Welch ’94

• Gilbert/Lynch ’02

Low Latency/Availability: - Increased Revenue - User Engagement

Strong Consistency:!- Easier for Programmers

- Less user “surprise”

Page 9: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

ConsistencyThere must exist a total order on all operations

such that each operation looks as if it were completed at a single instant. This is equivalent to requiring requests of

the distributed shared memory to act as if they were executing on a single node, responding to operations one at a time.

!

--Gilbert & Lynch

Page 10: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

ConsistencyOne important property of an atomic read/write shared memory is that

any read operation that begins after a write operation completes must return that value, or the result of a later write operation. This is the consistency guarantee that generally provides

the easiest model for users to understand, and is most convenient for those attempting to design a client application that uses the distributed service

!

--Gilbert & Lynch

Page 11: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

https://aphyr.com/posts/313-strong-consistency-models

Page 12: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Replica A Replica B Replica C

Client X Client Y

PUT “sue”PUT “bob”

NO!!!! :(

Consistent

Page 13: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

AvailabilityAny non-failing node can respond to any

request!!

--Gilbert & Lynch

Page 14: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Replica A Replica B Replica C

Client X Client Y

PUT “sue”PUT “bob”

NO!!!! :(

Consistent

Page 15: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Consensus for a total order of events

Page 16: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Requires a quorum

Page 17: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Coordination waits

Page 18: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Replica A Replica B Replica C

Client X Client Y

PUT “sue”PUT “bob”

Consistent

Page 19: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Client X put “BOB”

Client Y put “SUE”

Events put in a TOTAL ORDER

Page 20: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

https://aphyr.com/posts/313-strong-consistency-models

Page 21: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Eventual Consistency

Eventual consistency is a consistency model used in distributed computing that informally guarantees that, if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value. !!--Wikipedia!

Page 22: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Replica A Replica B Replica C

Client X Client Y

PUT “sue”

C’

PUT “bob”

A’ B’

Available

Page 23: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

AvailabilityWhen serving reads and writes matters more than consistency of data. Deferred consistency.

Page 24: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Fault Tolerance

Page 25: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Low Latency

Page 26: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Low Latency

Amazon found every 100ms of latency cost them 1% in sales.

Page 27: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Low Latency

Google found an extra 0.5 seconds in search page generation time dropped traffic by 20%.

Page 28: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Replica A Replica B Replica C

Client X Client Y

PUT “sue”

C’

PUT “bob”

A’ B’

Available

Page 29: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Optimistic replication

Page 30: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

No coordination - lower latency

Page 31: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Replica A Replica B Replica C

Client X Client Y

PUT “sue”PUT “bob”

Low Latency

[c1] “sue”

[c1] “sue”[a1] “bob”

Page 32: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

How Do We Order Updates?

Page 33: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

–Google Book Search p.148 “The Giant Anthology of Science Fiction”, edited by Leo Margulies and Oscar Jerome Friend, 1954

"'Time,' he said, 'is what keeps everything from happening at once.'"

Page 34: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Temporal Clocksposix time number line

Thursday, 1 January 1970

0 129880800 1394382600

Now-ishMy Birthday

Page 35: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Light Cone!By SVG version: K. Aainsqatsi at en.wikipediaOriginal PNG version: Stib at

en.wikipedia - Transferred from en.wikipedia to Commons.(Original text: self-made), CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=2210907

https://commons.wikimedia.org/w/index.php?curid=2210907
Page 36: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Physics Problem

4,148 km 14 ms Light 21 ms fibre

SF NY

PUT “bob” 1394382600000

PUT “sue” 1394382600020

Page 37: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

temporal clocks✴CAN

• A could NOT have caused B

• A could have caused B

✴CAN’T

• A caused B

Page 38: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

DynamoThe Shopping Cart

Page 39: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile
Page 40: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile
Page 41: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

1 CLIENT

2 REPLICAS

1 KEY

Page 42: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Optimistic replication

Page 43: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

No coordination - lower latency

Page 44: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

GETPUT

UPDATE

REPLICATE

Page 45: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

PUTPUT

Page 46: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Quorum

Page 47: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

GET

A

PUT

A

GET

B

PUT

B

TEMPORAL TIME

Page 48: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

>155196119890 155196118001

Timestamp - total order

Page 49: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile
Page 50: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

⨆

Logical clock - partial order

Page 51: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile
Page 52: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Clocks, Time, And the Ordering of Events

• Logical Time • Causality • A influenced B • A and B happened at the same

time • Per-process clocks, only tick when

something happensLeslie Lamport http://dl.acm.org/citation.cfm?id=359563

http://dl.acm.org/citation.cfm?id=359563
Page 53: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Detection of Mutual Inconsistency in

Distributed Systems

Version Vectors - updates to a data item

http://zoo.cs.yale.edu/classes/cs426/2013/bib/parker83detection.pdf

Page 54: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors or Vector Clocks?

http://haslab.wordpress.com/2011/07/08/version-vectors-are-not-vector-clocks/

version vectors - updates to a data item

http://haslab.wordpress.com/2011/07/08/version-vectors-are-not-vector-clocks/
Page 55: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Summary

• Distributed systems exist (scale out)

• There is a trade off (Consistency/Availability)

• To decide on a value we need to “order” updates

• Temporal time is inadequate

• Logical time can help

Page 56: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors

A CB

Page 57: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors

A CB

{a, 1}

{b, 1}

{c, 1}{a, 2}

[ ]{a, 2}, {b, 1}, {c, 1}

Page 58: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors

A CB

{a, 1}

{b, 1}

{c, 1}{a, 2}

[ ]{a, 2}, {b, 1}, {c, 1}

Page 59: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors

A CB

{a, 1} {b, 1} {c, 1}

{a, 2}

[ ]{a, 2}, {b, 1}, {c, 1}

Page 60: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors

A CB

{a, 1} {b, 1} {c, 1}

{a, 2}

[ ]{a, 2}, {b, 1}, {c, 1}

Page 61: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors

A CB

{a, 1} {b, 1} {c, 1}

{a, 2}

[ ]{a, 2}, {b, 1}, {c, 1}

Page 62: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors

[{a,2}, {b,1}, {c,1}]

Page 63: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors Update

[{a,2}, {b,1}, {c,1}]

Page 64: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors Update

[{a,2}, {b,2}, {c,1}]

Page 65: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors Update

[{a,2}, {b,3}, {c,1}]

Page 66: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors Update

[{a,2}, {b,3}, {c,2}]

Page 67: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors Descends

✴A descends B : A >= B

✴A has seen all that B has

✴A summarises at least the same history as B

Page 68: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors Descends

[{a,2}, {b,3}, {c,2}] [{a,2}, {b,3}, {c,2}]

[{a,2}, {b,3}, {c,2}] []

>=[{a,4}, {b,3}, {c,2}] [{a,2}, {b,3}, {c,2}]

Page 69: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors Dominates

✴A dominates B : A > B

✴A has seen all that B has, and at least one more event

✴A summarises a greater history than B

Page 70: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors Dominates

[{a,1}] []

>[{a,4}, {b,3}, {c,2}] [{a,2}, {b,3}, {c,2}]

[{a,5}, {b,3}, {c,5}, {d, 1}] [{a,2}, {b,3}, {c,2}]

Page 71: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors Concurrent

✴A concurrent with B : A | B

✴A does not descend B AND B does not descend A

✴A and B summarise disjoint events

✴A contains events unseen by B AND B contains events unseen by A

Page 72: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors Concurrent

[{a,1}] [{b,1}]

|[{a,4}, {b,3}, {c,2}] [{a,2}, {b,4}, {c,2}]

[{a,5}, {b,3}, {c,5}, {d, 1}] [{a,2}, {b,4}, {c,2}, {e,1}]

Page 73: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile
Page 74: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

happens before

concurrent ——— divergent

convergent

Logical Clocks

Page 75: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors Merge

✴A merge with B : A ⊔ B

✴A ⊔ B = C

✴C >= A and C >= B

✴ If A | B C > A and C > B

✴C summarises all events in A and B

✴Pairwise max of counters in A and B

Page 76: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors Merge

[{a,1}] [{b,1}]

⊔[{a,4}, {b,3}, {c,2}] [{a,2}, {b,4}, {c,2}]

[{a,5}, {b,3}, {c,5}, {d, 1}] [{a,2}, {b,4}, {c,2}, {e,1}]

Page 77: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Version Vectors Merge

[{a,1}{b,2}]

[{a,4}, {b,4}, {c,2}]

[{a,5}, {b,3}, {c,5}, {d, 1},{e,1}]

Page 78: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Syntactic Merging

✴Discarding “seen” information

✴Retaining concurrent values

✴Merging divergent clocks

Page 79: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Temporal vs Logical

[{a,4}, {b,3}, {c,2}]

[{a,2}, {b,3}, {c,2}]

A

B

“Bob”

“Sue”

?

Page 80: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Temporal vs Logical

[{a,4}, {b,3}, {c,2}]

[{a,2}, {b,3}, {c,2}]

A

B

“Bob”

“Sue”

Bob

Page 81: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Temporal vs Logical

1429533664000

A

B

“Bob”

“Sue”

?1429533662000

Page 82: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Temporal vs Logical

1429533664000

A

B

“Bob”

“Sue”

Bob?1429533662000

Page 83: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Temporal vs Logical

[{a,4}, {b,3}, {c,2}]

A

B

“Bob”

“Sue”

?[{a,2}, {b,4}, {c,2}]

Page 84: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Temporal vs Logical

[{a,4}, {b,3}, {c,2}]

A

B

“Bob”

“Sue”

[Bob, Sue][{a,2}, {b,4}, {c,2}]

Page 85: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Temporal vs Logical

1429533664000

A

B

“Bob”

“Sue”

?1429533664001

Page 86: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Temporal vs Logical

1429533664000

A

B

“Bob”

“Sue”

Sue?1429533664001

Page 87: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Summary

• Eventually Consistent Systems allow concurrent updates

• Temporal timestamps can’t capture concurrency

• Logical clocks (Version vectors) can

• Version Vectors are easy

Page 88: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

History Repeating“Those who cannot remember the past are condemned

to repeat it"

Page 89: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Terms• Local value - stored on disk at some replica!

• Incoming value - sent as part of a PUT or replication!

• Local clock - The Version Vector of the Local Value!

• Incoming clock - The Version Vector of the Incoming Value

Page 90: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Riak Version Vectors

Who’s the actor?

Page 91: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Riak 0.n Client Side IDs

• Client Code Provides ID

• Riak increments Clock at API boundary

• Riak syntactic merge and stores object

• Read, Resolve, Rinse, Repeat.

Page 92: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile
Page 93: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Client

Riak API

Riak Vnode

Page 94: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Riak Vnode

Page 95: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Conflict Resolution

• Client reads merged clock + sibling values

• sends new value + clock

• new clock descends old (eventually!)

• Store single value

Page 96: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Client Version Vector

What Level of Consistency Do We

Require?

Page 97: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

https://aphyr.com/posts/313-strong-consistency-models

Page 98: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

GET

A

PUT

A

GET

B

PUT

B

TEMPORAL TIME

Page 99: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

RYOW

• Invariant: strictly increasing events per actor.

• PW+PR > N

• Availability cost

• Bug made it impossible!

Page 100: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Client VClock

• Read not_found []

• store “bob” [{c, 1}]

• read “bob” [{c, 1}]

• store [“bob”, “sue”] [{c, 2}]

Page 101: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Client VClock

• Read not_found [] • store “bob” [{c, 1}] • read not_found [] • store “sue” [{c, 1}]

Page 102: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Riak Vnode

Page 103: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Client VClock

• If local clock: ([{c, 1}]) descendsincoming clock: ([{c,1}])

• discard incoming value

Page 104: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Client Side ID RYOW

• Read a Stale clock

• Re-issue the same OR lower event again

• No total order for a single actor

• Each event is not unique

• System discards as “seen” data that is new

Page 105: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Client Side IDs Bad

• Unique actor ID:: database invariant enforced by client!

• Actor Explosion (Charron-Bost)

• No. Entries == No. Actors

• Client Burden

• RYOW required - Availability Cost

Page 106: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Riak Version Vectors

Who’s the actor?

Page 107: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode Version Vectors Riak 1.n

• No more Version Vector, just say Context

• The Vnode is the Actor

• Vnodes act serially

• Store the clock with the Key

• Coordinating Vnode, increments clock

• Deliberate false concurrency

Page 108: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VClocks False Concurrency

C1 C2

RIAK

GET Foo GET Foo

Page 109: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VClocks False Concurrency

C1 C2

RIAK

[{a,1},{b4}]->”bob”

[{a,1},{b4}]->”bob”

Page 110: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VClocks False Concurrency

C1 C2

RIAK

PUT [{a,1},{b,4}]=“Rita”

PUT [{a,1},{b,4}]=“Sue”

Page 111: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VClocks False Concurrency

C1 C2

PUTFSM1 PUTFSM2

VNODE Q

RITASUE

VNODE

Page 112: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VClocks False Concurrency

VNODE Q

RITA

VNODE a [{a,2},{b,4}]=“SUE”

[{a,1},{b,4}]

Page 113: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VClocks False Concurrency

VNODE Q

[{a,3},{b,4}]=[RITA,SUE]

VNODE a

[{a,2},{b,4}]=“SUE”

Page 114: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Client

Riak API

Riak API

Coordinator

Page 115: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VV Coordinator

Page 116: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VV - Coordinator

• If incoming clock descends local

• Increment clock

• Write incoming as sole value

• Replicate

Page 117: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VV - Coordinator• If incoming clock does not descend local

• Merge clocks

• Increment Clock

• Add incoming value as sibling

• Replicate

Page 118: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VV - Replica

Page 119: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VClock GOOD

• Far fewer actors

• Way simpler

• Empty context PUTs are siblings

Page 120: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VClock BAD

• Possible latency cost of forward

• No more idempotent PUTs

• Store a SET of siblings, not LIST

• Sibling Explosion

• As a result of too much false concurrency

Page 121: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Sibling Explosion

• False concurrency cost

• Many many siblings

• Large object

• Death

Page 122: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Sibling Explosion

• Data structure • Clock + Set of Values

• False Concurrency

Page 123: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Sibling Explosion

Page 124: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Sibling Explosion

C1 C2

RIAK

GET Foo GET Foo

Page 125: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Sibling Explosion

C1 C2

RIAK

not_found

not_found

Page 126: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Sibling Explosion

C1

RIAK

PUT []=“Rita”[{a,1}]->”Rita”

Page 127: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Sibling Explosion

C2

RIAK

PUT []=“Sue”[{a,2}]->[”Rita”, “Sue”]

Page 128: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Sibling Explosion

C1

RIAK

PUT [{a, 1}]=“Bob”

[{a,3}]->[”Rita”, “Sue”, “Bob”]

Page 129: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Sibling Explosion

C2

RIAK

PUT [{a,2}]=“Babs”[{a,4}]->[”Rita”, “Sue”, “Bob”, “Babs”]

Page 130: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VClock

• Trick to “dodge’ the Charron-Bost result

• Engineering, not academic

• Tested (quickchecked in fact!)

• “Action at a distance”

Page 131: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Dotted Version VectorsDotted Version Vectors: Logical Clocks for Optimistic Replication

http://arxiv.org/abs/1011.5808

http://arxiv.org/abs/1011.5808
Page 132: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VClocks + Dots Riak 2.n

• What even is a dot?

• That “event” we saw back a the start

A

{a, 1}

{a, 2}

Page 133: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Oh Dot all the Clocks

✴Data structure

• Clock + List of Dotted Values

[{{a, 1}, “bob”}, {{a, 2}, “Sue”}]

Page 134: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VClock✴ If incoming clock descends local

• Increment clock

• Get Last Event as dot (eg {a, 3})

• Write incoming as sole value + Dot

• Replicate

Page 135: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VClock✴ If incoming clock does not descend local

• Merge clocks

• Increment Clock

• Get Last Event as dot (eg {a, 3})

• Prune siblings!

• Add incoming value as sibling

• Replicate

Page 136: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Oh drop all the dots

✴Prune Siblings

• Remove any siblings who’s dot is seen by the incoming clock

• if Clock >= [Dot] drop Dotted value

Page 137: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VClocks[{a, 4}]

Rita

Sue

Babs

Bob

[{a, 3}]

Pete

Page 138: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VClocks + Dots

[{a, 4}]

Rita

Sue

Babs

Bob

[{a, 3}]

Pete{a,1}

{a,2}

{a,3}

{a,4}

Page 139: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VClocks + Dots

[{a, 4}]

Babs

[{a, 3}]

Pete

{a,4}

Page 140: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Vnode VClocks + Dots

[{a, 5}]

Babs

Pete

{a,4}

{a,5}

Page 141: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Dotted Version Vectors

✴ Action at a distance

✴ Correctly capture concurrency

✴ No sibling explosion

✴ No Actor explosion

Page 142: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

KV679

Page 143: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Riak Overview Read Repair. Deletes.

Page 144: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Replica A Replica B Replica C

Client X

PUT “bob”

Page 145: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Read Repair

Replica A Replica B Replica C

ClientGET

“Bob”

“Bob”

not_found

Page 146: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Read Repair

Replica A Replica B Replica C

Client

“Bob”

“Bob”!!!

Page 147: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Replica A Replica B Replica C

Client X

DEL ‘k’ [{a, 4}, {b, 3}]

C’

Page 148: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Replica A Replica B Replica C

C’

Del FSM

GET

Page 149: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Replica A Replica B Replica C

C’

Del FSM

GET A=Tombstone, B=Tombstone, C=not_found

Page 150: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Read Repair

Replica A Replica B Replica C

“Tombstone”!!!

Page 151: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Replica A Replica B Replica C

C’

Client

GET A=Tombstone, B=Tombstone, C=Tombstone

FSMnot_found

Page 152: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Replica A Replica B Replica C

C’

REAP

FSM

Page 153: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Replica A Replica B Replica C

Client X

PUT “sue” []

Sue [{a, 1}]

C’

Page 154: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Replica A Replica B Replica C

C’

Hinted Hand off tombstone

Page 155: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Replica A Replica B Replica C

Client

GET A=Sue[{a,1}], B=Sue[{a,1}], C=Tombstone [{a,4}, {b1}]

FSMnot_found

Ooops!

Page 156: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

KV679 Lingering Tombstone

• Write Tombstone

• One goes to fallback

• Read and reap primaries

• Add Key again

• Tombstone is handed off

• Tombstone clock dominates, data lost

Page 157: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

KV679 Other flavours

• Back up restore

• Failed local read (disk error, operator “error” etc)

Page 158: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

KV679 RYOW?

• Familiar

• History repeating

Page 159: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

KV679 Per Key Actor Epochs

• Every time a Vnode reads a local “not_found”

• Increment a vnode durable counter

• Make a new actor ID

• <<VnodeId, Epoch_Counter>>

Page 160: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

KV679 Per Key Actor Epochs

• Actor ID for the vnode remains long lived

• No actor explosion

• Each key gets a new actor per “epoch”

• Vnode increments highest “Epoch” for it’s Id

• <<VnodeId, Epoch>>

Page 161: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Replica A Replica B Replica C

Client

GET A=Sue[{a:2,1}], B=Sue[{a:2,1}], C=Tombstone [{a:1,4}, {b1}]

FSM

[Sue, tombstone]

Page 162: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Per Key Actor Epochs BAD

• More Actors (every time you delete and recreate a key _it_ gets a new actor)

• More computation (find highest epoch for actor in Version Vector)

Page 163: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Per Key Actor Epochs GOOD

• No silent dataloss

• No actor explosion

• Fully backwards/forward compatible

Page 164: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Are we there yet?

?

Page 165: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Summary

• Client side Version Vectors

• Invariants, availability, Charron-Bost

• Vnode Version Vectors

• Sibling Explosion

Page 166: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Summary

• Dotted Version Vectors

• “beat” Charron-Bost

• Per-Key-Actor-Epochs

• Vnodes can “forget” safely

Page 167: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Summary

• Temporal Clocks can’t track causality

• Logical Clocks can

Page 168: A Brief History Of Time · Consistency One important property of an atomic read/write shared memory is that any read operation that begins after a write operation completes must returnFile

Summary

• Version Vectors are EASY!

• (systems using) Version Vectors are HARD!

• Mind the Gap!


7.0 inch TFT LCD · 1 Most of the data traffic is read operation to get the finger or fingers position 2 Read operations do need an initial write operation. 3 Write operations are

7.0 inch TFT LCD · 1 Most of the data traffic is read operation to get the finger or fingers position 2 Read operations do need an initial write operation. 3 Write operations are

RFID Read/Write Module, Serial (#28440) RFID Read/Write ... · The Parallax RFID Read/Write Module USB version can be connected ... The RFID Read/Write Module is controlled by the

RFID Read/Write Module, Serial (#28440) RFID Read/Write ... · The Parallax RFID Read/Write Module USB version can be connected ... The RFID Read/Write Module is controlled by the

Read, Write, & Run Relay

Read, Write, & Run Relay

AN98080 Read/write devices based on the HITAG read/write ... · Document information AN98080 Read/write devices based on the HITAG read/write IC HTRC110 Rev. 3.0 — 23 March 2010

AN98080 Read/write devices based on the HITAG read/write ... · Document information AN98080 Read/write devices based on the HITAG read/write IC HTRC110 Rev. 3.0 — 23 March 2010

SA800 Series Diskette Storage Drive Double Density Design ... · System Evaluation . . . . 9.1 General 9.2 Voltage Margins 9.3 Seek Operation 9.4 Write Operation 9.5 Read Margins

SA800 Series Diskette Storage Drive Double Density Design ... · System Evaluation . . . . 9.1 General 9.2 Voltage Margins 9.3 Seek Operation 9.4 Write Operation 9.5 Read Margins

Cassandra Read/Write Paths

Cassandra Read/Write Paths

Read+Write+Fce NEW

Read+Write+Fce NEW

LPDDR5 Clocking and Read/Write Operation

LPDDR5 Clocking and Read/Write Operation

Timestamps for Serializabilityibl/450/pdf/view-on-line... · T1(10) Read A Read B Write A T2(20) Read C Read B Write B Read D T3(30) Read C Write C Write E Timestamps in parentheses

Timestamps for Serializabilityibl/450/pdf/view-on-line... · T1(10) Read A Read B Write A T2(20) Read C Read B Write B Read D T3(30) Read C Write C Write E Timestamps in parentheses

Implementation of Read Write Operation for AMBA AXI4

Implementation of Read Write Operation for AMBA AXI4

Chapter 4 Memory - Free Web Hosting 4.pdfMemory Cell Operation The select terminal ,selects a memory cell for a read or write operation The control terminal indicates read or write

Chapter 4 Memory - Free Web Hosting 4.pdfMemory Cell Operation The select terminal ,selects a memory cell for a read or write operation The control terminal indicates read or write

Mag Pro V4.1 Supports Car List · 2015-03-01 · mag pro v4.1 supports car list 1.read write read/write read/write read/write automatic read/write abarth 500 ii 1400 16v tjet 135cv

Mag Pro V4.1 Supports Car List · 2015-03-01 · mag pro v4.1 supports car list 1.read write read/write read/write read/write automatic read/write abarth 500 ii 1400 16v tjet 135cv

A Read and write

A Read and write

Read/Write Modules (RWM)

Read/Write Modules (RWM)

Turing Machine Read/Write – Move Left/Right BB Read/Write Head 001000 State Qi.

Turing Machine Read/Write – Move Left/Right BB Read/Write Head 001000 State Qi.

CSCI2510 Computer Organization Lecture 08: Cache ...mcyang/csci2510/2020F/Lec08 Cache...Recall: Cache Usage • Cache Read (or Write) Hit/Miss: The read (or write) operation can/cannot

CSCI2510 Computer Organization Lecture 08: Cache ...mcyang/csci2510/2020F/Lec08 Cache...Recall: Cache Usage • Cache Read (or Write) Hit/Miss: The read (or write) operation can/cannot

Memory and Programmable Logic Chapter 7. Introduction RAM: Random Access Memory ROM: Read Only Memory Write operation: Storing info into memory Read operation:

Memory and Programmable Logic Chapter 7. Introduction RAM: Random Access Memory ROM: Read Only Memory Write operation: Storing info into memory Read operation:

Read to Write Write to Read Slide Show by Vicky G. Mather.

Read to Write Write to Read Slide Show by Vicky G. Mather.

Read Write Now 5

Read Write Now 5

Read & Write Gold v10

Read & Write Gold v10