CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND...
Transcript of CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND...
![Page 1: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/1.jpg)
CS5412:
REPLICATION, CONSISTENCY
AND CLOCKS
Ken Birman
1 CS5412 Spring 2014 (Cloud Computing: Birman)
Lecture X
![Page 2: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/2.jpg)
Recall that clouds have tiers
CS5412 Spring 2014 (Cloud Computing: Birman)
2
Up to now our focus has been on client systems and the network, and the way that the cloud has reshaped both
We looked very superficially at the tiered structure of the cloud itself
Tier 1: Very lightweight, responsive “web page builders” that can also route (or handle) “web services” method invocations. Limited to “soft state”.
Tier 2: (key,value) stores and similar services that support tier 1. Basically, various forms of caches.
Inner tiers: Online services that handle requests not handled in the first tier. These can store persistent files, run transactional services. But we shield them from load.
Back end: Runs offline services that do things like indexing the web overnight for use by tomorrow morning’s tier-1 services.
![Page 3: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/3.jpg)
Replication
CS5412 Spring 2014 (Cloud Computing: Birman)
3
A central feature of the cloud
To handle more work, make more copies
In the first tier, which is highly elastic, data center management layer pre-positions inactive copies of virtual machines for the services we might run
Exactly like installing a program on some machine
If load surges, creating more instances just entails
Running more copies on more nodes
Adjusting the load-balancer to spray requests to new nodes
If load drops... just kill the unwanted copies!
Little or no warning. Discard any “state” they created locally.
![Page 4: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/4.jpg)
Replication is about keeping copies
CS5412 Spring 2014 (Cloud Computing: Birman)
4
The term may sound fancier but the meaning isn’t
Whenever we have many copies of something we say
that we’ve replicated that thing
But usually replica does connote “identical”
Instead of replication we use the term redundancy for things
like alternative communication paths (e.g. if we have two
distinct TCP connections from some client system to the cloud)
Redundant things might not be identical. Replicated things
usually play identical roles and have equivalent data.
![Page 5: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/5.jpg)
Things we can replicate in a cloud
CS5412 Spring 2014 (Cloud Computing: Birman)
5
Files or other forms of data used to handle requests
If all our first tier systems replicate the data needed for end-user requests, then they can handle all the work!
Two cases to consider: in one the data itself is “write once” like a photo. Either you have a replica, or don’t
In the other the data evolves over time, like the current inventory count for the latest iPad in the Apple store
Computation
Here we replicate some request and then the work of computing the answer can be spread over multiple programs in the cloud
We benefit from parallelism by getting a faster answer
Can also provide fault-tolerance
![Page 6: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/6.jpg)
Many things “map” to replication
CS5412 Spring 2014 (Cloud Computing: Birman)
6
As we just saw, data (or databases), computation
Fault-tolerant request processing
Coordination and synchronization (e.g. “who’s in charge of the air traffic control sector over Paris?”)
Parameters and configuration data
Security keys and lists of possible users and the rules for who is permitted to do what
Membership information in a DHT or some other service that has many participants
![Page 7: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/7.jpg)
So... focus on replication!
CS5412 Spring 2014 (Cloud Computing: Birman)
7
If we can get replication right, we’ll be on the road
to a highly assured cloud infrastructure
Key is to understand what it means to correctly
replicate data at cloud scale...
... then once we know what we want to do, to find
scalable ways to implement needed abstraction(s)
![Page 8: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/8.jpg)
Concept of “consistency”
CS5412 Spring 2014 (Cloud Computing: Birman)
8
We would say that a replicated entity behaves in a
consistent manner if mimics the behavior of a non-
replicated entity
E.g. if I ask it some question, and it answers, and then
you ask it that question, your answer is either the same
or reflects some update to the underlying state
Many copies but acts like just one
An inconsistent service is one that seems “broken”
![Page 9: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/9.jpg)
Consistency lets us ignore implementation
A consistent distributed system will often have many
components, but users observe behavior
indistinguishable from that of a single-component
reference system
9
Reference Model Implementation
CS5412 Spring 2014 (Cloud Computing: Birman)
![Page 10: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/10.jpg)
Dangers of Inconsistency
Inconsistency causes bugs
Clients would never be able to trust servers… a free-for-all
Weak or “best effort” consistency?
Common in today’s cloud replication schemes
But strong security guarantees demand consistency
Would you trust a medical electronic-health records system or a bank that used “weak consistency” for better scalability?
10 My rent check bounced?
That can’t be right!
Jason Fane Properties 1150.00
Sept 2009 Tommy Tenant
CS5412 Spring 2014 (Cloud Computing: Birman)
![Page 11: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/11.jpg)
Leslie Lamport’s insight
CS5412 Spring 2014 (Cloud Computing: Birman)
11
To formalize notions of consistency, start
by formalizing notions of time
Once we do this we can be rigorous about notions
like “before” or “after” or “simultaneously”
If we try to write down conditions for correct replication
these kinds of terms often arise
![Page 12: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/12.jpg)
What time is it?
In distributed system we need practical ways to
deal with time
E.g. we may need to agree that update A occurred
before update B
Or offer a “lease” on a resource that expires at time
10:10.0150
Or guarantee that a time critical event will reach all
interested parties within 100ms
CS5412 Spring 2014 (Cloud Computing: Birman)
12
![Page 13: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/13.jpg)
But what does time “mean”?
Time on a global clock?
E.g. on Cornell clock tower?
... or perhaps on a GPS receiver?
… or on a machine’s local clock
But was it set accurately?
And could it drift, e.g. run fast or slow?
What about faults, like stuck bits?
… or could try to agree on time
CS5412 Spring 2014 (Cloud Computing: Birman)
13
![Page 14: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/14.jpg)
Lamport’s approach
Leslie Lamport suggested that we should reduce
time to its basics
Time lets a system ask “Which came first: event A or
event B?”
In effect: time is a means of labeling events so that…
If A happened before B, TIME(A) < TIME(B)
If TIME(A) < TIME(B), A happened before B
CS5412 Spring 2014 (Cloud Computing: Birman)
14
![Page 15: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/15.jpg)
Drawing time-line pictures:
p
m
sndp(m)
q
rcvq(m) delivq(m)
D
CS5412 Spring 2014 (Cloud Computing: Birman)
15
![Page 16: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/16.jpg)
Drawing time-line pictures:
A, B, C and D are “events”.
Could be anything meaningful to the application
So are snd(m) and rcv(m) and deliv(m)
What ordering claims are meaningful?
p
m
A
C
B
sndp(m)
q
rcvq(m) delivq(m)
D
CS5412 Spring 2014 (Cloud Computing: Birman)
16
![Page 17: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/17.jpg)
Drawing time-line pictures:
A happens before B, and C before D
“Local ordering” at a single process
Write and
p
q
m
A
C
B
rcvq(m) delivq(m)
sndp(m)
BAp
DCq
D
CS5412 Spring 2014 (Cloud Computing: Birman)
17
![Page 18: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/18.jpg)
Drawing time-line pictures:
sndp(m) also happens before rcvq(m)
“Distributed ordering” introduced by a message
Write
p
q
m
A
C
B
rcvq(m) delivq(m)
sndp(m)
)m(rcv)m(snd q
M
p
D
CS5412 Spring 2014 (Cloud Computing: Birman)
18
![Page 19: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/19.jpg)
Drawing time-line pictures:
A happens before D
Transitivity: A happens before sndp(m), which happens
before rcvq(m), which happens before D
p
q
m
D
A
C
B
rcvq(m) delivq(m)
sndp(m)
CS5412 Spring 2014 (Cloud Computing: Birman)
19
![Page 20: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/20.jpg)
p
q
m
D
A
C
B
rcvq(m) delivq(m)
sndp(m)
Drawing time-line pictures:
B and D are concurrent
Looks like B happens first, but D has no way to know. No information flowed…
CS5412 Spring 2014 (Cloud Computing: Birman)
20
![Page 21: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/21.jpg)
Happens before “relation”
We say that “A happens before B”, written AB, if
1. APB according to the local ordering, or
2. A is a snd and B is a rcv and AMB, or
3. A and B are related under transitive closure of rules (1) and (2)
Notice that, so far, this is just a mathematical
notation, not a “systems tool”
Given a trace of what happened in a system we
could use these tools to talk about the trace
But need a way to “implement” this idea
CS5412 Spring 2014 (Cloud Computing: Birman)
21
![Page 22: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/22.jpg)
Logical clocks
A simple tool that can capture parts of the happens
before relation
First version: uses just a single integer
Designed for big (64-bit or more) counters
Each process p maintains LTp, a local counter
A message m will carry LTm
CS5412 Spring 2014 (Cloud Computing: Birman)
22
![Page 23: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/23.jpg)
Rules for managing logical clocks
When an event happens at a process p it increments LTp.
Any event that matters to p
Normally, also snd and rcv events (since we want receive to occur “after”
the matching send)
When p sends m, set
LTm = LTp
When q receives m, set
LTq = max(LTq, LTm)+1
CS5412 Spring 2014 (Cloud Computing: Birman)
23
![Page 24: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/24.jpg)
Time-line with LT annotations
LT(A) = 1, LT(sndp(m)) = 2, LT(m) = 2
LT(rcvq(m))=max(1,2)+1=3, etc…
p
q
m
D
A
C
B
rcvq(m) delivq(m)
sndp(m)
LTq 0 0 0 1 1 1 1 3 3 3 4 5 5
LTp 0 1 1 2 2 2 2 2 2 3 3 3 3
CS5412 Spring 2014 (Cloud Computing: Birman)
24
![Page 25: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/25.jpg)
Logical clocks
If A happens before B, AB,
then LT(A)<LT(B)
But converse might not be true:
If LT(A)<LT(B) can’t be sure that AB
This is because processes that don’t communicate still
assign timestamps and hence events will “seem” to have
an order
CS5412 Spring 2014 (Cloud Computing: Birman)
25
![Page 26: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/26.jpg)
Can we do better?
One option is to use vector clocks
Here we treat timestamps as a list
One counter for each process
Rules for managing vector times differ from what
did with logical clocks
CS5412 Spring 2014 (Cloud Computing: Birman)
26
![Page 27: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/27.jpg)
History of vector clocks?
CS5412 Spring 2014 (Cloud Computing: Birman)
27
Originated in work at UCLA on file systems that
allowed updates from multiple sources concurrently
Jerry Popek’s FICUS system
Today version systems (e.g. SVN, CVS) use the idea
Also gradually adopted in distributed systems
Most of the “formal” work was done by Fidge and
Mattern in Europe, long after idea was in wide use
![Page 28: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/28.jpg)
Vector clocks
Clock is a vector: e.g. VT(A)=[1, 0]
We’ll just assign p index 0 and q index 1
Vector clocks require either agreement on the numbering, or that the actual process id’s be included with the vector
Rules for managing vector clock
When event happens at p, increment VTp[indexp]
Normally, also increment for snd and rcv events
When sending a message, set VT(m)=VTp
When receiving, set VTq=max(VTq, VT(m))
CS5412 Spring 2014 (Cloud Computing: Birman)
28
![Page 29: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/29.jpg)
Time-line with VT annotations
p
q
m
D
A
C
B
rcvq(m) delivq(m)
sndp(m)
VTq 0 0
0 0
0 0
0 1
0 1
0 1
0 1
2 2
2 2
2 2
23
2 3
2 4
VTp 0 0
1 0
1 0
2 0
2 0
2 0
2 0
2 0
2 0
3 0
30
3 0
3 0
VT(m)=[2,0]
Could also be [1,0] if we decide not to increment the clock on a snd event. Decision depends on how the timestamps will be used.
CS5412 Spring 2014 (Cloud Computing: Birman)
29
![Page 30: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/30.jpg)
Rules for comparison of VTs
We’ll say that VTA ≤ VTB if
I, VTA[i] ≤ VTB[i]
And we’ll say that VTA < VTB if
VTA ≤ VTB but VTA ≠ VTB
That is, for some i, VTA[i] < VTB[i]
Examples?
[2,4] ≤ [2,4]
[1,3] < [7,3]
[1,3] is “incomparable” to [3,1]
CS5412 Spring 2014 (Cloud Computing: Birman)
30
![Page 31: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/31.jpg)
Time-line with VT annotations
VT(A)=[1,0]. VT(D)=[2,4]. So VT(A)<VT(D)
VT(B)=[3,0]. So VT(B) and VT(D) are incomparable
p
q
m
D
A
C
B
rcvq(m) delivq(m)
sndp(m)
VTq 0 0
0 0
0 0
0 1
0 1
0 1
0 1
2 2
2 2
2 2
23
2 3
2 4
VTp 0 0
1 0
1 0
2 0
2 0
2 0
2 0
2 0
2 0
3 0
30
3 0
3 0
VT(m)=[2,0]
CS5412 Spring 2014 (Cloud Computing: Birman)
31
![Page 32: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/32.jpg)
Vector time and happens before
If AB, then VT(A)<VT(B)
Write a chain of events from A to B
Step by step the vector clocks get larger
If VT(A)<VT(B) then AB
Two cases: if A and B both happen at same process p, trivial
If A happens at p and B at q, can trace the path back by
which q “learned” VTA[p]
Otherwise A and B happened concurrently
CS5412 Spring 2014 (Cloud Computing: Birman)
32
![Page 33: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/33.jpg)
Temporal distortions
Things can be complicated because we can’t predict
Message delays (they vary constantly)
Execution speeds (often a process shares a machine
with many other tasks)
Timing of external events
Lamport looked at this question too
CS5412 Spring 2014 (Cloud Computing: Birman)
33
![Page 34: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/34.jpg)
Temporal distortions
What does “now” mean?
p 0 a
f
e
p 3
b
p 2
p 1 c
d
CS5412 Spring 2014 (Cloud Computing: Birman)
34
![Page 35: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/35.jpg)
Temporal distortions
What does “now” mean?
p 0 a
f
e
p 3
b
p 2
p 1 c
d
CS5412 Spring 2014 (Cloud Computing: Birman)
35
![Page 36: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/36.jpg)
Temporal distortions
Timelines can “stretch”…
… caused by scheduling effects, message
delays, message loss…
p 0 a
f
e
p 3
b
p 2
p 1 c
d
CS5412 Spring 2014 (Cloud Computing: Birman)
36
![Page 37: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/37.jpg)
Temporal distortions
Timelines can “shrink”
E.g. something lets a machine speed up
p 0 a
f
e
p 3
b
p 2
p 1 c
d
CS5412 Spring 2014 (Cloud Computing: Birman)
37
![Page 38: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/38.jpg)
Temporal distortions
Cuts represent instants of time.
But not every “cut” makes sense
Black cuts could occur but not gray ones.
p 0 a
f
e
p 3
b
p 2
p 1 c
d
CS5412 Spring 2014 (Cloud Computing: Birman)
38
![Page 39: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/39.jpg)
Consistent cuts and snapshots
Idea is to identify system states that “might” have
occurred in real-life
Need to avoid capturing states in which a message is
received but nobody is shown as having sent it
This the problem with the gray cuts
CS5412 Spring 2014 (Cloud Computing: Birman)
39
![Page 40: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/40.jpg)
Temporal distortions
Red messages cross gray cuts “backwards”
p 0 a
f
e
p 3
b
p 2
p 1 c
d
CS5412 Spring 2014 (Cloud Computing: Birman)
40
![Page 41: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/41.jpg)
Temporal distortions
Red messages cross gray cuts “backwards”
In a nutshell: the cut includes a message that
“was never sent”
p 0 a
e
p 3
b
p 2
p 1 c
CS5412 Spring 2014 (Cloud Computing: Birman)
41
![Page 42: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/42.jpg)
Application: Deadlock detection
p worries: perhaps we have a deadlock
p is waiting for q, so sends “what’s your state?”
q, on receipt, is waiting for r, so sends the same
question… and r for s…. And s is waiting on p.
CS5412 Spring 2014 (Cloud Computing: Birman)
42
![Page 43: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/43.jpg)
Suppose we detect this state
We see a cycle…
… but is it a deadlock?
p q
s r
Waiting for
Waiting for
Waiting for Waiting for
CS5412 Spring 2014 (Cloud Computing: Birman)
43
![Page 44: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/44.jpg)
Phantom deadlocks!
Suppose system has a very high rate of locking.
Then perhaps a lock release message “passed” a
query message
i.e. we see “q waiting for r” and “r waiting for s” but in fact,
by the time we checked r, q was no longer waiting!
In effect: we checked for deadlock on a gray cut – an
inconsistent cut.
CS5412 Spring 2014 (Cloud Computing: Birman)
44
![Page 45: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/45.jpg)
One solution is to “freeze” the system
X
Y
Z
A
B
STOP!
CS5412 Spring 2014 (Cloud Computing: Birman)
45
![Page 46: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/46.jpg)
One solution is to “freeze” the system
X
Y
Z
A
B
STOP!
Ok…
Yes sir!
I’ll be late!
Was I speeding?
Sigh…
CS5412 Spring 2014 (Cloud Computing: Birman)
46
![Page 47: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/47.jpg)
One solution is to “freeze” the system
X
Y
Z
A
B
Sorry to trouble you, folks. I just
need a status snapshot, please
CS5412 Spring 2014 (Cloud Computing: Birman)
47
![Page 48: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/48.jpg)
One solution is to “freeze” the system
X
Y
Z
A
B
No problem
Hey, doesn’t a guy have a right to privacy?
Done…
Here you go…
Sigh…
CS5412 Spring 2014 (Cloud Computing: Birman)
48
![Page 49: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/49.jpg)
One solution is to “freeze” the system
X
Y
Z
A
B
Ok, you can go now
CS5412 Spring 2014 (Cloud Computing: Birman)
49
![Page 50: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/50.jpg)
Why does it work?
When we check bank accounts, or check for deadlock, the system is idle
So if “P is waiting for Q” and “Q is waiting for R” we really mean “simultaneously”
But to get this guarantee we did something very costly because no new work is being done!
CS5412 Spring 2014 (Cloud Computing: Birman)
50
![Page 51: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/51.jpg)
Consistent cuts and snapshots
Goal is to draw a line across the system state such
that
Every message “received” by a process is shown as
having been sent by some other process
Some pending messages might still be in communication
channels
And we want to do this while running
CS5412 Spring 2014 (Cloud Computing: Birman)
51
![Page 52: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/52.jpg)
Turn idea into an algorithm
To start a new snapshot, pi …
Builds a message: “Pi is initiating snapshot k”.
The tuple (pi, k) uniquely identifies the snapshot
Writes down its own state
Starts recording incoming messages on all channels
CS5412 Spring 2014 (Cloud Computing: Birman)
52
![Page 53: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/53.jpg)
Turn idea into an algorithm
Now pi tells its neighbors to start a snapshot
In general, on first learning about snapshot (pi, k), px
Writes down its state: px’s contribution to the snapshot
Starts “tape recorders” for all communication channels
Forwards the message on all outgoing channels
Stops “tape recorder” for a channel when a snapshot message for (pi, k) is received on it
Snapshot consists of all the local state contributions and all the tape-recordings for the channels
CS5412 Spring 2014 (Cloud Computing: Birman)
53
![Page 54: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/54.jpg)
Chandy/Lamport
Outgoing wave of requests… incoming wave of
snapshots and channel state
Snapshot ends up accumulating at the initiator, pi
Algorithm doesn’t tolerate process failures or
message failures.
CS5412 Spring 2014 (Cloud Computing: Birman)
54
![Page 55: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/55.jpg)
Chandy/Lamport
p
q r
s
t
u
v
w
x y
z
A network
CS5412 Spring 2014 (Cloud Computing: Birman)
55
![Page 56: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/56.jpg)
Chandy/Lamport
p
q r
s
t
u
v
w
x y
z
A network
I want to start a
snapshot
CS5412 Spring 2014 (Cloud Computing: Birman)
56
![Page 57: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/57.jpg)
Chandy/Lamport
p
q r
s
t
u
v
w
x y
z
A network
p records local state
CS5412 Spring 2014 (Cloud Computing: Birman)
57
![Page 58: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/58.jpg)
Chandy/Lamport
p
q r
s
t
u
v
w
x y
z
A network
p starts monitoring
incoming channels
CS5412 Spring 2014 (Cloud Computing: Birman)
58
![Page 59: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/59.jpg)
Chandy/Lamport
p
q r
s
t
u
v
w
x y
z
A network
“contents of channel p-y”
CS5412 Spring 2014 (Cloud Computing: Birman)
59
![Page 60: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/60.jpg)
Chandy/Lamport
p
q r
s
t
u
v
w
x y
z
A network
p floods message on
outgoing channels…
CS5412 Spring 2014 (Cloud Computing: Birman)
60
![Page 61: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/61.jpg)
Chandy/Lamport
p
q r
s
t
u
v
w
x y
z
A network
CS5412 Spring 2014 (Cloud Computing: Birman)
61
![Page 62: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/62.jpg)
Chandy/Lamport
p
q r
s
t
u
v
w
x y
z
A network
q is done
CS5412 Spring 2014 (Cloud Computing: Birman)
62
![Page 63: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/63.jpg)
Chandy/Lamport
p
q r
s
t
u
v
w
x y
z
A network
q
CS5412 Spring 2014 (Cloud Computing: Birman)
63
![Page 64: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/64.jpg)
Chandy/Lamport
p
q r
s
t
u
v
w
x y
z
A network
q
CS5412 Spring 2014 (Cloud Computing: Birman)
64
![Page 65: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/65.jpg)
Chandy/Lamport
p
q r
s
t
u
v
w
x y
z
A network
q
z s
CS5412 Spring 2014 (Cloud Computing: Birman)
65
![Page 66: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/66.jpg)
Chandy/Lamport
p
q r
s
t
u
v
w
x y
z
A network
q
v
z
x
u
s
CS5412 Spring 2014 (Cloud Computing: Birman)
66
![Page 67: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/67.jpg)
Chandy/Lamport
p
q r
s
t
u
v
w
x y
z
A network
q
v
w
z
x
u
s
y
r
CS5412 Spring 2014 (Cloud Computing: Birman)
67
![Page 68: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/68.jpg)
Chandy/Lamport
p
q r
s
t
u
v
w
x y
z
A snapshot of a network
q
x
u
s
v
r
t
w
p
y
z
Done!
CS5412 Spring 2014 (Cloud Computing: Birman)
68
![Page 69: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/69.jpg)
Chandy/Lamport “snapshot”
CS5412 Spring 2014 (Cloud Computing: Birman)
69
Once we collect the state snapshots plus the channel
contents we have a consistent cut from the system
It “could” have occured as a concurrent instant in the
system execution (although in fact, it obviously didn’t)
Processing such a snapshot requires understanding the
state in this form
But many algorithms use this pattern of messages
without necessarily writing down the whole state or
logging all the messages in the channels
![Page 70: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/70.jpg)
Relation to vector time?
CS5412 Spring 2014 (Cloud Computing: Birman)
70
In book the connection of consistent cuts to notion of logical time is explored
A consistent cut is a snapshot taken at a set of concurrent points in a system trace
In effect, all the members of the system concurrently write down their states
We can restate Chandy/Lamport to implement it precisely in this manner!
But out of time today, so we’ll leave that for you to read about in Chapter 10 of the text
![Page 71: CS5412: REPLICATION, CONSISTENCY AND CLOCKS - Replica… · CS5412: REPLICATION, CONSISTENCY AND CLOCKS Ken Birman ... Either you have a replica, or don’t In the other the data](https://reader034.fdocuments.us/reader034/viewer/2022052519/5f13f0a3e9d9d6243933a4e8/html5/thumbnails/71.jpg)
Conclusions
CS5412 Spring 2014 (Cloud Computing: Birman)
71
By formalizing notion of time we can build tools for thinking about fancier ideas such as consistency of replicated data
Today we looked more closely at time than at consistency.
We introduced idea of consistency to motivate need to look closely at time
But didn’t tie the logical or vector timestamp ideas back to implementation of replicated data
Next lectures will make this connection explicit