Incremental Consistency Guarantees - USENIX · Incremental Consistency Guarantees Dragos-Adrian...
Transcript of Incremental Consistency Guarantees - USENIX · Incremental Consistency Guarantees Dragos-Adrian...
Incremental Consistency Guarantees For Replicated Objects
Rachid Guerraoui, Matej Pavlovic, Dragos-Adrian Seredinschi
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
2
replicate
SOCIAL MEDIA APPLICATION
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
2
replicate
SOCIAL MEDIA APPLICATION
2220
17Number of recent eventson the user’s timeline
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
• Returns the correct data • Latency: ~200 ms • Can become unavailable
2
Strong Consistency
[CAP], [PACELC]
replicate
SOCIAL MEDIA APPLICATION
2220
17
22
Number of recent eventson the user’s timeline
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
• Latency: ~100 ms • High availability • Allows inconsistencies: can return
• Returns the correct data • Latency: ~200 ms • Can become unavailable
2
Strong Consistency
or or[CAP], [PACELC]
Weak Consistency
replicate
SOCIAL MEDIA APPLICATION
2220
17
22
2022 17
Number of recent eventson the user’s timeline
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
• Latency: ~100 ms • High availability • Allows inconsistencies: can return
• Returns the correct data • Latency: ~200 ms • Can become unavailable
2
Strong Consistency
or or[CAP], [PACELC]
Weak Consistency
replicate
SOCIAL MEDIA APPLICATION
2220
17
22
2022 17
Number of recent eventson the user’s timeline
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Neither model is ideal!
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
• Latency: ~100 ms • High availability • Allows inconsistencies: can return
• Returns the correct data • Latency: ~200 ms • Can become unavailable
2
Strong Consistency
or or[CAP], [PACELC]
Weak Consistency
replicate
SOCIAL MEDIA APPLICATION
2220
17
22
2022 17
Number of recent eventson the user’s timeline
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Neither model is ideal!
We use both models.
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
3
Multiple models1. Weak consistency
2. Strong consistency
100ms
300ms
20
22
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
3
Multiple models1. Weak consistency
2. Strong consistency
100ms
300ms
20
22
Dynamo [SOSP’07]
Pileus [SOSP’13]
Increasingly many systems exposemultiple consistency models:
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
3
Multiple models
1. Send multiple requests?
2. How to leverage individual responses?
3. Semantics?
4. …
1. Weak consistency
2. Strong consistency
100ms
300ms
20
22
Issues
Dynamo [SOSP’07]
Pileus [SOSP’13]
Increasingly many systems exposemultiple consistency models:
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
3
Multiple models
How do you program with ★ inconsistencies? ★ multiple values?
Problem1. Send multiple requests?
2. How to leverage individual responses?
3. Semantics?
4. …
1. Weak consistency
2. Strong consistency
100ms
300ms
20
22
Issues
Dynamo [SOSP’07]
Pileus [SOSP’13]
Increasingly many systems exposemultiple consistency models:
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
4
ABSTRACTION FOR REPLICATED OBJECTS
Weak
Consistency
Strong
Consistency
…
SOCIAL MEDIA APPLICATION
20
17
22
????
Consistency
…
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
4
ABSTRACTION FOR REPLICATED OBJECTS
Weak
Consistency
Strong
Consistency
…CORRECTABLE
SOCIAL MEDIA APPLICATION
20
17
22
????
Consistency
…
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
4
ABSTRACTION FOR REPLICATED OBJECTS
Weak
Consistency
Strong
Consistency
…CORRECTABLE
Incremental Consistency Guarantees (ICG)
provides
SOCIAL MEDIA APPLICATION
20
17
22
????
Consistency
…
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Correctables / Design
5
PROMISE
resolveasynchronously
➤ Starting point: Promises
➤ Placeholders for values
➤ Becoming mainstream
value
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Correctables / Design
5
CORRECTABLEPROMISE
resolveasynchronously
➤ Starting point: Promises
➤ Placeholders for values
➤ Becoming mainstream
value
value1
value2
valuen
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Correctables / Design
5
CORRECTABLEPROMISE
resolveasynchronously
PROGRESSIVELY STRONGER CONSISTENCY
(INCREMENTAL)
PROGRESSIVELY HIGHER LATENCY
➤ Starting point: Promises
➤ Placeholders for values
➤ Becoming mainstream
value
value1
value2
valuen
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Consistency Models are Complementary
6
Con
sisten
cy
Weak
Strong
LowHigh PerformanceEventual
Linearizable
Sequential
Causal
(Ideal protocol)
(Worst protocol)
INCREM
ENTAL
Weak consistency: ★ Fast ★ (Often correct)
Strong consistency: ★ Slower ★ (Correct with certainty)Weak
Consistency
Strong Consistency
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Consistency Models are Complementary
6
Con
sisten
cy
Weak
Strong
LowHigh PerformanceEventual
Linearizable
Sequential
Causal
(Ideal protocol)
(Worst protocol)
INCREM
ENTAL
Weak consistency: ★ Fast ★ (Often correct)
Strong consistency: ★ Slower ★ (Correct with certainty)Weak
Consistency
Strong Consistency
So what?
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Consistency Models are Complementary
6
Con
sisten
cy
Weak
Strong
LowHigh PerformanceEventual
Linearizable
Sequential
Causal
(Ideal protocol)
(Worst protocol)
INCREM
ENTAL
Weak consistency: ★ Fast ★ (Often correct)
Strong consistency: ★ Slower ★ (Correct with certainty)Weak
Consistency
Strong Consistency
So what?Latency optimizations
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
7
Speculating with Correctables
SOCIAL MEDIA APPLICATION
read timeline
CORRECTABLE
Lower latency of strong consistency
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
7
Speculating with Correctables
SOCIAL MEDIA APPLICATION
Weak
Consistency
value1
Strong
Consistency
value2
read timeline
CORRECTABLE
Lower latency of strong consistency
Latency gap: ~100 ms
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
7
Speculating with Correctables
SOCIAL MEDIA APPLICATION
Weak
Consistency
value1
Strong
Consistency
value2
read timeline
Speculative execution
• value1 is often correct
• Speculatively execute any further stepse.g., prefetch dependent data
[Existential Consistency. SOSP’15][PBS. VLDB 5(8)’12]
Verify based on value2
CORRECTABLE
Lower latency of strong consistency
Latency gap: ~100 ms
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
7
Speculating with Correctables
SOCIAL MEDIA APPLICATION
Weak
Consistency
value1
Strong
Consistency
value2
read timeline
Speculative execution
• value1 is often correct
• Speculatively execute any further stepse.g., prefetch dependent data
[Existential Consistency. SOSP’15][PBS. VLDB 5(8)’12]
Verify based on value2
CORRECTABLE
Lower latency of strong consistency
Latency gap: ~100 ms
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Strongly-consistent timeline
Fetch timeline items
200 ms
100 ms
Traditional operation:
8
300ms
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Strongly-consistent timeline
Fetch timeline items
200 ms
100 ms
Traditional operation:
8
Speculative operation with ICG:
value2
100 ms
value1
100 ms
100 ms
300ms
Fetch timeline items
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Strongly-consistent timeline
Fetch timeline items
200 ms
100 ms
Traditional operation:
8
Speculative operation with ICG:
value2
value1
matches
value2100 ms
value1
100 ms
100 ms
yes
300ms
200ms
Fetch timeline items
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Strongly-consistent timeline
Fetch timeline items
200 ms
100 ms
Traditional operation:
8
Speculative operation with ICG:
value2
value1
matches
value2100 ms
value1
100 ms
100 ms
yes
no
100 msRe-fetch
300ms
300ms
200ms
Fetch timeline items
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Speculation case-study
9
➤ Application: Twissandra ➤ Workload generated via YCSB ➤ Clients in Ireland ➤ Geo-replication on Amazon’s EC2
VRGCAL
ORG
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Speculation case-study
9
★ Advertising System— Speculation case-study
★ Ticket-selling System— Exploiting application semantics
★ Overheads evaluation& Optimizations
★ Latency gaps between consistencymodels
➤ Application: Twissandra ➤ Workload generated via YCSB ➤ Clients in Ireland ➤ Geo-replication on Amazon’s EC2
VRGCAL
ORG
check the paper
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Decreasing latency of strong consistency
10
Workload A (50:50 read/write) Workload C (read-only)
What is the latency of the fetch_timeline() operation?
Workload B (95:5 read/write)
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Decreasing latency of strong consistency
10
Workload A (50:50 read/write) Workload C (read-only)
What is the latency of the fetch_timeline() operation?
Workload B (95:5 read/write)
Baseline
Read using a quorum of 2/3 replicasvs.
ICG 1. Weak: Read with 1/3 replicas 2. “Strong:” Read with quorum of 2/3 replicas
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Decreasing latency of strong consistency
10
Workload A (50:50 read/write) Workload C (read-only)
What is the latency of the fetch_timeline() operation?
Workload B (95:5 read/write)
Baseline
Read using a quorum of 2/3 replicasvs.
ICG 1. Weak: Read with 1/3 replicas 2. “Strong:” Read with quorum of 2/3 replicas
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
0
50
100
150
200
250
0 50 100 150 200 250 300
Late
ncy
(m
s)
Workload A
Ad
s S
yste
mT
wis
san
dra
C2 (R=2) CC2 (R=1,2)
0
50
100
150
200
250
0 50 100 150 200 250 300Throughput (ops/sec)
Workload B
Ad
s S
yste
mT
wis
san
dra
0
50
100
150
200
250
0 50 100 150 200 250 300
Workload C
Ad
s S
yste
mT
wis
san
dra
0
150
300
450
600
0 50 100 150 200 250 300
Late
ncy
(m
s)A
ds
Sys
tem
Tw
issa
nd
ra
0
150
300
450
600
0 50 100 150 200 250 300Throughput (ops/sec)
Ad
s S
yste
mT
wis
san
dra
0
150
300
450
600
0 50 100 150 200 250 300
Ad
s S
yste
mT
wis
san
dra
Decreasing latency of strong consistency
10
Workload A (50:50 read/write) Workload C (read-only)
What is the latency of the fetch_timeline() operation?
Workload B (95:5 read/write)
Baseline
Read using a quorum of 2/3 replicasvs.
ICG 1. Weak: Read with 1/3 replicas 2. “Strong:” Read with quorum of 2/3 replicas
Baseline
ICG
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
0
50
100
150
200
250
0 50 100 150 200 250 300
Late
ncy
(m
s)
Workload A
Ad
s S
yste
mT
wis
san
dra
C2 (R=2) CC2 (R=1,2)
0
50
100
150
200
250
0 50 100 150 200 250 300Throughput (ops/sec)
Workload B
Ad
s S
yste
mT
wis
san
dra
0
50
100
150
200
250
0 50 100 150 200 250 300
Workload C
Ad
s S
yste
mT
wis
san
dra
0
150
300
450
600
0 50 100 150 200 250 300
Late
ncy
(m
s)A
ds
Sys
tem
Tw
issa
nd
ra
0
150
300
450
600
0 50 100 150 200 250 300Throughput (ops/sec)
Ad
s S
yste
mT
wis
san
dra
0
150
300
450
600
0 50 100 150 200 250 300
Ad
s S
yste
mT
wis
san
dra
Decreasing latency of strong consistency
10
Workload A (50:50 read/write) Workload C (read-only)
What is the latency of the fetch_timeline() operation?
★ Latency decrease by 40%
★ Throughput drop by 6%
★ Same consistency model (2/3 replicas)
Workload B (95:5 read/write)
Baseline
Read using a quorum of 2/3 replicasvs.
ICG 1. Weak: Read with 1/3 replicas 2. “Strong:” Read with quorum of 2/3 replicas
Baseline
ICG
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
11
The Correctables abstraction enables you to: 1. Leverage consistency models incrementally 2. Lower latency of strong consistency
CORRECTABLE
value1
value2
valuen
Incremental Consistency Guarantees
@adizere
Conclusion
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Legacy code vs. Correctables
14
semantics
execution details
Correctable
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Legacy code vs. Correctables
14
semantics
execution details
Correctable
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Overheads
➤ Cassandra
➤ YCSB workload, various configurations
➤ Client in Ireland
➤ Replicas in Virginia, Frankfurt, and Ireland
15
0.8 1
1.2 1.4 1.6 1.8
2 2.2 2.4
30 60 120
180
240
300
Effic
ienc
y (k
B/op
)
Workload A
C1
C1CC2
CC2*CC2
*CC2
#Total client threads
Latest distribution:Zipfian distribution:
+27%
+77% +15%
+90%
30 60 120
180
240
300
Workload B
#Total client threads
Latest distribution:Zipfian distribution:
+27%
+77% +15%
+90%
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Overheads
➤ Cassandra
➤ YCSB workload, various configurations
➤ Client in Ireland
➤ Replicas in Virginia, Frankfurt, and Ireland
15
0.8 1
1.2 1.4 1.6 1.8
2 2.2 2.4
30 60 120
180
240
300
Effic
ienc
y (k
B/op
)
Workload A
C1
C1CC2
CC2*CC2
*CC2
#Total client threads
Latest distribution:Zipfian distribution:
+27%
+77% +15%
+90%
30 60 120
180
240
300
Workload B
#Total client threads
Latest distribution:Zipfian distribution:
+27%
+77% +15%
+90%
0 2 4 6 8
10 12 14 16
1 2 4 6 8 10 12Effic
ienc
y (K
B/op
)
ZooKeeper Correctable ZooKeeper
# Clients
500 tickets 1000 tickets
-71%
-44%
-81%
-60%
1 2 4 6 8 10 12# Clients
500 tickets 1000 tickets
-71%
-44%
-81%
-60%
➤ ZooKeeper queue implementation
➤ Wasteful implementation (by default)
➤ We were able to improve — negative overhead
500 elements 1000 elements
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Exploiting application semantics
➤ Ticket selling application
➤ Implemented through a ZooKeeper queue
➤ Buy ticket = dequeue operation
16
Ticket 1 Ticket 2 … Ticket 500
update
close
ticket X
ticket Y | NULL
(weak)
(strong)
if ( X > threshold) confirm()else wait for strong
if (! NULL) confirm()
invokedequeue()
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Exploiting application semantics
➤ Ticket selling application
➤ Implemented through a ZooKeeper queue
➤ Buy ticket = dequeue operation
16
Ticket 1
0
100
200
300
400 420 440 460 480 500La
ten
cy t
o b
uy
ticke
t (
mse
c)
Ticket number
Correctable ZooKeeper ZooKeeper
Last 20 tickets
Ticket 2 … Ticket 500
update
close
ticket X
ticket Y | NULL
(weak)
(strong)
if ( X > threshold) confirm()else wait for strong
if (! NULL) confirm()
invokedequeue()
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Divergence between weak and strong consistency
17
0 5
10 15 20 25 30
30 60 120 180 240 300
%D
ive
rge
nce
#Total client threads
Workload A-LatestWorkload A-Zipfian
Workload B-LatestWorkload B-Zipfian
➤ Extremely high
➤ Unusual
0.1% — 5% typical range}
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Latency gaps between consistency models
18
0 50
100 150 200 250
0 200 400 600 800 1000
Late
ncy
(ms)
Workload A (50:50 read/write)C1 (R=1) C2 (R=2) CC2 preliminary (R=1) CC2 final (R=2)
0 50
100 150 200 250
0 200 400 600 800 1000Throughput (ops/sec)
Workload B (95:5 read/write)
0 50
100 150 200 250
0 200 400 600 800 1000
Workload C (read-only)
0
50
100
150
Aver
age
Rea
d L
aten
cy (m
s)
CC preliminaryCC final
C99th %ile latency
R=3 R=2 R=1
Late
ncy
gap
0
50 100
150 200
Aver
age
Late
ncy
(ms)
CZK preliminaryCZK final
ZK99th %ile latency
Leader in IRL Leader in VRG
Follower (FRK)
Leader (IRL)
Follower (IRL)
Leader(VRG)
Clientconnection:
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
19
value1
?
value2
Efficiency of Multiple Responses
CORRECTABLE
read timeline
SOCIAL MEDIA APPLICATION
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Binding
Request
Replicated Storage
Response(final)
Response(preliminary)
Weak consistency Strong consistency
Coordination
19
value1
?
value2
Efficiency of Multiple Responses
CORRECTABLE
read timeline
SOCIAL MEDIA APPLICATION
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Binding
Request
Replicated Storage
Response(final)
Response(preliminary)
Weak consistency Strong consistency
Coordination
19
value1
(quorum gathering)
value2
Efficiency of Multiple Responses
CORRECTABLE
read timeline
SOCIAL MEDIA APPLICATION
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Correctables / Library
20
Application
Cache
Storage
binding binding binding binding
CorrectablesLIBRARY
RPC
invoke
API
(Weak / Strong)
RPC
Co
nsi
stency-
base
d
inte
rface
Sys
tem
-sp
ecifi
c
inte
rface
DesktopApplication
WebFrontend
MobileApp
CachingDaemon
Correctable
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Correctables / Library
20
Application
Cache
Storage
binding binding binding binding
CorrectablesLIBRARY
RPC
invoke
API
(Weak / Strong)
RPC
Co
nsi
stency-
base
d
inte
rface
Sys
tem
-sp
ecifi
c
inte
rface
DesktopApplication
WebFrontend
MobileApp
CachingDaemon
Correctable
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
★ Calendar
★ Social networks
★ Online shopping
★ Ad serving
★ News reading
★ Collaborative editing
★ Computation on static content
★ Cold data analysis
★ Disconnected operations in mobile applications
★ Infrastructure services
★ Stock tickers
★ Trading applications
21
Demand for Performance HighLowWeak
Strong
Dem
and
for
Cor
rect
ness
Weaker Consistency
Gray Zone
(no single choice is ideal)
Stro
nger
Con
sist
ency
Perform
ance is a
second-order
concern
Semantics allow
to bypass the need for strong
consistency
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Replicated storage systems
22
Dynamo [SOSP’07]
Pileus [SOSP’13]
Time
Richness of consistency models (CMs)
2 CMs
(weak and str
ong)
1 CM (st
rong) 10 CMs
Caching
(clien
t & se
rver)
CAP theorem
6 CMs
dynamic
quoru
ms
Incremental Consistency Guarantees Dragos-Adrian Seredinschi
Replicated storage systems
22
Dynamo [SOSP’07]
Pileus [SOSP’13]
Time
Richness of consistency models (CMs)
2 CMs
(weak and str
ong)
1 CM (st
rong) 10 CMs
Caching
(clien
t & se
rver)
CAP theorem
No single consistency model is ideal → expose multiple choices
6 CMs
dynamic
quoru
ms