Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be...
Transcript of Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be...
![Page 2: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/2.jpg)
![Page 3: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/3.jpg)
![Page 4: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/4.jpg)
![Page 5: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/5.jpg)
![Page 6: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/6.jpg)
This project is funded by the European Union,
7th Research Framework Programme, ICT call 10,
grant agreement n°609551.
![Page 7: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/7.jpg)
1.What are CRDTs (good for)?
2.History of CRDT Sets
3.Sets in Riak
4.Bigger Sets in Riak
4 Sections
![Page 8: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/8.jpg)
Why CRDTs?
![Page 9: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/9.jpg)
Fundamental Trade Off
• Lipton/Sandberg ’88
• Attiya/Welch ’94
• Gilbert/Lynch ’02
Low Latency/Availability: - Increased Revenue - User Engagement
Strong Consistency:- Easier for Programmers
- Less user “surprise”
![Page 10: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/10.jpg)
![Page 11: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/11.jpg)
![Page 12: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/12.jpg)
1 CLIENT
2 REPLICAS
1 KEY
![Page 13: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/13.jpg)
GETPUT
UPDATE
REPLICATE
![Page 14: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/14.jpg)
PUTPUT
![Page 15: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/15.jpg)
Quorum
![Page 16: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/16.jpg)
GET
A
PUT
A
GET
B
PUT
B
TEMPORAL TIME
![Page 17: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/17.jpg)
happens before
concurrent ——— divergent
convergent
Logical Clocks
![Page 18: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/18.jpg)
happens before
concurrent ——— divergent
convergent
Logical Clocks
![Page 19: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/19.jpg)
![Page 20: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/20.jpg)
>155196119890 155196118001
Timestamp based reconciliation
![Page 21: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/21.jpg)
![Page 22: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/22.jpg)
⨆
Business Logic/Semantic Reconciliation
![Page 23: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/23.jpg)
![Page 24: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/24.jpg)
happens before
concurrent ——— divergent
convergent
Logical Clocks
![Page 25: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/25.jpg)
Removes?
![Page 26: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/26.jpg)
GET
A
PUT
A
GET
B
PUT
B
TEMPORAL TIME
![Page 27: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/27.jpg)
⨆
![Page 28: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/28.jpg)
![Page 29: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/29.jpg)
Removes?
![Page 30: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/30.jpg)
Google F1“Designing applications to cope with concurrency anomalies in their data is very error-prone, time-consuming, and ultimately not worth the performance gains.”
![Page 31: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/31.jpg)
http://www.infoq.com/articles/key-lessons-learned-from-transition-to-nosql
“…writing merge functions was likely to confuse the hell out of
all our developers and slow down development…”
![Page 32: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/32.jpg)
CRDTsDATA TYPESThat CONVERGE
![Page 33: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/33.jpg)
CRDTsOff the shelfMERGE functions
![Page 34: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/34.jpg)
CRDTsCRDTs are Data Types (maps/sets/booleans/graphs/etc)
THAT CONVERGE
![Page 35: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/35.jpg)
![Page 36: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/36.jpg)
CRDT SETS
![Page 37: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/37.jpg)
http://www.infoq.com/articles/key-lessons-learned-from-transition-to-nosql
“…after some analysis we found that much of our data could be modelled
within sets so by leveraging CRDT’s our developers don't have to worry about
writing bespoke merge functions for 95% of carefully selected use cases…”
![Page 38: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/38.jpg)
Evolution of a CRDT Set
![Page 39: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/39.jpg)
Evolution of a Set
G-SET
![Page 40: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/40.jpg)
Bob
Replica BReplica AShelly
AnnaPete
AlexShelly
![Page 41: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/41.jpg)
Bob
Replica BReplica AShelly
AnnaPete
AlexShelly⨆
![Page 42: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/42.jpg)
BobShelly
AnnaPete
AlexG-SET
![Page 43: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/43.jpg)
Removes?
![Page 44: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/44.jpg)
Evolution of a Set
G-SET2P-SET
![Page 45: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/45.jpg)
Adds
BobPete
Shelly
Anna
Removes
BobPete
Shelly
2P-SET
![Page 46: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/46.jpg)
Adds
BobPete
Shelly
Anna
Removes
BobPete
Shelly
= Anna
![Page 47: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/47.jpg)
Value /= Structure
![Page 48: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/48.jpg)
Adds
BobPete
Shelly
Anna
Removes
BobPete
Shelly
= Anna
![Page 49: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/49.jpg)
I changed my mind!
![Page 50: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/50.jpg)
Adds
BobPete
Shelly
Anna
Removes
BobPete
Shelly
= AnnaShelly
![Page 51: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/51.jpg)
Evolution of a Set
U-SET
![Page 52: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/52.jpg)
Bob
Replica BReplica A
Shelly
AnnaPete
AlexShelly⨆
4321 5
6
![Page 53: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/53.jpg)
BobShelly
AnnaPete
Alex432
1,6
5
U-SET
![Page 54: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/54.jpg)
Evolution of a Set
U-SETOR-SET
![Page 55: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/55.jpg)
Evolution of a Set
OR-SETAW-SET
![Page 56: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/56.jpg)
Adds
Shelly
Removes
BobShelly
AnnaPete
432
1,5ShellyBob
Shelly
Pete321
AW-SET
![Page 57: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/57.jpg)
Adds
2
3
Bob
Pete
1 Shelly
Replica A
Removes
2
3
Bob
Pete
1 Shelly
Replica B
4 Anna
5 Shelly
Adds
⨆
![Page 58: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/58.jpg)
Adds
2
3
Removes
Bob
Pete
1,5 Shelly
2
3
Bob
Pete
1 Shelly
4 Anna
Anna=Shelly
![Page 59: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/59.jpg)
ObservedRemove
![Page 60: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/60.jpg)
Semantics
AddWins
![Page 61: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/61.jpg)
Evolution of a Set
AW-SET
![Page 62: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/62.jpg)
Adds
2
3
Removes
Bob
Pete
1,5 Shelly
2
3
Bob
Pete
1,5 Shelly
=Shelly
4 Anna
[]4 Anna
![Page 63: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/63.jpg)
![Page 64: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/64.jpg)
Evolution of a Set
OR-SWOT
![Page 65: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/65.jpg)
Evolution of a Set
OR-SWOTOptimised
AW-SET
![Page 66: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/66.jpg)
Version Vectors
A B
![Page 67: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/67.jpg)
Version Vectors
A B
{a, 1}
{b, 1}
{a, 2}
[ ]{a, 2}, {b, 1}
![Page 68: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/68.jpg)
EVENTS/TAGS
A
{a, 1}
{a, 2}
![Page 69: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/69.jpg)
[{a, 1}]
{a, 1} Shelly
Replica A
![Page 70: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/70.jpg)
[{a, 1}] [{a, 1}
{a, 1} Shelly {a, 1} Shelly
Replica B Replica A
![Page 71: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/71.jpg)
[{a, 1}] [{a, 1}, {b, 3}]
{a, 1} Shelly
{b, 1}
{b, 2}
{b, 3}
Bob
Pete
Phil
{a, 1} Shelly
Replica A Replica B
![Page 72: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/72.jpg)
[{a, 1}, {b,3}] [{a, 1}, {b, 3}]
{a, 1} Shelly
{b, 1}
{b, 2}
{b, 3}
Bob
Pete
Phil
{a, 1} Shelly
{b, 1}
{b, 2}
{b, 3}
Bob
Pete
Phil
Replica B Replica A
![Page 73: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/73.jpg)
[{a, 2}, {b, 3}]
{b, 1}
{b, 3}
[{a, 1}, {b, 4}]
Bob
Pete
{a, 1} Shelly
{b, 1}
{b, 2}
{b, 3}
Bob
Pete
Phil
{a, 1} Shelly
{a, 2} Anna {b, 4} John
Replica A Replica B
{b, 2} Phil
![Page 74: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/74.jpg)
[{a, 2}, {b, 3}]
{b, 1}
{b, 3}
[{a, 1}, {b, 4}]
Bob
Pete
{a, 1} Shelly
{b, 1}
{b, 2}
{b, 3}
Bob
Pete
Phil
{a, 1} Shelly
{a, 2} Anna {b, 4} John
Replica A Replica B
![Page 75: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/75.jpg)
[{a, 2}, {b, 3}]
{b, 1}
{b, 3}
[{a, 1}, {b, 4}]
Bob
Pete
{a, 1} Shelly
{b, 1}
{b, 2}
{b, 3}
Bob
Pete
Phil
{a, 1} Shelly
{a, 2} Anna {b, 4} John
Replica A Replica B MERGE
![Page 76: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/76.jpg)
[{a, 2}, {b, 3}]
{b, 1}
{b, 3}
[{a, 1}, {b, 4}]
Bob
Pete
{a, 1} Shelly
{b, 1}
{b, 2}
{b, 3}
Bob
Pete
Phil
{a, 1} Shelly
{a, 2} Anna {b, 4} John
Replica A Replica B MERGE
{a, 1} Shelly
{b, 1} Bob
![Page 77: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/77.jpg)
[{a, 2}, {b, 3}]{b, 2} ∈Phil
![Page 78: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/78.jpg)
[{a, 2}, {b, 3}]{b, 2} ∈Phil
![Page 79: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/79.jpg)
[{a, 2}, {b, 3}]
{b, 1}
{b, 3}
[{a, 1}, {b, 4}]
Bob
Pete
{a, 1} Shelly
{b, 1}
{b, 2}
{b, 3}
Bob
Pete
Phil
{a, 1} Shelly
{a, 2} Anna {b, 4} John
Replica A Replica B MERGE
{a, 1} Shelly
{b, 1} Bob
{b, 3} Pete
![Page 80: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/80.jpg)
∉{a, 2}Anna
[{a, 1}, {b, 4}]
![Page 81: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/81.jpg)
[{a, 2}, {b, 3}]
{b, 1}
{b, 3}
[{a, 1}, {b, 4}]
Bob
Pete
{a, 1} Shelly
{b, 1}
{b, 2}
{b, 3}
Bob
Pete
Phil
{a, 1} Shelly
{a, 2} Anna {b, 4} John
Replica A Replica B MERGE
{a, 1} Shelly
{b, 1} Bob
{b, 3} Pete
{a, 2} Anna
{b, 4} John
⨆ =
![Page 82: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/82.jpg)
[{a, 2}, {b, 4}]
{b, 1}
{a, 2}
{b, 3}
Bob
Pete
Anna
{a, 1} Shelly
{b, 4} John
= Bob
Pete
Anna
Shelly
John
![Page 83: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/83.jpg)
CRDT Sets
a semantic of “Add-Wins” via
“Observed Remove”
![Page 84: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/84.jpg)
SETS in RIAK 2.0+
![Page 85: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/85.jpg)
Riak 2.0riak_dt -> Riak Data Types
![Page 86: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/86.jpg)
Sets in Riak
An optimized conflict-free replicated set Annette Bieniusa et al
http://arxiv.org/abs/1210.3368
![Page 87: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/87.jpg)
WHO USES THE LIB?
![Page 88: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/88.jpg)
A B
Client 1
who’s the actor?
Client 10000Client 2 ………
![Page 89: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/89.jpg)
A B
Client 1 Client 10000Client 2 ………
{c1, 1} {cX, 1}{c2, 1}
![Page 90: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/90.jpg)
A B
Client 1
[]
![Page 91: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/91.jpg)
A B
Client 1
{c1, 1} Shelly
![Page 92: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/92.jpg)
A B
Client 1
[]
![Page 93: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/93.jpg)
A B
Client 1
{c1, 1} Bob
![Page 94: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/94.jpg)
Read Your Own
Writes
![Page 95: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/95.jpg)
A B
REPLICAS ACTORS
![Page 96: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/96.jpg)
HOW TO USE THE LIB?
![Page 97: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/97.jpg)
A B
[HAIRDRYER, PENCIL CASE]
SHOPPING CART
![Page 98: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/98.jpg)
A B
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …, …, …, …, …, …, …, …], […,
ZUCK’s FOLLOWERS?
![Page 99: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/99.jpg)
Client X Client Y
Add “Shelly” Add “Bob”
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
![Page 100: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/100.jpg)
Client X Client Y
remove “Shelly” remove “Bob”
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
![Page 101: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/101.jpg)
ObservedRemove
![Page 102: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/102.jpg)
Client X Client Y
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
![Page 103: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/103.jpg)
Client X Client Y
remove “Shelly”[{A3, B4}]
remove “Bob”[{A1, B5}]
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
![Page 104: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/104.jpg)
Sets in Riak• Operation Based API
• With causal Context for removes!
• Vnode As Actor/Replica
• Action-at-a-distance
• Full state replication
![Page 105: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/105.jpg)
Sets in Riak
An optimized conflict-free replicated set Annette Bieniusa et al
http://arxiv.org/abs/1210.3368
![Page 106: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/106.jpg)
Sets in Riak
![Page 107: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/107.jpg)
Sets in Riak
![Page 108: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/108.jpg)
Sets in Riak
PHOTO © 2011 J. RONALD LEE, CC ATTRIBUTION 3.0. https://www.flickr.com/photos/jronaldlee/5566380424
![Page 109: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/109.jpg)
Client X
Add “Shelly”
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
![Page 110: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/110.jpg)
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
![Page 111: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/111.jpg)
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
![Page 112: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/112.jpg)
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
![Page 113: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/113.jpg)
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
![Page 114: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/114.jpg)
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
[{a, 34}, {b, 1000}…]
![Page 115: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/115.jpg)
[{a35}shelly, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
[{a, 35}, {b, 1000}…]
![Page 116: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/116.jpg)
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
![Page 117: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/117.jpg)
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
![Page 118: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/118.jpg)
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
REPLICATE
![Page 119: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/119.jpg)
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
U
![Page 120: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/120.jpg)
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
![Page 121: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/121.jpg)
[…, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …, …,
…, …, …], […, …, …, …, …, …, …, …,
ZUCK’s FOLLOWERS
![Page 122: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/122.jpg)
Problem?
• 1key -> 1 Set • Poor Write speed • Can’t have “big” sets
![Page 123: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/123.jpg)
Every time we change the set we read and write the
whole set!
![Page 124: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/124.jpg)
Sets in Riak
![Page 125: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/125.jpg)
Sets in Riak
10k sets, 100k elements, 50 workers - write
![Page 126: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/126.jpg)
Sets in Riak
Small : riak object 1MB limit
![Page 127: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/127.jpg)
Bigsets:Make writes faster
andsets bigger
![Page 128: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/128.jpg)
Bigset Design: Overview
![Page 129: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/129.jpg)
Bigset Design: Overview
![Page 130: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/130.jpg)
Initial Results
![Page 131: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/131.jpg)
Initial Results
10k sets, 100k elements, 50 workers - write
![Page 132: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/132.jpg)
10k sets, 100k elements, 50 workers - write
![Page 133: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/133.jpg)
Client X
Add “Shelly”
ZUCK’s FOLLOWERS CLOCK
bob a 1
[{a, 10}, {b 99}…{z, 89}]
bob c 12
Zooey b 97
![Page 134: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/134.jpg)
ZUCK’s FOLLOWERS CLOCK
bob a 1
[{a, 10}, {b 99}…{z, 89}]
bob c 12
Zooey b 97
![Page 135: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/135.jpg)
ZUCK’s FOLLOWERS CLOCK
bob a 1
[{a, 10}, {b 99}…{z, 89}]
bob c 12
Zooey b 97
ZUCK’s FOLLOWERS CLOCK
bob a 1
[{a, 10}, {b 99}…{z, 89}]
bob c 12
Zooey b 97
![Page 136: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/136.jpg)
ZUCK’s FOLLOWERS CLOCK
bob a 1
[{a, 11}, {b 99}…{z, 89}]
bob c 12
Zooey b 97
Shelly a 11
[{a, 11}, {b 99}…{z, 89}]
ZUCK’s FOLLOWERS CLOCK
bob a 1
[{a, 10}, {b 99}…{z, 89}]
bob c 12
Zooey b 97
![Page 137: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/137.jpg)
[{a, 11},
Shelly a 11
[{a, 11}, {b 99}…{z, 89}]
ZUCK’s FOLLOWERS CLOCK
bob a 1
[{a, 11}, {b 99}…{z, 89}]
bob c 12
Zooey b 97
![Page 138: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/138.jpg)
REPLICATEZUCK’s FOLLOWERS CLOCK
bob a 1
[{a, 11}, {b 99}…{z, 89}]
bob c 12
Zooey b 97
Shelly a 11
Shelly a 11
![Page 139: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/139.jpg)
ZUCK’s FOLLOWERS CLOCK
bob a 1
[{a, 10}, {b 99}…{z, 89}]
bob c 12
Zooey b 97
Shelly a 11
![Page 140: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/140.jpg)
ZUCK’s FOLLOWERS CLOCK
bob a 1
[{a, 10}, {b 99}…{z, 89}]
bob c 12
Zooey b 97
Shelly a 11[{a, 10}, {b 99}…{z, 89}]
![Page 141: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/141.jpg)
ZUCK’s FOLLOWERS CLOCK
bob a 1
[{a, 10}, {b 99}…{z, 89}]
bob c 12
Zooey b 97
Shelly a 11
[{a, 10}, {b 99}…{z, 89}]
![Page 142: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/142.jpg)
ZUCK’s FOLLOWERS CLOCK
bob a 1
[{a, 10}, {b 99}…{z, 89}]
bob c 12
Zooey b 97
Shelly a 11
[{a, 11}, {b 99}…{z, 89}][{a, 11}, {b 99}…{z, 89}]
![Page 143: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/143.jpg)
ZUCK’s FOLLOWERS CLOCK [{a, 10}, {b 99}…{z, 89}]
bob a 1
bob c 12
Zooey b 97
Shelly a 11
[{a, 11}, {b 99}…{z, 89}]
![Page 144: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/144.jpg)
THAT’S IT!
![Page 145: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/145.jpg)
THAT’S IT?• Reads! • Version Vector! • Hand-Off! • AAE!
![Page 146: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/146.jpg)
Reads?
![Page 147: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/147.jpg)
Initial Read Results
![Page 148: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/148.jpg)
10k sets, 100k elements, 20 workers - read
![Page 149: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/149.jpg)
clock
element-N
end_key
set-tombstone
element-1
clock
element-N
end_key
set-tombstone
element-1
Read Clock
Iterate keys
![Page 150: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/150.jpg)
C++clock
element-N
end_key
set-tombstone
element-1
Read Clock
Iterate each key
ERLANG
![Page 151: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/151.jpg)
C++clock
element-N
end_key
set-tombstone
element-1
Read Clock
100k buffer of keys
ERLANG
![Page 152: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/152.jpg)
<<Set, \0, $c, \0, Actor, \0>>
<<Set, \0, %z, \0, \0>>
<<Set, \0, $e, \0, Element-1, \0, Actor, Cnt:/64/big-unsigned-
integer>>
<<Set, \0, $e, \0, Element-1, \0, Actor, Cnt:/64/big-unsigned-
integer>>
No Sext
No T2B
![Page 153: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/153.jpg)
Bigset Design: read
Client X
read fsm
block1
replica A replica C
block1
blockN
block1
blockN
incremental merge
blockN
![Page 154: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/154.jpg)
Reads Today
10k sets, 100k elements, 20 workers - read
![Page 155: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/155.jpg)
Reads Today
10k sets, 100k elements, 20 workers - read
![Page 156: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/156.jpg)
Full Set Read or Queries?
Why read the whole set?
‘Cos you HAVE TO!
![Page 157: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/157.jpg)
Full Set Read or Queries?
Why read the whole set?
![Page 158: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/158.jpg)
Full Set Read or Queries?
Why read the whole set?
‘Cos you HAVE TO!
![Page 159: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/159.jpg)
Bigset Queries• Subset
• Is Member? • Range queries SORTED! • Pagination
![Page 160: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/160.jpg)
Removes• Observed-Remove - context
• Requires _some kind_ of read
• cheap membership check
![Page 161: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/161.jpg)
Is Member(X)
Client X
read fsm
is_member(<<bob>>).
replica A replica C
read(<<bob>>)
![Page 162: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/162.jpg)
Is Member(X)Replica A
{ReplicaA, clock} [{a, 3}, {b,3}]
{<<bob>>, a, 1} <<>>
{<<bob>>, b, 2} <<>>
{<<shelly>>, b, 1} <<>>
{ReplicaA, end_key} <<>>
read clock
{<<anne>>, a, 3} <<>>
Seek <<bob>>
![Page 163: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/163.jpg)
Is Member(X)Replica A
{ReplicaA, clock} [{a, 3}, {b,3}]
{<<bob>>, a, 1} <<>>
{<<bob>>, b, 2} <<>>
{<<shelly>>, b, 1} <<>>
{ReplicaA, end_key} <<>>
read clock
{<<anne>>, a, 3} <<>>
Seek <<bob>>
![Page 164: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/164.jpg)
Is Member(X)Replica A
[{a, 3}, {b,3}]
<<bob>> [{a,1}, {b,2}]
read fsm
<<bob>> Set
Replica C
[{a,3}, {c, 1}]
<<bob>> [{c, 1}]
![Page 165: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/165.jpg)
Is Member(X)
⊔=[{a, 3}, {b,3}]
<<bob>> [{a,1}, {b,2}]
[{a,3}, {c, 1}]
<<bob>> [{c, 1}]
==<<bob>> [{b,2}, {c,1}]
Read FSM
{true, [{b,2}, {c, 1}]}Client X
![Page 166: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/166.jpg)
Next?
• Other “Big” Types - Maps
• Quorum Read Secondary Indexes
• Big Sets of Maps - Tables
• Joins? SQL?
![Page 167: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/167.jpg)
Summary
• CRDTs make eventual consistency easier on developers
• There exists an Optimised Add-Wins Set…
• It takes more more than a lib
• A little engineering goes a long way
![Page 168: Bigsets…CRDT sets but BIGGER... “…after some analysis we found that much of our data could be modelled within sets so ...](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f0563c87e708231d412b8ae/html5/thumbnails/168.jpg)
THANK YOU!
WE’RE HIRING! UK Client Service Engineer Developer Advocate EMEA
bashojobs.theresumator.com
VISIT OUR STAND Experience our Riak TS demo and be
entered to win a Scalextric set! Get your invitation to our IoT Riak TS
Roadshow
Bigset Paper https://arxiv.org/abs/1605.06424