Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5...
Transcript of Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5...
![Page 1: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/1.jpg)
Workshop: ImplementingDistributed Consensus
Dan Lüdtke [email protected]
Disclaimer This work is not affiliated with any company (including Google). This talk is the result of a personal education project!
Kordian Bruck [email protected]
![Page 2: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/2.jpg)
Agenda● Part I - Hot Potato at Scale
○ Why we need Distributed Consensus
● Part II - Experiments○ Introduction to Skinny, an educational distributed lock service
○ How Skinny reaches consensus
○ How Skinny deals with instance failure
● Part III - Implementation○ A simple Paxos-like protocol
○ Making our protocol more reliable
![Page 3: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/3.jpg)
Part IHot Potato at Scale
![Page 4: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/4.jpg)
![Page 5: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/5.jpg)
![Page 6: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/6.jpg)
![Page 7: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/7.jpg)
![Page 8: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/8.jpg)
![Page 9: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/9.jpg)
![Page 10: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/10.jpg)
Same potato!
![Page 11: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/11.jpg)
![Page 12: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/12.jpg)
Protocols● Paxos
○ Multi-Paxos○ Cheap Paxos
● Raft
● ZooKeeper Atomic Broadcast● Proof-of-Work Systems
○ Bitcoin
● Lockstep Anti-Cheating○ Age of Empires
Implementations● Chubby
○ Coarse grained lock service● etcd
○ A distributed key value store
● Apache ZooKeeper○ A centralized service for
maintaining configuration information, naming, providing distributed synchronization
Raft Logo: Attribution 3.0 Unported (CC BY 3.0) Source: https://raft.github.io/#implementationsEtcd Logo: Apache 2 Source: https://github.com/etcd-io/etcd/blob/master/LICENSEZookeeper Logo: Apache 2 Source: https://zookeeper.apache.org/
![Page 14: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/14.jpg)
Part IIDistributed Consensus
Hands-on
![Page 15: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/15.jpg)
Introducing Skinny● Paxos-based● Minimalistic● Educational● Lock Service
The “Giraffe”, “Beaver”, “Alien”, and “Frame” graphics on the following slides have been released under Creative Commons Zero 1.0 Public Domain License
![Page 16: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/16.jpg)
12
3
4
5
Instances
![Page 17: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/17.jpg)
12
3
4
5
Quorum
![Page 18: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/18.jpg)
12
3
4
5
Majority
![Page 19: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/19.jpg)
12
3
4
5
Also a majority
![Page 20: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/20.jpg)
12
3
4
5
NOT a majority
![Page 21: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/21.jpg)
12
3
4
5
Name catbusIncr. 1ID 0Promised 0Holder
Name kantaIncr. 2ID 0Promised 0Holder
Name meiIncr. 3ID 0Promised 0Holder Name totoro
Incr. 5ID 0Promised 0Holder
Name satsukiIncr. 4ID 0Promised 0Holder
InstanceState Information
![Page 22: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/22.jpg)
3
Name meiIncr. 3ID 0Promised 0Holder foo
Instance Name
Unique "Increment"
Current Paxos Round Number (ID)
Promised Paxos Round Number
Agreed-on value (Lock Holder)
![Page 23: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/23.jpg)
12
3
4
5
Name catbusIncr. 1ID 0Promised 0Holder
Name kantaIncr. 2ID 0Promised 0Holder
Name meiIncr. 3ID 0Promised 0Holder Name totoro
Incr. 5ID 0Promised 0Holder
Name satsukiIncr. 4ID 0Promised 0Holder
Client asking for the lock
![Page 24: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/24.jpg)
Let's get used to the lab...
![Page 25: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/25.jpg)
Lab Machine Folder Structure/home/ubuntu/ └── skinny ├── bin ├── cmd ├── config ├── doc │ ├── ansible │ ├── examples │ ├── img │ ├── plots │ ├── terraform │ └── workshop │ ├── ami │ ├── code │ ├── configs │ └── scripts ├── proto ├── skinny └── vendor
Our working directory
BinariesSource of the Skinny CLI tools
Source of the Skinny config parser module
Protocol buffer definitions (API definitions)
Main Skinny source code3rd party source code
Lab virtual machine disk image
Lab code/config/scripts for our experiments
![Page 26: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/26.jpg)
How Skinny reaches consensus
![Page 27: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/27.jpg)
Lock please?
SKINNY QUORUM
12
3
4
5
Name catbusIncr. 1ID 0Promised 0Holder
Name kantaIncr. 2ID 0Promised 0Holder
Name meiIncr. 3ID 0Promised 0Holder Name totoro
Incr. 5ID 0Promised 0Holder
Name satsukiIncr. 4ID 0Promised 0Holder
![Page 28: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/28.jpg)
Name catbusIncr. 1ID 0Promised 1Holder
Lock please?
ProposalID 1
Name kantaIncr. 2ID 0Promised 0Holder
ProposalID 1
ProposalID 1
ProposalID 1
PHASE 1A: PROPOSE
12
3
4
5
Name meiIncr. 3ID 0Promised 0Holder Name totoro
Incr. 5ID 0Promised 0Holder
Name satsukiIncr. 4ID 0Promised 0Holder
![Page 29: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/29.jpg)
Name catbusIncr. 1ID 0Promised 1Holder
PromiseID 1
PromiseID 1
PromiseID 1
PromiseID 1
PHASE 1B: PROMISE
12
3
4
5
Name kantaIncr. 2ID 0Promised 1Holder
Name meiIncr. 3ID 0Promised 1Holder Name totoro
Incr. 5ID 0Promised 1Holder
Name satsukiIncr. 4ID 0Promised 1Holder
![Page 30: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/30.jpg)
Name catbusIncr. 1ID 1Promised 1Holder Beaver
CommitID 1Holder Beaver
CommitID 1Holder Beaver
CommitID 1Holder Beaver
CommitID 1Holder Beaver
PHASE 2A: COMMIT
12
3
4
5
Name kantaIncr. 2ID 0Promised 1Holder
Name meiIncr. 3ID 0Promised 1Holder Name totoro
Incr. 5ID 0Promised 1Holder
Name satsukiIncr. 4ID 0Promised 1Holder
![Page 31: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/31.jpg)
Name catbusIncr. 1ID 1Promised 1Holder Beaver
Lock acquired!Holder is Beaver.
Committed
Committed
Committed
Committed
PHASE 2B: COMMITTED
12
3
4
5
Name kantaIncr. 2ID 1Promised 1Holder Beaver
Name meiIncr. 3ID 1Promised 1Holder Beaver Name totoro
Incr. 5ID 1Promised 1Holder Beaver
Name satsukiIncr. 4ID 1Promised 1Holder Beaver
![Page 32: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/32.jpg)
Experiment One1.) Inspect Quorum
skinnyctl status
2.) Acquire Lock for "beaver" (using instance catbus)skinnyctl acquire --instance=catbus beaver
3.) Inspect Quorumskinnyctl status
4.) Release Lock (using random instance)skinnyctl release
5.) Inspect Quorumskinnyctl status
Note: Reset the Quorum to initial state to start over!./scripts/reset-experiment-one.sh
Important: Run all commands in folder ~/skinny/doc/workshop/
![Page 33: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/33.jpg)
How Skinny deals withInstance Failure
![Page 34: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/34.jpg)
Name catbusIncr. 1ID 9Promised 9Holder Beaver
SCENARIO
12
3
4
5
Name kantaIncr. 2ID 9Promised 9Holder Beaver
Name meiIncr. 3ID 9Promised 9Holder Beaver Name totoro
Incr. 5ID 9Promised 9Holder Beaver
Name satsukiIncr. 4ID 9Promised 9Holder Beaver
![Page 35: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/35.jpg)
Name totoroIncr. 5ID 9Promised 9Holder Beaver
Name meiIncr. 3ID 9Promised 9Holder Beaver
TWO INSTANCES FAIL
12
3
4
5
Name catbusIncr. 1ID 9Promised 9Holder Beaver
Name kantaIncr. 2ID 9Promised 9Holder Beaver
Name satsukiIncr. 4ID 9Promised 9Holder Beaver
![Page 36: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/36.jpg)
INSTANCES ARE BACKBUT STATE IS LOST
Lock please?
12
3
4
5
Name catbusIncr. 1ID 9Promised 9Holder Beaver
Name kantaIncr. 2ID 9Promised 9Holder Beaver
Name meiIncr. 3ID 0Promised 0Holder Name totoro
Incr. 5ID 0Promised 0Holder
Name satsukiIncr. 4ID 9Promised 9Holder Beaver
![Page 37: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/37.jpg)
INSTANCES ARE BACKBUT STATE IS LOST
Lock please? Proposal
ID 3
ProposalID 3
ProposalID 3
ProposalID 3
12
3
4
5
Name catbusIncr. 1ID 9Promised 9Holder Beaver
Name kantaIncr. 2ID 9Promised 9Holder Beaver
Name meiIncr. 3ID 3Promised 3Holder Name totoro
Incr. 5ID 0Promised 0Holder
Name satsukiIncr. 4ID 9Promised 9Holder Beaver
![Page 38: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/38.jpg)
PROPOSAL REJECTED
PromiseID 3
NOT PromisedID 9Holder Beaver
NOT PromisedID 9Holder Beaver
NOT PromisedID 9Holder Beaver
12
3
4
5
Name catbusIncr. 1ID 9Promised 9Holder Beaver
Name kantaIncr. 2ID 9Promised 9Holder Beaver
Name meiIncr. 3ID 3Promised 3Holder Name totoro
Incr. 5ID 0Promised 3Holder
Name satsukiIncr. 4ID 9Promised 9Holder Beaver
![Page 39: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/39.jpg)
START NEW PROPOSALWITH LEARNED VALUES
ProposalID 12
ProposalID 12
ProposalID 12
ProposalID 12
12
3
4
5
Name catbusIncr. 1ID 9Promised 9Holder Beaver
Name kantaIncr. 2ID 9Promised 9Holder Beaver
Name meiIncr. 3ID 9Promised 12Holder Beaver Name totoro
Incr. 5ID 0Promised 3Holder
Name satsukiIncr. 4ID 9Promised 9Holder Beaver
![Page 40: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/40.jpg)
PROPOSAL ACCEPTED
PromiseID 12
PromiseID 12
PromiseID 12 Promise
ID 12
12
3
4
5
Name catbusIncr. 1ID 9Promised 12Holder Beaver
Name kantaIncr. 2ID 9Promised 12Holder Beaver
Name meiIncr. 3ID 12Promised 12Holder Beaver Name totoro
Incr. 5ID 0Promised 12Holder
Name satsukiIncr. 4ID 9Promised 12Holder Beaver
![Page 41: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/41.jpg)
COMMIT LEARNED VALUE
CommitID 12Holder Beaver
CommitID 12Holder Beaver
CommitID 12Holder Beaver
CommitID 12Holder Beaver
12
3
4
5
Name catbusIncr. 1ID 9Promised 12Holder Beaver
Name kantaIncr. 2ID 9Promised 12Holder Beaver
Name meiIncr. 3ID 12Promised 12Holder Beaver Name totoro
Incr. 5ID 0Promised 12Holder
Name satsukiIncr. 4ID 9Promised 12Holder Beaver
![Page 42: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/42.jpg)
Name kantaIncr. 2ID 12Promised 12Holder Beaver
Name meiIncr. 3ID 12Promised 12Holder Beaver
COMMIT ACCEPTEDLOCK NOT GRANTED
Committed
Committed
Committed Committed
Lock NOT acquired!Holder is Beaver.
12
3
4
5
Name catbusIncr. 1ID 12Promised 12Holder Beaver
Name totoroIncr. 5ID 12Promised 12Holder Beaver
Name satsukiIncr. 4ID 12Promised 12Holder Beaver
![Page 43: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/43.jpg)
Experiment Two1.) Inspect Quorum
skinnyctl status
2.) Stop instances mei and totorosudo systemctl stop skinny@meisudo systemctl stop skinny@totoro
3.) Inspect Quorum. Verify that instances mei and totoro are down!skinnyctl status
4.) Start instances mei and totoro againsudo systemctl start skinny@meisudo systemctl start skinny@totoro
5.) Inspect Quorum. Verify that instances mei and totoro are out of sync!skinnyctl status
6.) Acquire Lock for "alien" using instance meiskinnyctl acquire --instance=mei alien
Important: Run all commands in folder ~/skinny/doc/workshop/
Screwed up? No worries!Reset the Quorum to initial state via:./scripts/reset-experiment-two.sh
![Page 44: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/44.jpg)
Part IIIImplementing Distributed Consensus
![Page 45: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/45.jpg)
Skinny APIs
![Page 46: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/46.jpg)
Skinny APIs
● Consensus API○ Used by Skinny
instances to reach consensus
client
Consensus API
admin
Lock API
Control API
● Lock API○ Used by clients to
acquire or release a lock
● Control API○ Used by us to observe
what's happening
![Page 47: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/47.jpg)
Lock APImessage AcquireRequest {
string Holder = 1;
}
message AcquireResponse {
bool Acquired = 1;
string Holder = 2;
}
message ReleaseRequest {}
message ReleaseResponse {
bool Released = 1;
}
service Lock {
rpc Acquire(AcquireRequest) returns (AcquireResponse);
rpc Release(ReleaseRequest) returns (ReleaseResponse);
}
client
admin
![Page 48: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/48.jpg)
Consensus API// Phase 1: Promise
message PromiseRequest {
uint64 ID = 1;
}
message PromiseResponse {
bool Promised = 1;
uint64 ID = 2;
string Holder = 3;
}
// Phase 2: Commit
message CommitRequest {
uint64 ID = 1;
string Holder = 2;
}
message CommitResponse {
bool Committed = 1;
}
service Consensus {
rpc Promise (PromiseRequest) returns (PromiseResponse);
rpc Commit (CommitRequest) returns (CommitResponse);
}
![Page 49: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/49.jpg)
Control APImessage StatusRequest {}
message StatusResponse {
string Name = 1;
uint64 Increment = 2;
string Timeout = 3;
uint64 Promised = 4;
uint64 ID = 5;
string Holder = 6;
message Peer {
string Name = 1;
string Address = 2;
}
repeated Peer Peers = 7;
}
service Control {
rpc Status(StatusRequest) returns (StatusResponse);
}
admin
![Page 50: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/50.jpg)
Reaching Out...
![Page 51: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/51.jpg)
// Instance represents a skinny instance
type Instance struct {
mu sync.RWMutex
// begin protected fields
...
peers []peer
// end protected fields
}
type peer struct {
name string
address string
conn *grpc.ClientConn
client pb.ConsensusClient
}
Skinny Instance● List of peers
○ All other instances in the quorum
● Peer○ gRPC Client Connection○ Consensus API Client
![Page 52: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/52.jpg)
for _, p := range in.peers {
// send proposal
resp, err := p.client.Promise(
context.Background(),
&pb.PromiseRequest{ID: proposal})
if err != nil {
continue
}
if resp.Promised {
yea++
}
learn(resp)
}
Propose Function1. Send proposal to all peers2. Count responses
○ Promises
3. Learn previousconsensus (if any)
![Page 53: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/53.jpg)
Resulting Behavior● Sequential Requests● Waiting for IO
Propose P1
count
Propose P2 Propose P3 Propose P4
● Instance slow or down...?
Propose P1 Propose P2 Propose P3 Propose P4 Propose P5
t
t
learn
![Page 54: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/54.jpg)
Improvement #1● Limit the Waiting for IO
Propose P1 Propose P2 Propose P3 Propose P4
tcancel
![Page 55: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/55.jpg)
for _, p := range in.peers {
// send proposal
ctx, cancel := context.WithTimeout(
context.Background(),
time.Second*10)
resp, err := p.client.Promise(ctx,
&pb.PromiseRequest{ID: proposal})
cancel()
if err != nil {
continue
}
if resp.Promised {
yea++
}
learn(resp)
}
Timeouts● WithTimeout()
○ Here: Hardcoded○ Real world: Configurable
● Cancel() to prevent context leak
![Page 56: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/56.jpg)
Improvement #2
● Concurrent Requests● Synchronized Counting● Synchronized Learning
Propose P1
Propose P2
Propose P3
Propose P4
t
![Page 57: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/57.jpg)
for _, p := range in.peers {
// send proposal
go func(p *peer) {
ctx, cancel := context.WithTimeout(
context.Background(),
time.Second*10)
defer cancel()
resp, err := p.client.Promise(ctx,
&pb.PromiseRequest{ID: proposal})
if err != nil { return }
// now what?
}(p)
}
Concurrency● Goroutine!● Context with timeout● But how to handle
success?
![Page 58: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/58.jpg)
type response struct {
from string
promised bool
id uint64
holder string
}
responses := make(chan *response)
for _, p := range in.peers {
go func(p *peer) {
...
responses <- &response{
from: p.name,
promised: resp.Promised,
id: resp.ID,
holder: resp.Holder,
}
}(p)
}
Synchronizing● Define response data
structure● Channels to the rescue!● Write responses to
channel as they come in
![Page 59: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/59.jpg)
// count the votes
yea, nay := 1, 0
for r := range responses {
// count the promises
if r.promised {
yea++
} else {
nay++
}
learn(r)
}
Synchronizing● Counting● yea := 1
○ Because we always vote for ourselves
● Learning
![Page 60: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/60.jpg)
responses := make(chan *response)
for _, p := range in.peers {
go func(p *peer) {
...
responses <- &response{...}
}(p)
}
// count the votes
yea, nay := 1, 0
for r := range responses {
// count the promises
...
learn(r)
}
What's wrong?● We did not close
the channel● range is blocking
forever
![Page 61: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/61.jpg)
responses := make(chan *response)
wg := sync.WaitGroup{}
for _, p := range in.peers {
wg.Add(1)
go func(p *peer) {
defer wg.Done()
...
responses <- &response{...}
}(p)
}
// close responses channel
go func() {
wg.Wait()
close(responses)
}()
// count the promises
for r := range responses {...}
Solution: Moresynchronizing!
● Use WaitGroup● Close channel when all
requests are done
![Page 62: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/62.jpg)
Result
Propose P1
Propose P2
Propose P3
Propose P4
t
![Page 63: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/63.jpg)
Experiment Three1.) Copy source code of experiment three
cp code/consensus.go.experiment-three ../../skinny/consensus.go
2.) Build Skinny from sourcemage -d ../../ build
3.) Restart Quorum./scripts/restart-quorum.sh
4.) Inspect Quorumskinnyctl status
5.) Acquire Lock for "beaver" and stop the timeskinnyctl acquire beaver
6.) Repeat previous step a couple of times.How long does it take Beaver to acquire the lock on average (estimated)?Do you have an idea why it took the amount of time it took?What could be changed to improve lock acquisition times withoutviolating the majority requirement?
Important: Run all commands in folder ~/skinny/doc/workshop/
Hint: Specify an instance when acquiring/releasing and Inspect the instance's logs (cheat-sheet.pdf)
![Page 64: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/64.jpg)
Ignorance Is Bliss?
![Page 65: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/65.jpg)
Early Stopping
Propose P1
Propose P2
Propose P3
Propose P4
tReturn
Yea:
Majority
![Page 66: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/66.jpg)
type response struct {
from string
promised bool
id uint64
holder string
}
responses := make(chan *response)
ctx, cancel := context.WithTimeout(
context.Background(),
time.Second*10)
defer cancel()
Early Stopping (1)● One context for all
outgoing promises● We cancel as soon as
we have a majority● We always cancel
before leaving the function to prevent a context leak
![Page 67: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/67.jpg)
wg := sync.WaitGroup{}
for _, p := range in.peers {
wg.Add(1)
go func(p *peer) {
defer wg.Done()
resp, err := p.client.Promise(ctx,
&pb.PromiseRequest{ID: proposal})
... // ERROR HANDLING. SEE NEXT SLIDE!
responses <- &response{
from: p.name,
promised: resp.Promised,
id: resp.ID,
holder: resp.Holder,
}
}(p)
}
Early Stopping (2)● Nothing new here
![Page 68: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/68.jpg)
resp, err := p.client.Promise(ctx,
&pb.PromiseRequest{ID: proposal})
if err != nil {
if ctx.Err() == context.Canceled {
return
}
responses <- &response{from: p.name}
return
}
responses <- &response{...}
...
Early Stopping (3)● We don't care about
cancelled requests● We want errors which
are not the result of a canceled proposal to be counted as a negative answer (nay) later.
● For that we emit an empty response into the channel in those cases.
![Page 69: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/69.jpg)
go func() {
wg.Wait()
close(responses)
}()
Early Stopping (4)● Close responses
channel once all responses have been received, failed, or canceled
![Page 70: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/70.jpg)
yea, nay := 1, 0
canceled := false
for r := range responses {
if r.promised { yea++ } else { nay++ }
learn(r)
if !canceled {
if in.isMajority(yea) || in.isMajority(nay) {
cancel()
canceled = true
}
}
}
Early Stopping (5)● Count the votes● Learn previous
consensus (if any)● Cancel all in-flight
proposal if we have reached a majority
![Page 71: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/71.jpg)
Homework1.) Copy source code of experiment three (start from there)
cp code/consensus.go.experiment-three ../../skinny/consensus.go
2.) Implement "early stopping"a.) Use a global contextb.) Distinguish between context errors and other errors
Handle them differentlyc.) Make sure to stop as soon as you have a majority
Note: A majority of negative answers is still a majority!
Hint: You can find a reference implementation in skinny/consensus.go
![Page 72: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/72.jpg)
Sources
![Page 73: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/73.jpg)
Further Reading
https://lamport.azurewebsites.net/pubs/reaching.pdf
![Page 74: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/74.jpg)
Further Reading
https://research.google.com/archive/chubby-osdi06.pdf
Naming of "Skinny" absolutely not inspired by "Chubby" ;)
![Page 75: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/75.jpg)
Further Watching
The Paxos AlgorithmLuis Quesada TorresGoogle Site Reliability Engineeringhttps://youtu.be/d7nAGI_NZPk
Paxos Agreement - ComputerphileDr. Heidi HowardUniversity of Cambridge Computer Laboratoryhttps://youtu.be/s8JqcZtvnsM
![Page 76: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/76.jpg)
Further WatchingSREcon19 APAC - Implementing Distributed ConsensusYours trulyhttps://youtu.be/nyNCSM4vGF4
![Page 77: Workshop: Implementing Distributed Consensus · Incr. 3 ID 0 Promised 0 Holder Name totoro Incr. 5 ID 0 Promised 0 Holder Name satsuki Incr. 4 ID 0 Promised 0 Holder Instance ...](https://reader034.fdocuments.us/reader034/viewer/2022042315/5f03da427e708231d40b1556/html5/thumbnails/77.jpg)
Try, Play, Learn!● The Skinny lock server is open source software!● Terraform module● Ansible playbook● Packer config
Find me on Twitter @danrl_comI blog about SRE and technology: https://danrl.com
github.com/danrl/skinny