MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log...

42
MetaSync File Synchronization Across Multiple Untrusted Storage Services Seungyeop Han, Haichen Shen, Taesoo Kim*, Arvind Krishnamurthy, Thomas Anderson, and David Wetherall University of Washington *Georgia Institute of Technology 1

Transcript of MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log...

Page 1: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

MetaSyncFileSynchronizationAcrossMultipleUntrustedStorageServices

Seungyeop Han,HaichenShen,Taesoo Kim*,ArvindKrishnamurthy,ThomasAnderson,and

DavidWetherall

UniversityofWashington *Georgia InstituteofTechnology1

Page 2: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

Filesyncservicesarepopular

400M ofDropbox usersreachedinJune20152

Page 3: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

Baidu(2TB)

Manysyncserviceproviders

Dropbox (2GB) GoogleDrive(15GB)

MSOneDrive (15GB) Box.net (10GB)

3

Page 4: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

Canwerelyonanysingleservice?

4

Page 5: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

ExistingApproaches

• Encryptfilestopreventmodification– Boxcryptor

• Rewritefilesyncservicetoreducetrust– SUNDR(Lietal.,04),DEPOT(Mahajan etal.,10)

5

Page 6: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

MetaSync:Canwebuildabetter filesynchronizationsystemacrossmultipleexistingservices?

MetaSync

6

Higheravailability,greatercapacity,higherperformanceStrongerconfidentiality &integrity

Page 7: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

Goals

• Higheravailability• Strongerconfidentiality&integrity• Greatercapacityandhigherperformance

• Noservice-service,client-clientcommunication

• Noadditionalserver• Opensourcesoftware

7

Page 8: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

Overview

• Motivation&Goals• MetaSync Design• Implementation• Evaluation• Conclusion

8

Page 9: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

KeyChallenges

• Maintainagloballyconsistentview ofthesynchronizedfilesacrossmultipleclients

• Usingonlytheserviceproviders’unmodifiedAPIs withoutanycentralizedserver

• Eveninthepresenceofservicefailure

9

Page 10: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

OverviewoftheDesign

Synchronization ReplicationObjectStore

MetaSync

BackendabstractionsLocalStorage

Dropbox GoogleDrive OneDrive Remote

Services

10

1.FileManagement

Page 11: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

ObjectStore

• Similardatastructurewithversioncontrolsystems(e.g.,git)

• Content-basedaddressing– Filename=hashofthecontents– De-duplication– Simpleintegritychecks

• Directoriesformahashtree– Independent&concurrentupdates

11

Page 12: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

ObjectStore

12

head =f12…

Dir1

abc…

Dir2

4c0…

Large.bin

20e…

blob blobblob

small1 small2

• Filesarechunkedorgroupedintoblobs• Theroothash=f12…uniquely identifiesasnapshot

Page 13: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

ObjectStore

13

old=f12…

Dir1

abc…

Dir2

4c0…

Large.bin

20e…

blob blobblob

small1 small2

• Filesarechunkedorgroupedintoblobs• Theroothash=f12…uniquely identifiesasnapshot

1ae…

blob

head =07c…

Large.bin

Page 14: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

OverviewoftheDesign

Synchronization ReplicationObjectStore

MetaSync

BackendabstractionsLocalStorage

Dropbox GoogleDrive OneDrive Remote

Services

14

2.Consistentupdate

Page 15: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

UpdatingGlobalView

GlobalView

v0ab1…

Client1 Prev

Head

Prev

Head

Client2

master

15

Previouslysynchronizedpoint

Currentroothash

Page 16: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

UpdatingGlobalView

GlobalView

v0ab1…

Client1 Prev Head

PrevClient2

v1c10…

master

16

Head

Page 17: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

UpdatingGlobalView

GlobalView

v0ab1…

Client1 Prev

Head

PrevClient2

v1c10…

master

17

Head

Page 18: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

UpdatingGlobalView

GlobalView

v0ab1…

Client1 Prev

Head

PrevClient2

v1c10…

master

18

Head

Page 19: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

UpdatingGlobalView

GlobalView

v0ab1…

Client1 Prev

Prev HeadClient2

v1c10…

v27b3…

master

19

Head

v2f13…

Page 20: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

UpdatingGlobalView

GlobalView

v0ab1…

Client1 Prev

Prev

Head

Client2

v1c10… v27b3…

master

20

Head

v2f13…

Page 21: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

UpdatingGlobalView

GlobalView

v0ab1…

Client1 Prev

Prev

Head

Client2

v1c10… v27b3…

master

21

Head

v3a31…

Page 22: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

ConsistentUpdateofGlobalView

• Needtohandleconcurrentupdates,unavailableservicesbasedonexistingAPIs

MetaSync

Dropbox

MetaSync

GoogleDrive OneDrive

root=f12… root=b05…

22

Page 23: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

Paxos

• Multi-roundnon-blockingconsensusalgorithm– Saferegardlessoffailures– Progressifmajorityisalive

Proposer Acceptor23

Page 24: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

Metasync:SimulatePaxos• Useanappend-only list tologPaxos messages– ClientsendsnormalPaxos messages– Uponarrivalofmessage,serviceappendsitintoalist– Clientcanfetchalistoftheorderedmessages

• EachserviceproviderhasAPIstobuildappend-onlylist– GoogleDrive,OneDrive,Box:Commentsonafile– Dropbox:Revisionlistofafile– Baidu:Filesinadirectory

24

Page 25: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

Metasync:PassivePaxos (pPaxos)

• Backendservicesworkaspassiveacceptor• Acceptordecisionsaredelegatedtoclients

Clients Passive StorageServices

S2

S1

S3

25

P1

P2

propose(3)

Page 26: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

Metasync:PassivePaxos (pPaxos)

• Backendservicesworkaspassiveacceptor• Acceptordecisionsaredelegatedtoclients

Clients Passive StorageServices

S2

S1

S3

26

P1

P2

propose(2)

Page 27: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

Metasync:PassivePaxos (pPaxos)

• Backendservicesworkaspassiveacceptor• Acceptordecisionsaredelegatedtoclients

Clients Passive StorageServices

S2

S1

S3

27

P1

P2

fetch(S1)

fetch(S2)

fetch(S3)

Page 28: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

Metasync:PassivePaxos (pPaxos)

• Backendservicesworkaspassiveacceptor• Acceptordecisionsaredelegatedtoclients

Clients Passive StorageServices

S2

S1

S3

28

P1

P2

accept(3,v1)

fetch

Page 29: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

DiskPaxos

29

Disk1 Disk2 Disk3

P1 P2 P3

Propose

Page 30: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

DiskPaxos

30

Disk1 Disk2 Disk3

P1 P2 P3

Fetch

Page 31: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

Paxos vs.DiskPaxos vs.pPaxos

• DiskPaxos:maintainsablockperclient

Proposer

Acceptor

computation

Propose Accept

Paxos

Proposer

Acceptor

diskblocks

Propose Check

DiskPaxos

…

Proposer

Acceptor

append-only

Propose Check

pPaxos

Gafni &Lamport ’02

RequiresacceptorAPIO(clientsxacceptors)O(acceptors) O(acceptors)

31

require#msgs

Page 32: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

OverviewoftheDesign

Synchronization ReplicationObjectStore

MetaSync

BackendabstractionsLocalStorage

Dropbox GoogleDrive OneDrive Remote

Services

32

3.Replicateobjects

Page 33: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

StableDeterministicMapping

• MetaSync replicatesobjectsRtimesacrossSstorageproviders(R<S)

• Requirements– Shareminimalinformationamongservices/clients– Supportvariationinstoragesize–Minimizerealignmentuponconfigurationchanges

• Deterministicmapping

– E.g.,map(7a1…)=Dropbox,Google

33

Page 34: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

DeterministicMappingExample

• Service={A(1),B(2),C(2),D(1)}• N={A1,B1,B2,C1,C2,D1}(normalized)• Map(i)=Sorted(N,key=md5(i,serviceID,vID))

Capacity

map[0]=[A1,C2,D1,B1,B2,C1]=[A,C]map[1]=[B2,B1,C1,C2,A1,D1]=[B,C]…map[19]=[C2,B1,D1,A1,B2,C1]=[C,B]

H=20

R=2

bc1…mod20=1=>ReplicateontoBandC

34

Page 35: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

DeterministicMappingExample

• WhenCisremovedmap[0]=[A1,C2,D1,B1,B2,C1]=[A,C]map[1]=[B2,B1,C1,C2,A1,D1]=[B,C]…map[19]=[C2,B1,D1,A1,B2,C1]=[C,B]

H=20

R=2

map[0]=[A1,D1,B1,B2]=[A,D]map[1]=[B2,B1,A1,D1]=[B,A]…map[19]=[B1,D1,A1,B2]=[B,D]

H=20

Thesortedorderismaintained=>Minimizerealignments 35

Page 36: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

Implementation

• PrototypedwithPython– ~8klinesofcode

• Currentlysupports5backendservices– Dropbox,GoogleDrive,OneDrive,Box.net,Baidu

• Twofront-endclients– Commandlineclient– Syncdaemon

36

Page 37: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

Evaluation

• Howistheend-to-endperformance?

• What’stheperformancecharacteristicsofpPaxos?

• HowquicklydoesMetaSync reconfiguremappings?

37

Page 38: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

Evaluation

• Howistheend-to-endperformance?

• What’stheperformancecharacteristicsofpPaxos?

• HowquicklydoesMetaSync reconfiguremappings?

38

Page 39: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

End-to-EndPerformance

Dropbox Google MetaSyncLinuxKernel920 directories15kfiles,166MB

2h45m > 3hrs 12m18s

Pictures50 files,193MB

415s 143s 112s

Synchronizethetargetbetweentwocomputers

39

Performancegainsarefrom:• Parallelupload/downloadwithmultipleproviders• Combinedsmallfilesintoablob

(S=4,R=2)

Page 40: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

LatencyofpPaxos

Latencyisnotdegradedwithincreasingconcurrentproposersoraddingslowbackendstorageservice

40

0

5

10

15

20

25

30

35

1 2 3 4 5

Latency(s)

#ofProposers

Google

Dropbox

OneDrive

Box

Baidu

Page 41: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

LatencyofpPaxos

Latencyisnotdegradedwithincreasingconcurrentproposersoraddingslowbackendstorageservice

41

0

5

10

15

20

25

30

35

1 2 3 4 5

Latency(s)

#ofProposers

Google

Dropbox

OneDrive

Box

Baidu

All

Page 42: MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

Conclusion

• MetaSync providesasecure,reliable,andperformant filessyncserviceontopofpopularcloudproviders– Toachieveaconsistentupdate,wedeviseanewclient-basedPaxos

– Tominimizeredistribution,wepresentastabledeterministicmapping

• Sourcecodeisavailable:– http://uwnetworkslab.github.io/metasync/

42


Paxos introduction

Paxos introduction

Flexible Paxos: Reaching agreement without majorities

Flexible Paxos: Reaching agreement without majorities

Paxos Quorum Leases Sayed Hadi Hashemi. BACKGROUND.

Paxos Quorum Leases Sayed Hadi Hashemi. BACKGROUND.

Bringing Paxos Consensus in Multi-agent Systems

Bringing Paxos Consensus in Multi-agent Systems

Greek luxury villa rentals in paxos glyfada beach

Greek luxury villa rentals in paxos glyfada beach

Welcome to the luxury paxos villas in greece

Welcome to the luxury paxos villas in greece

Paxos made Live - Carnegie Mellon School of Computer Sciencepavlo/courses/fall2013/static/slides/paxos_made_live.pdf · Paxos made Live How Google employs paxos to build a replicated

Paxos made Live - Carnegie Mellon School of Computer Sciencepavlo/courses/fall2013/static/slides/paxos_made_live.pdf · Paxos made Live How Google employs paxos to build a replicated

CS5412: Paxos

CS5412: Paxos

Luxury greek villas holidays in paxos

Luxury greek villas holidays in paxos

Paxos Made Moderately Complex - Cornell University

Paxos Made Moderately Complex - Cornell University

Paxos Made Wireless: Consensus in the Airbeshr/docs/poirot-ewsn19.pdf · failures (e.g., message losses, network segmentation, node crashes). 2.2 The Paxos Basics Paxos is a fault-tolerant

Paxos Made Wireless: Consensus in the Airbeshr/docs/poirot-ewsn19.pdf · failures (e.g., message losses, network segmentation, node crashes). 2.2 The Paxos Basics Paxos is a fault-tolerant

Holiday villas on paxos glyfada beach villas

Holiday villas on paxos glyfada beach villas

Paxos Made Wireless: Consensus in the Air · sensus and concurrent transmissions. We begin with an overview of consensus. Then, we present Paxos and Multi-Paxos. Finally, we introduce

Paxos Made Wireless: Consensus in the Air · sensus and concurrent transmissions. We begin with an overview of consensus. Then, we present Paxos and Multi-Paxos. Finally, we introduce

Paxos Made Simple

Paxos Made Simple

Glyfada beach villas paxos

Glyfada beach villas paxos

the Paxos Commit algorithm

the Paxos Commit algorithm

Fault-tolerance techniques RSM, Paxos Jinyang Li.

Fault-tolerance techniques RSM, Paxos Jinyang Li.

Paxos - Distributed Consensus - Department of Computer

Paxos - Distributed Consensus - Department of Computer

PAXOS Lecture by Avi Eyal Based on: Deconstructing Paxos – by Rajsbaum Paxos Made Simple – by Lamport Reconstructing Paxos – by Rajsbaum.

PAXOS Lecture by Avi Eyal Based on: Deconstructing Paxos – by Rajsbaum Paxos Made Simple – by Lamport Reconstructing Paxos – by Rajsbaum.

C*SC301 - Paxos, Tuples and UDTs

C*SC301 - Paxos, Tuples and UDTs