April 11th, 2011 1EuroSys 2011
DepSky – Dependable and Secure Storage in a
Cloud-of-Clouds
Alysson Bessani, Miguel Correia, Bruno Quaresma, Fernando André, Paulo Sousa
University of Lisbon, Faculty of Sciences
April 11th, 2011 2EuroSys 2011
Moving to Clouds
• Data is moving to the cloud
• Main reason: costs (pay-per-use model)
Servers Storage Servers Cloud
Storage
e.g., Amazon S3
April 11th, 2011 3EuroSys 2011
One Cloud is Not Enough
• Critical data storage
– Medical records
– Company financial data
– Critical infrastructures data
Critical
System
Cloud
Storage Failure?
April 11th, 2011 4EuroSys 2011
Trusted Clouds
• Two options:
1) Improve the cloud infrastructure
2) Use several cloud providers (cloud-of-clouds)
April 11th, 2011 5EuroSys 2011
Benefits of Replication
• Datacenter and cloud outages
• Vendor lock-in
• Better read performance
• Data corruption
– Bugs
– Malicious insiders
– Attacks and intrusions
Critical
System
Amazon S3
Windows Azure
Rackspace
April 11th, 2011 6EuroSys 2011
Cloud-of-CloudsObject Storage
Amazon S3
Nirvanix
Rackspace
Windows
Azure
April 11th, 2011 7EuroSys 2011
DepSky Design
April 11th, 2011 8EuroSys 2011
DepSky Design Principles
1. No trust on individual cloud providers
Distributed trust is built by using multiple clouds
2. Use storage clouds as they are
No server-side code on the replication protocols
3. Data is updatable
Quorum replication protocols for consistency
April 11th, 2011 9EuroSys 2011
Key Challenges
• How to implement an efficient replication protocol using only passive storage nodes?
• How to make it affordable?
read
write
April 11th, 2011 10EuroSys 2011
DepSky Interface• write(data_unit, data)
• read(data_unit)
• create(data_unit)
• destroy(data_unit)
• lock(data_unit, …)
• unlock(data_unit)
• garbageCollect(data_unit, …)
• reconfigure(data_unit,…)
See details on the paper!
Object Storage
April 11th, 2011 11EuroSys 2011
System Model
• Asynchronous distributed system
• Faults
– Clouds can be unavailable, corrupt or destroy data
– Readers can do whatever they want
– Writers can crash and recover
• n = 3f +1 clouds to tolerate f faults
– In practice: f = 1
• Symmetric and asymmetric cryptography
Byzantine faults
April 11th, 2011 12EuroSys 2011
Data Model
…
Version
Verification
DUSingle Writer
Multiple Readers
Priv
Pub
Pub
Pub
single-writer multi-reader regular register(but multiple writers are supported through a locking algorithm)
Data
Sign (Version+Data)
April 11th, 2011 13EuroSys 2011
Data Model Implementation
April 11th, 2011 14EuroSys 2011
Read/Write Protocols
• f-dissemination Byzantine quorum systems [Malkhi & Reiter 1998]
– quorums of 2f+1 servers out-of 3f+1 servers
– data is self-verifiable (signed)
Cloud A Cloud B Cloud C Cloud D
write quorum read quorum
f+1 servers in
the intersection
April 11th, 2011 15EuroSys 2011
Cloud A
Cloud B
Cloud C
Cloud D
DD
DD
DD
DD
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd
DepSky Write
WRITE
DATA
DD
ACK
DD
DD
DD
DD
WRITE
METADATA
qwjda
sjkhd
ahsd
ACK
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd
April 11th, 2011 16EuroSys 2011
Cloud A
Cloud B
Cloud C
Cloud D
DD
DD
DD
DD
DepSky Read
READ
DATA
DD
DATA
DD
DD
DD
DD
READ
METADATA
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd
qwjda
sjkhd
ahsd
METADATA
qwjda
sjkhd
ahsd
Data will be fetched from other cloudsif needed.
highest version number
April 11th, 2011 17EuroSys 2011
Why does it work?
DDqwjda
sjkhd
ahsd
READ
WRITE
METADATA DATA
Key property: if metadata pointing to D is read, D can be read
April 11th, 2011 18EuroSys 2011
Cloud A Cloud B Cloud C Cloud D
Confidentiality
DataDataLimitations:
1. Data is accessible by cloud providers
2. Requires n×|Data| storage space
DataData DataData DataDataDataData
April 11th, 2011 19EuroSys 2011
Cloud A Cloud B Cloud C Cloud D
DepSky ConfidentialityCombining Erasure Codes and Secret Sharing [Krawczyk 1993]
S1 S2 S3 S4
share
K
generated
keyDataData
disperse
F1 F2 F3 F4
F1 S1 F2 S2 F3 S3 F4 S4
Secret sharing not needed if key distribution is available.
encrypt
Inverse process for reading from f+1 shares/fragments.
April 11th, 2011 20EuroSys 2011
Consistency Proportionality
• The consistency provided by DepSky is the same as the base storage clouds
– If the weakest consistency cloud provides eventual consistency, DepSky provides eventual consistency
– If the weakest consistency cloud provides “read your writes”, DepSky provides “read your writes”
– If the weakest consistency cloud provides regular storage, DepSky provides regular storage
• This notion may be useful for other systems
April 11th, 2011 21EuroSys 2011
DepSky Evaluation
April 11th, 2011 22EuroSys 2011
DepSky Performance
• Prototype: 3K locs (Java), REST/HTTPS
• Experimental Setup
– Two DepSky setups: A (DepSky) and CA (DepSky with confidentiality)
– Four commercial storage Clouds: S3 (Amazon S3), WA(Windows Azure), NX (Nirvanix SDN) and RS (Rackspace)
– Clients spread through 8 PlanetLab sites around the world
– Three clients on each site, reading/writing data units of three sizes (100kb, 1Mb and 10Mb)
– 437000+ reads/writes between Sep. 10th and Oct. 7th 2010
• Experiments cost: ~400€
April 11th, 2011 23EuroSys 2011
DepSky Operations Costs ($)
• Monetary costs (in USD) for 1Mb data unity and four clouds
– Read cost is the same of reading from the less expensive cloud
– Write cost is the sum of writing 50% of the DU size on each cloud
• These costs don’t include data storage!
Operation DepSky-CA Amazon S3 Rackspace Win. Azure Nirvanix
10K Reads 1.47 1.46 2.15 1.46 1.46
10K Writes 3.08 1.46 0.78 0.98 2.93
April 11th, 2011 24EuroSys 2011
DepSky Storage Costs ($)
• DepSky-CA storage cost (1M data unit):
2×(Avg. individual cloud cost per GB/month)
Plain replication
Erasure code
April 11th, 2011 25EuroSys 2011
DepSky Latency (100kb DU)DepSky read latency is close to the cloud with the best latency
DepSky write latency is close to the cloud with the worst latency
April 11th, 2011 26EuroSys 2011
DepSky Performance
• Secret sharing latency overhead < 0.1%
• Effectiveness of read optimization
– Fetch data first from the clouds that returned metadata faster
– Effective in 83% (A) and 68% (CA) of reads
• Throughput per client:
– 65-1480 kb/s (read) and 3-108 kb/s (write)
• Orders of magnitude smaller than LAN BFT storage systems [Hendricks et al 2007]
– Cloud aggregate throughput may be “infinite”
April 11th, 2011 27EuroSys 2011
DepSky Perceived Availability
• Apparently, some clouds don’t provide the promised 5 or 6 9’s of availability
• Internet availability plays an important role
April 11th, 2011 28EuroSys 2011
Conclusions
• DepSky: Cloud-of-clouds storage system for untrusted clouds
– Techniques: Byzantine quorum systems (integrityand availability), erasure codes (storage efficiency) and secret sharing (confidentiality)
– Can be used on storage clouds as they are
– Can be a foundation for more advanced storage systems
– A use case for Byzantine fault tolerance
April 11th, 2011 29EuroSys 2011
Conclusions
• Costs × Benefits
– Four clouds are needed to tolerate a single “faulty cloud”
– Reads are faster than single cloud reads
– Writes are slower than single cloud writes
– Monetary costs roughly twice the average costs of individual clouds
– It can be improved: data doesn’t need to be in all 3f+1 clouds!
April 11th, 2011 30EuroSys 2011
Questions?
We are hiring!
http://www.tclouds-project.eu
April 11th, 2011 31EuroSys 2011
Tools for Confidentiality & Storage-Efficiency
• Information-optimal erasure codes
– encode(D) generates n fragments F1, …, Fn
– decode(…), uses f+1 fragments to recover D
– Remark: |Fi| = ((f+1)/n)×|D|, i.e., |D|/2 if f=1
• Secret sharing
– share(s) generates n secrets S1, …, Sn
– combine(…) uses f+1 secrets to recover s
– Remarks:
• |si| = |s|
• no information about s can be obtained with f or less shares
April 11th, 2011 32EuroSys 2011
Related Work
• Data storage on diverse clouds
– HAIL [Bowers et al 2009]: no confidentiality, no update, and requires code running on the clouds
– RACS [Abu-Libdeh et al 2010]: no confidentiality, no integrity, no updates
• Byzantine Quorum Protocols
– Most of them require servers running some protocol code
– Byzantine Disk Paxos [Abraham 2006] is similar, but satisfies a weak livenesscondition (finite writes)
• Untrusted Clouds
– Depot [Mahajan et al 2010], SPORC [Feldman et al 2010], Venus [Shraer et al 2010]
– Doesn’t improve availability and require code running on the clouds
• None of these did experiments using multiple clouds
April 11th, 2011 33EuroSys 2011
DepSky Latency (100kb DU)
Top Related