S3Guard: What's in your consistency model?

1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

S3Guard: What’s in Your Consistency Model? Mingliang Liu @liuml07Steve Loughran @steveloughran

December 2016

Steve LoughranHadoop committer & PMC, ASF Member

Mingliang Liu Apache Hadoop committer

Chris Nauroth, Hadoop committer & PMC, ASF member

Rajesh BalamohanTez Committer & PMC


S3A:Hadoop File System for S3(EMR: use Amazon's s3:// )


Storage Use Evolution

HDFS

Application

HDFS

Application

GoalEvolution towards cloud storage as the primary Data Lake

Input Output

Backup Restore

InputOutput

Copy

HDFS

Application

Input

Output

tmp


org.apache.hadoop.fs.FileSystem

hdfs s3awasb adlswift gs

Hadoop File System - One Interface Fits All


/

work

pending

part-00

part-01

00

00

00

01

0101

complete

part-01

rename("/work/pending/part-01", "/work/complete")

A FileSystem: Directories, Files Data


S3A: Object Store Pretending A FileSystem

Cloud Object Stores designed for– Scale– Cost– Geographic Distribution– Availability

Cloud apps dedicatedly deal with cloud storage semantics and limitations Hadoop apps should work on cloud storage transparently

– S3A partially adheres to the FileSystem specification


00

00

00

01

01

s01 s02

s03 s04

hash("/work/pending/part-01") ["s02", "s03", "s04"]

01

010101


hash(name)->blob


What Is The Problem?

Performance– separated from compute– cloud storage not designed for file-like access patterns

Limitations in APIs– delete(path, recursive=true)– rename(source, dest)

Eventual consistency– Create Consistency– Update– Delete– Listing

• take time to list created objects• lag in changed metadata about existing objects• lag in observing deleted objects


00

00

00

01

01

s01 s02

s03 s04


copy("/work/pending/part-01", "/work/complete/part01")

01

010101

delete("/work/pending/part-01")


rename(): A Series of Operations on The Client


Eventual Consistency From FileSystem’s View

When listing "a directory”– Newly created files may not yet be visible, deleted ones still present

After updating an object– Opening and reading the object may still return the previous data

After deleting an object– Opening the object may succeed, returning the data

While reading an object– If object is updated or deleted during the process


00

00

00

01

01

s01 s02

s03 s04

01

DELETE /work/pending/part-00

HEAD /work/pending/part-00

GET /work/pending/part-00

200

200

200

Eventually Consistent – Seeing Deleted Data


S3Guard:Fast, Consistent S3 Metadata(EMR: use Amazon's EMRFS)


S3Guard: Fast, Consistent S3 Metadata

Inspired by Apache licensed S3mper project from Netflix Using DynamoDB as the consistent metadata store Mutating file system operations

– Update both S3 and DynamoDB Read operations

– Return results to callers as sourced from S3– First check their results against the metadata in DynamoDB– S3A waits and rechecks both S3 and DynamoDB until they agree

Goals– Provide consistent list and get status operations on S3 objects written with S3Guard enabled

• listStatus() after put and delete• getFileStatus() after put and delete

– Provide tools to manage associated metadata and caching policies.– Configurable error handling when inconsistency is detected– Performance improvements that impact real workloads.


00

00

00

01

01

s01 s02

s03 s04

01

DELETE part-00200

HEAD part-00200

HEAD part-00404

PUT part-00200

00

DynamoDB As The Consistent Metadata Store


Demo


https://issues.apache.org/jira/browse/HADOOP-13345


Backup Slides


00

00

00

01

01

s01 s02

s03 s04

HEAD /work/complete/part-01

PUT /work/complete/part01x-amz-copy-source: /work/pending/part-01

01

DELETE /work/pending/part-01

PUT /work/pending/part-01... DATA ...

GET /work/pending/part-01Content-Length: 1-8192

GET /?prefix=/work&delimiter=/

REST APIs

S3Guard: What's in your consistency model?

Technology

Transcript of S3Guard: What's in your consistency model?