SparkResilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

Presentation by Antonio Lupher[Thanks to Matei for diagrams & several of the nicer slides!]October 26, 2011

The world today…Most current cluster programming models are based on acyclic data flow from stable storage to stable storage

Map

Map

Map

Reduce

Reduce

Input Output

The world today…

Benefits: decide at runtime where to run tasks and automatically recover

from failures

Most current cluster programming models are based on acyclic data flow from stable storage to stable storage

… butInefficient for applications that repeatedly reuse working set of data:»Iterative machine learning, graph algorithms • PageRank, k-means, logistic regression, etc.

»Interactive data mining tools (R, Excel, Python)• Multiple queries on the same subset of data

Reload data from disk on each query/stage of execution

Distributedmemory

Input

iteration 1

iteration 2

iteration 3

. . .

iter. 1 iter. 2 . . .

Input

Goal: Keep Working Set in RAM

one-timeprocessing

RequirementsDistributed memory abstraction must be

»Fault-tolerant»Efficient in large commodity clusters

How to provide fault tolerance efficiently?

RequirementsExisting distributed storage abstractions offer an interface based on fine-grained updates

»Reads and writes to cells in a table»E.g. key-value stores, databases,

distributed memory

Have to replicate data or logs across nodes for fault tolerance

»Expensive for data-intensive apps, large datasets

Resilient Distributed Datasets (RDDs)

»Immutable, partitioned collection of records

»Interface based on coarse-grained transformations (e.g. map, groupBy, join)

»Efficient fault recovery using lineage• Log one operation to apply to all elements• Re-compute lost partitions of dataset on failure• No cost if nothing fails

RDDs, cont’d»Control persistence (in RAM vs. on disk)• Tunable via persistence priority: user specifies

which RDDs should spill to disk first

»Control partitioning of data• Hash data to place data in convenient

locations for subsequent operations

»Fine-grain reads

ImplementationSpark runs on Mesos

=> share resources with Hadoop & other apps

Can read from any Hadoop input source (HDFS, S3, …)

Spark Hadoop MPI

Mesos

Node Node Node Node

…

Language-integrated API in Scala~10,000 lines of code, no changes to Scala Can use interactively from interpreter

Spark OperationsTransformations

»Create new RDD by transforming data in stable storage using data flow operators• Map, filter, groupBy, etc.

»Lazy: don’t need to be materialized at all times• Lineage information is enough to compute

partitions from data in storage when needed

Spark OperationsActions

»Return a value to application or export to storage• count, collect, save, etc.

»Require a value to be computed from the elements in the RDD => execution plan

Spark Operations

Transformations

(define a new RDD)

mapflatMap

filtersample

groupByKeyreduceByKey

unionjoin

cogroupcrossProductmapValues

sortpartitionBy

Actions(return a result

to driver program)

countcollectreducelookupsave

RDD RepresentationCommon interface:

»Set of partitions»Preferred locations for each partition»List of parent RDDs»Function to compute a partition given

parents»Optional partitioning info (order, etc.)

Capture a wide range of transformations

»Scheduler doesn’t need to know what each op does

Users can easily add new transformations

»Most transformations implement in ≤ 20 lines

RDD RepresentationLineage & Dependencies

»Narrow dependencies• Each partition of parent RDD is used by at most

one partition of child RDD– e.g. map, filter

• Allow pipelined execution

RDD RepresentationLineage & Dependencies

»Wide dependencies• Multiple child partitions may depend on parent

RDD partition– e.g. join

• Require data from all parent partitions & shuffle

SchedulerTask DAG (like Dryad)Pipelines functionswithin a stageReuses previouslycomputed dataPartitioning-awareto avoid shuffles

join

union

groupBy

map

Stage 3

Stage 1

Stage 2

A: B:

C: D:

E:

F:

G:

= previously computed partition

RDD RecoveryWhat happens if a task fails?

»Exploit coarse-grained operations• Deterministic, affect all elements of collection

– Just re-run the task on another node if parents available

– Easy to regenerate RDDs given parent RDDs + lineage

»Avoids checkpointing and replication• but you might still want to (and can)

checkpoint:– long lineage => expensive to recompute – intermediate results may have disappeared, need

to regenerate– Use REPLICATE flag to persist

Example: Log MiningLoad error messages from a log into memory, then interactively search for various patternslines = spark.textFile(“hdfs://...”)

errors = lines.filter(_.startsWith(“ERROR”))messages = errors.map(_.split(‘\t’)(2))messages.persist()

Block 1

Block 2

Block 3

Worker

Worker

Worker

Driver

messages.filter(_.contains(“foo”)).countmessages.filter(_.contains(“bar”)).count. . .

tasksresults

Msgs. 1

Msgs. 2

Msgs. 3

Base RDDTransformed

RDD

Action

Result: full-text search of Wikipedia in <1 sec (vs 20

sec for on-disk data)

Result: scaled to 1 TB data in 5-7 sec

(vs 170 sec for on-disk data)

Fault Recovery Results

1 2 3 4 5 6 7 8 9 10020406080

100120140

119

57 56 58 58

81

57 59 57 59

No Failure

Iteration

Iter

atri

on t

ime

(s)

k-means

PerformanceOutperforms Hadoop by up to 20x

»Avoiding I/O and Java object [de]serialization costs

Some apps see 40x speedup (Conviva)Query a 1TB dataset w/5-7 sec. latencies

PageRank Results

30 600

20406080

100120140160180 17

1

8072

2823 14

HadoopBasic SparkSpark + Con-trolled Partition-ing

Number of machines

Iter

atio

n ti

me

(s)

Behavior with Not Enough RAM

Cache disabled

25% 50% 75% Fully cached

020406080

10068

.8

58.1

40.7

29.7

11.5

% of working set in memory

Iter

atio

n ti

me

(s)

Example: Logistic RegressionGoal: find best line separating two sets of points

+

–

+ ++

+

+

++ +

– ––

–

–

–– –

+

target

–

random initial line

Logistic Regression Codeval points = spark.textFile(...).map(parsePoint).persist()

var w = Vector.random(D)

for (i <- 1 to ITERATIONS) { val gradient = points.map(p => (1 / (1 + exp(-p.y*(w dot p.x))) - 1) * p.y * p.x ).reduce((a,b) => a+b) w -= gradient}

println("Final w: " + w)

Logistic Regression Performance

1 5 10 20 300

50010001500200025003000350040004500

Hadoop

Number of Iterations

Runn

ing

Tim

e (s

) 127 s / iteration

first iteration 174 s

further iterations 6 s

More ApplicationsEM alg. for traffic prediction (Mobile Millennium)In-memory OLAP & anomaly detection (Conviva)Twitter spam classification (Monarch)Pregel on Spark (Bagel)Alternating least squares matrix factorization

Mobile MillenniumEstimate traffic using GPS on taxis

Conviva GeoReport

Aggregations on many keys w/ same WHERE clause40× gain comes from:

»Not re-reading unused columns or filtered records»Avoiding repeated decompression»In-memory storage of deserialized objects

Spark

Hive

0 2 4 6 8 10 12 14 16 18 200.5

20

Time (hours)

Use transformations on RDDs instead of Hadoop jobs

»Cache RDDs for similar future queries»Many queries re-use subsets of data• Drill-down, etc.

»Scala makes integration with Hive (Java) easy… or easier

(Cliff, Antonio, Reynold)

SPARK

ComparisonsDryadLINQ, FlumeJava

»Similar language-integrated “distributed collection” API, but cannot reuse datasets efficiently across queries

Piccolo, DSM, Key-value stores (e.g. RAMCloud)»Fine-grained writes but more complex fault recovery

Iterative MapReduce (e.g. Twister, HaLoop), Pregel»Implicit data sharing for a fixed computation pattern

Relational databases»Lineage/provenance, logical logging, materialized views

Caching systems (e.g. Nectar)»Store data in files, no explicit control over what is

cached

Comparisons: RDDs vs DSMConcern RDDs Distr. Shared

Mem.Reads Fine-grained Fine-grainedWrites Bulk

transformationsFine-grained

Consistency Trivial (immutable) Up to app / runtime

Fault recovery Fine-grained and low-overhead using lineage

Requires checkpoints and program rollback

Straggler mitigation

Possible using speculative execution

Difficult

Work placement

Automatic based on data locality

Up to app (but runtime aims for transparency)

Behavior if not enough RAM

Similar to existing data flow systems

Poor performance (swapping?)

SummarySimple & efficient model, widely applicable

»Express models that previously required a new framework efficiently, i.e. same optimizations

Achieve fault tolerance efficiently by providing coarse-grained operations and tracking lineageExploit persistent in-memory storage + smart partitioning for speed

Thoughts: Tradeoffs

»No fine-grain modifications of elements in collection• Not the right tool for all applications

– E.g. storage system for web site, web crawler, anything where you need incremental/fine-grain writes

»Scala-based implementation• Probably won’t see Microsoft use it anytime

soon– But concept of RDDs is not language-specific

(abstraction doesn’t even require functional language)

Thoughts: InfluenceFactors that could promote adoption

»Inherent advantages • in-memory = fast, RDDs = fault-tolerant

»Easy to use & extend»Already supports MapReduce, Pregel

(Bagel)• Used widely at Berkeley, more projects coming

soon• Used at Conviva, Twitter

»Scala means easy integration with existing Java applications• (subjective opinion) More pleasant to use than

Java

VerdictShould spark enthusiasm in cloud crowds