Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible...

74
Berlin Buzzwords, May 26th Stephan Ewen [email protected] / @stratosphere_eu Stratosphere / Flink Next-Gen Data Analytics Platform

Transcript of Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible...

Page 1: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

1

Berlin Buzzwords, May 26th

Stephan [email protected] / @stratosphere_eu

Stratosphere / FlinkNext-Gen Data Analytics Platform

Page 2: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

About me

2

Stephan EwenLast days Ph.D. student at TU Berlin

Stratosphere core developer

Page 3: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

3

Apache Flink

Page 4: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

What is Stratosphere?

4

An efficient distributed general-purpose data

analysis platform.

Built on top ofHDFS and YARN.

Focusing on ease of programming.

Page 5: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Project status• Research project started

in 2009 by TU Berlin,HU Berlin, HPI

• Now a growing open sourceproject with first industrialinstallations

• Moving to Apache as "Flink"

• v0.4 - stable & documented, v0.5 release candidate 2 out

Page 6: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Introducing StratosphereGeneral Purpose Data Analytics Platform.

● Declarativity for SQL

● Optimizer● Efficient Runtime

● Scalability● User-defined

functions (UDFs)● Complex data types● Schema on read

● Iterations● Advanced

Dataflows● Declarativity

Stratosphere

6

Database Technology MapReduce-style Technology

Page 7: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Stratosphere Stack

7

JavaAPI

Storage

Stratosphere Runtime

HDFS Local Files

S3

ClusterManager

YARN EC2 Direct

Stratosphere Optimizer

ScalaAPI

Spargel

(graphs)

SQL,Python

...JDBC

HadoopMapRedu

ce

Hive ...Mete

or(scripti

ng)

Page 8: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Key Features

8

Easy to use developer APIs

• Java, Scala, Graphs, Nested Data(Python & SQL under development)

• Flexible composition of large programs

Automatic Optimization• Join algorithms• Operator chaining• Reusing

partitioning/sorting

High Performance Runtime• Complex DAGs of operators• In memory & out-of-core• Data streamed between operations

Native Iterations• Embedded in the APIs• Data streaming / in-

memory• Delta iterations speed up

many programs by orders of mag.

Page 9: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Stratosphere Features

9

Page 10: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

DataSet<String> text = env.readTextFile(input);

DataSet<Tuple2<String, Integer>> result = text.flatMap(new Splitter()).groupBy(0).aggregate(SUM, 1);

// map function implementationclass Splitter extends FlatMap<String, Tuple2<String, Integer>> {

public void flatMap(String value, Collector out){ for (String token : value.split("\\W")) { out.collect(new Tuple2<String, Integer>(token, 1)); } }}

Concise & rich APIs

10

Word Count in Stratosphere, new Java API

Can use regular POJOs!

Page 11: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Concise & rich APIs

11

val input = TextFile(textInput)val w ords = input flatM ap { line = > line.split("\\W + ") }val counts = w ords groupBy { w ord = > w ord } count()

Word Count in Stratosphere

Scala API

Page 12: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Concise & rich APIs

12

Union

JoinReduceMap CoGroup

Cross IterateIterateDelta

Basic Operators

Derived Operators• Filter, FlatMap, Project• Aggregate, Distinct• Outer-Join, Semi-Join, Anti-Join• Vertex-Centric Graphs computation (Pregel

style)• ....

Page 13: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Flexible Data Pipelines

13

Reduce

Join

Map

Reduce

Map

Iterate

Source

Sink

Source

Page 14: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

DataSet<Tuple...> large = env.readCsv(...);DataSet<Tuple...> medium = env.readCsv(...);DataSet<Tuple...> small = env.readCsv(...);

DataSet<Tuple...> joined1 = large.join(medium).where(3).equals(1) .with(new JoinFunction() { ... });

DataSet<Tuple...> joined2 = small.join(joined1).where(0).equals(2) .with(new JoinFunction() { ... });

DataSet<Tuple...> result = joined2.groupBy(3).aggregate(MAX, 2);

Joins in Stratosphere

14

Built-in strategies include partitioned join and replicated join with local sort-merge or hybrid-hash algorithms.

⋈⋈

γ

large

medium

small

Page 15: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

DataSet<Tuple...> large = env.readCsv(...);DataSet<Tuple...> medium = env.readCsv(...);DataSet<Tuple...> small = env.readCsv(...);

DataSet<Tuple...> joined1 = large.join(medium).where(3).equals(1) .with(new JoinFunction() { ... });

DataSet<Tuple...> joined2 = small.join(joined1).where(0).equals(2) .with(new JoinFunction() { ... });

DataSet<Tuple...> result = joined2.groupBy(3).aggregate(MAX, 2);

Automatic Optimization

15

Possible execution 1) Partitioned hash-join

3) Grouping /Aggregation reuses the partitioningfrom step (1) No shuffle!!!

2) Broadcast hash-join

Partitioned ≈ Reduce-sideBroadcast ≈ Map-side

Page 16: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Running Programs

16

> bin/stratosphere run prg.jar

Packaged ProgramsRemote EnvironmentLocal Environment

Program JAR file

JVM

master master

RPC &Serialization

RemoteEnvironment.execute()LocalEnvironment.execute()

Spawn embeddedmulti-threaded environment

Page 17: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Stratosphere Runtime

17

Page 18: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Distributed Runtime

18

• Master (Job Manager) handles job submission, scheduling, and metadata

• Workers (Task Managers) execute operations

• Data can be streamed between nodes

• All operators startin-memory and graduallygo out-of-core

Page 19: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Runtime Architecture(comparison)

19

DistributedCollection

List[WC]

public class WC { public String word; public int count;}

emptypage

Pool of Memory Pages• Collections of objects• General-purpose

serializer (Java / Kryo)• Limited control over

memory & less efficientspilling

• Deserialize all or nothing

• Works on pages of bytes• Maps objects transparently to these pages• Full control over memory, out-of-core

enabled• Algorithms work on binary representation• Address individual fields (not deserialize

whole object)

Page 20: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Iterative Programs

20

Page 21: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Why Iterative Algorithms

• Algorithms that need iterationso Clustering (K-Means, Canopy, …)

o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

o Graph Algorithms (e.g., PageRank, Line-Rank, components, paths, reachability, centrality, )

o Graph communities / dense sub-components

o Inference (believe propagation)

o …

• Loop makes multiple passes over the data

21

Page 22: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Iterations in other systems

22

Step Step Step Step Step

ClientLoop outside the system

Step Step Step Step Step

ClientLoop outside the system

Page 23: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Iterations in Stratosphere

23

Streaming dataflowwith feedback

map

join

red.

join

System is iteration-aware, performs automatic optimization

Page 24: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Automatic Optimization for Iterative Programs

24

Caching Loop-invariant DataPushing work„out of the loop“

Maintain state as index

Page 25: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Unifies various kinds of Computations

25

ExecutionEnvironment env = getExecutionEnvironment();

DataSet<Long> vertexIds = ...DataSet<Tuple2<Long, Long>> edges = ...

DataSet<Tuple2<Long, Long>> vertices = vertexIds.map(new IdAssigner());

DataSet<Tuple2<Long, Long>> result = vertices .runOperation( VertexCentricIteration.withPlainEdges( edges, new CCUpdater(), new CCMessager(), 100));

result.print();env.execute("Connected Components");

Pregel/Giraph-style Graph Computation

Page 26: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Delta Iterations speed up certain problems by

a lot

26

0 2 4 6 8 101214161820222426283032340

200000

400000

600000

800000

1000000

1200000

1400000

Iteration

# V

ert

ices (

thousands)

Bulk

Delta

Twitter Webbase (20)0

1000

2000

3000

4000

5000

6000

Computations performed in each iteration for connected communities of a social graph

Runtime (secs)

Cover typical use cases of Pregel-like systems with comparable performance in a generic platform and developer API.

Page 27: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Program Optimization

27

Page 28: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Why Program Optimization ?

28

Do you want to hand-optimize that?

Page 29: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

What is Automatic Optimization

29

Run on a sampleon the laptop

Run a month laterafter the data evolved

Hash vs. SortPartition vs. BroadcastCachingReusing partition/sortExecution

Plan A

ExecutionPlan B

Run on large fileson the cluster

ExecutionPlan C

Page 30: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Using Stratosphere

30

Page 31: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

31

www.stratosphere.eu

Page 32: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Its easy to get started…

32

wget http://stratosphere-bin.s3-website-us-east-1 .amazonaws.com/stratosphere-dist-0.5-SNAPSHOT-yarn.tar.gztar xvzf stratosphere-dist-0.5-SNAPSHOT-yarn.tar.gz ./stratosphere-yarn-0.5-SNAPSHOT/bin/yarn-session.sh -n 4 -jm 1024 -tm 3000

If you have YARN, deploy a full stratosphere setup in 3 commands:

Also works on Amazon Elastic MapReduce ;-)

Quickstart projects set up a program skeleton, includingembedded local execution/debugging environment…

For the experts…

trying it out…

$ wget https://.../stratosphere-0.4.tgz$ tar xzf stratosphere-*.tgz$ stratosphere/bin/start-local.sh

running a local pseudo-clusterAlso available as aDebian package

Page 33: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

33

Page 34: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Roadmap• Last pre-Apache release 0.5 coming now, moving to Apache

• Mid-query fault tolerance

• Interactive queries / Cross query data caching

• Adding Tez as a distributed runtime backend

• Add support for the new Mahout Scala DSL

• Streaming – Initial Storm-like API coming up (SZTAKI Budapest)

• Add "logical" operations to Java API

34

Page 35: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

The infamous WordCount in Stratosphere

35

Java API – Expression Variant (prototype)

DataSet<String> text = env.readTextFile(input);

DataSet<WC> words = text.flatMap(

new FlatMapFunction<String, WC>() { public void flatMap(String value, Collector<WC> out){ for (String token : value.toLowerCase().split("\\W")) { out.collect(new WC(token, 1)); } } });

words.groupBy("word").aggregate(SUM, "count");

public class WC { public String word; public int count;}

Page 36: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

stratosphere.eu

github.com/stratosphere/stratosphere

@stratosphere_eu

"Big Data looks tinyfrom Stratosphere"

Page 37: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Appendix

37

Page 38: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Fits into Hadoop Stack

38

• Analyzes HDFS data directly• Runs on top of YARN

Page 39: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Overview

39

Distributed Collections

(RDD)

WritableKey/Value

pairs

MapReduce

Iterative Data Flows

StratoSphereAbove the Clouds

Paradigm

Data Model

Runtime

Java/Scala types

as key/value pairsBatch

processingin memory

BatchParallel

Sort

Java/Scalatype

system

Streamingin-memory

& out of core

Compilation/Optimization

nonenoneholistic planning

fordata exchange,

sort/hash, caching, ...

Page 40: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Data ModelStratosphere vs. Spark

40

StratoSphereAbove the Clouds

Arbitrary Java Objects

Arbitrary Java Objects

Tuples asfirst class citizens

Key/value pairs asfirst class citizens

Joins / Grouping viaKey/value pairs

Joins / Grouping viafield references

(tuple position, selector-function)

(coming: field name)

Page 41: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

The infamous WordCount in Stratosphere

41

DataSet<String> text = env.readTextFile(input);

DataSet<WC> words = text.flatMap( new FlatMapFunction<String, WC>() { public void flatMap(String value, Collector<WC> out){ for (String token : value.toLowerCase().split("\\W")) { out.collect(new WC(token, 1)); } } });

words.groupBy( (WC v) -> return v.word; )

.reduce( new ReduceFunction<WC>() { public WC reduce(WC val1, WC val2) { return new WC(val1.word, val1.count + val2.count); } });

public class WC { public String word; public int count;}

Java API – POJO Variant

Page 42: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Simple and self contained

Programming/Testing

42

ExecutionEnvironment env = getExecutionEnvironment();

DataSet<String> text = env.fromElements("To be", "or not to be","or to be still", "and certainly not to be not at all", …);

DataSet<Tuple2<String, Integer>> result = text.flatMap(new Tokenizer()) .groupBy(0).aggregate(SUM, 1);

List<Tuple2<String, Integer>> list = new ArrayList<>();result.output(new CollectingOutput(list));env.execute();

// validate the list contents

Page 43: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Stratosphere offerstwo types of iterations

Bulk Iterations

DeltaIterations(aka. Workset

Iterations)

IterativeFunction

Initial Dataset

Result

InitialWorkset

InitialSolutionset

IterativeFunction

State

Result

43

Page 44: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

A Sample Bulk Iteration

44

Page 45: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

A Sample Delta Iteration

45

Connected Components of a Graph

def step = (s: DataSet[Vertex], ws: DataSet[Vertex]) => {

val min = ws groupBy {_.id} reduceGroup { x => x.minBy { _.component } }

val delta = s join minNeighbor where { _.id } isEqualTo { _.id } flatMap { (c,o) => if (c.component < o.component) Some(c) else None }

val nextWs = delta join edges where {v => v.id} isEqualTo {e => e.from} map { (v, e) => Vertex(e.to, v.component) } (delta, nextWs)}

val components = vertices.iterateWithWorkset(initialWorkset, {_.id}, step)

Page 46: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

A Sample Delta Iteration

46

def step = (s: DataSet[Vertex], ws: DataSet[Vertex]) => {

val min = ws groupBy {_.id} reduceGroup { x => x.minBy { _.component } }

val delta = s join minNeighbor where { _.id } isEqualTo { _.id } flatMap { (c,o) => if (c.component < o.component) Some(c) else None }

val nextWs = delta join edges where {v => v.id} isEqualTo {e => e.from} map { (v, e) => Vertex(e.to, v.component) } (delta, nextWs)}

val components = vertices.iterateWithWorkset(initialWorkset, {_.id}, step)

Define Step function

Return Delta andnext Workset

Invoke Iteration

46

Connected Components of a Graph

Page 47: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Spargel: The Graph API

47

A Giraph-style API in < 500 lines of code!

Vertex Centic computation

is a special case of a

Delta iteration

Page 48: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Workset Algorithm Illustrated

48

4 2

1 3

7

8

9

5 6

42

1 3

7 9

8

65

Algorithm: Find connected components of a graph.

Start: Vertices have IDs that respresent the component they belong to. Initially, every vertex has its own id (is its own component).

Step: Each vertex tells its neighbors its component id. Vertices take the min-ID of all candidates from their neighbors. A vertex that did not adopt a new ID needs not participate in the next step, as it has nothing new to tell its neighbors.

Page 49: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Workset Algorithm Illustrated

49

4 2

1 3

7

8

9

5 6

42

1 3

7 9

8

65

Solution Set Workset

Solution Set Delta

1 (2,2)(3,3)

2 (1,1)(3,3)(4,4)

3 (1,1)(2,2)

4 (2,2)

5 (6,6)

6 (5,5)

8 (7,7)(9,9)

9 (7,7)(8,8)

Messages sent to neighbors:

1 (4, 3) means that vertex 1 receives a candidate id of 3 from vertex 4

Page 50: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Workset Algorithm Illustrated

50

4 2

1 3

7

8

9

5 6

42

1 3

7 9

8

65

Solution Set Workset

Solution Set Delta

1 (2,2)(3,3)

2 (1,1)(3,3)(4,4)

3 (1,1)(2,2)

4 (2,2)

5 (6,6)

6 (5,5)

8 (7,7)(9,9)

9 (7,7)(8,8)

(2,1)(3,1)(4,2)

(6, 5)(8,7)(9,7)

Page 51: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Workset Algorithm Illustrated

51

4 2

1 3

7

8

9

5 6

21

1 1

7 7

7

55

Solution Set Workset

Solution Set Delta

1 (2,2)(3,3)

2 (1,1)(3,3)(4,4)

3 (1,1)(2,2)

4 (2,2)

5 (6,6)

6 (5,5)

8 (7,7)(9,9)

9 (7,7)(8,8)

(2,1)(3,1)(4,2)

(6, 5)(8,7)(9,7)

Page 52: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Workset Algorithm Illustrated

52

4 2

1 3

7

8

9

5 6

21

1 1

7 7

7

55

Solution Set Workset

Solution Set Delta

1 (2,1)(3,1)

2 (3,1)(4,2)

3 (2,1)

4 (2,1)

5 (6,5)

7 (8,7)(9,7)

8 (9,7)

9 (8,7)

(2,1)(3,1)(4,2)

(6, 5)(8,7)(9,7)

Page 53: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Workset Algorithm Illustrated

53

4 2

1 3

7

8

9

5 6

21

1 1

7 7

7

55

Solution Set Workset

Solution Set Delta

1 (2,1)(3,1)

2 (3,1)(4,2)

3 (2,1)

4 (2,1)

5 (6,5)

7 (8,7)(9,7)

8 (9,7)

9 (8,7)

(4,1)

Page 54: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Workset Algorithm Illustrated

54

4 2

1 3

7

8

9

5 6

11

1 1

7 7

7

55

Solution Set Workset

Solution Set Delta

(4,1)

1 (2,1)(3,1)

2 (3,1)(4,2)

3 (2,1)

4 (2,1)

5 (6,5)

7 (8,7)(9,7)

8 (9,7)

9 (8,7)

Page 55: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Workset Algorithm Illustrated

55

4 2

1 3

7

8

9

5 6

11

1 1

7 7

7

55

Solution Set Workset

Solution Set Delta

(4,1)

2 (4,1)

Page 56: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Optimization

val orders = DataSource(...)val items = DataSource(...)

val filtered = orders filter { ... }

val prio = filtered join items where { _.id } isEqualTo { _.id } map {(o,li) => PricedOrder(o.id, o.priority, li.price)}

val sales = prio groupBy {p => (p.id, p.priority)} aggregate ({_.price},SUM)

Filter

Grp/AggJoin

Orders Items

partition(0)

sort (0,1)

partition(0)

sort (0)

Filter

Join

Grp/Agg

Orders Items

(0,1)

(0) = (0)

(∅)

case class Order(id: Int, priority: Int, ...)case class Item(id: Int, price: double, )case class PricedOrder(id, priority, price)

56

Page 57: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Type Analysis/Code Gen

• Types and Key Selectors are mapped to flat schema

• Generated code for interaction with runtimePrimitive Types,

Arrays, Lists

Single Value

TuplesTuples /Classes

NestedTypes

Recursivelyflattened

recursivetypes

Tuples(w/ BLOB

for recursion)

Int, Double, Array[String], ...

(a: Int, b: Int, c: String)class T(x: Int, y: Long)

class T(x: Int, y: Long)class R(id: String, value: T)

(a: Int, b: Int, c: String)(x: Int, y: Long)

class Node(id: Int, left: Node, right: Node)

(id:Int, left:BLOB, right:BLOB)

(x: Int, y: Long)(id:String, x:Int, y:Long)

57

Page 58: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Type Analysis/Code Gen

• Implemented in the Scala API via Scala Macros

• Lift the AST, analyze types/code

• Generate type serializers/ accessors and glue code around UDF and type

• Implementation of type analysis via reflection

• Implementation of code generation in Java API in progress

58

Page 59: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Optimizing Programs• Program optimization happens in two phases

1. Data type and function code analysis inside the Scala Compiler

2. Relational-style optimization of the data flow

Run Time

Scala Compiler

ParserProgramType

Checker

Execution

CodeGeneration

Stratosphere Optimizer

InstantiateFinalize

Glue CodeCreate

ScheduleOptimize

AnalyzeData Types

GenerateGlue Code

Instantiate

59

Page 60: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Stratosphere APIsby Example

What does it look like using the system?

60

Page 61: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Scala API

• The infamous word count example

val input = TextFile(textInput)

val words = input flatMap { line => line.split("\\W+") }

val counts = words groupBy { word => word } count()

In-situ data source Transformation

function

Group by entire data type (the words)

Count per group

61

Page 62: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

DataSet Transformations

• Operations are methods on DataSet[A].

• Working with DataSet[A] feels like working with Scala collections.

• DataSet[A] is not an actual collection, but represents computation on a collection.

• Stringing together operations creates a data flow that can be executed.

62

Page 63: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Scala API by Example

• Graph Triangles (Friend-of-a-Friend problem)o Recommending friends, finding important connections

• 1) Enumerate candidate triads

• 2) Close as triangles

63

Page 64: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Scala API by Example

case class Edge(from: Int, to: Int)case class Triangle(apex: Int, base1: Int, base1: Int)

val vertices = DataSource("hdfs:///...", CsvFormat[Edge])

val byDegree = vertices map { projectToLowerDegree } val byID = byDegree map { (x) => if (x.from < x.to) x else Edge(x.to, x.from) } val triads = byDegree groupBy { _.from } reduceGroup { buildTriads }

val triangles = triads join byID where { t => (t.base1, t.base2) } isEqualTo { e => (e.from, e.to) } map { (triangle, edge) => triangle }

64

Page 65: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Scala API by Example

case class Edge(from: Int, to: Int)case class Triangle(apex: Int, base1: Int, base1: Int)

val vertices = DataSource("hdfs:///...", CsvFormat[Edge])

val byDegree = vertices map { projectToLowerDegree } val byID = byDegree map { (x) => if (x.from < x.to) x else Edge(x.to, x.from) } val triads = byDegree groupBy { _.from } reduceGroup { buildTriads }

val triangles = triads join byID where { t => (t.base1, t.base2) } isEqualTo { e => (e.from, e.to) } map { (triangle, edge) => triangle }

Custom Data Types In-situ data

source

65

Page 66: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Scala API by Example

case class Edge(from: Int, to: Int)case class Triangle(apex: Int, base1: Int, base2: Int)

val vertices = DataSource("hdfs:///...", CsvFormat[Edge])

val byDegree = vertices map { projectToLowerDegree } val byID = byDegree map { (x) => if (x.from < x.to) x else Edge(x.to, x.from) } val triads = byDegree groupBy { _.from } reduceGroup { buildTriads }

val triangles = triads join byID where { t => (t.base1, t.base2) } isEqualTo { e => (e.from, e.to) } map { (triangle, edge) => triangle }

Relational

Join

Non-relationallibrary function

Non-relational function

66

Page 67: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Scala API by Example

case class Edge(from: Int, to: Int)case class Triangle(apex: Int, base1: Int, base2: Int)

val vertices = DataSource("hdfs:///...", CsvFormat[Edge])

val byDegree = vertices map { projectToLowerDegree } val byID = byDegree map { (x) => if (x.from < x.to) x else Edge(x.to, x.from) } val triads = byDegree groupBy { _.from } reduceGroup { buildTriads }

val triangles = triads join byID where { t => (t.base1, t.base2) } isEqualTo { e => (e.from, e.to) } map { (triangle, edge) => triangle }

Key Reference

s

67

Page 68: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Iterative Program (Java)

68

Page 69: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Bulk Iteration Performance

69

Wikipedia Webbase Twitter0

500

1000

1500

2000

2500

3000

3500

4000

4500

SparkGiraphStratosphere Part.Stratosphere BC

Execu

tion

Tim

e (

sec)

Other systems ran out of memory

Cf. VLDB 2012 Paper "Spinning Fast Iterative Data Flows"

Page 70: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Delta Iteration Performance

70

Wikipedia Hollywood0

100

200

300

400

500

600

Spark Giraph Stratosphere Full Stratosphere Incr.

Execu

tion

Tim

e (

sec)

Twitter Webbase (20)0

1000

2000

3000

4000

5000

6000

Other systems ran out of memory

Cf. VLDB 2012 Paper "Spinning Fast Iterative Data Flows"

Page 71: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Per-Iteration Times

71

Spark Full Spark Sim. Incr. Giraph

Stratosphere Full Stratosphere Incr.Iteration

Execu

tion

Tim

e (

msecs)

Delta

Bulk

(Bagle)

Cf. VLDB 2012 Paper "Spinning Fast Iterative Data Flows"

Page 72: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Architecture: Front-Ends

Stratosphere Front-Ends

• Multiple Font-Ends for different target audiences

• Supports operations from both shallow analytics(SQL, Hadoop MapReduce) and deep analytics(Machine Learning, Data Mining)

• Supports custom user-defined operations as required to support the diverse Big Data use cases

• API may be used to create connectors to other application tools (visualization, dashboards, …)

• Shared compiler/optimizer allows easy creation of new front-ends (unlike Hadoop eco-system).

Sky JavaAPI

Storage

Stratosphere Runtime

HDFS

Local

Files S3

Cluster YARN EC2 Direct

Stratosphere Compiler/Optimizer

Sky ScalaAPI

Meteor

...

.

.

.

Page 73: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Architecture: Compiler

Sky JavaAPI

Storage

Stratosphere Runtime

HDFS

Local

Files S3

Cluster YARN EC2 Direct

Stratosphere Compiler/Optimizer

Sky ScalaAPI

Meteor

...

.

.

.

Stratosphere Compiler

• Inspired by Relational Database Optimizer(optimizes SQL, enables efficient complex queries)

• Extended to non-relational use cases through code analysis techniques

• Eliminates costly and time-intensive manual tuning of analysis tasks to the data, automatically adapts programs when the data characteristics change

• Extended to iterative algorithms, optimizes machine learning algorithms and subsumes specialized systems

• Code generation techniques efficiently support custom operations in the system

Page 74: Stratosphere / Flink - Berlin Buzzwords · PDF fileStratosphere / Flink ... 15 Possible execution 1) Partitioned hash- ... o Gradient descent (e.g., Logistic Regression, Matrix Factorization)

Architecture: Runtime

Sky JavaAPI

Storage

Stratosphere Runtime

HDFS

Local

Files S3

Cluster YARN EC2 Direct

Stratosphere Compiler/Optimizer

Sky ScalaAPI

Meteor

...

.

.

.

Stratosphere Runtime

• Hybrid between a Parallel Database and aMapReduce engine.

• Supports both database operations, and custom operation defined by users for specialized use cases

• Streaming Engine Fast, low latency queries

• Support for stateful multi-pass algorithms Very efficient for machine learning and graph analysis algorithms

• Heavily in-memory Fast on modern computers

• Out-of-core capabilities Scales beyond main memory