1

http://flink.incubator.apache.org@ApacheFlink

Apache FlinkNext-Gen Data Analytics Platform

2

What is Apache Flink

An efficient distributed general-purpose data analysis platform.

Runs on top ofHDFS and YARN.

Focusing on ease of programming.

Project status• Incubation at Apache Software Foundation

• Version 0.6 stable, 0.7-SNAPSHOT

• Came out of the Stratosphere research project

• Growing community of contributors

4

Flink Stack

JavaAPI

Storage

Flink Runtime

HDFS Local Files S3

ClusterManager

YARN EC2 Direct

Flink Optimizer

ScalaAPI

Spargel(graphs)

SQL,Python

...JDBC

HadoopMapReduce

Hive ...

5

Key Features

Easy to use developer APIs• Java, Scala, Graphs, Nested Data

(Python & SQL under development) • Flexible composition of large

programs

Automatic Optimization• Join algorithms• Operator chaining• Reusing partitioning/sorting

High Performance Runtime• Complex DAGs of operators• In memory & out-of-core• Data streamed between operations

Native Iterations• Embedded in the APIs• Data streaming / in-memory• Delta iterations speed up many

programs by orders of mag.

6

Flink Overview

7

DataSet<String> text = env.readTextFile(input);

DataSet<Tuple2<String, Integer>> result = text.flatMap(new Splitter()).groupBy(0).aggregate(SUM, 1);

// map function implementationclass Splitter extends FlatMap<String, Tuple2<String, Integer>> {

public void flatMap(String value, Collector out){ for (String token : value.split("\\W")) { out.collect(new Tuple2<String, Integer>(token, 1)); } }}

Concise & rich APIs

Word Count, Java API

Can use regular POJOs!

8

Concise & rich APIs

val input = TextFile(textInput)val words = input flatMap { line => line.split("\\W+") }val counts = words groupBy { word => word } count()

Word Count, Scala API

9

Flexible Data Pipelines

Reduce

Join

Map

Reduce

Map

Iterate

Source

Sink

Source

10

Rich Building Blocks

• Map, FlatMap, MapPartition• Filter, Project• Reduce, ReduceGroup, Aggregate, Distinct• Join• CoGoup• Cross• Iterate• Iterate Delta• Graph Vertex-Centric (Pregel style)

11

DataSet<Tuple...> large = env.readCsv(...);DataSet<Tuple...> medium = env.readCsv(...);DataSet<Tuple...> small = env.readCsv(...);

DataSet<Tuple...> joined1 = large.join(medium).where(3).equals(1) .with(new JoinFunction() { ... });

DataSet<Tuple...> joined2 = small.join(joined1).where(0).equals(2) .with(new JoinFunction() { ... });

DataSet<Tuple...> result = joined2.groupBy(3).aggregate(MAX, 2);

Example: Joins in Flink

Built-in strategies include partitioned join and replicated join with local sort-merge or hybrid-hash algorithms.

⋈⋈

γ

large medium

small

12

DataSet<Tuple...> large = env.readCsv(...);DataSet<Tuple...> medium = env.readCsv(...);DataSet<Tuple...> small = env.readCsv(...);

DataSet<Tuple...> joined1 = large.join(medium).where(3).equals(1) .with(new JoinFunction() { ... });

DataSet<Tuple...> joined2 = small.join(joined1).where(0).equals(2) .with(new JoinFunction() { ... });

DataSet<Tuple...> result = joined2.groupBy(3).aggregate(MAX, 2);

Automatic Optimization

Possible execution 1) Partitioned hash-join

3) Grouping /Aggregation reuses the partitioningfrom step (1) No shuffle!!!

2) Broadcast hash-join

Partitioned ≈ Reduce-sideBroadcast ≈ Map-side

13

Running Programs

> bin/flink run prg.jar

Packaged ProgramsRemote EnvironmentLocal Environment

Program JAR file

JVM

master master

RPC &Serialization

RemoteEnvironment.execute()LocalEnvironment.execute()

Spawn embeddedmulti-threaded environment

14

Flink Runtime

15

Distributed Runtime• Master (Job Manager)

handles job submission, scheduling, and metadata

• Workers (Task Managers) execute operations

• Data can be streamed between nodes

• All operators startin-memory and graduallygo out-of-core

16

Runtime Architecture(comparison)

DistributedCollection

List[WC]

public class WC { public String word; public int count;}

emptypage

Pool of Memory Pages• Collections of objects• General-purpose

serializer (Java / Kryo)• Limited control over

memory & less efficientspilling

• Deserialize all or nothing

• Works on pages of bytes• Maps objects transparently to these pages• Full control over memory, out-of-core enabled• Algorithms work on binary representation• Address individual fields (not deserialize whole object)

Flink

17

Iterative Programs

18

Why Iterative Algorithms

• Algorithms that need iterationso Clustering (K-Means, Canopy, …)o Gradient descent (e.g., Logistic Regression, Matrix Factorization)o Graph Algorithms (e.g., PageRank, Line-Rank, components, paths,

reachability, centrality, )o Graph communities / dense sub-componentso Inference (believe propagation)o …

• Loop makes multiple passes over the data

19

Iterations in other systems

Step Step Step Step Step

ClientLoop outside the system

Step Step Step Step Step

ClientLoop outside the system

Iterations in Flink

20

Streaming dataflowwith feedback

map

join

red.

join

System is iteration-aware, performs automatic optimization

Flink

21

Automatic Optimization for Iterative Programs

Caching Loop-invariant DataPushing work„out of the loop“

Maintain state as index

22

Unifies various kinds of Computations

ExecutionEnvironment env = getExecutionEnvironment();

DataSet<Long> vertexIds = ...DataSet<Tuple2<Long, Long>> edges = ...

DataSet<Tuple2<Long, Long>> vertices = vertexIds.map(new IdAssigner());

DataSet<Tuple2<Long, Long>> result = vertices .runOperation( VertexCentricIteration.withPlainEdges( edges, new CCUpdater(), new CCMessager(), 100));

result.print();env.execute("Connected Components");

Pregel/Giraph-style Graph Computation

23

Delta Iterations speed up certain problems

by a lot

0

200000

400000

600000

800000

1000000

1200000

1400000

Iteration

# V

ert

ices

(th

ou

san

ds)

Bulk

Delta

Twitter Webbase (20)0

1000

2000

3000

4000

5000

6000

Computations performed in each iteration for connected communities of a social graph

Runtime (secs)

Cover typical use cases of Pregel-like systems with comparable performance in a generic platform and developer API.

24

Program Optimization

25

Why Program Optimization ?

Do you want to hand-optimize that?

26

What is Automatic Optimization

Run on a sampleon the laptop

Run a month laterafter the data evolved

Hash vs. SortPartition vs. BroadcastCachingReusing partition/sortExecution

Plan A

ExecutionPlan B

Run on large fileson the cluster

ExecutionPlan C

27

Using Flink

28

http://flink.incubator.apache.org

29

Its easy to get started…

wget http://www.apache.org/dyn/closer.cgi/incubator/flink/flink-0.6-incubating-bin-hadoop2-yarn.tgz

tar xvzf flink-0.6-incubating-bin-hadoop2-yarn.tgz ./flink-0.6-incubating/bin/yarn-session.sh -n 4 -jm 1024 -tm 3000

If you have YARN, deploy a full Flink setup in 3 commands:

Also works on Amazon Elastic MapReduce ;-)

Quickstart projects set up a program skeleton, includingembedded local execution/debugging environment…

For the experts…

trying it out…

$ wget https://.../flink-0.6-incubating.tgz$ tar xzf flink-*.tgz$ flink/bin/start-local.sh

running a local pseudo-clusterAlso available as aDebian package

30

Tutorial Example

flink.incubator.apache.org

github.com/apache/incubator-flink

@ApacheFlink

Where to find us