Apache Flink: Real-World Use Cases for Streaming Analytics

Slim Baltagi @SlimBaltagi

Brazil - Sao Paulo Apache Flink MeetupMarch 17th, 2016

Agenda

I. What is Apache Flink Stack?II. Movement from Batch Analytics to Streaming AnalyticsIII. Key Differentiators of Apache Flink for Streaming AnalyticsIV. Real-World Use Cases with Flink for Streaming AnalyticsV. Who is using Flink?VI. Where do you go from here?

2

I. What is Apache Flink stack?

Gel

lyTa

ble

Had

oop

M/R

SAM

OA

DataSet (Java/Scala/Python)Batch Processing

DataStream (Java/Scala)Stream Processing

Flin

kML

LocalSingle JVMEmbedded

Docker

ClusterStandalone YARN, Mesos (WIP)

CloudGoogle’s GCEAmazon’s EC2IBM Docker Cloud, …

Apa

che

Bea

m

Apa

che

Bea

m

MR

QL

Tabl

e

Cas

cadi

ng

Runtime : Distributed Streaming Dataflow

Zepp

elin

DEP

LOY

SYST

EMA

PIs

& L

IBR

AR

IES

STO

RA

GE Files

LocalHDFS

S3, Azure StorageTachyon

DatabasesMongoDB HBaseSQL …

Streams FlumekafkaRabbitMQ…

Batch Optimizer Stream Builder

Stor

m

Flin

kCEP

Gel

ly-S

trea

m3

I. What is Apache Flink stack?See First Apache Flink meetup in South America that I

gave as a webinar on February 24th 2016. It is titled: Introduction to Apache Flink: What, How, Why, Who, Where? https://www.youtube.com/watch?v=YAKdD1rHCxs (Part 1)

See similar talk on February 2nd 2016 that I previously gave a at the New York City Apache Flink which. Now, the world’s largest Flink meetup• Slideshttp

://www.slideshare.net/sbaltagi/apacheflinkwhathowwhywhowherebyslimbaltagi-57825047

• Video recording https://www.youtube.com/watch?v=G77m6Ou_kFA

Flink Knowledge Base: all resources related to Flink http://sparkbigdata.com/component/tags/tag/27-flink 4

Agenda

I. What is Apache Flink Stack?II. Movement from Batch Analytics to Streaming AnalyticsIII. Key Differentiators of Apache Flink for Streaming AnalyticsIV. Real-World Use Cases with Flink for Streaming AnalyticsV. Who is using Flink?VI. Where do you go from here?

5

II. Movement from Batch Analytics to Streaming Analytics

Batch StreamingHigh-latency apps Low-latency apps

Static Files Event Streams

Process-after-store Sense-and-respondBatch processors Stream processors

6

What is batch processing? Many big data sources represent series of events that

are continuously produced. Example: tweets, web logs, user transactions, system logs, sensor networks, …

Batch processing:  These events are collected together based on the number of records or a certain period of time (a day for example) and stored somewhere to be processed as a finite data set.

What’s the problem with ‘process-after-store’ model: • Unnecessary latencies between data generation and

analysis & actions on the data. • Implicit assumption that the data is complete after a

given period of time and can be used to make accurate predictions for example.

7

What is stream processing? Most data is available as series of events (click

streams, mobile apps data, .. ) continuously produced by a variety of applications and systems in the enterprise.

Data sources are not anymore typical enterprise sources but new ones such as social media data, sensor data …

Data from disparate systems (internally and externally) can be integrated in a central hub and: Made available as low-latency data streams

required for real-time stream processing. Loaded into your data warehouse for offline

analysis.

8

Factors behind the movement from Batch Analytics to Streaming Analytics

There is a movement in Big Data processing from Batch Analytics to Streaming Analytics driven by many factors:• Data streams: Sensors networks, mobile apps data, .. • Technology: Rapidly growing open source streaming

analytics tools, vendors innovating in this space, more mobile devices than human beings, cloud services for real-time stream processing…

• Business: Organizations are more and more embracing streaming analytics for faster time to insight and competitive advantages.

• Customers: Costumers are becoming more and more demanding for instant responses in the way they are used to in social networks: twitter, facebook, linkedin… 9

Agenda

I. What is Apache Flink Stack?II. Batch vs. Streaming AnalyticsIII. Key Differentiators of Apache Flink for Streaming AnalyticsIV. Real-World Use Cases with Flink for Streaming AnalyticsV. Who is using Flink?VI. Where do you go from here?

10

III. Key Differentiators of Apache Flink for Streaming AnalyticsThe 8 Requirements of Real-Time Stream Processing,

Stonebraker et al. 2005 • Original paper http://cs.brown.edu/~ugur/8rulesSigRec.pdf• A short summaryhttp

://blog.acolyer.org/2014/12/03/the-8-requirements-of-real-time-stream-processing/

Apache Flink fulfills all these requirements and more!• http://data-artisans.com/real-time-stream-processing-the-next-step-for-apache-flink/• http

://data-artisans.com/flink-0-10-a-significant-step-forward-in-open-source-stream-processing/• http://data-artisans.com/flink-1-0-0/• https://cloud.google.com/dataflow/blog/dataflow-beam-and-spark-comparison• https

://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JKXm-778DAD7eqw5GANwE/edit

• http://www.slideshare.net/robertmetzger1/january-2016-flink-community-update-roadmap-2016/9

11

III. Key Differentiators of Apache Flink for Streaming AnalyticsTrue Low latency streaming engine: fast results in milliseconds

High throughput: handle large data amounts (millions of events per second)• http://data-artisans.com/extending-the-yahoo-streaming-benchmark/

Exactly once guarantees: Correct results, also in failure cases• http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-pr

ocessing-with-apache-flink/

Programmability: Higher level, Intuitive and easy to use APIs

Backpressure refers to the situation where a system is receiving data at a higher rate than it can process during a temporary load spike.• http://data-artisans.com/how-flink-handles-backpressure/

Event time and out of order stream processing • http://data-artisans.com/how-apache-flink-enables-new-streaming-applications-pa

rt-1/

Stateful stream processing and versioning state• http://data-artisans.com/how-apache-flink-enables-new-streaming-applications/

12

Agenda

I. What is Apache Flink Stack?II. Batch vs. Streaming AnalyticsIII. Key Differentiators of Apache Flink for Streaming AnalyticsIV. Real-World Use Cases with Flink for Streaming AnalyticsV. Who is using Flink?VI. Where do you go from here?

13

IV. Real-World Use Cases with Flink for Streaming Analytics

Stonebraker et al. make the case in 2005 that stream processing is going to become increasingly important. Not just for the usual finance, fraud, and command-and-control use cases, but also….… “as the “sea change” caused by cheap micro-sensor technology takes hold, we expect to see everything of material significance on the planet get “sensor-tagged” and report its state or location in real time. This sensorization of the real world will lead to a “green field” of novel monitoring and control applications with high-volume and low-latency processing requirements.”Reference:http://blog.acolyer.org/2014/12/03/the-8-requirements-of-real-time-stream-processing/

14

Shift from Reactive approach to proactive approach

Capturing new data and providing the ability to process streams of this data is allowing organizations to shift • From: taking a REACTIVE, post transaction

approach

• To: more of a PROACTIVE, pre decision approach to interactions with their customers, suppliers and employees.

Again, no matter the vertical, this transition is happening.

15

…to real-time personalization

From static branding

…to repair before break

From break then fix

…to designer medicine

From mass treatment

…to automated algorithms

From educated investing

…to 1x1 targetingFrom mass branding

A shift in Advertising

A shift in Financial Services

A shift in Healthcare

A shift in Retail

A shift in Manufacturing

Big Data Analytics Frameworks enable shifting the business from…

Reactive

Proactive

Shift from Reactive approach to proactive approach

16

Real-Time Monitoring of Customer Activity Events

17

Generic Streaming Analytics Architectural pattern. Ev

ent

Prod

ucer

s

Even

t Col

lect

or

Even

t Bro

ker

Even

t Pro

cess

or

Inde

xer

Visu

aliz

er/S

earc

h

• Kafka• RabitMQ• JMS

• Flink• Spark• Storm• Samza

• ElasticSearch• Solr• Cassandra• NoSQL DB

• Kibana• Custom

GUI

• Flume• SpringXD• Logstash• Nifi• Fluentd

• Apps• Devices• Sensors

18

IV. Real-World Use Cases with Flink for Streaming AnalyticsBelow is list several use cases, taken from real

industrial situations: Financial Services

– Real-time fraud detection.– Real-time mobile notifications.

Healthcare– Smart hospitals - collect data and readings from hospital

devices (vitals, IVs, MRI, etc.) and analyze and alert in real time.

– Biometrics - collect and analyze data from patient devices that collect vitals while outside of care facilities.

Ad Tech– Real-time user targeting based on segment and preferences.

Oil & Gas• Real-time monitoring of pumps/rigs. 19

IV. Real-World Use Cases with Flink for Streaming Analytics

Retail• Build an intelligent supply chain by placing sensors or RFID

tags on items to alert if items aren’t in the right place, or proactively order more if supply is low.

• Smart logistics with real-time end-to-end tracking of delivery trucks.

Telecommunications• Real-time antenna optimization based on user location data.• Real-time charging and billing based on customer usage,

ability to populate up-to-date usage dashboards for users.• Mobile offers.• Optimized advertising for video/audio content based on what

users are consuming.

20

Agenda

I. What is Apache Flink Stack?II. Batch vs. Streaming AnalyticsIII. Key Differentiators of Apache Flink for Streaming AnalyticsIV. Real-World Use Cases with Flink for Streaming AnalyticsV. Who is using Flink?VI. Where do you go from here?

21

V. Who is using Flink?ho is using Apache Flink? How companies are using Flink as presented at Flink

Forward 2015. Kostas Tzoumas and Stephan Ewen. http://www.slideshare.net/stephanewen1/flink-use-cases-bay-area-meetup-october-2015

Powered by Flink page: https://cwiki.apache.org/confluence/display/FLINK/Powered+by+Flink 22

V. Who is using Flink? is using Apache Flink? has its hack week and the winner was

a Flink based streaming project! December 18, 2015• Extending the Yahoo! Streaming Benchmark and Winning

Twitter Hack-Week with Apache Flink. Posted on February 2, 2016 by Jamie Grier http://data-artisans.com/extending-the-yahoo-streaming-benchmark/

did some benchmarks to compare performance of their use case implemented on Apache Storm against Spark Streaming and Flink. Results posted on December 18, 2015• http://yahooeng.tumblr.com/post/135321837876/benchmarking-stream

ing-computation-engines-at

• http://data-artisans.com/extending-the-yahoo-streaming-benchmark/• https://github.com/dataArtisans/yahoo-streaming-benchmark

23

Agenda

I. What is Apache Flink Stack?II. Batch vs. Streaming AnalyticsIII. Key Differentiators of Apache Flink for Streaming AnalyticsIV. Real-World Use Cases with Flink for Streaming AnalyticsV. Who is using Flink?VI. Where do you go from here?

24

VI. Where do you go from here?

A few resources for you:

• Flink at the Apache Software Foundation: flink.apache.org/

• Free ebook from MapR: Streaming Architecture: New Designs Using Apache Kafka and MapR Streams https://www.mapr.com/streaming-architecture-using-apache-kafka-mapr-streams

• Free Apache Flink training from data Artisans http://dataartisans.github.io/flink-training/ Still version 0.10.1 and not latest 1.0

• Flink Knowledge Base: One-Stop for everything related to Apache Flink http://sparkbigdata.com/component/tags/tag/27-flink

• Apache Flink in Action is probably the First book on Apache Flink! It will be published by Manning. I am co-authoring this book! Please stay tuned for the MEAP: Manning Early Access Program! 25

VI. Where do you go from here? A few takeaways :

• Organizations are more and more embracing streaming analytics for:• Use cases requiring lower latency: monitoring,

altering, … • Faster time to insight • Competitive advantages

• By leveraging streaming analytics, new startups are challenging established companies. Example: Pay-As-You-Go insurance or Usage-Based Auto Insurance

• Speed is said to have become the new currency of business.

26

Thanks!To all of you for attending!Let’s keep in touch!

• sbaltagi@gmail.com• @SlimBaltagi• https://www.linkedin.com/in/slimbaltagi

Any questions?

27