Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

33
WEBINAR Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End Dr. Dean Wampler (@deanwampler)

Transcript of Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Page 1: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

WEBINAR

Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Dr. Dean Wampler (@deanwampler)

Page 2: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Upgrade your grey matterGet Dean’s free O’Reilly book from Lightbend

bit.ly/fastdata-ORbook

http://bit.ly/fastdata-ORbook
Page 3: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Streaming Engines in Context…

Page 4: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Classic Batch Architecture: Hadoop

Page 5: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Logs

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

YARN

ResourceManager

NodeManager

NM

Batch

MapReduce

…

Spark

Flume

SqoopDBs

Page 6: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Logs

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

YARN

ResourceManager

NodeManager

NM

Batch

MapReduce

…

Spark

Flume

SqoopDBs

Page 7: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Logs

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

YARN

ResourceManager

NodeManager

NM

Batch

MapReduce

…

Spark

Flume

SqoopDBs

Page 8: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Logs

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

YARN

ResourceManager

NodeManager

NM

Batch

MapReduce

…

Spark

Flume

SqoopDBs

Page 9: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Logs

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

YARN

ResourceManager

NodeManager

NM

Batch

MapReduce

…

Spark

Flume

SqoopDBs

Page 10: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

New Streaming, “Fast Data” Architecture (but it also supports batch)

Page 11: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Mesos, YARN, Cloud, …

Logs

Sockets

RESTZooKeeper Cluster

ZK

Mini-batch

SparkStreaming

Batch

Spark

…

Low Latency

Flink

Ka9aStreamsAkkaStreams

Gearpump Beam

…

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

1

5

6

311

KaEa Cluster

Ka9a

Microservices

RP Go

Node.js …

24

7

8

9

10

Beam

Page 12: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Mesos, YARN, Cloud, …

Logs

Sockets

RESTZooKeeper Cluster

ZK

Mini-batch

SparkStreaming

Batch

Spark

…

Low Latency

Flink

Ka9aStreamsAkkaStreams

Gearpump Beam

…

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

1

5

6

311

KaEa Cluster

Ka9a

Microservices

RP Go

Node.js …

24

7

8

9

10

Beam

Page 13: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Mesos, YARN, Cloud, …

Logs

Sockets

RESTZooKeeper Cluster

ZK

Mini-batch

SparkStreaming

Batch

Spark

…

Low Latency

Flink

Ka9aStreamsAkkaStreams

Gearpump Beam

…

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

1

5

6

311

KaEa Cluster

Ka9a

Microservices

RP Go

Node.js …

24

7

8

9

10

Beam

Page 14: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Mesos, YARN, Cloud, …

Logs

Sockets

RESTZooKeeper Cluster

ZK

Mini-batch

SparkStreaming

Batch

Spark

…

Low Latency

Flink

Ka9aStreamsAkkaStreams

Gearpump Beam

…

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

1

5

6

311

KaEa Cluster

Ka9a

Microservices

RP Go

Node.js …

24

7

8

9

10

Beam

Page 15: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Mesos, YARN, Cloud, …

Logs

Sockets

RESTZooKeeper Cluster

ZK

Mini-batch

SparkStreaming

Batch

Spark

…

Low Latency

Flink

Ka9aStreamsAkkaStreams

Gearpump Beam

…

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

1

5

6

311

KaEa Cluster

Ka9a

Microservices

RP Go

Node.js …

24

7

8

9

10

Beam

Page 16: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Mesos, YARN, Cloud, …

Logs

Sockets

RESTZooKeeper Cluster

ZK

Mini-batch

SparkStreaming

Batch

Spark

…

Low Latency

Flink

Ka9aStreamsAkkaStreams

Gearpump Beam

…

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

1

5

6

311

KaEa Cluster

Ka9a

Microservices

RP Go

Node.js …

24

7

8

9

10

Beam

Page 17: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Mesos, YARN, Cloud, …

Logs

Sockets

RESTZooKeeper Cluster

ZK

Mini-batch

SparkStreaming

Batch

Spark

…

Low Latency

Flink

Ka9aStreamsAkkaStreams

Gearpump Beam

…

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

1

5

6

311

KaEa Cluster

Ka9a

Microservices

RP Go

Node.js …

24

7

8

9

10

Beam

Page 18: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Mesos, YARN, Cloud, …

Logs

Sockets

RESTZooKeeper Cluster

ZK

Mini-batch

SparkStreaming

Batch

Spark

…

Low Latency

Flink

Ka9aStreamsAkkaStreams

Gearpump Beam

…

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

1

5

6

311

KaEa Cluster

Ka9a

Microservices

RP Go

Node.js …

24

7

8

9

10

Beam

Page 19: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Streaming Engines

Page 20: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Features to Consider

Page 21: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

• Low latency? How low?• …

Page 22: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

• Low latency? How low?• High volume? How high?• …

Page 23: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

• Low latency? How low?• High volume? How high?• Integration with other tools? Which ones?• …

Page 24: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

• Low latency? How low?• High volume? How high?• Integration with other tools? Which ones?• Kinds of data processing, analytics? Which ones?•Bulk processing of records?•Individual processing of events?

Page 25: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Example Streaming Engines

Page 26: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Mesos, YARN, Cloud, …

Logs

Sockets

RESTZooKeeper Cluster

ZK

Mini-batch

SparkStreaming

Batch

Spark

…

Low Latency

Flink

Ka9aStreamsAkkaStreams

Gearpump Beam

…

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

1

5

6

311

KaEa Cluster

Ka9a

Microservices

RP Go

Node.js …

24

7

8

9

10

Beam

Page 27: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Mesos, YARN, Cloud, …

Logs

Sockets

RESTZooKeeper Cluster

ZK

Mini-batch

SparkStreaming

Batch

Spark

…

Low Latency

Flink

Ka9aStreamsAkkaStreams

Gearpump Beam

…

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

1

5

6

311

KaEa Cluster

Ka9a

Microservices

RP Go

Node.js …

24

7

8

9

10

Beam

Page 28: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End
Page 29: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Mesos, YARN, Cloud, …

Logs

Sockets

RESTZooKeeper Cluster

ZK

Mini-batch

SparkStreaming

Batch

Spark

…

Low Latency

Flink

Ka9aStreamsAkkaStreams

Gearpump Beam

…

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

1

5

6

311

KaEa Cluster

Ka9a

Microservices

RP Go

Node.js …

24

7

8

9

10

Beam

Page 30: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Mesos, YARN, Cloud, …

Logs

Sockets

RESTZooKeeper Cluster

ZK

Mini-batch

SparkStreaming

Batch

Spark

…

Low Latency

Flink

Ka9aStreamsAkkaStreams

Gearpump Beam

…

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

1

5

6

311

KaEa Cluster

Ka9a

Microservices

RP Go

Node.js …

24

7

8

9

10

Beam

Page 31: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Mesos, YARN, Cloud, …

Logs

Sockets

RESTZooKeeper Cluster

ZK

Mini-batch

SparkStreaming

Batch

Spark

…

Low Latency

Flink

Ka9aStreamsAkkaStreams

Gearpump Beam

…

Persistence

S3

HDFS

DiskDiskDisk

SQL/NoSQLSearch

1

5

6

311

KaEa Cluster

Ka9a

Microservices

RP Go

Node.js …

24

7

8

9

10

Beam

Page 32: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

Upgrade your grey matterGet Dean’s free O’Reilly book from Lightbend

http://bit.ly/fastdata-ORbook

http://bit.ly/fastdata-ORbook
Page 33: Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End

For more information on Lightbend Fast Data Platform:

lightbend.com/fast-data-platform

https://www.lightbend.com/fast-data-platform

Streaming Data in R

Streaming Data in R

Module 3: Selecting Weather Data - Welcome to FarmD€¦ · Mombasa, Kenya Module 3: Selecting Weather Data + ... Wind speed Rainfall ... Module 3 –Selecting Weather Data

Module 3: Selecting Weather Data - Welcome to FarmD€¦ · Mombasa, Kenya Module 3: Selecting Weather Data + ... Wind speed Rainfall ... Module 3 –Selecting Weather Data

Lab Streaming Layer-Enabled Myo Data Collection Software ... · Lab Streaming Layer-Enabled Myo Data Collection Software User Manual . ... Lab Streaming Layer-Enabled Myo Data ...

Lab Streaming Layer-Enabled Myo Data Collection Software ... · Lab Streaming Layer-Enabled Myo Data Collection Software User Manual . ... Lab Streaming Layer-Enabled Myo Data ...

STREAMING VIDEO DATA - Home - AMD

STREAMING VIDEO DATA - Home - AMD

Streaming Data Mining - Computer Science Liberty , Jelani Nelson : Streaming Data Mining 3 / 111. The need for Streaming Data Mining ... Until the stream is consumed Edo Liberty ,

Streaming Data Mining - Computer Science Liberty , Jelani Nelson : Streaming Data Mining 3 / 111. The need for Streaming Data Mining ... Until the stream is consumed Edo Liberty ,

Data in Motion: Streaming Static Data Efficiently 2

Data in Motion: Streaming Static Data Efficiently 2

Streaming Data with scalaz-stream

Streaming Data with scalaz-stream

Data in Motion: Streaming Static Data Efficiently

Data in Motion: Streaming Static Data Efficiently

Kafka Tutorial: Streaming Data Architecture

Kafka Tutorial: Streaming Data Architecture

Data Streaming for Metabolomics: Accelerating Data ...Data Streaming for Metabolomics: Accelerating Data Processing and ... J. Rafael Montenegro-Burke,†, Aries E. Aisporna,†, ...

Data Streaming for Metabolomics: Accelerating Data ...Data Streaming for Metabolomics: Accelerating Data Processing and ... J. Rafael Montenegro-Burke,†, Aries E. Aisporna,†, ...

Querying Streaming XML Data

Querying Streaming XML Data

New Case-Base Maintenance: A Streaming Approachceur-ws.org/Vol-1815/paper22.pdf · 2017. 3. 17. · streaming is a knowledge discovery method for selecting representative elements

New Case-Base Maintenance: A Streaming Approachceur-ws.org/Vol-1815/paper22.pdf · 2017. 3. 17. · streaming is a knowledge discovery method for selecting representative elements

Building Fast Applications for Streaming Data

Building Fast Applications for Streaming Data

Streaming real time data with Vibe Data Stream

Streaming real time data with Vibe Data Stream

Selecting Data Security Technology

Selecting Data Security Technology

Streaming Data and Concurrency in R

Streaming Data and Concurrency in R

Dell EMC Streaming Data Platform: Gettingtelcos ready for the … · Dell EMC Streaming Data Platform is a holistic data streaming solution designed to deliver a cohesive, scalable,

Dell EMC Streaming Data Platform: Gettingtelcos ready for the … · Dell EMC Streaming Data Platform is a holistic data streaming solution designed to deliver a cohesive, scalable,

· Web viewLesson 01. Selecting Data----- Total Training for SQL Server 2005-- -- Part 2 - Selecting Data

· Web viewLesson 01. Selecting Data----- Total Training for SQL Server 2005-- -- Part 2 - Selecting Data

Viewing Streaming Spatially-Referenced Data at …hjs/pubs/acmgis14-labeling-header.pdfViewing Streaming Spatially-Referenced Data ... streaming queries such as windowing realized

Viewing Streaming Spatially-Referenced Data at …hjs/pubs/acmgis14-labeling-header.pdfViewing Streaming Spatially-Referenced Data ... streaming queries such as windowing realized

Categorical Skylines for Streaming Data

Categorical Skylines for Streaming Data