Streaming Visualization...Source: adapted from Tibco Edge. Apache Kafka –A Streaming Platform...

http://guidoschmutz.wordpress.com@gschmutz

Streaming VisualizationDOAG Konferenz 2019Guido Schmutz

Agenda

1. Motivation / Introduction

2. Stream Data Integration & Stream Analytics Ecosystem

3. Three Blueprints for Streaming Visualization

End-to-End Demo available here:https://github.com/gschmutz/various-demos/tree/master/streaming-visualization

GuidoWorking at Trivadis for more than 22 yearsConsultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast DataOracle Groundbreaker Ambassador & Oracle ACE Director

@gschmutz guidoschmutz.wordpress.com

edition

Motivation / Introduction

Timely decisions require new data immediately

Keep the data in motion …

Data at Rest Data in Motion

(Re)Act

Visualize/Analyze

StoreAct

Analyze

111010101010110

Visualize

Hadoop ClusterdHadoop ClusterBig Data

Reference Architecture for Data Analytics Solutions

Search

Service

BI Tools

Enterprise Data Warehouse

Search / Explore

File Import / SQL Import

Event Hub

Data Flow

Data FlowChange DataCapture Parallel

Processing

Storage

Results

SQL Export

Microservice State

StreamProcessor

EventStream

Search

Service

Stream Analytics

MicroservicesEnterprise Apps

Edge Node

Event Hub

Storage

Bulk Source

Event Source

Location

DBExtract

IoTData

MobileApps

Social

Event Stream

Telemetry

Two Types of Stream Processing(by Gartner)

Stream Data Integration• focuses on the ingestion and processing of

data sources targeting real-time extract-transform-load (ETL) and data integration use cases

• filter and enrich the data

Stream Analytics• targets analytics use cases

• calculating aggregates and detecting patterns to generate higher-level, more relevant summary information (complex events)

• Complex events may signify threats or opportunities that require a response from the business

Gartner: Market Guide for Event Stream Processing, Nick Heudecker, W. Roy Schulte

Stream Data Integration & Stream Analytics Ecosystem

Stream Analytics

Event Hub

Open Source Closed Source

Stream Data Integration

Source: adapted from Tibco

Apache Kafka – A Streaming Platform

Kafka Cluster

Consumer 1 Consume 2r

Broker 1 Broker 2 Broker 3Zookeeper Ensemble

ZK 1 ZK 2ZK 3

Schema Registry

Service 1

Management

Control Center

Kafka Manager

KAdmin Producer 1 Producer 2

kafkacat

Data Retention:• Never• Time (TTL) or Size-based• Log-Compacted based

Producer3Producer3

ConsumerConsumer 3

Apache Kafka – A Streaming Platform

SourceConnector

SinkConnector

trucking_driver

KSQL Engine

Kafka Streams

Kafka Broker

Demo using Kafka Stack for Stream Data Integration

Stream Analytics

Event Hub

Stream Data Integration & Stream Analytics

StreamingVisualization

Data Flow

ConsumerData Sources

Data Flow ??

Filter: #doag2019,….User: @gschmutz

Demo: Kafka Connect to retrieve Tweets

curl -X "POST" "$DOCKER_HOST_IP:8083/connectors" \-H "Content-Type: application/json" \--data '{

"name": "twitter-source","config": {

"connector.class": "com.github.jcustenborder.kafka.connect.twitter.TwitterSourceConnector",

"twitter.oauth.consumerKey": "xxxxx","twitter.oauth.consumerSecret": "xxxxx","twitter.oauth.accessToken": "xxxx","twitter.oauth.accessTokenSecret": "xxxxx","process.deletes": "false","filter.keywords": "#doag2019","filter.userIds": "15148494","kafka.status.topic": "tweet-raw-v1","tasks.max": "1"}

Demo: KSQL for Streaming ETL

CREATE STREAM tweet_sWITH (KAFKA_TOPIC='tweet-v1', VALUE_FORMAT='AVRO', PARTITIONS=8) AS SELECT id , createdAt , text , user->screenNameFROM tweet_raw_s;

CREATE STREAM tweet_raw_s WITH (KAFKA_TOPIC='tweet-raw-v1', VALUE_FORMAT='AVRO');

SELECT id, lang, removestopwords(split(LCASE(text), ' ')) AS word FROM tweet_raw_sWHERE lang = 'en' or lang = 'de';

SELECT id, LCASE(hashtagentities[0]->text) FROM tweet_raw_sWHERE hashtagentities[0] IS NOT NULL;

Demo using Kafka Stack for Stream Data Integration

Stream Analytics

Event Hub

Data Flow

Data Flow ??

Filter: #voxxeddaysbanff,#java,#kafka,….User: @VoxxedDaysBanff, @gschmutz

Visualization: many many options!

But do they all support Streaming Data?

Three Blueprints forStreaming Visualization

BP1: Fast datastore with regular polling from consumer

Storage

Stream Analytics

Event Hub

Data Store

Data Flow

Data In Motion Data at Rest

Data Flow

BP1-1: Elasticsearch / Kibana

Storage

Stream Analytics

Event Hub

Data Store

Data Flow

Alternatives:SOLR & Banana

BP1-2: InfluxDB / Grafana or Chronograf

Storage

Stream Analytics

Event Hub

Data Store

Data Flow

Alternatives:Prometheus & GrafanaDruid & Superset

BP1-3: NoSQL & Custom Web

Storage

Stream Analytics

Event Hub

Data Store

Data Flow

BP-1: Demo Redis NoSQL & Custom Web

https://opensky-network.org/

BP1-4: Kafka Streams Interactive Query & Custom App

Storage

Stream Analytics

Event Hub

Data Store

Data Flow

Alternatives:Flink…

BP2: Direct Streaming to the Consumer

Stream Analytics

Event Hub

Data Flow

Data In Motion

Data Flow

Channel/Protocol

BP2-1: Kafka Connect to Slack / WhatsApp

Stream Analytics

Event Hub

Data Flow

Data In Motion

Data Flow

Channel/Protocol

Alternatives:TwitterSMS…

BP-2-1: Demo Kafka Connect to Slack

curl -X "POST" "$DOCKER_HOST_IP:8083/connectors" \-H "Content-Type: application/json" \--data '{

"name": "slack-sink","config": {"connector.class": "net..SlackSinkConnector","tasks.max": "1","topics":"slack-notify","slack.token":”XXXX","slack.channel":"general","message.template":"tweet by ${USER_SCREENNAME} with ${TEXT}",

BP2-2: Kafka to Tipboard (Dashboard Solution)

Stream Analytics

Event Hub

Data Flow

Data In Motion

Data Flow

Channel/Protocol

Alternatives:DashingGeckoboard…

BP2-2: Demo Kafka to Tipboard (Dashboard Solution)

http://allegro.tech/tipboard/

BP2-2: Demo Kafka to Tipboard (Dashboard Solution) c.subscribe(['DASH_TWEET_COUNT_BY_HOUR_T'])

while True:msg = c.poll(1.0)

data = json.loads(msg.value().decode('utf-8'))data_selected = data.get('NOF_TWEETS’)data_prepared = prepare_for_just_value(data_selected)data_jsoned = json.dumps(data_prepared)data_to_push = { 'tile': TILE_NAME, 'key': TILE_KEY

, 'data': data_jsoned }resp = requests.post(API_URL_PUSH, data=data_to_push)

def prepare_for_just_value(data):# data={"title": "Number of Tweets:", "description": "(1 hour)", "just-value": "23"

data_prepared = datadata_prepared = {'title': '# Tweets:', 'description': 'per hour’,

'just-value': data_prepared}return data_prepared

BP2-3: Web Sockets / SSE & Custom Modern Web App

Stream Analytics

Event Hub

Data Flow

Data In Motion

Data Flow

Channel/Protocol

Sever Sent Event (SSE)

BP3: Streaming SQL Result to Consumer

Stream Analytics

Event Hub

Stream Data Integration & Stream Analytics ConsumerData Sources

Data In Motion

Data Flow

API StreamingVisualization

BP3-1: KSQL and Arcadia Data

Stream Analytics

Event Hub

Data In Motion

Data Flow

BP3-1: Demo KSQL and Arcadia Data

https://www.arcadiadata.com/

BP3-2: KSQL with REST API to Custom Web App

Stream Analytics

Event Hub

Data In Motion

Data Flow

BP3-2: Demo KSQL with REST API

curl -X POST -H 'Content-Type: application/vnd.ksql.v1+json’ -i http://analyticsplatform:8088/query --data '{

"ksql": "SELECT text FROM tweet_raw_s;","streamsProperties": { "ksql.streams.auto.offset.reset": "latest” }

{"row":{"columns":["The latest The Naji Filali Daily! https://t.co/9E6GonrySE Thanks to @Xavier_Porter1 @ClouMedia #ai #bigdata"]},"errorMessage":null,"finalMessage":null}

{"row":{"columns":["RT @Futurist_Invest: This robot can copy your face! Creepy \n\n#SaturdayThoughts#SaturdayMorning #creepy #bots #bot #AI #bigdata #robotics #…"]},"errorMessage":null,"finalMessage":null}

{"row":{"columns":["She’s back telling us all about why datathons are exciting now :) Catch her while you can! �@ARUKscientist� �@S_Bauermeister� #bigdata #ARUKConfhttps://t.co/Br484db5ut"]},"errorMessage":null,"finalMessage":null}

{"row":{"columns":["Blockchain Competitive Innovation Advantage"]},"errorMessage":null,"finalMessage":null}

BP3-3: Spark Streaming & Oracle Stream Analytics

Stream Analytics

Event Hub

Data In Motion

Data Flow

BP3-3: Demo Spark Streaming & Oracle Stream Analytics

https://www.oracle.com/middleware/technologies/complex-event-processing.html

Summary

BP1: Fast Store & Polling

• “classic” pattern

• Not end-to-end “data-in-motion” -> “Data-at-rest” before visualization

• Slight delay might not be acceptable for monitoring dashboard

• Can use full power of data store(s) => NoSQL

• In-memory reduces overhead

BP2: Stream to Consumer

• minimal latency

• More difficult on “client side”

• good if stream holds directly what should be displayed

• More difficult if data in stream needs to be analyzed before visualization

• No historical info available

BP3: Streaming SQL

• Minimal latency

• Power of SQL query engine available for visualization

• possibility for “self-service” style visualization

• Some analytics are more difficult on streaming data

• No historical info available

Streaming Visualization...Source: adapted from Tibco Edge. Apache Kafka –A Streaming Platform...

Documents

Transcript of Streaming Visualization...Source: adapted from Tibco Edge. Apache Kafka –A Streaming Platform...

Streaming in Practice - Putting Apache Kafka in Production

Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Cloudera Distribution of Apache Kafka · PDF fileinstancesofthesamerole.AserviceinstancemightbeKafka-1.Kafka-1mighthosttheroleinstancesBroker-1, Broker-2,andBroker-3. Kafkabrokersprocessrecordsorganizedintotopics.

ÿ ô ¢ @ £ 8 ú Û ¾ | Y a t / Z < i ? Flamingo · MQTT FTP HDFS Kafka MQTT FTP HTTP Kafka Realtime Distributed Message Broker Spark Streaming 1;17,Q 2 0 ...

Data Streaming with Apache Kafka & MongoDB - EMEA

Streaming Data and Stream Processing with Apache Kafka

Fast, Scalable, Streaming Applications with Spark Streaming, the Kafka API and the HBase API

Kafka as Message Broker

Exactly-Once Streaming from Kafka-(Cody Koeninger, Kixer)

Introduction to streaming and messaging flume,kafka,SQS,kinesis

Streaming Big Data with Spark Streaming, Kafka, Cassandra ...chariotsolutions.com/wp-content/uploads/2015/04/HelenaEdelson_ETE... · Streaming Big Data with Spark Streaming, Kafka,

Streaming Data Ingest and Processing with Apache Kafka

Apache Kafka with Spark Streaming: Real-time Analytics Redefined

Confluent: Streaming operational data with Kafka – Couchbase Connect 2016

Real Time Aggregation with Kafka ,Spark Streaming and ...

WITSML data processing with Kafka and Spark Streaming

Data Streaming with Apache Kafka & MongoDB

Spark streaming with apache kafka

Kafka Tutorial: Streaming Data Architecture

Real-time streaming and data pipelines with Apache Kafka