Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.

IOT, STREAMING ANALYTICS AND MACHINE LEARNINGDelivering Real-Time Intelligence With Apache NiFi

Paul Kent, VP of Big Data, Platform R&DDan Zaratsian, Sr. Solutions Architect

Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.

Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.

• Drop-and-Drag Interface

• Secure/Encrypted

• Bi-Directional Communication

• Data Provenance

Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.

Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.

SAS ESP + HDF WHY IS THIS IMPORTANT?

RAPID PROTOTYPING OF MACHINE LEARNING MODELS

ANALYTICS WITHIN AN OPEN FRAMEWORK

Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.

Events(Data In)

Events(Data Out)

FilteringAggregation

Pattern DetectionComputation

Merging / JoinsFunctions

Retention WindowText Analytics

Unsupervised LearningPredictive Modeling

…more…

Detect Events & Patterns of Interest

Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.

KEY CONCEPTSESP MODEL - PROCESS FLOW

SAS EVENT STREAM PROCESSING ENGINE

DATA IN (Events)

DATA OUT(Events)

Design of the rule model (called “Continuous Query”)using components (called “Windows”)

DATA IN (Events)

DATA IN (Events)

DATA OUT(Events)

SOURCE1

WINDOW

SOURCE2

WINDOW

SOURCE3

WINDOW

FILTERWINDOW

CALCULATIONSWINDOW

JOINWINDOW

JOINWINDOW

NOTIFICATIONWINDOW

PREDICTIVE MODEL

(SCORING)

WINDOW

Copyr igh t © 2012, SAS Ins t i tute Inc . A l l r i gh ts r es erved.

EVENT STREAM PROCESSING ESP STUDIO INTERFACE

Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.

DESIGNING ESP MODELS

DESIGN COMPONENTSDS2 PROCEDURAL WINDOW

In-Stream Analytics:

1. Build analytical model using EM,VA, etc.• Decision Tree• Neural Network• Regression• Rule Induction• And more

2. Use PROC DSTRANS to convert code to DS23. Deploy model to procedural window

Only when the existing model is additive in nature and can process one event at a time.

Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.

DESIGNING ESP MODELS

DESIGN COMPONENTSPATTERN WINDOW OVERVIEW

BUILD COMPLEX NETWORK OF EVENTS USING TEMPORAL CONDITIONSMULTIPLE EVENTS IN CAN PRODUCE ONE EVENT OUT

E1 E2And

FollowedBy

E4 E5AndNot

E6

E3Or

5 min

1 hourFollowed

By

“Detect when event A is followed by event B and not Event C in a 3min time frame”

Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.

DESIGNING ESP MODELS

DESIGN COMPONENTSTEXT ANALYTICS WINDOWS

• Process unstructured text fields• 3 dedicated Text Analytics windows

• Text Context (.liti files)• Text Category (.mco files)• Text Sentiment (.sam files)

• An appropriate Text Analytics license is required.

Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.

SAS EVENT STREAM PROCESSING HTML5 STREAMVIEWER

• HTML5 interface• Uses HTTP (RESTful) XML server• 2 Modes:

• Streaming mode: display all events• Update : events processed with opcode

• Google charts• Subscribe & Publish

Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.

INTEGRATION SAS EVENT STREAM PROCESSING & HORTONWORKS DATA FLOW (NIFI)

&SAS Event Stream Processing Hortonworks Data Flow (Nifi)

Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.

SAS® EVENT STREAM PROCESSING CONCEPTUAL OVERVIEW

SAS-generated Insights

Enrichment Data

Event Actions

SAS In-Memory

SAS® Event Stream Processing Model

Continuous QueryPu

blis

h

Subs

crib

e

Streaming Events

Analytic Models

Business Rules

Nifi

Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.

SAS EVENT STREAM PROCESSING CONNECTORS & ADAPTERS

PUB/SUB API Connect to any system with Java or CPublic, documented and easy to useAdapters are standalone processes and can be networkedPublish to ESP Source windows – Subscribe to any ESP windowAll Connectors & Adapters are built using the Pub/Sub API

•File/Socket•XML / JSON•Database (odbc)•SAS® LASR™•Hadoop•SAS® Dataset

OUT OF THE BOX

*Publish only**Subscribe only

•ESP Project•RabbitMQ•Solace•Tervela•Google Protobuff•Twitter*

•SAS® HDAT•JMS•IBM WebSphere MQ•Tibco RendezVous•Syslog *•Network Sniffer*

•HTTP RESTful•OSIsoft PI•Axeda•Teradata•SMTP **•ESP to ESP

Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.

DIRECTION: ADAPTERS & CONNECTORS

•Flume: Integrate ESP with streaming log data•Kafka: Integration with large scale message processing•MQTT: Support within IoT and Connected things•Cassandra (adaptor only): integration with large-scale, distributed data source•HortonWorks Data Flow (NiFi) Processor: support NiFI streams•MapR: MapR Streams support•Boardreader: Blogs, News, Boards, Reviews•Spryware: Market data through direct exchange feeds•IOT Gateways and devices

PUB/SUB APIConnect to any system with Java or CDocumented and easy to useAdapters are standalone processes and can be networkedPublish to ESP Source windows – Subscribe to any ESP windowAll Connectors & Adapters are built using the Pub/Sub API

SAS® EVENT STREAM PROCESSING 4.1

Copyr igh t © 2015, SAS Ins t i tute Inc . A l l r i gh ts r es erved.

SAS® EVENT STREAM PROCESSING 4.1 FLEXIBILITY AND INTEGRATION

• Python Pub/sub API: drive ESP using Python• Leverage Analytic Decisions within ESP

• Decision/Rules/Analytical Model Integration via SAS Micro Analytic Service• Lightweight, fast service for decision deployment

• Leverage Languages in ESP (In-process Event Stream Handlers)• DATAstep (native)• DS2 (current)• Python• Future: R language (post-16w48)

Company Confidential - For Internal Use OnlyCopyright © 2015, SAS Insti tute Inc. Al l r ights reserved.

Streaming Analytics Ecosystem

EdgeAnalytics

In-MotionAnalytics

At-RestAnalytics

Connected Systems, Devices

Monitor equipment on for failures and safety

issues, and take action.

Identify fraudulent transactions and be alerted in real-time.

Intelligently integrate customerinformation with real-time

streaming data

Strategic Data IntegrationTransactions, Logs, Clickstreams

Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.

STREAMING ANALYTICS Where are the Opportunities?

• Competitive Pressure (Technology, Sensors, Analytics)

• Risk

• Safety

• Security

• Personalization

Extend the existing analytical footprint!Capture value otherwise lost through information lag

Copy r igh t © 2012, SAS Ins t i tute Inc . A l l r i ghts r eserved.

INTEGRATION SAS EVENT STREAM PROCESSING & HORTONWORKS DATA FLOW (NIFI)

&SAS Event Stream Processing Hortonworks Data Flow (Nifi)

Copyright © 2012, SAS Institute Inc. All rights reserved.Copyright © 2012, SAS Institute Inc. All rights reserved.

Paul Kent, VP of Big Data, Platform R&DDan Zaratsian, Sr. Solutions Architect

Copyright © 2012, SAS Institute Inc. All rights reserved.Copyright © 2012, SAS Institute Inc. All rights reserved.

Demo

Company Confidential - For Internal Use OnlyCopyright © 2015, SAS Insti tute Inc. Al l r ights reserved.

Company Confidential - For Internal Use OnlyCopyright © 2015, SAS Insti tute Inc. Al l r ights reserved.

Company Confidential - For Internal Use OnlyCopyright © 2015, SAS Insti tute Inc. Al l r ights reserved.

Company Confidential - For Internal Use OnlyCopyright © 2015, SAS Insti tute Inc. Al l r ights reserved.