Make Streaming Analytics work for you: The Devil is in the Details

1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Make Streaming Analytics Work for you: The Devil is in the DetailsKanishk Mahajan

June 2016

Director Product Management, Hortonworks

Ryan MedlinDirector Software Engineering, Neustar


Agenda• Building a Stream Analytics Platform:

Requirements

• Building a Stream Analytics Platform: Capabilities

• Use Case -• CyberSecurity - Threat Protection

• Q&A

Building a Stream Analytics PlatformRequirements


5 Attributes of a Streaming Platform

Ingest Process Analyze Visualize Respond


Data Flow versus Stream Analytics

Data Flow– Ingest and route terabytes of data into a ”unified firehose”–Actively performance manage the latency and quality of these data flows -

– across high variability of data formats, size of data and speed of data

–Produces Streams - ”unbounded sequence of events” Real Time Stream Analytics

–Sub second event processing with linear scalability to billions of events –Predictive Analytics at Scale

– Real time data aggregation across edge nodes while processing 10s of millions of events and 100s of gigabytes per second

–Guaranteed no data loss and events processed in order


6 key focus areas for building a Streaming Platform

Common Abstraction Layer Latency Lambda Architecture

–“Orchestrate” over static and real time data Scale-out Rapid Application Development Data Visualization


Focus Areas for Streaming Platform

Common Abstraction Layer–Select one or more streaming engines–Select one or more cloud providers–Select one or more event sources–…Future Proof



Latency–500 millisecond or less- Dashboards, Security Incidents, Asset Performance–20 milliseconds or less - Ad Networks, Preventive Maintenance–…



Lambda Architecture–Integrate static and real time data–Enrichment–Orchestration of Batch workflows–Predictive Classification and Scoring



Scale Out–Linear Scale out or scale down–Resource Management–Handle Transient workloads


Focus Areas for Streaming Platform Rapid Application Development

–Aggregations–Filters–Multi Stream Correlations–Splits–Joins–Normalizations–Business Rules Editor


Focus Areas for Streaming Platform Data Visualization

–Time Series Visualizations–Metrics Dashboarding–Trends–Comparisons–Thresholds–Custom UI extensions


Real Time Stream Analytics versus Streaming Engine

Abstracts underlying Streaming Engine- Storm, Spark Streaming, Flink.. OOTB Support for multiple (cloud) event sources - Kafka, AWS Kinesis, Azure Event

Hub Built-in Operators for Complex Event Processing Built-in Real Time Dashboarding- Metrics and Events Pluggable Workflow Management Business Rules Editor and Rapid Application Development Framework Cloud Deployment Scalable Architecture Handles different latency requirements

Hortonworks Inc Confidential 2016

Building a Stream Analytics PlatformCapabilities


Stream Analytics Platform - Capabilities

• Continuous, Real Time Processing–Continuous Query and near real time processing of event data–Distributed, scalable platform for processing continuous unbounded streams of data–Abstraction across multiple open source stream engine technologies– Large Scale Event Processing–No coding approach for event processing logic

• Event Source–Multiple event source support–Multiple data format support–Drag and Drop, no coding approach

• Event Storage–Multiple event storage support– Event Retention time period support–Drag and Drop, no coding approach


Streaming Platform Capabilities - Analytics

Detect Trends Alert on probable Failures

Descriptive Proactive Predictive Prescriptive

• Rules Calculations on predefined settings

• High Throughput, Low Latency Event Processing Framework

• Train Models Offline• Provide Query interface and

storage of Models• Integrate Real Time Data

with Batch Data

• Connect with Enterprise Applications

Reporting and DashboardingVisual Pattern Recognition

Monitor System Alert before Failure

Take Action on Data Avoid Failure and

Drive Operational Efficiency



• Filtering–Event Selection based on pre-defined criteria

• Transformations–Aggregations–Event Prioritization–Event Deduplication–Event Time Stamping–Custom User Defined Functions

• Enrichment–Pull in reference data for additional context



• Correlations–Correlation of in-flight event data from multiple event streams

• Temporal Calculations–Time Based Calculations on in-flight data in an event stream for e.g. a windowing

computation with a time based window

• Pattern Detection–Calculating trends on in-flight data in an event stream–Outlier detection : Pattern Matching with historical data



• Serialization/Deserialization–Marshalling/Unmarshalling event data into a series of bytes–Drag and Drop, no coding support

• Rule Engine–Support for business rules (business directives that impact a decision) –Support for invoking external business rule engine

• Visualization–Reporting and Dashboarding–Visual Pattern Recognition


Stream Analytics - Capabilities Summary

# Capability

1. Filtering

2. Transformations

3. Enrichment

4. Correlations

5. Temporal Calculations

6. Pattern Detection

# Capability

1. Continuous Real Time Processing

2. Event Sources

3. Event Storage

4. Temporal Calculations

5. Rules Engine

6. Visualization

# Capability

7. Filtering

8. Transformation

9. Event Storage

10. Enrichment

11. Correlations

12. Pattern Detection


Use CasesCyberSecurity - Threat Protection


CyberSecurity - Threat Protection Use Case


CyberSecurity - Threat Protection - StepsApplying Streaming Analytics

• Expose• Provide Visibility into your enterprise• Passively watch network traffic and construct events to describe the activity it sees.• “Ingest”

• Transmit• Compress, Encrypt, De-identify event data• Send to the cloud for centralized log retention, real-time threat analysis and incident investigation



• Prioritize• Aggregate intelligence across all control points to identify and prioritize those systems that remain

compromised and require immediate remediation.• Allows analysts to “zero in” on just those events of most importance.• Significantly reduce the number of incidents that security analysts need to investigate• Combine global telemetry from cyber intelligence networks with local customer context across

endpoints, networks and email, to uncover attacks that would otherwise evade detection.• “Correlate” and “Enrich”



• Analyze• Create comprehensive detection rules, behavioral analytics and guided investigations to detect the

latest threats.• Enable quick and nimble data exploration across billions of events to proactively hunt for hidden

indicators of compromise (IOCs).• Provide agile investigation tooling/UI to pivot from one indicator to the next, reconstruct the attack

storyline and plan a forceful response to disrupt the attack.• Allow Security analysts to visualize all related attack components, e.g., all files used in the attack,

all email addresses, all malicious IP addresses, etc.• “Proactive”, “Proactive”, “Prescriptive” and “Descriptive” Analytics• “Real Time Analytics across large volumes of data at scale”



• Remediate• Generate alerts to expedite validating and scoping the incident.• Enrich alerts with supporting data. This data could include threat intelligence, point-in-time

context regarding users impacted, actions taken and hosts involved help you validate and scope the incident.

• Allow a Security Investigator to hunt for Indicators-of-Compromise across all your endpoints from a single console.

• Leverage your existing installations of Endpoint Protection and Email Security for enforcement without having to install any new endpoint agents.

• Export rich security intelligence into third-party security incident and event management systems (SIEMs).

• “Notifications”, “Event Storage”, “Workflow”


Make Streaming Analytics Work for you: The Devil is in the DetailsKanishk Mahajan

June 2016

Director Product Management, Hortonworks

Ryan MedlinDirector Software Engineering, Neustar

AREAS OF FOCUS AND INNOVATION FOR NEUSTAR

Security

User

Device

Anomaly Detection

Multi-Factor Authentication (PKI, etc)

Policy Creation/Decisioning/Enforcement

Local / Edge / Distributed functionality

Openness and Standardization

(OCF/OneM2M/MQTT/CoAP/GitHub)

Partnerships


DEVICE AND IOT PLATFORM

Devices

Data Processing Services

Local & Offline Functionality

Device and User Authentication

Device Directory & Identity

User and Device Policy

Device Certification & Public / Private Key Distribution/MFA

OIC/OCF and IoTivity Standards Based Platform for Interoperability & device definition

Absolute, Group/Collection, Local & cloud

Verification / Fraud /

Anomaly detection

Location Services

Future

Peer-to-Peer communication & Discovery. decentralization

Secure Discovery & Communication OCF based security & communication protocols

Data pipeline, enrichment and analytics

CORE DATA PLATFORM REQUIREMENTS

Provide Ingestion for all Machine Data Exhaust from all system components. (ex: Anomaly Detection, Monitoring/Alerting)

Deserializing avro messages using given schema(s) and by default store a message in hdfs.

Given a message attribute (for example device type and customer Id or device Id) execute a rule to determine next course of action which should comprise of one or more of these actions:– Notify a user via email or sms or 3rd party api – Notify or pass message to registry service/API– Store data in hdfs– Store data in a relational data store (hbase phoenix or other?)



Outgoing Messages


Turn a Light on ManuallyInbound Message Flow


Incoming Messages

INTERESTING PROBLEMS We need to work in offline and temporary offline mode so data replication is the hardest

problem to solve and is based on unique use cases. Also need to run a subset of the Cloud Inbound real time service rules and processing.

How to reuse this logic both in the cloud and locally on a Local Gateway is a design challenge with tradeoffs.



Q & A

Make Streaming Analytics work for you: The Devil is in the Details

Technology

Transcript of Make Streaming Analytics work for you: The Devil is in the Details