Complex Event Processing:

Power your middleware with

StreamInsight

Mahesh Patel (Microsoft)

Amit Bansal (PeoplewareIndia.com)

Agenda

The Value of Timely Analytics

The challenges / Scenarios

Introduction to Complex Event Processing

Introducing Microsoft StreamInsight

Brief introduction to BizTalk Server

Understand the key scenarios that StreamInsight can unlock for BizTalk

Map out the architectural, integration requirements for BizTalk + StreamInsight.

The value of timely analytics

Current Products for analytics

Current Products for analytics

Introduction to CEP

Complex Event Processing (CEP) is an architecture style based on the principles of Event Driven Architecture

(EDA).

Continuously processing a high-volume stream of events from various event sources

Events are logically grouped in sequences (typically known as streams) based on a time defined criteria such

as an interval.

Event streams are processed through a series of queries

Capabilities of a CEP Application

Event Driven

Continuous Queries

Time Sensitive

Low Latency

High Volume

The value of timely analytics

Scenarios

Stream Data Store & Archive

Event Processing Engine

Dat

a St

ream

Asset Specs & Parameters

Power, Utilities:

Energy consumption Outages Smart grids 100,000 events/sec

Visual trend-line and KPI monitoring Batch & product management Automated anomaly detection Real-time customer segmentation Algorithmic trading Proactive condition-based maintenance

Web Analytics:

Click-stream data Online customer behavior Page layout 100,000 events /sec

Manufacturing:

Sensor on plant floor React through device

controllers

Aggregated data 10,000 events/sec

Threshold queries Event correlation from multiple

sources Pattern queries

Lookup

Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds

Financial Services:

Stock & news feeds Algorithmic trading Patterns over time Super-low latency 100,000 events /sec

Dat

a St

ream

Introducing StreamInsight

DEMO

CEP using StreamInsight

Traditional messaging applications:

High signal every message is business relevant

Rich data schemas sets of related information

Workflow centric and transactional

Emerging data sources and trends:

Machine born data is growing at a very rapid rate

Not all of this data is business relevant (low signal/noise ratio)

How do we:

Identify and extract business relevant events from streaming data?

Take action on these business insights (line of business, human workflow, etc)

Messages and Events

LOB

Large Messages Low Volume

High Volume Small Events

Web Services

B 2 2B SWIFT

RFID

Stock Feeds

Electrical Grids

Operational Data

Health Care

CRM

Send Port

MessageBox

Orchestration Receive Port

Message

Maps

Receive Pipeline

Adapter (HTTP)

Message

LOB

Host Host

Send Port

Adapter (File)

Map

Send Pipeline

Component

Basic BizTalk Architecture

Component

Host

Management DB Tracking DB

Message Agent

Message Agent

Message Agent

Scenarios

Good BizTalk + StreamInsight scenarios:

Streaming data/event sources high noise to signal ratio

Need to analyze and extract business relevant events

Integrate and transact with line of business applications.

Scenarios

Manufacturing (Oil and Gas) Monitor plant floor processes Identify business relevant events Transact with LOB apps

Utilities Automated metering handle flood of data from

utility meters

Identify business relevant events Transact with LOB apps

The data feeds change, the question changes the core scenario doesnt!

OSIsoft PI & StreamInsight

OSIsoft PI - Industry leading process data management and historian.

Integrating and leveraging StreamInsight

OSIsoft shipping StreamInsight adapter for PI.

Stream data in and out of PI

Extract powerful business insights from PI data

Integrate with LOB applications using BizTalk Server

OPC & StreamInsight (with

MatrikonOPC)

OPC Industry standard for process automation and data exchange

MatrikonOPC Industry leader in OPC interfaces

Developed StreamInsight adapters for their OPC implementation

Extract powerful business insights from OPC data

Integrate with LOB applications using BizTalk Server

Automated Metering

Contoso Utilities is rolling out smart electrical meters to their customers.

Meters report power usage (wattage, voltage) at a point in time.

Data is consolidated into a back end application used for billing and forecasting.

The volume of data has forced Contoso to only read values from meters every hour

They want to:

Monitor the data more closely, compressing the point stream to an interval stream before loading into the back end application.

i.e. if the data does not change by more than 2% over a 10 minute window, keep the average value.

Capture spikes and threshold events, route to a process control application for further inspection.

Automated Metering Collect high volume streaming data from smart meters

Data consists of timestamp, source ID and payload (power in Watts)

Sample operations on the streaming data: Consolidate stream of points events into 5 minute chunks; record

average, count and standard deviation.

Detect anomalies (threshold 10%), collect last 5 minutes worth of data for that meter and all meters fed from that substation

Need to link meter ID with reference data (location, substation)

Results must be integrated into line of business application

Timestamp 3/5/2010 14:23

Power 1020

Meter ID 112322

Data consolidation

(point -> interval)

Anomaly Detection

Reference data (location, hierarchy)

Batch to Message

Batch to Message

Line of Business

Application

Smart Meters

Automated Metering

Timestamp 3/5/2010 14:23

Power 1020

Meter ID 112322

Data consolidation

(point -> interval)

Anomaly Detection

Reference data (location, hierarchy)

Batch to Message

Batch to Message

Line of Business

Application

Smart Meters

StreamInsight

Meter Input Adapter

(StreamInsight)

ConsolidationQuery

Threshold Detection Query

MessageBuilderOutputAdapter

BizTalk Server

WCFMSMQAdapter

ReceivePort

OrchestrationSendPort

LOBAdapter

QueryResults (XML)

QueryResults (XML)

Query Result Format (XSD)

Query Result Format (XSD)

1

1

2

2

3

3

4

5

4

5

Architecture Message Integration

WC

F A

dapte

rs

WC

F A

dapte

rs

BizTalk Host

Orchestration

Business

Activity

Monitoring

Business

Rules

Engine

Message Box

StreamInsight

XmlPayload Output

Adapter

Queries

Input Adapters

XSD Schema

XSD Schema

Batched QueryOutput

Batched QueryOutput

1

2

3

1 StreamInsight extracts insights from real time data feeds, feeds aggregate information to BizTalk.

2 A strongly typed XML schema is inferred from the StreamInsight query

3 Batched and correlated query results are delivered to a BizTalk receive port.

StreamInsight is an in-memory event processing engine.

Do not deploy StreamInsight instance on the same hardware as your BizTalk host instances.

Leverage StreamInsight-local MSMQ for message transfer.

BizTalk

Hosts

SQL Servers

for BizTalk

MSM

QStreamInsight

Instance

Deployment Message Integration

Summary BizTalk Server and StreamInsight together can unlock powerful scenarios:

High volume data processing

Message integration

Leverage StreamInsight to:

Consolidation and analyze incoming data streams

convert event sets into messages

Deliver to BizTalk receive port

Leverage BizTalk Server to:

Integrate these insights into line of Business applications

Other potential alignment points:

Business Activity Monitoring (BAM). Tap into BAM event events stream for rich analytics and event detections

Business Rules Engine (BRE). Leverage BRE to define event pattern rules

2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in

the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft

must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any

information provided after the date of this presentation.

MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.