Complex Event Processing - A brief overview
-
Upload
istvan-david -
Category
Education
-
view
309 -
download
2
description
Transcript of Complex Event Processing - A brief overview
![Page 1: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/1.jpg)
Budapesti Műszaki és Gazdaságtudományi EgyetemMéréstechnika és Információs Rendszerek Tanszék
Complex Event ProcessingA brief overview
Dávid István
![Page 2: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/2.jpg)
Key concepts
Event
o „An immutable record of a past occurrence of an action or state change.”
o An observable entity triggered by a component of the system (IT infrastructure, software, etc).
Event stream
o A time ordered sequence of events in time.
Atomic event vs. Complex event
o A complex event is a compound structure of several atomic events.
![Page 3: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/3.jpg)
Typical use-cases of CEP
Stock market trading
o Efficient exploitation of the window-of-opportunity
IT infrastructure monitoring
o Evaluating compound metrics
Security
o Prevention of dDOS attacks
Online fraud detection
o UEFA online betting system
Automotive vehicle architectures
o Timing constraint modeling for ECU signals
Processing modes:• Low-latency online
processing• Batch processing
![Page 4: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/4.jpg)
A slightly different use-case
„a computer system which uses information gleaned from omnipresentsurveillance to predict future terrorist attacks and ordinary crimes”
![Page 5: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/5.jpg)
Background
Employed techniques
o Event abstraction
o Pattern matching
o Algebraic representation of processing logic(a.k.a. the process algebra)
o …
Assumptions and requirements
o An observable event stream
o A CEP engine which includes the proper implementation of the processing algebra
o Scalable behavior (because of the high-volume data)
Event representation (POJO, JSON, XML…)
Detecting matches on the event stream
A formal framework for defining
interrelationships among events
![Page 6: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/6.jpg)
Custom processing algebra?
Carlson: An Intuitive and Resource-Efficient Event Detection AlgebraBroy, Stefanescu: The Algebra Of Stream Processing FunctionsDe Lathauwer: Signal Processing Based on Multilinear Algebra
Go ahead!
![Page 7: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/7.jpg)
Infrastructure alternatives
Components themselves publish events into a central repository
Cf. Event-driven architectureDB (in-memory), [email protected],
log file…
Executed asynchronously
![Page 8: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/8.jpg)
Infrastructure alternatives
A monitoring framework is employed to collect events from components (instrumentation)
Monitoring framework with
agents (e.g. Nagios)
![Page 9: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/9.jpg)
Infrastructure alternatives
In general: we assume a messaging channel, which…
o carries the event stream generated by the monitoring logic (i.e. the components OR a 3rd-party framework)
o is accessible for the CEP platform
![Page 10: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/10.jpg)
Notable implementations
ESPER
Drools Fusion (JBoss)
IBM InfoSphere Streams (formerly: System S)
Oracle CEP
Microsoft StreamInsight
StreamBase
Lots of academic frameworks
![Page 11: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/11.jpg)
Typical objection #1
Q: „I’m a coder guy. Can’t I just implement the event processing logic in my source code?”
A: Yes, you can, but:
o you will have to implement a custom processing algebra
o as your event processing component evolves, first you will encounter the benefits of using business rules over if-else structures, then you will employ business roles with timing extensions for event processing
this is exactly how JBoss Drools Fusion works
![Page 12: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/12.jpg)
Typical objection #2
Q: „I’m all about databases. Can’t I just put my events into a DB and write some queries and triggers to handle the event stream?”
A: Yes, you can, but:
o you will have to implement a custom processing algebra
o in order to allow low-latency on-line processing, you will have to employ an in-memory DB and explore the benefits of the active database model
this is exactly how Esper works
![Page 13: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/13.jpg)
Examples
Drools Fusionrule „Rule1”
when w:Withdrawal (amount>=200)
then println(„Withdrawal over 200!");
end
Esperselect * from
Withdrawal(amount>=200).WIN:LENGTH(5)
![Page 14: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/14.jpg)
IBM Continuous Insight
CEP
BI
High-volume data analysis
![Page 15: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/15.jpg)
ESPER
Java-based implementation
There is also an implementation for .NET (NEsper)
Open-source
Embeddable into your Java code
You call it as a service, runtime
Esper Processing Language (EPL)
o SQL-like
o Implements an expansive processing algebra
• 400 pages of language definition, 100 pages of configuration
![Page 16: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/16.jpg)
ESPER: Processing model
EPL expression
A proper Java bean
SELECT * FROMWithdrawal.WIN:LENGTH(5)
Executable action
![Page 17: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/17.jpg)
ESPER: Processing model
SELECT * FROMWithdrawal(amount>=200).WIN:LENGTH(5)
![Page 18: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/18.jpg)
ESPER: Processing model
SELECT * FROMWithdrawal.WIN:LENGTH(5) WHERE amount>=200
![Page 19: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/19.jpg)
Further Esper Pointers
The full documentation:http://esper.codehaus.org/esper/documentation/documentation.html
Clauses, subqueries, patterns: http://esper.codehaus.org/esper-4.7.0/doc/reference/en-US/html/epl_clauses.html
Complex patterns: http://esper.codehaus.org/esper-
4.7.0/doc/reference/en-US/html/event_patterns.html
EPL operators in depth: http://esper.codehaus.org/esper-
4.7.0/doc/reference/en-US/html/epl-operator.html
Design patterns: http://esper.codehaus.org/tutorials/solution_patterns/solution_patterns.html (see section „EPL Questions”)
![Page 20: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/20.jpg)
Steps for CEP @ESPER
Install the distribution
o Download and unzip
In your Java project, import esper.jar
Define Filters and Listeners
Get an instance of the engine service provider
Configuration
Register Filters and Listeners
Connect to the engine
Send events
![Page 21: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/21.jpg)
ESPERDEMO
![Page 22: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/22.jpg)
Emerging problems
No modeling support
Inconvenient coding
Hard to maintain the codebase
No validation for the defined patterns
o Possible interference, overlaps, correlations, etc.
Lack of traceability
![Page 23: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/23.jpg)
The CEPWorkbench
![Page 24: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/24.jpg)
The CEPWorkbench
![Page 25: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/25.jpg)
The CEPWorkbench
![Page 26: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/26.jpg)
The CEPWorkbench
![Page 27: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/27.jpg)
CEPWorkbenchDEMO
![Page 28: Complex Event Processing - A brief overview](https://reader034.fdocuments.us/reader034/viewer/2022052622/5594788c1a28ab93388b4666/html5/thumbnails/28.jpg)
Future directions
Enormous amount of data is produced day by day
o The last 50 years: processing data at rest
o The next 50 years: processing information in motion
(Mark Palmer)
“The world is changing fast. Big will not beat small anymore. It will be the fast beating the slow.”(Rupert Murdoch)