Complex event processing Seminar : HKUST – September 2004
description
Transcript of Complex event processing Seminar : HKUST – September 2004
Active Technologies - HRL
| 2003 | © 2002 IBM Corporation
Complex event processingSeminar : HKUST – September 2004
Opher EtzionIBM Research Laboratory in Haifa
© 2002 IBM Corporation
YES!! It works every time.The Gartner quote always gets their attention.
As stated in the Gartner report, apples are ….
Wow, Gartner! Perhaps this wouldn’t be a total waste of time after all
© 2002 IBM Corporation
Active Behavior – scenario 1
Your Refrigerator called me..
© 2002 IBM Corporation
Active Behavior - Scenario 2
At least N people of my team are in this building now
© 2002 IBM Corporation
Active technologies – scenario 3
An information about a certain target arrived from two different sources within the last hour AND a contradictory information did not arrive within the last hour, and there have been at least five active sources in the same area within the last hour then handle the target.
© 2002 IBM Corporation
Active behavior : Scenario 4
Whenever a traffic congestion occurs, re-evaluate the traffic-lights timing policies and change it.
© 2002 IBM Corporation
What is the common denominator ?
All of these are event-driven They are not driven by a single event, so some
processing of the events is needed.We can have some hints about the types of
processing needed in each case…
© 2002 IBM Corporation
What is the talk about ?
Events --- what are they ? How do they relate to the rest of
the universe ? What are the types of event
processes ? AMIT – a CEP example Other examples Some use cases Some research and pragmatic
challenges
© 2002 IBM Corporation
What is an event ?
An event is a significant (in some context) instantaneous (happens in a specific point in time) atomic (happens completely or not at all) occurrence
Are those events ?
© 2002 IBM Corporation
Event collection middleware
A platform to communicate events Event sources: application, workflow, database, IT resource –
needs instrumentation and normalization. Event transfer: publish/subscribe (with content).
Many technical issues: Scalability
Standard protocols
Event store
Event bus
© 2002 IBM Corporation
What is the connection between events and data ?
Database does not have an inherent semantics, it can represent everything (including events)
But – semantically there is a distinction between:State: snapshot of the state of (the appropriate subset of) the universe at a certain point in time.
Transition: transfer from state to state.Database processing (e.g. SQL) is state
processing.So – does event represent transitions ?
Sometimes.
o A transition is an event.
oNot all events change states that are of interest.
© 2002 IBM Corporation
Is event processing different from data processing ?
In database processing – a snapshot that includes multiple entities is processed. All the processing is a function of values that exists within this snapshot.
In event processing --- the history of events is processed, and can also relate to the state information. Temporal processing is dominant.
Event processing subsumes data processing (but usually not all the power of SQL is needed).
Event processing can be expressed in SQL (the data-stream people are working on it) but in many cases it is not easy (exercise: write in SQL query that finds if a sequence of at least 8 events of 8 different event types has occurred in a sequence within an hour anytime).
© 2002 IBM Corporation
Complex event processing – a model based approach
context
data
event
state
entity
activity
flow
situation
© 2002 IBM Corporation
What is an event model ?
Data model does not talk only about data, but also about things in the world and dependencies among them (e.g ER, semantic data model).
Event model is an extended data model in which events play role as a first class citizen.
Events are connected to : Other events
Context (what’s that ???)
Messages
Entities
Databases or other state keepers
Flows
© 2002 IBM Corporation
Event processing - Situation
Situation is defined as a transition in the universe that requires reaction (either “reactive” situation or “proactive” situation).
One of the main event processing goals is situation detection An approximation for situation is an inferred causality event within a
context, the function can optionally contain other players (data, state etc..) S = F (e1,…en, context, [state information]).
This approximation equals the situation when the function is deterministic; in other cases we shall need to operate uncertainty measures.
Other types of event processing relate to relationships of events with everything else.
© 2002 IBM Corporation
Situation detection
© 2002 IBM Corporation
Some examples from various domains
Alert if the IBM stock has gone up in 3 percent within two hours, and the Dow Jones did not go up in more than 1 percent at the same period.
Alert if three memory problems occurred during the last hour. Alert if the same request was reassigned to three agents, and no answer
was given to the requester.
© 2002 IBM Corporation
Relationships among events
Causality: Observed causality : two events that has statistical correlation that
indicates that one of them is an antecedent of the other (example: getting out of the car, locking the car).
Inferred causality: An event that is signaled as a result of processing in which the other event participated in (example: traffic jam identified, traffic-lights policy re-calculated).
Subset hierarchy : possibly conditional generalization/specialization relations (printer
problem, hardware problem) Cross-section :
Events with certain conditions are considered as other virtual events (e.g. all events that relate to the same object).
© 2002 IBM Corporation
Context
Event is instantaneous – occurs in a certain period of time.
Sometimes event processing is done within a context Examples:
Within the working hours
The last 30 minutes of trade
within 60 minutes from the time that the request was sent
From transaction start to transaction end
Context has : Validity interval.
possibly multiple instances by some partition criterion (e.g. by customer).
possibly spatial characteristics. Events are related to contexts:
Start and end of contexts are events.
Event processing can be a function of context.
© 2002 IBM Corporation
Context Awareness
Composite perspectives of the environment Semantic perspective designates environment information about a specific
object or entity (e.g. users that are members of the same group).
Temporal perspective designates environment information within a specific temporal element (e.g. network overload in one hour).
Spatial perspective designates environment information within a specific location or area (e.g. vehicles near a traffic problem).
State perspective designates environment information within a specific state (e.g. low market volume).
© 2002 IBM Corporation
Temporal Context
free
busydysfunctional
busy free
free
free
freefree
busy
Designate a collection of events that occur within a temporal interval
Bounded by initiator and terminator
Has maximal length, initiation and termination policies
Multiple lifespan may exists simultaneously
© 2002 IBM Corporation
Spatial context
Express the spatial perspective of Amit context
Designate a collection of events that are originated from the same region
Either a circle specified by a center coordinate and a radius or a polygon specified by a set of coordinates.
Fixed or moving
dysfunctionaldysfunctional
© 2002 IBM Corporation
Events and Messages
There is common misconception that events and messages are the same.
In fact there are some connections: A message may represent a description of event occurrence – this is
one of the common ways to report events.
There are events associated with messages (created, sent, received, acknowledged…) like any other entity.
Complex event processing is part of message brokering/mediation.
© 2002 IBM Corporation
Events and entities
An event can refer to one or more entities,Example: John sell the bike to Jim (type of event sale, entities:
John, bike, Jim with different roles).An entity has a role in an event (seller, buyer, merchandise)An event may have a role in the entity (starting, ending, disrupting –
can be interpreted as change the entity status)
© 2002 IBM Corporation
Events and databases – again
An event may (but does not have to) change state (or results in a database update)
A database operation occurrence is an event Event processing can look at both events and states.
© 2002 IBM Corporation
Events and activities (and flows…)
Event/situation can trigger activitiesActivity can signal events
A flow can contain activities, event, data and messages with all interactions.
In case of a flow --- an event/situation can add/modify/delete sub-flows dynamically and can interrupt running flows
Transition in flow states are events.
© 2002 IBM Corporation
Short history of event processing ..
In early days of programming languages it was used for interrupts/exception handling
Real-time applications raised requirements for reaction based on time.
Active databases started in the late 1980-ies System and network management tools emerged in the early
1990-ies Publish/Subscribe system appeared in the lat 1990-ies Currently: monitoring, management application, business
process integration, straight-through-processing… Gartner’s : CEP, BAM, RTE.
© 2002 IBM Corporation
Active behavior specification:
Embedded in regular programming languages. Rule-based approaches (reactive/proactive programming):
Condition-Action rules (event is hidden)
Event-Condition-Action rules
Situation-[Condition]- Action rules Model based approaches (reflective programming)
Semantic net approach.
Self-stabilization approach (data-driven).
The “action” can be a “business rule”
© 2002 IBM Corporation
Event-Condition-Action rules
Making the processing event driven The paradigm :
when event occurred if a condition is satisfied perform action
A variation can be E(CA)*
Example: When a message about Microsoft stock quote has arrived, If the value is less than 25, then notify all customers in the subscription-list.
© 2002 IBM Corporation
Situation based processing
Next step in the evolution – from event based to situation based. The concept of situation is what triggers the action from the
user’s point of view (may not be a single event) This is an abstraction over the universe of transitions in the
same way that SQL query or view is an abstraction over the universe of states
It is roughly equivalent to the term “composite event within a context”, but may have uncertainty associated with it.
© 2002 IBM Corporation
Situation examples
A client has withdrawn at least 3 times a sum of more than $100,000 that he deposited at most 2 days before the withdrawal [looking for money exceptional movement]
A client has withdrawn within 2 hours from two ATM machines that are more then 200 KM apart [fraud detection]
A client wishes to be notified when IBM stock is up more than 3 percent if he is in the office [personalized location-aware publish/subscribe]
At the end of the day at least 2% of the orders have not completed [monitoring]
© 2002 IBM Corporation
Relationship to Publish/Subscribe Technology
Publish/subscribe is : event - action. Publish/subscribe with filtering is: event - condition – action
(current state of the art). Situation based publish/subscribe is the next generation – enable
“personalization of push technology”.
© 2002 IBM Corporation
Relationship to event correlation
The term “event correlation” has originated from network management.
The idea is to find event that have statistical correlation among them (and hence the name) and possibly filter out some of them to cope with “event storm”
It has some notion of causality (usually – two events occurring together within a fixed time interval).
Used in system management applications, and as an event filtering tool.
© 2002 IBM Corporation
AmitActive Middleware Technology
© 2002 IBM Corporation
Amit – Technologies Context diagram
Amit Metadata
(Definition Manager)
ADI Model
Templates
Events
Rules
GUI
Model-based
Rule-based
Rule-based
Situation
ManagerARAD
External (pub/sub)
Amit
Tooling
Action
Manager
Exception
Handling
Events
Instances
Situations
Conclusions
Alerts
Exceptions
Definitions
Exception Handling Metadata
Exceptions
Exceptions
DetectionsCreate new
Events
Change Definitions
tools
© 2002 IBM Corporation
The situation concept
e1.id
e2.name
e3.key
KeyConditions )where...(Attributes )retain, override.,..(
Detection Mode )immediate, delayed, deferred(
Repeat Mode )always, once(
Contexte5
e8
Initiator Terminator
Situation
e1
e2
e3
Events Operators
JOINING
COUNTING
TEMPORAL
ABSENCE
)all, sequence(
)atleast, atmost, nth(
)every, after, at(
)not, unless(
3
© 2002 IBM Corporation
Amit Tooling Today
© 2002 IBM Corporation
© 2002 IBM Corporation
Selected additional complex event processing solutions
Apama Elity Actimize Ispheres …
Active Technologies - HRL
| 2003 | © 2002 IBM Corporation
Some applications
of Complex Event Processing
© 2002 IBM Corporation
Transaction Flow Monitoring
Transaction Level Alerts: Acknowledgements is not
consistent with the sending order. 15 minutes before a settlement has
to close, a message is waiting to be handled by a compliance officer due to insufficient credit.
No acknowledgement/reject received from the stock exchange within one hour from sending a message.
Business Level Alerts: Abnormal number of payments
received from a specific bank (account) within the first two hours or business
Three rejects received within a single working day, for FED messages of the same platinum client.
Order Management System Execution Management
Open Order
23
SM CP
validate the security
validate the Party & account
4
Exchange
Transaction ManagementCalculationsFigurationAllocationSetlement Details*
AM = Account MasterCP = CounterpartySM = Security MasterFC = Fees and ChargesFR= Fees and charges rulesSI = Setlement Instructions
AM
Reference DataOperation &Transaction Data
Order1
Investment Manger
Open Order
updates open orders with fills
Request for ExecutionNotification of Completion
Block trade
Message/Transaction/Flow
5
6
7
8Allocations, Net proceeds, affirms , confirms
Confirmation and Allocation
9Fully figure trade
Purchase & SalesClearing
SM3 CP3AM3
Settlement
SI2
10 Settlement Instruction and Payment
CP2 FC FRAM2SM2 SI1*
Front-Office
Middle-Office
Back-Office
© 2002 IBM Corporation
Finance Scenarios
Trade Regulation BreachCEPS verifies that sells and subsequent purchases of large quantities of stock meet regulatory requirements
Credit BreachCEPS initially allows a credit limit to be exceeded (by no more than 10%). The second time the limit is exceeded, CEPS routes the order to a credit officer for approval
Trade Execution DelayNotify if a purchase order was sent for processing and no response was received within the time specified by the SLA.
Fraud DetectionReport when multiple credit card purchases are performed within an hour or (any given time frame) at a distance greater than 300 km (or any given distance).
Finance
© 2002 IBM Corporation
Server
CEP SERVER
Action Manager
Wireless server
Data Warehouse
CRM
On Demand eCRM Architecture and Dataflow Example:
Hand-Held
Database stores all historical customer information
A CEP server receives relevant customer data from database when customer enters the store
Each event is processed when customer is in the store
Action managers performs actions – recommendation of complementary products, sales on frequently purchased items etc..
Retail
© 2002 IBM Corporation
Homeland security
CEP
Unusual activity
Security Scenario
Security
© 2002 IBM Corporation
Insurance Scenario
CEP helps improve claim processing by automating key activities such as:
Identifying invalid claims
Detection of potential fraudulent claims
We can help release delayed payments according to pre-specified conditions.
Identifying problematic health insurance providers that do not comply with regulations.
Insurance
Automatic Exception Resolution
ECS Automatic Exception Resolution
ECS
WBI Message Broker
StartQueue
XML
HTML claim form
EDSroutingnode
Stage 1
(log)
Stage 1Queue
Suspendqueue 1
Effectqueue 1
Stage n
(log)
Finalqueue
ECSroutingnode
Suspendqueue 1
Effectqueue 1
Claimdatabase
© 2002 IBM Corporation
Higher level technologies that use CEP
The “sense and respond” loop Real-time analytics Autonomic computing
© 2002 IBM Corporation
On-demand control loop (sense and respond) :
06nAFS -
© 2002 IBM Corporation
Real-time analytics
Departure from traditional use of analytic tools that may not be time constrained.
The traffic light problem is an exampleOther examples:
Re-calculation of network configuration policies when part of the network is disabled (e.g. due to “denial of service attack”.
Re-establishing of queue priorities policies Trade-off between time and quality of solution (cannot get to the
optimal solution in 1 minutes, how should I get to the best possible solution given these time constraints)..
© 2002 IBM Corporation
Autonomic computingAlerts, events and problem analysis request interface
Interface to real and virtualized resources and components that regulate control.
SLA/Policy interface, interprets and translates into "control logic"
Plan
Policy Transforms
Plan Generators
Policy InterpreterAnalyze
Execute
Service Dispatcher
Distribution Engine
Scheduler Engine
Workflow Engine
Monitor
Metric Managers
Filters
Simple Correlators
Knowledge
Policy
CalendarTopology
Recent Activity Log
Sensors Effectors
Rules Engines
Analysis Engines
Policy Validations
Policy Resolution
© 2002 IBM Corporation
Additional research topics
Real-time aspects Distribution and parallelism Transactional support Temporal issues Uncertainty in complex event processing. Software engineering aspects.
© 2002 IBM Corporation
Real-time aspects:
The end-to-end process involving the CEP may have real-time constraints.
This may inflict real-time Real-time awareness built-in operations: Scheduling
Prioritization
Relevance of “late” events.
© 2002 IBM Corporation
Distribution and Parallelism
For scalability reasons: The ability to apply N engines and balance the processing load
Note, that there can be many nested situations, thus full partition may not be possible
Establishing minimal traffic among the various engines.
Reference: M. Shmueli, O. Etzion - Parallel Implementation of Composite Events. ICDCS Workshops 2002: 579-580
For high availability: Requires support in clustering and failover.
Usually done by basing on middleware services
© 2002 IBM Corporation
Transactional support
May require non-ACID transactional support : Total rollback may not be possible, since we cannot say that the “event
did not happen”, even if we fail to process its consequences.
Events that are part of the process may belong to different transactions, only the last of them “closes the loop”.
However, some of the process can be “atomic”.
© 2002 IBM Corporation
Temporal issues
Events may not arrive in the same order they are produced The time-stamps on different events may not be consistent with
relative timings. There may be a communication delay to report events
How long should we wait for an event ?
What happens we get event beyond this time-out ?In general – how do we process:
Retroactive events (events about the past)
Predictive events (“certain” events about the future)
© 2002 IBM Corporation
Uncertainty aspects :
There are several sources for uncertainty: Uncertainty that the event has (or has not) occurred
Uncertainty in the details of the events itself
Uncertainty that the SITUATION is equivalent to this specific function of events and contexts
Uncertainty that the right context is identified,
…
© 2002 IBM Corporation
Software Engineering aspects:
A programming paradigm that deserves: Methodologies - as a programming tool, and as part of a bigger picture
Modeling tools, automatic creations of rules
Debugging --- all the known difficulties of rule debugging --- interactions among situations, halting problem, determinism and sequencing.
© 2002 IBM Corporation
Future --- stand-alone vs. embedded technology
Complex event processing is being developed in two main contexts: As a central service collecting events from various sources (will be
part of all middleware products)..
As an embedded technology inside other frameworks/products/solutions (seem to grow more rapidly)
According to Gartner’s hype-cycle, time to full maturity is 5-7 years.
© 2002 IBM Corporation
References D. Luckham – The power of events: An Introduction to Complex Event Processing. Addison-Wesley, 2002 A. Adi, O. Etzion – Amit, the situation manager. VLDB Journal, 13(2), 173-203, 2004. Gartner reports:
“Events will transform application servers” “Hype cycle for application integration middleware and platforms 2003” …
Vendors URLs : http://www.apama.com http://www.elity.com http://www.actimize.com http://www.ispheres.com
Foundations : Chakravarthy-S, and Mishra-D. "Snoop: an expressive event specification language for active
databases." Data and Knowledge Engineering 14.1 (1994): 1-26. Tombros-D, Geppert-A, and Dittrich-KR. "Semantics of Reactive Components in Event-Driven
Workflow Execution". CAiSE 1997: 409-422. Yemini-SA, Kliger-S, Mozes-E, Yemini-Y, and Ohsie-D. "High speed and robust event correlation."
IEEE Communications Magazine. 34.5 (1996): 82-90. Zimmer-D, and Unland-R. "A General Model for Specification of the Semantics of Complex Events in
Active Database Management Systems." C-LAB Report. 1998. Zimmermann-J, and Buchmann-A. "REACH." Active Rules in Database Systems. Springer Verlag,
1999. 263-277.
.