Post on 26-May-2015
description
Lucas Jellema
Oracle OpenWorld 2014, San Francisco, CA, USA
How Fast Data Is Turned into Fast Information and Timely Action
2
Audience Challenge
3
Audience Challenge
4
Audience Challenge
5
Audience Challenge
6
Audience Challenge
7
Audience Challenge
8
Filter
Pattern Detection
Agregate
9
Fast Data Example
14,016,114,116,116,013,114,016,013,113,014,116,014,113,014,116,013,114,0
Smart Processing• Information• Conclusion• Alert• Recommendation• Action
10
Demonstration: Live Tennis
• Tennis Tournament• Many matches played in parallel• The data that is produced:
– At a rate of up to 10 events/minute
Match Id, Player [who scored]14,016,114,116,116,013,114,0
11
Demonstration: Live Tennis
• The information, conclusions &actions we are looking for:– Scoreboard per game, set, match– Match start and completion (action:
inform next players for that court)– Interrupted match (action: go and check
out the reason for the interruption)
Fast Data
Smart Processing
• Scoreboard• Match start and
completion • Interrupted match
12
Real Time – from event to UIPush through Web Sockets
Fast Data
Smart Processing
Oracle Event Processor
WebLogic
eventWebSocket
Servermsg
msg
CQL queries
13
WebSocket Powered Scoreboard
14
OEP application to process fast tennis data
• Preparation– Define event definitions– Create local, in memory cache with static, enriching data– Gather (in this case generate) tennis data through adapter– Create Event Sink to consume all findings and publish to console
TennisMatchEvent
matchIdplayer
15
Match Level events
16
Rally’s to games
- The first player to have won more than 4 points
- and have won two or more points more than his opponent
TennisMatchEvent
matchIdplayer
17
Detect interrupted matches by ‘finding’ missing events
• When a match is interrupted, obviously no more ‘rally point events’ are produced
• Detecting the absence of these events for a match [that has begun] is equivalent to detecting an interruption of the match– Unless the match is complete because someone won
18
Detect interrupted matches by ‘finding’ missing events
19
Complete EPN diagram for Tennis Tournament Processor
• A single OEP application that consumes fine grained rally point events and performs three-stage aggregation and enrichment
TennisMatchEvent
matchIdplayer
New Match
Match Finish
InterruptedMatch
Set WonGame
Won
Overview
• What is [special about] Fast Data?– Continuous, Volume|Velocity|Variety, Real Time
• Challenges– Volatile, non persistent– Data => Information, Conclusion, Alert, Recommendation, Action
• Strategies– Smart gathering– Discard – filter, aggregate, pattern
(and also look for missing events!)– Promote (process, enrich)– Visualize
• Technology/Tools• Demonstration/Cases
21
Fast Data
• Tweet• Feed• Beat• Signal• Measurement• Message• Mail• Notification• Tick• Pulse
22
New theme (that brings it all together)
23
Some event producing devices
24
Most of these events….
26
27
Fast Data Processing
Fast Data
Smart Processing
• Information• Conclusion• Alert• Recommendation• Action
28
Fast Data ProcessingMulti-stage cleansing & aggregation
Fast Data
Smart Processing
• Information• Conclusion• Alert• Recommendation• Action
29
Typical Flow and Additional Challenge…
Business event
Bu
sin
ess
Val
ue
Data captured
Analysis completedAction taken
Fragmented event entities
TIME
30
The V-factor
VolumeVelocityVariety
VALUE
31
Key strategy
• Discard – as early as possible (close to the source)– Ignore irrelevant events– Filter out unneeded attributes– Takes samples instead of entire stream– Aggregate: merge multiple, correlated events into one
32
Fast Data Processing:Oracle Event Processor
Fast Data
Smart Processing
Oracle Event Processor
RMIFile
RESTHTTP Channel
JMSDatabase
Custom (Java)SOA Suite EDN
CoherenceJMX
QuickFix (financial)
RMIFileRESTHTTP ChannelJMSBusiness RuleDatabaseCustom (Java)SOA Suite EDNCoherenceJMX
33
Oracle Event Processor
• Light weight, real-time (sub-sub-second), in-memory, continuous query engine– Available in embedded form – with corresponding licence
• Interacts with many different channels – inbound and outbound• Has internal caches to enrich events and temporarily retain events• Uses CQL to:
– Filter, aggregate, enrich and detect patterns (including missing events)
events
Event Processor
35
Fast Data ProcessingFusion Middleware Tooling
Fast Data
Smart Processing
Oracle Event Processor
Coherence
SOA Suite 12c EDN
RMIFile
RESTHTTP Channel
JMSDatabase
JMXCustom (Java)
RMIFileRESTHTTP ChannelJMSBusiness RuleJMXDatabaseCustom (Java)
36
Fast Data ProcessingFusion Middleware Tooling
Fast DataSmart Processing
• Information• Conclusion• Alert• Recommendation• Action
OEP
BAM
ADFCoherence SOA Suite
EDNBPMSuite
BPELTask
BIRTD
ODIGolden Gate
NoSQL
37
Business User Friendly Exploration of Fast Data: Stream Explorer
38
Credit Card Theft Detection
• Several situations in the past– Credit card is stolen in the main terminal building– Several purchases are made in shops on the way from that area to the main exit
• Purchases between $200-$500 dollar• Purchases made within 5 minutes of each other• Sometimes the purchases are made in not entirely the direct route to the exit
EXIT
Main Terminal
39
Credit Card Theft Detection
• Several situations in the past– Credit card is stolen in the main terminal building– Several purchases are made in shops on the way from that area to the main exit
• Purchases between $200-$500 dollar• Purchases made within 5 minutes of each other• Sometimes the purchases are made in not entirely the direct route to the exit
• To catch the perpetrator– Consume the credit card purchase event stream for airport shops– Spot situations where three or more purchases of $200-$500 are made within 5
minutes from each other and roughly in the terminal => exit physical order– Publish an event to alert security staff
• To watch for any further purchases with that credit card• To inform show staff for that credit card• To send staff to the exit to try and apprehend the thief
(perhaps based on the shopping bags he is carrying from the shops he bought stuff at)
40
Catch me if you can
EXIT
Main Terminal
41
Catch me if you can
EXIT
Main Terminal
$440$300
$380
$250
42
CQL to detect ‘funny string of transactions’
43
Real Time – from Event to TaskOEP => SOA Suite 12c EDN
Fast Data
Smart Processing
Oracle Event Processor
SOA Suite 12c
EDNevent event
44
Real Time – from Event to TaskOEP => SOA Suite 12c EDN
Fast Data
Smart Processing
Oracle Event Processor
SOA Suite 12c
EDN
BPEL Task
BPMN
Medi-ator
event event
event
45
From OEP finding to EDN Business Event triggering the SOA Suite
46
Human consumers
• Slow at data processing• Not electronically connected• Visually oriented (1 picture > 1000 words)• Frequently (though perhaps decreasingly so) the actor or decision maker
• Interact along human communication channels• Use visualization to present findings, conclusions, recommended actions
– And as a second tier of fast data processing:Highlight (filter), aggregate, patterns, extrapolate/interpolate, missing elements
• Sometimes take over from humans and just take action
47
Visualize and Aggregate
48
Real Time – from event to UIBusiness Activity Monitoring
Fast Data
Smart Processing
Oracle Event Processor
WebLogic
JMS
event
msgBAMmsg
49
Real Time – from event to UIADF DVT Visualizations
Fast Data
Smart Processing
Oracle Event Processor
WebLogic
JMS
event
msg
ADF DVT
msg
50
Visualize physical locations of [string of] suspicious transactions
51
Summary
• Fast Data (events): Vast, Continuous, Velocity, Variety– Wanted: Near real time conclusions, recommendations, alerts, actions
• Strategy:– Discard – as early as possible (Filter, Aggregate)– Enrich, Correlate and Pattern Match, Missing Events, Retain, Publish higher level,
more coarse grained business event– Repeat this cycle multiple times (such as rally point, game, set, match)
• Technology for Fast Data processing: Oracle Event Processor & CQL– Interacts with JMS, EDN, RMI, HTTP (/REST), JMX, Database, Coherence
• New: Stream Explorer – business friendly, industry pattern based fast data explorations and visualization
• To assist humans in Fast Data and Information Processing: Visualization– Filter, Aggregate, Enrich, Pattern Match (1 picture > 1000 words)– Technology: BAM (Dashboard and Rule processing), ADF Data Visualization– Also: turn findings into actions using Human Task, BPEL and BPM via the SOA Suite
12c Event Delivery Network (EDN)
53
Fast Data Example
14,016,114,116,116,013,114,016,013,113,014,116,014,113,014,116,013,114,0
Smart Processing
Oracle Event Processor
54
Demonstration: Live Tennis
• Tennis Tournament• Many matches played in parallel• The data that is produced:
– At a rate of up to 10 events/minute
Match Id, Player [who scored]14,016,114,116,116,013,114,0
55
Demonstration: Live Tennis
• The information, conclusions &actions we are looking for:– Scoreboard per game, set, match– Match start and completion (action:
inform next players for that court)– Interrupted match (action: go and check
out the reason for the interruption)
Fast Data
Smart Processing
• Scoreboard• Match start and
completion • Interrupted match
56
OEP application to process fast tennis data
• Preparation– Define event definitions– Create local, in memory cache with static, enriching data– Gather (in this case generate) tennis data through adapter– Create Event Sink to consume all findings and publish to console
TennisMatchEvent
matchIdplayer
57
Simple Time-slice Aggregation
• Produce aggegrates once every 30 seconds– Count number of matches going on currently (meaning: in the last 30 seconds)– Calculate average time per rally (over the last 30 seconds)– Count total number of points played (over the last 30 seconds)
60
Match Level events
61
Rally’s to games
- The first player to have won more than 4 points
- and have won two or more points more than his opponent
TennisMatchEvent
matchIdplayer
62
Games to Sets
- The first player to have won more than 5 games
- and have won two or more games more than his opponent
63
Detect interrupted matches by ‘finding’ missing events
• When a match is interrupted, obviously no more ‘rally point events’ are produced
• Detecting the absence of these events for a match [that has begun] is equivalent to detecting an interruption of the match– Unless the match is complete because someone won
64
Detect interrupted matches by ‘finding’ missing events
65
Complete EPN diagram for Tennis Tournament Processor
• A single OEP application that consumes fine grained rally point events and performs three-stage aggregation and enrichment
TennisMatchEvent
matchIdplayer
New Match
Match Finish
InterruptedMatch
Set WonGame
Won