Drive Smarter Decisions with Big Data Using Complex Event Processing
-
Upload
perficient-inc -
Category
Technology
-
view
1.898 -
download
3
description
Transcript of Drive Smarter Decisions with Big Data Using Complex Event Processing
1
Drive Smarter Decisions with Big Data
Using Complex Event Processing
Eric Roch ▪ July 16, 2013
Our Speaker
Eric Roch
• Principal for Perficient’s Connected Solutions Practice –
SOA – Cloud - Mobile
• 25+ years of experience in Information Technology
• Previous roles include: executive level management,
technical architect, and software development in top tier
technology organizations including TIBCO Software and
Deloitte Consulting
• Strategic planning and commercialization of
methodologies and software
• Technical architecture for multi-platform application and
systems integration at organizations
• Guest speaker and author
2
3
Perficient is a leading information technology consulting firm serving clients
throughout North America.
We help clients implement business-driven technology solutions that integrate
business processes, improve worker productivity, increase customer loyalty and
create a more agile enterprise to better respond to new business opportunities.
About Perficient
4
• Founded in 1997
• Public, NASDAQ: PRFT
• 2012 revenue $327 million
• Major market locations throughout North America
• Atlanta, Boston, Charlotte, Chicago, Cincinnati, Cleveland, Columbus, Dallas, Denver,
Detroit, Fairfax, Houston, Indianapolis, Los Angeles, Minneapolis, New Orleans, New York
City, Northern California, Philadelphia, Southern California, St. Louis, Toronto and
Washington, D.C.
• Global delivery centers in China, Europe and India
• ~2,000 colleagues
• Dedicated solution practices
• ~85% repeat business rate
• Alliance partnerships with major technology vendors
• Multiple vendor/industry technology and growth awards
Perficient Profile
5
Business Solutions
• Business Intelligence
• Business Process Management
• Customer Experience and CRM
• Enterprise Performance Management
• Enterprise Resource Planning
• Experience Design (XD)
• Management Consulting
Technology Solutions
• Business Integration/SOA
• Cloud Services
• Commerce
• Content Management
• Custom Application Development
• Education
• Information Management
• Mobile Platforms
• Platform Integration
• Portal & Social
Our Solutions Expertise
Agenda
• Big Data Trends and Categories
• Analysis of large amounts of complex, unstructured and semi-
structured data
• Harnessing the power big data, social/mobile data stores and BI
projects for real-time decision-making
• Predictive Analytics and Event Processing for decision management
6
Evolution of Big-Data
• Mainframe
• Client-Server
• Web
• Mobile
• Cloud
• Social
• Internet of Things
7
Source: Go-Globe.com
State of Technology Adoption
8
Categories of Big-Data
9
Source: splunk
10
Characteristics of Big Data
• Data in motion
analyzes data
before storage
• Data at rest
analytics are
based on a
historic snapshot Source: IBM
Big Data Technologies
• MapReduce frameworks implements pattern recognition though classification algorithms – what happened?
• Data Visualization presents information views graphically and/or statistically – what happened and what might happen?
• Predictive Analytics uses mathematical pattern recognition in historical data – what’s going to happen?
• Complex Event Processing uses pattern recognition on event streams and can apply rules to predict logical events – what is going to happen and what do we do about it?
11
Source: TIBCO Spotfire
Log Analysis vs. Business Analytics
• Ingest – Versus ETL
• Big Data – Bidirectional integration with Hadoop
• Query language – MapReduce function on unstructured data
• Drill anywhere – Investigate on all the data versus a predefined schema or cube
• Information discovery – Discover relationships based on patterns in the data
• Ad-hoc versus dimensional – Log analysis is not based a predefined structure based a point-in-time set of requirements
12
Source: splunk Implementation
Predictive Analysis
• Predict the future state of
variables associated with
business goals
• Describe human
detectable patterns
• Data mining techniques • Rule Discovery – describe
• Pattern Discovery – describe
• Clustering – describe
• Classification – predict
• Regression – predict
• Deviation – predict
13
Source: InformationBuilders
Event-driven Architecture
• Event-driven architecture
(EDA) is a software
architecture pattern promoting
the production, detection,
consumption of, and reaction
to events
• Complex event processing
(CEP) consists in processing
many events happening
across all the layers of an
organization, identifying the
most meaningful events within
the event cloud, analyzing their
impact, and taking subsequent
action in real time.
14
14
A Holist View of Decision Optimization
15
Source: James Taylor
http://www.decisionmanagementsolutions.com
Barriers to Big Data Analytics
• Information throughout the
enterprise
• Silos of data
• Decentralized control
• No one single solution
• No cohesive strategy
• Legacy systems difficult to
make part of the strategy
16
SOA and Integration
17
HT
TP
HT
TP
/S
SO
AP
/HT
TP
SO
AP
/JM
S
FT
P
SM
TP
EM
S/J
MS
ED
I
Enterprise Service Bus (ESB)
Credit
Check Place
Order
Check
Quantity Issue
Invoice
Alert
Large
Order
Notify
Customer
Process
Order
Check
Customer
Account
• Connect
• Transport
• Route
Services Backbone Enterprise Service Bus
(ESB)
• Mediate
• Event notification
• Exception Handling
Abstract the data format and the behavior of legacy systems to publish events
The SOA Information Gap
″SOA by itself does nothing to address the question
of how data should be managed within this
architecture. ... data remains fragmented despite
the best efforts to rationalize it. This issue is
motivating the creation of a new class of
middleware that Forrester calls the information
fabric.”
The Forrester Report Information Fabric:
Enterprise Data Virtualization
18
″ You will waste your investment in SOA unless you have enterprise
information that SOA can exploit.”
Gartner
Data Virtualization Layer
19
Data Warehouse
Packaged Application
Legacy Application
• Master Data Management and Data Virtualization • Data federation for consistent packaging of data • Leverages understanding of metadata relationships • Applies consistent rules to data • Centralized control and maintenance • Flexibility to change information sources and formatsar
Create Quote
Process Flow
Trigger
Create Estimate
Process Flow
Trigger
Information as a Service (Shared Metadata)
Business Process Management and
Workflow
• The term Business Process Management refers to activities performed by businesses to optimize and adapt their processes.
• Although it can be said that organizations have always been using BPM, a new impetus based on the advent of software tools which allow for
• Direct execution of the business processes without a costly and time intensive development of the required software.
• In addition, these tools can also monitor the execution of the business processes, providing managers of an organization with the means to analyze their performance and make changes to the original processes in real-time
• BPM has a tight link to componentized and service oriented IT architecture
20
BPM and Services
21
Service
X
Service
U
Service
Y
Service
Z
Human Task
A
Human Task
D
Human Task
F
Human Task
B
Human Task
C
Workflow
Invoke
Invoke Invoke Invoke
• Workflows implement business processes
• Workflow engine navigates the network of activities
• Typically invoking automatic (service choreography) or manual activities
• Mostly visual programming/modeling
Process Orchestration Layer - BPMS
• Designer and repository
• Execution engine
• Database – case state
• Database – case history
• Case history reporting – KPIs,
task timings, timings by role
• Starting a new case is
resource intensive
22
State Management
Design Repository
Process History
Execution Engine
BRMS Architecture
• Manages the lifecycle of the
rules
• Author rules
• Execute stateless rules
• Statistical reports about rule
execution
• Rule execution is embedded in
business applications – e.g. a
decision service
23
Source: IBM
Using BRMS in BPMS
• Lifecycle of rules are
external to the BPMS
• Business processes “call”
rules e.g. via services
• Rules make a stateless
decision
• Rules have to have a
driving workflow or
application
24
Rule Repository
Rule Engine
Rule
Authoring
BPMS
CEP Architecture
25
• Consistent operational
rules applied to business
events
• Declarative rules and
implicit state management
• Event driven, non-linear,
closed-loop, agile business
processes
• Component failure (fine
grain) – outage (logical
/predictive)
Concept State
Rule Bases
BPMS
CEP Engine
Logical Events – Notifications, Consequences Actions
SOA
Business Applications
Fine-grain Business Events
System(s) of Record
Integration and Business Components
Flexible Workflows
ESB Event
Channel(s)
CEP High-level Architecture
Patterns
• Situation awareness is
about "knowing" the state of
the product, person,
document, or entity of interest
at any point in time.
• Sense and respond is about
detecting some significant
fact about the product,
person, document or entity of
interest, and responding
accordingly
• Track and trace is about
tracking the product, person,
document or entity of interest
over time and tracing
pertinent facts
Source: TIBCO Software
26
CEP Benefits
• Manage events, state transitions, and event correlation reducing code in the application layer
• Control logic
• Persistence logic
• Business Rules
• Drive business processes with correlated events
• Create operational efficiencies with the same events and drive longer-term strategic decision support
• Less complex rules with the event driven concepts
• Persistent business objects
• Known context of the event
27
CEP Roadmap and Methodology
• Target critical business events for process automation and decision optimization
• Inventory relevant events, rules and concepts
• Identify candidate business (sub)process to automate
• Project LoE(s) and Roadmap
• Integrate systems used in key business processes
• Instrument applications to emit events
• Define process activities
• Mine candidate rules – code and predictive analytics
• Model events, rules and concepts
• Iterate through business processes
28
Telco CEP Case Study
• Provisioning
• Track missing provisioning notifications and sends complex events to Billing Ops on
missing notifications
• Open Orders
• Used to track Orders that have not been closed due to a missing event. CEP detects
the missing event and auto closes the Order in the Payment Processing system.
• Pending Payments
• Used to process payments that are pended by the payment processing system. CEP
stores the payment data within the cache and closes the payment at a later via
SOA.
• Customer Coupon Offers
• CEP is used to monitor, alert and prevent Stores from going over a threshold of the
discount funds that they are allocated.
• Logistics Alerts
• CEP is used to track location and Product updates from logistics and to invoke GEH
to republish failed messages
• CEP Framework
• Created CEP developer guide and logging framework to log and search events in
Splunk
29
TIBCO BusinessEvents is a CEP Platform
• Platform Features • Stateful Rule Engine
• State for Temporal Rules
• Persistence Object Manager
• High Performance Rules Engine
• TIBCO integration platform • 150+ Adapters
• Channels approach
• Continuous queries
and Event Stream Patterns
• Decision Manager for Business
User Rules Authoring (can write
can upload rules from Excel!)
• Distributed Agents Architecture
for dynamic scalability
• Data Grid
• BE Views (Dashboard)
30
Source: TIBCO Software
Event Enabled Enterprise
31
3
1
Transformation
Projects
2009-2011
Business
Solutions
2011-2012
• Last minute addition • Concept to launch in 6 weeks • Decoupled architecture – no risk
Implementation
• Customers: Ensure timely activations
• Operations: Immediate visibility into order provisioning times
Customer Service
• Stores: Reduce inventory issues • Operations: Automate fall out of
shipping notices
Supply Chain
• Customers: Added security to account access
• Operations: Report/alert on suspicious access attempts
Security/CPNI
• Customers: More access to discounts • Revenue: Manage discount limits by
individual location
Retail Sales
• Customers: Use IVR to set up payment agreements
• Customer Service: Reduced call center volumes
Self-Service
Event
Enabled 2013
• Proven success in real-time, value-based activities – ready for prime-time!
• Sense. Model. Respond.
The Tipping Point
• Adapt and respond to real-time customer behaviors/experiences
• Example: Proactive retention offers
Fast Response
• Abandon one-size-fits-all customer limitations
• Enable event-driven decisions for best customer experience
Customer Flexibility
CEP Solution Architecture
32
CEP References
• http://it.toolbox.com/blogs/the-soa-blog/complex-
event-processing-reference-materials-48348
• http://it.toolbox.com/blogs/the-soa-blog/complex-
event-processing-patterns-message-routing-
48987
• http://complexevents.com/wp-
content/uploads/2008/02/1-a-short-history-of-cep-
part-1.pdf
• http://complexevents.com/wp-
content/uploads/2008/07/2-final-a-short-history-of-
cep-part-2.pdf
33
Questions
34
Daily unique content
about content
management, user
experience, portals
and other enterprise
information technology
solutions across a
variety of industries.
Perficient.com/SocialMedia
Facebook.com/Perficient
Twitter.com/Perficient
35
Thank you for your time
and attention today. Please visit us at Perficient.com
36