Advanced Logging and Analysis for SOA, Social, Cloud and Big Data
-
Upload
perficient-inc -
Category
Technology
-
view
2.110 -
download
3
description
Transcript of Advanced Logging and Analysis for SOA, Social, Cloud and Big Data
Advanced Logging and Analysis for SOA, Social, Cloud and Big Data
About Perficient
Perficient is a leading information technology consulting firm serving clients throughout North America.
We help clients implement business-driven technology solutions that integrate business processes, improve worker productivity, increase customer loyalty and create a more agile enterprise to better respond to new business opportunities.
Perficient Profile Founded in 1997
Public, NASDAQ: PRFT
2011 Revenue of $260 million
Major market locations throughout North America— Atlanta, Austin, Charlotte, Chicago, Cincinnati, Cleveland,
Columbus, Dallas, Denver, Detroit, Fairfax, Houston, Indianapolis, Los Angeles, Minneapolis, New Orleans, Philadelphia, San Francisco, San Jose, Southern California,St. Louis and Toronto
Global delivery centers in China, Europe and India
2,000+ colleagues
Dedicated solution practices
87% repeat business rate
Alliance partnerships with major technology vendors
Multiple vendor/industry technology and growth awards
Perficient brings deep solutions expertise and offers a complete set of flexible services to help clients implement business-driven IT solutions
Our Solutions Expertise & Services
Business-Driven Solutions• Enterprise Portals• SOA and Business Process Mgmt• Business Intelligence• User-Centered Custom Applications• Interactive Design• CRM Solutions• Enterprise Performance Management• Customer Self-Service• eCommerce & Product Information
Management• Enterprise Content Management• Enterprise Resource Planning• Management Consulting• Industry-Specific Solutions• Mobile Technology• Security Assessments
Perficient Services End-to-End Solution Delivery IT Strategic Consulting IT Architecture Planning Business Process & Workflow
Consulting Usability and UI Consulting Custom Application Development Offshore Development Package Selection, Implementation
and Integration Architecture & Application Migrations Education Interactive Design
Our Speakers
Eric Roch• Principal SOA | Mobile | Cloud • 30+ years of experience in various aspects of
Information Technology including:• IT executive level management within industry and
consulting• technical architecture• application and systems development.
• He has also been an IT industry speaker and author for many years.
Ben Hahn• Sr. Technical Architect • 20+ years of experience in various aspects of
Information Technology. • Software Solutions Architect• Enterprise Infrastructure Architect• Product Management
• Software community contributor
6
Agenda
• Glitches and outages drive the need for more comprehensive logging and monitoring
• Big Data and Logging • Events and Logging • Logging and exception management as a
service• Explicit Transaction Monitoring• Instrumentation for Logging • Logging and Exception Management
Framework and Demo
7
The Need for Application Monitoring
• Systems are distributed, increasingly more complex and we are progressively more dependent upon them
• Application glitches are costly in dollar terms, customer loyalty and brand image:– Department of Commerce study found software errors cost U.S. economy
$59.5 billion annually– Information Week says IT downtime costs $26.5 billion in lost revenue– The 2003 North America blackout was triggered when a local outage was
not detected by monitoring software – Glitches in the reporting of prices on the NYSE may have caused the “flash
crash”– Knight Capital Group Inc. lost $440 million from software trading error –
about $10M a minute for 45 minutes – Programming error at Visa Debit Processing Services caused a billing error of
$23,148,855,308,184,500
8
Humans Monitoring for Glitches
Tourist follows GPS into Bay UK woman follows GPS into river
9
Application Monitoring
“Oh yeah, well I’m watching you watch him”Source: E*TRADE Baby
10
Machine Monitoring
• Log analysis is well established for IT operations, security and compliance
• Enterprise management software standards to detect platform and network problems
• LogLogic appliance can ingest up to 250,000 events per second with high speed filtering and routing capabilities
• Splunk provides general-purpose search language for analysis and reporting for time-series data using the MapReduce framework
11
Finding Glitches in the Data
Source: splunk
12
Log Analysis vs. Business Analytics
• Ingest – Versus ETL • Big Data – Bidirectional integration with Hadoop• Query language – MapReduce function on unstructured data • Drill anywhere – Investigate on all the data versus a predefined schema
or cube• Information discovery – Discover relationships based on patterns in the
data • Ad-hoc versus dimensional – Log analysis is not based a predefined
structure based a point-in-time set of requirements
Source: splunk Implementation
1313
Business Events and Event Processing
• Event-driven architecture (EDA) is a software architecture pattern promoting the production, detection, consumption of, and reaction to events.
• Complex event processing (CEP) consists in processing many events happening across all the layers of an organization, identifying the most meaningful events within the event cloud, analyzing their impact, and taking subsequent action in real time.
14
CEP High-Level Use Cases
• Situation awareness is about "knowing" the state of the product, person, document, or entity of interest at any point in time.
• Sense and respond is about detecting some significant fact about the product, person, document or entity of interest, and responding accordingly
• Track and trace is about tracking the product, person, document or entity of interest over time and tracing pertinent facts
Source: TIBCO Software
1515
CEP Architecture
• Millions of raw events can be represented in one complex event
• Component status (fine grain) – outage (logical /predictive)
• Events are process with declarative rules and implicit state management
• Events drive agile business processes
15
ConceptState
RuleBases
BPMS
CEPEngine
Logical Events – Notifications, Consequences Actions
SOA
Business Applications
Fine-grainBusiness Events
System(s) of Record
Integration andBusiness Components
Flexible Workflows
ESBEvent
Channel(s)
16
Business Event Logging with CEP
17
Application and Business Event Logging
• Monitoring logical transaction state vs. monitoring transaction through-put • Exception management vs. error logging • Explicit application event logging versus machine logging • Application instrumentation to a framework• Exception management and log correlation
– Transaction and conversation IDs – User defined fields – XML processing - XSD, XSLT and XPath– Agents and appenders – Time and dependencies
17
18
Logging and Exception Management Design Goals
• Highly configurable and user friendly GUI • High-speed • Non-blocking asynchronous calls • Open architecture / standards support • Technology agnostic • Service-oriented
19
What is GAL?
Generic Audit Logger• A configurable message logger to
record, view or reply message• Fully configurable via a GUI • Map logs to custom fields• Filter and query logs• Supports message query from GEH• Works standalone and supports our
exception handler
GAL DEMO
21
What is GEH?
• Generic Exception Handling• Provides
– Centralized Exception Handling– Process/Workflow Exception Resolution– Issue Resolution Knowledge Base– Centralized Audit Trail– Centralized Transaction Monitor
• Benefits– Standardized Issue Resolution– Minimal Project Based Exception Handling– Dynamic Views / Dynamic Workflows– Faster Learning Curve– External to Internal Exception Mappings
GEH DEMO
Transaction Monitoring via Logging
23
• Leveraging GAL and GEH to define a transaction
Using recognition rules, each log entry can define a• Transaction demarcation – begin or end• Transaction context – using a conversation identifier• Transactional checkpoints – define completed stages in a transaction.• Transactional errors
Once a transaction is defined we get• Monitoring• Instrumentation• Error tracking via an error handler like GEH• State and through-put
Transaction Capturing via Logs
24
Events via Transactions
25
Now that we have transactions we can also define events• Each log entry can now also define an event• Events can be aggregated to make more logical events• Events can also be generated for “non-events” e.g. if number of
transactions do not reach a certain threshold (sales quotas)
Transaction Event Generation
26
And eventually published out to a CEP
TRANSACTION MONITOR DEMO
Q & A