Developer and Fusion Middleware 1 _ Christian Ledwidge _ Application Performance Monitoring -...
-
Upload
insync2011 -
Category
Documents
-
view
627 -
download
0
description
Transcript of Developer and Fusion Middleware 1 _ Christian Ledwidge _ Application Performance Monitoring -...
The most comprehensive Oracle applications & technology content under one roof The most comprehensive Oracle applications & technology content under one roof
Applica'on Performance Monitoring: Improve user experience, reduce user abandonment
Chris&an Ledwidge Director Solu&ons Architecture Monitoring Quest So9ware
The most comprehensive Oracle applications & technology content under one roof
Going to tell you a li<le story before we begin
• Established well established travel company wants to get into the online travel booking engine business – They have an established brand in the package holiday market with
their own fleet of aircra9 to service – Margins are under the gun and they need to find new areas for
revenue genera&on • Its seems an obvious choice they know the travel business a9er all • Have a good name that should help drive users to their new website • They ramp up a big team of marke&ng, IT, and business owners to kick this
project off • Some solid ini&al success and word of mouth spreads and site hit counts
move in the direc&on they had hoped for
The most comprehensive Oracle applications & technology content under one roof
Story con&nued • BUT! • Booking numbers are missing forecasts by a solid margin • Calls to their call center are going thru the roof
– Business: these call are ea&ng up our margin aren't they?
• Nobody in IT seems to be able to tell the business why they are not conver&ng the solid site hit rate into completed transac&ons
• Lots of finger poin&ng and conspiracy theories are geSng thrown around – It’s the network! – No it’s the DB! – Its that new Java applica&on that THEY wrote! – Why did we ever go Virtual?
• Only thing the business cares about it why are they not conver&ng all of these poten&al customers into paid clients?
The most comprehensive Oracle applications & technology content under one roof
Part One
User Experience Management (UEM) Indicators
• What are they and what do they do
• Five key indicators and how they can be used
• What sources can be used to obtain indicators
• Important points to remember
The most comprehensive Oracle applications & technology content under one roof
UEM Indicators: Bridge the Gap between Business and IT
Business
Useable? Stable? Right Content? Right Features? Fast Enough?
Available? Fast Enough? Short MTTR? Proactive Problem Solving?
IT
Back-end Systems Web User Web App
UEM Indicators • User counts • Changes in user behavior • Geographical variations • Problem impact • Service level agreements
The most comprehensive Oracle applications & technology content under one roof
UEM Indicator: User Counts
Business IT
Use Case Twenty-five users have registered in the last 15
minutes, but normally at this time, I see 200 or more people registering.
Is the reduc&on in registra&ons isolated to a specific region or is it across the board?
Are there any outages impac&ng registra&ons that we need to communicate to the business?
How many users are being impacted and from which region(s)?
Did our promo&onal campaign stop producing results or is the link broken?
The most comprehensive Oracle applications & technology content under one roof
UEM Indicator: Changes in User Behavior
Business IT
Use Case Typically 25% of users complete this transaction, but today we see fewer than 15%; why the changes in
user behavior?
Did the new site rollout hide the submit bu<on to where some users can find it anymore?
Where along the path are we losing the users and when did it change?
Are problems with the applica&on causing problems with this transac&on to where users can’t complete it?
How can we help the business restore service and measure the business impact of the change?
The most comprehensive Oracle applications & technology content under one roof
UEM Indicator: Geographical Varia&ons
Business IT
Use Case We see unexpected geographical variations in user
volumes from our EMEA region.
How many users are being impacted and for how long?
Is this a problem we are causing or is it one of our applica&on delivery partners?
Is our European proxy server having performance issues?
Is there a specific service provider that we or an ISP that our users are using experiencing issues?
The most comprehensive Oracle applications & technology content under one roof
UEM Indicator: Problem Impact
Business IT
Use Case A user just called in and reported a problem that our monitors didn’t pick up; what is the problem impact?
When did the problem start and what area of the site is being impacted?
Is the error being displayed to end users or is it just an image failing that they don’t really see?
Is there a specific web server throwing these excep&ons?
Is the right web administrator involved to get this fixed or diagnosed?
The most comprehensive Oracle applications & technology content under one roof
UEM Indicator: Service Level Agreements
Do the performance agreements that are in place guarantee a sa&sfactory user experience?
Are the IT systems mee&ng the agreed-‐upon delivery of services to end users?
Business IT
Use Case SLA goals are not being met?
When the SLA viola&ons started occurring were there devia&ons in transac&on conversions?
Were our Web, App, or Database systems at fault or was it an external interface?
The most comprehensive Oracle applications & technology content under one roof
User Experience Management Data Sources
Data Sources
User Experience Management Indicators
• Web Log Files • Synthetic Transaction Robots • Client-Side Tagging • Web Traffic Analysis Appliances • Content Capture Appliances • User Surveys
• User Counts • Changes in User Behavior • Geographical Variations • Problem Impact • Service Level Agreements
The most comprehensive Oracle applications & technology content under one roof
Data Source
Usefulness • Error tracking • User click stream analysis • Demographic analysis • Performance
Limitations • Difficult to identify
individual user sessions. • Data set is partial due to
caching.
Web Log Files
The most comprehensive Oracle applications & technology content under one roof
Usefulness • Measures transaction
performance and availability via scripted transactions executed from remote locations at set intervals.
• Sometimes referred to as a “clean room” measurement.
• Good for service level reporting.
Limitations • Measures only a small
percentage of locations. • Exercises a small
percentage of the technology stack in larger environments.
Synthetic Transaction Robots Data Source
The most comprehensive Oracle applications & technology content under one roof
Usefulness
• Good for measuring true client performance
• Gathering browser-based events needed to understand Web 2.0 user interactions
Limitations • Inserting and managing
the scripts adds complexity and overhead to the application
• Data security concerns
Client-Side Tagging Data Source
The most comprehensive Oracle applications & technology content under one roof
Usefulness
• 100% sampling of transaction performance, volume, and errors.
• Analysis has many perspectives including by server, by geography, by ISP, and even by user.
Limitations • Asynchronous
applications make it difficult to see cause and effect.
• Analyzed data set is partial due to external caching.
Web Traffic Analysis Appliances Data Source
The most comprehensive Oracle applications & technology content under one roof
Usefulness
• Stores the full user sessions, including the data returned from the web server.
• Session replay and other perspectives on the session that are attainable only when full payloads are accessible
Limitations
• Storage capacity concerns
• Privacy and security concerns
Content Capture Appliances Data Source
The most comprehensive Oracle applications & technology content under one roof
Usefulness
• True voice-of-the-customer
• Can validate metric-based hypotheses
• Answer questions like: o Why are customers
happy? o Why are customers
ready to leave?
Limitations
• Typically only small samples are gathered, and small samples may represent extremes.
User Surveys Data Source
The most comprehensive Oracle applications & technology content under one roof
Part Two
• Measuring user experience in web 1.0 and 2.0 environments
• Top 5 performance measurement challenges
• Applying approaches to address the challenges
• Key elements to remember
The most comprehensive Oracle applications & technology content under one roof
User Experience Management Performance Measurements
• Web 1.0 – Simple requests and responses
• Web 2.0 – Asynchronous requests and responses
• CDNs and content hos&ng – third-‐party effects
Processing Time Network Time Client Time
Request
Response
Back-end Systems Web App
Internet
The most comprehensive Oracle applications & technology content under one roof
Web 1.0 – Simple Page Request No Third-‐party
Client Server Client Server
1 2
Connection 1 Connection 3
GET main.html
GET image1.gif
GET frame1.html
Client Server
4
3
Connection 2
GET style.css
GET news.asp
1. Request ini&ated
2. Server gets first packet
3. Server sends last packet
4. Final response received by client
The most comprehensive Oracle applications & technology content under one roof
Web 2.0 – Asynchronous Requests
Client Server
1
2
4
3
Connection 1
GET item.js
1. Request ini&ated
2. Server gets first packet
3. Server sends last packet
4. Final response received by client
The most comprehensive Oracle applications & technology content under one roof
CDNs and Remote Content – Third-‐Party Effects
Client Server Client Server Client Akamai Client Akamai
2
4
3
Connection 1 Connection 2 Connection 3 Connection 4
GET main.html
GET perf.js GET frame1.html
GET style.css
GET news.asp
1
The most comprehensive Oracle applications & technology content under one roof
User Experience Management Measuring Performance to Solve IT Challenges
IT
Web User Back-end Systems Web App
“We learn about most problems only when a user
calls in”
“The user tries to explain her problem, but it really doesn’t
help me resolve it.” “We initially can’t tell if the
issue is on the client, in the network, or buried in the backend infrastructure.”
“We don’t know how many and which users are having
this experience.”
“We lack historical data to proactively detect unwanted performance trends and their
root cause.”
Internet
The most comprehensive Oracle applications & technology content under one roof
Challenge One “We learn about most problems when users call”
Management Approach Minimal Coverage BeCer Coverage Best Coverage
• Log parser-‐-‐ alerts on errors or performance
• Log parser-‐-‐ alerts on errors or performance
• Synthe&c package-‐-‐to add transac&on perspec&ve
• Synthe&c package-‐-‐to add transac&on perspec&ve
• Selec&ve page instrumenta&on-‐-‐captures third party content provider issues
• Passive sniffer-‐-‐low overhead capture all solu&on
The most comprehensive Oracle applications & technology content under one roof
Challenge Two “User explana'on is not useful for resolu'on”
Management Approach Minimal Coverage BeCer Coverage Best Coverage
• Call the user(s) back and inquire about issue
• Enlist support teams to recreate issue
• Selec&ve page instrumenta&on-‐-‐captures desktop events
• Scrape web logs for traces of informa&on about the issue
• Use online desktop sharing to record customer scenario
• Passive sniffer-‐-‐captures full content and recreates scenario
The most comprehensive Oracle applications & technology content under one roof
Challenge Three “Can’t isolate issue to client, network, or backend”
Management Approach Minimal Coverage BeCer Coverage Best Coverage
• Look for anomalies in web logs-‐-‐dismiss problem if no anomalies
• Look for anomalies in web logs-‐-‐dismiss problem if no anomalies
• Selec&ve page instrumenta&on-‐-‐captures third-‐party content performance
• Use synthe&c solu&on to isolate issue-‐-‐if it shows up
• Passive sniffer-‐-‐differen&ates client from network &me for requests
The most comprehensive Oracle applications & technology content under one roof
Challenge Four “Don’t know how many users experiencing issue”
Management Approach Minimal Coverage BeCer Coverage Best Coverage
• Count anomalies in web logs-‐-‐es&ma&ng impact from counts
• Count anomalies in web logs-‐-‐es&ma&ng impact from counts
• Selec&ve page instrumenta&on when third-‐party content being served
• Use synthe&c solu&on to judge impact according to bad loca&ons
• Passive sniffer-‐-‐assess impact of all first-‐party requests
The most comprehensive Oracle applications & technology content under one roof
Challenge Five “No historic info to detect performance anomalies”
Management Approach Minimal Coverage BeCer Coverage Best Coverage
• Customer sa&sfac&on surveys
• Web log analysis tools with long term reten&on
• Selec&ve page instrumenta&on when third-‐party content being served
• Web log analysis tools with long term reten&on
• Synthe&c solu&on with base lining capabili&es
• Passive sniffer-‐-‐for first party base-‐lining and anomaly detec&on
The most comprehensive Oracle applications & technology content under one roof
Important Points to Remember
• User Experience Management encourages a common understanding between IT and the business
• Common indicators enable collaboration
• Sources of User Experience Management information are many—use those that best fit your needs
The most comprehensive Oracle applications & technology content under one roof
Part Three • Why Manage Transac&on Conversions
• Web Analy&cs vs. Transac&on Conversion Analysis
• Measuring Transac&on Conversions
• Top 5 Challenges to Managing Conversions
• Points to Remember
The most comprehensive Oracle applications & technology content under one roof
Why Manage Transac&on Conversions
• Why do we manage transac&ons the way we do?
• Example Transac&on Types
ü Shopping Carts
ü Ad campaigns
ü Customer registra&on
ü Login
The most comprehensive Oracle applications & technology content under one roof
Complementary solutions
Why Transac&on Analysis—We already have Web AnalyCcs? Web Analy'cs • Primary goal: How effec&ve your marke&ng is…
• Answers “How effec&ve is my marke&ng?”
• Observe how users use your website:
– What transac&ons they use and how o9en
– Click stream analysis
Transac'on Analysis • Primary goal: What kind of experience your users have…
• Answers “What influenced user transac&on conversions?”
• Observe how user transac&on conversions are affected by:
– Content
– Applica&on func&onality
– Applica&on useability
– Applica&on stability
The most comprehensive Oracle applications & technology content under one roof
User Experience Management TransacCon Conversion Analysis
The most comprehensive Oracle applications & technology content under one roof
Measuring Transac&on Conversions
Start
Transaction
Step 2 End Step 3
Hit Hit Async Hit Hit Async Hit Hit Async Hit Hit Async
Page 1 Page 2 Page 3 Page 4
Session
Capturing Measurements
• Unique Request ID (Off the wire) – URL, Page Title, Content, Variable, Loca&on,
etc.., – Persists current transac&on state per session in
memory
• JavaScript Instrumenta&on – Captures page or request level
informa&on per session
– Reports transac&on progress back to central monitoring server
– Result sent to a third or first party
The most comprehensive Oracle applications & technology content under one roof
User Experience Management Increasing Conversion Rates through Conversion Analysis
“How much money have we lost because of this
problem?”
“We know where people are abandoning the website but we don’t
understand why.”
“I need to understand bad conversion trends immediately to reduce lost business.”
“We can’t identify how many and which users are associated with conversion issues.”
“When people abandon the site, they often call customer service – which drives up our cost of service.” Business
The most comprehensive Oracle applications & technology content under one roof
“Revenue loss -‐ a Key Indicator example”
My Transac'on Record
Transac&on Name Buy Funnel
Username Mike Smith
Browser IE 8
Cart Amount $625
Transac&on State Ac&ve
Sale Amount N/A
Lost Sale Amount N/A
1. Recognized add-‐to-‐cart page 2. Page analyzed
• Parse $$ amount 3. Updates Transac&on Record
• Add to current Cart Amount 4. Transac&on Completes Successfully 5. Increment Sale Amount
• Set Lost Sale Amount to 0
My Transac'on Record
Transac&on Name Buy Funnel
Username Mike Smith
Browser IE 8
Cart Amount ….
Transac&on State Complete Success
Sale Amount $625
Lost Sale Amount 0
The most comprehensive Oracle applications & technology content under one roof
“Understanding Bad Conversion Trends”
• Quan&ta&ve measures available in Analy&cs and UEM • Allows us to form a hypothesis about conversion trends • Qualita&ve valida&on is the next step to validate • Session replay is that valida&on
My Transac'on Record
Transac&on Name Buy Funnel
Username Mike Smith
Browser IE 8
Cart Amount $625
Transac&on State Ac&ve
Sale Amount N/A
Lost Sale Amount N/A
The most comprehensive Oracle applications & technology content under one roof
“The 5 W’s of Transac&on Abandonment”
• Where are users dropping? • Which users are dropping? • Why are users dropping? • When did users start dropping? • What were users doing when they dropped?
My Transac'on Record Transac&on Name Buy Funnel
Username Mike Smith
Browser IE 8
Cart Amount $625
Transac&on State Ac&ve
Sale Amount N/A
Lost Sale Amount N/A
My Transac'on Record Transac&on Name Buy Funnel
Username Mike Smith
Browser IE 8
Cart Amount $625
Transac&on State Ac&ve
Sale Amount N/A
Lost Sale Amount N/A
My Transac'on Record Transac&on Name Buy Funnel
Username Mike Smith
Browser IE 8
Cart Amount $625
Transac&on State Ac&ve
Sale Amount N/A
Lost Sale Amount N/A
My Transac'on Record Transac&on Name Buy Funnel
Username Mike Smith
Browser IE 8
Cart Amount $625
Transac&on State Ac&ve
Sale Amount N/A
Lost Sale Amount N/A
My Transac'on Record Transac&on Name Buy Funnel
Username Mike Smith
Browser IE 8
Cart Amount $625
Transac&on State Ac&ve
Sale Amount N/A
Lost Sale Amount N/A
My Transac'on Record Transac&on Name Buy Funnel
Username Mike Smith
Browser IE 8
Cart Amount $625
Transac&on State Ac&ve
Sale Amount N/A
Lost Sale Amount N/A
The most comprehensive Oracle applications & technology content under one roof
“People calling customer service drive up costs”
• Relying on customers to call in when problems with transac&ons is costly – Lost revenue – Damage to brand—reduces repeat purchases
– Call center personnel are expensive
• Transac&on Conversion Analysis provides insight to: – Spot unwanted conversion trends before
they impact the business – Increases efficiency of call-‐center personnel
—look at a user session to recreate the problem
The most comprehensive Oracle applications & technology content under one roof
Points to Remember
ü Transaction analysis is insurance for achieving and maintaining desired business outcomes
ü Weigh the costs of not doing versus doing conversion analysis
ü Is analytics enough or do you need something more to make you analytics actionable?
The most comprehensive Oracle applications & technology content under one roof
Thank you for your &me
Please drop by the Quest booth in the vendor hall if you would like to have a look at our approach to Applica&on Performance Management