IDC & Gomez Webinar --Best Practices: Protect Your Online Revenue Through Web Performance Testing
-
Upload
compuware-apm -
Category
Technology
-
view
1.172 -
download
1
description
Transcript of IDC & Gomez Webinar --Best Practices: Protect Your Online Revenue Through Web Performance Testing
Best PracticesBest PracticesProtect your Online Revenue through Web Performance Testing
Melinda Carol-Ballou - Application Life-Cycle Management
Imad Mouline CTO Gomez
pp y g& IT Executive Strategies, IDC
Imad Mouline - CTO, Gomez
Optimizing Application Quality and Performance to Drive the BusinessOptimizing Application Quality and Performance to Drive the Business
Melinda BallouApplication Life-Cycle Management IT E ti St t iIT Executive Strategies September 2009
Copyright 2009 IDC. Reproduction is forbidden unless authorized. All rights reserved.
AgendaAgenda
Understanding the drivers and challenges for taking aUnderstanding the drivers and challenges for taking a life-cycle approach to application quality and performance
Coordinating quality and performance from requirements building through development and deploymentdeployment
Establishing process maturity and organizational strategies for successful application deployments
© 2009 IDC 3
Disruptive Trends Driving Automated Software Quality Adoption (ASQ)Disruptive Trends Driving Automated Software Quality Adoption (ASQ)• Evolving, flexible development paradigm with services creation will increasingly demand technology and business collaboration and drive for service quality within and outside the firewall strong agile emergenceservice quality within and outside the firewall – strong agile emergence
• Changing approaches to application testing as application packages evolve to new architectures
• End-user experience and business impact challenges of emerging web technologies (rich Internet), virtualization, cloud computing and Web 2.0
Global economic competition plus local compliance initiatives across• Global economic competition plus local compliance initiatives across geographies demand quality, adaptability and rigor
• Complex sourcing/off-shoring necessitate strong teaming and effective testing and quality management; limited infrastructure creates barriers
• License and delivery mechanisms with SaaS for new models, choices, customer support, speed adoption, limit complexity (human and systems)
© 2009 IDC 4
pp , p p , p y ( y )
“Quality Gap”: High Cost of Failure “Quality Gap”: High Cost of Failure
Poor Quality = Increased Business Risky
($$$$$)Lost Revenue
($$$$$)Lost Customers Increased CostsLost Revenue Lost Customers Increased Costs
L t P d ti it L P fitDamaged Brand
© 2009 IDC 5
Lost Productivity Lower ProfitsDamaged Brand
Two Goals of Effective IT/Business AlignmentTwo Goals of Effective IT/Business Alignment
Innovation:Innovation:Maximize Upside
Through Technology-Enabled Business
New Business Value
Processes
ReducedExposure
Compliance:Compliance:Minimize Downside
Through Risk Management
© 2009 IDC 6
g
Key IT Challenges: Business-Criticality
Key IT Challenges: Business-Criticality
Today’s applications are high-visibility, and carry a high cost of failure:high cost-of-failure:
Customer “self-serve”
Supplier/channel integrationSupplier/channel integration
Internal applications that automate critical business processes
“Network effect”: failure in one applicationcan mean loss of service in others.
“Outside-in” Testing is Increasingly Important Due to Web Application Delivery Chain Complexity
© 2009 IDC 7
Performance Problems: a Leading Cause of Failure for New ApplicationsPerformance Problems: a Leading Cause of Failure for New Applications
Q. When your software fails to satisfy users, the top 3 reasons are:
Flawed specification
Poor performance /capacity/response time
Under-estimated resources /time required
p
Inability to integrate with other software
Failure of app. to evolve with business needs
Flawed implementation of a correct spec
Data quality issues
Dissatisfaction with user interface
Inefficient or incorrect configuration
Quality of code /lack of testing
Flawed implementation of a correct spec
© 2009 IDC 8
0 5 10 15 20 25 30 35(% of respondents)Source: IDC n=467
IT Business Challenges: Silos, Gaps IT Business Challenges: Silos, Gaps
The need for a quality life-cycle is key since G2000 organizations are split across interdependent groups:
Business/users stakeholdersDevelopersQA professionalsOperational staff
Increased complexity for both technology and source (offshore, etc.) drives need for coordination
Must extend the Quality life-cycle across phases andMust extend the Quality life cycle across phases and groups
© 2009 IDC 9
Rise of Global Teams and BusinessesRise of Global Teams and Businesses
EuropeanUI Team EuropeanBusiness Unit
APACBusiness UnitDev/QA/
Dev/QA/Support
US Business Sponsors
Content
SupportContent
Dev/QA/Support
BusinessAnalysts
© 2009 IDC 10
Coordinating across the Life-CycleCoordinating across the Life-Cycle
• Coordinating requirements, testing, and operational performance are key across core emerging technologies
• Slow response times for key business areas are problematic
• Organizations should target quality life-cycle approaches th h i t it t t t i t ti dthrough requirements, unit test, system integration and pre-deployment and performance testing and change management
• Leveraging test automation for synthetic transactionLeveraging test automation for synthetic transaction monitoring enables users to manage user experience with applications
• As business requirements change, a cogent life-cycle approach enables adaptive testing and monitoring for flexible responses
© 2009 IDC 11
Establishing Process and Org StrategiesEstablishing Process and Org Strategies
Engage Upfront Business input with requirements and QA and change activities via organizational and process strategy
“Customer advocates” – attuned to business and IT– Design a test plan based on how real users interact with
your applications• Where they do it
– LA, Edmonton, London, Bangkok, NY, etc• What they do
– Key tasks, transactions, and business processes• How they do it
– Browsers galore, mobile devices• When they do it (peak times, spikes and how frequently)
Better estimate effort better coverageEarly requirements/test/design helps reveal design problems –
© 2009 IDC 12
y q g p g pmanaged change key
Establishing Process and Org StrategiesEstablishing Process and Org Strategies
Involve LoB & Ops in projects earlier
Refine test plansRefine test plans– Identify key geographies and third-party components to test– Help define third-party SLAs, and overall “business SLAs”
Coordinate testing of external components and services delivered from outside the datacenter
Anticipate deployment and configuration issuesAnticipate deployment and configuration issues Leverage tools, tests and skill sets across the life-cycle
to ensure functionality, availability, and performance y y pgoals are met
© 2009 IDC 13
Application Quality – Design and DeliveryApplication Quality – Design and Delivery
Quality Evaluation Matrix
•Easy to Use
U di t bl
•Easy to Use
B h
esig
n
•Unpredictable Behavior
•Behaves as Designed Quality of
Customer ExperienceD •Difficult to Use
•Unpredictable Beha ior
•Difficult to Use
•Behaves as D i d
Experience
Behavior Designed
© 2009 IDC 14
Delivery
App Deployment & Support: Old ViewApp Deployment & Support: Old View
Define
DEV
SLAs, Provision
Test/Tune Monitor Support
Issues
Result:Little input into specifications or developmentLittle input into specifications or developmentLittle leverage between development/ops of testing/monitoring investmentsTesting/tuning LATE in the cycle!
© 2009 IDC 15
Testing/tuning LATE in the cycle!
Closing the Loop: Leverage Skills & Tools for Agile, Iterative ApproachClosing the Loop: Leverage Skills & Tools for Agile, Iterative Approach
•Shared goals,
DesignDefine
g ,•Shared metrics,•Share tests, tools and skills Define
Develop
tools,and skills•Shared systems
Support
Deploy & Monitor
Test & Tune
© 2009 IDC 16
Establish a Common LanguageEstablish a Common Language
“Network statistics look great.”“All servers are available ”
“Sales says customers can’t
“Call Center’s swamped with All servers are available.
“All databases are available.”“99% of our monitored apps are OK”
“Service levels are within limits.”
customers can t find product in the
database.”
swamped with Order Status
inquiries.”
“We’re not “Order Service levels are within limits.meeting our first call resolution metric today.”
volume is down.”
Business
Bridge the communication gap!IT
© 2009 IDC 17
Bridge the communication gap! Establish a common language!
Bridging the Customer Experience GapBridging the Customer Experience Gap
Business M
“Is the end-to-end business processManagement
Customer End-to-End Usage Scenarios
business process working?”
Customer ExperienceManagement
g
Web Voice Web Services email FAX
“Can customers do what they want to do?”
Application
Web, Voice, Web Services, email, FAX…
“Can I proactively fix things before
want to do?
Server, Network & Element Management
Management fix things before they impact the business?”
© 2009 IDC 18
Element Management
Closing a ‘Customer Experience’ GapClosing a ‘Customer Experience’ Gap
1. Establish a common language between IT and the business, designate a liaison
2. Agree on business process measurement priorities
3. Implement testing/management for end-to-end p g gbusiness processes – from the customer/end-userperspective
4. Eliminate silos and close the loop between development and operations
S f5. Share goals, metrics, feedback with the entire team: make Customer Experience Management part of the culture
© 2009 IDC 19
SummarySummary
•Coordinate a Quality Life-Cycle approach from requirements through to operations to enable adaptive businesses
•Schisms between business, development, testers and operations must be addressed
•IT groups and the business must build a common language, common metrics, and common tools and practices
•Focusing on the customer experience will help IT achieve this along with core incentives to collaboration
•Making “customer experience” the culture is key to help bridge the gap and to enable successful collaboration across groups
© 2009 IDC 20
Best Practices: Protecting your Online Revenue
BP #1: Establish a Common Languagebetween IT and the business
BP #2: Make Customer Experience Management part of the culture
BP #3: Implement testing & management for end‐to‐end business processes – from the customer/end‐processes from the customer/enduser perspective
BP #1: Establish a Common Language Between IT & the Business:Customer Satisfaction Index
Apdex (Application P f I d )
Quickly isolate and prioritize i i f fPerformance Index)
Single measurement of user satisfaction with W b f
opportunities for performance improvement based on real customer satisfaction
h fWeb performance Based on response time thresholds
Why is customer satisfaction decreasing? What do users have in
?common? How do I compare against my competitors?Satisfied Tolerating Frustrated
http://www.apdex.org 2007
Customer Satisfaction Index: An Example
A l d i i f i dAs load increases customer satisfaction decreases
Customer satisfactionsatisfaction decreases
Load increases
BP #2: Make Customer Experience Management Part of your Culture ‐ Design your Web site for a Multi‐browser World
ill i k ll b ?Will it work across all browsers?Functions correctlyFunctions correctly
Does not function correctly = brand damageDoes not function correctly = brand damage
Make Customer Experience Management Part of your Culture ‐ Test & Monitor your Key Transactions
Further analysis shows sequence 0 of the transaction - the login page - is
However sequence 1, the authentication step, is i t d b junaffected impacted by a major performance issue
Make Customer Experience Management Part of your Culture – Ensure that your Applications Will Scale Under Load
Obama family exclusively wears leading fashion brand for inauguration ceremony
Leading fashion brand’s web application crashes leading to short and long term loss of revenue and brand equity
BP #3: Implement Testing/Management for End‐to‐end Business Process from the Customer/End‐user Perspective
Behind‐the‐firewall testing
tools: “OK”
…user is NOT happy
Load Balancing
Users
The traditional view of Web Application Delivery
Web Servers
App Servers The traditional view of Web Application Delivery
Storage
DB Servers
Mobile Components
Traditional zone of control
Critical to Test Across Entire Web Application Delivery Chain
The Web Application Delivery Chain
Behind‐the‐firewall testing
tools: “OK”
…user is NOT happy
Local ISPLoad Balancing
Users3rd Party/
Cloud ServicesBrowsers
and Devices
• Inconsistent geo performance• Bad performance under load
• Network peering problems
• Poorly performing
Web Servers
App Servers • Network peering
problems
Bad performance under load• Blocking content delivery• Incorrect geo‐targeted content
p• Bandwidth throttling• Inconsistent connectivity
• Configuration errors
p gJavaScript
• Browser/device incompatibility
• Page size too bigT
Internet
MajorISP
Storage
DB Servers
problems• Outages
• Configuration issues• Oversubscribed POP
• Application design issues
• Code defects• Insufficient infrastructure
• Too many objects
• Low cache hit rate
• Network resource shortage
• Faulty content transcoding SMS i /
Content DeliveryNetworks
Mobile Carrier
Mobile Components
• Poor routing optimization• Low cache hit rate
• SMS routing / latency issues
Zone of customer expectationZone of customer expectationTraditional zone of control
Load Testing 2.0Performance Testing For A Web 2.0 World
Self‐service testing gives you control to test as needed
T t i t l & t l bTest internal & external web application components to ensure transactions perform under load
Test cloud‐based applications &Test cloud‐based applications & services
SaaS based Load Testing 2.0 solutions enable on‐demand, ,scalable testing without associated hardware investment & maintenance costs
C bi hi h l Test the full web application delivery chain all the way to your end users’ desktops
Combine high volume application load testing with broad geographic experience testing from the outside-in
Backbone
Gomez Reality Load XF: On‐Demand Realistic Load Testing from Browser to Data Center
Last MileBackboneHigh volume load (HTTP, Browser)
Find infrastructure breaking pointsDefine capacity headroom
Last MileReal‐world load
Find user experience breaking pointsAccurately measure response time
Local ISPLoad Balancing
Users3rd Party/
Cloud ServicesBrowsers
and Devices100,000+ consumer‐
d
100,000+ consumer‐
d
100,000+ consumer‐
dWeb Servers
App Servers
grade desktops
168+ countries
grade desktops
168+ countries
grade desktops
168+ countries100+
commercial‐100+ commercial‐100+ commercial‐
Internet
MajorISP
Storage
DB Servers
2,500+ ISPs
Major mobile
2,500+ ISPs
Major mobile
2,500+ ISPs
Major mobile
grade nodes & data centers
grade nodes & data centers
grade nodes & data centers
Content DeliveryNetworks
Mobile Carrier
Mobile Components
carriers around the globe
carriers around the globe
carriers around the globe
Ensuring Performance of All 3rd Party Components
Company: Online Retailer• Several 3rd Parties now involved in serving up key content
G l t lid t f f ti li ti• Goal was to validate performance of entire application
3rd Party/ Browsers Local ISP
y/Cloud Services and devices
Load Balancing
Web
Load Balancing
Web
Users*
Internet
Servers
App Servers
DB
Servers
App Servers
DB MajorISP
Storage
Servers
Mobile Components
Storage
Servers
Mobile Components
Content DeliveryNetworks
Mobile Carrier
ComponentsComponents
Response Times Rise Due To 3rd Party Object Error
The load increases throughout the test
The transaction rate increases and then falls off as responsefalls off as response times climb
Errors are seen, all on a 3rd party objecton a 3rd party object
3 d t h d i ffi i t f ll d d li ti• 3rd party hardware was insufficient for overall demands on application• Based on SLAs 3rd party had to improve performance to get paid
Ensuring Performance in Key Markets
Company: Regional Online News Source• Began testing for the 2008 election season
G l t lid t ll f f i i 2 k i• Goal was to validate overall performance focusing in 2 key regions
3rd Party/ Browsers Local ISP
y/Cloud Services and devices
Load Balancing
Web
Load Balancing
Web
Users
*
Internet
Servers
App Servers
DB
Servers
App Servers
DB MajorISP
Storage
Servers
Mobile Components
Storage
Servers
Mobile Components
Content DeliveryNetworks
Mobile Carrier
ComponentsComponents
No Performance Issues Detected From Data-Center
There was only 1 page error and
Increase and hold load and not exceed response times of 4 seconds and Success Rate of 99%
Page response times stayed under
11 errors total out of 60000+ transactions
times stayed under 4 seconds, outside of one brief blip
By traditional test standards the test passed
Performance Issues Detected From Real User Desktops
Key geographies for this customer are New York and PennsylvaniaNew York and Pennsylvania
Last Mile data showing substantial number ofLast Mile data showing substantial number of measurements greater than 4 seconds
The Gomez Platform: Web Application Experience Management
SelfSelf‐‐Service SaaS PortalService SaaS Portal
20• Compatibility• Compatibility
Web Cross‐Browser
Testing
Web Cross‐Browser
Testing
• High‐volume• High‐volume
Web Load and Performance
Testing
Web Load and Performance
Testing
• Real‐user monitoring• Real‐user monitoring
Web Performance Management
Web Performance Management
• Web performance• Web performance
Web Performance
Business Analysis
Web Performance
Business Analysis
20Compatibility• Functionality• Performance
Compatibility• Functionality• Performance
High volume backbone load
• Real‐world Last Mile load
High volume backbone load
• Real‐world Last Mile load
Real user monitoring• Last Mile monitoring• Backbone monitoring
Real user monitoring• Last Mile monitoring• Backbone monitoring
Web performance analytics
• Benchmarks• Business dashboard
Web performance analytics
• Benchmarks• Business dashboard
Web 2 0 RIA Streaming Mobile
Dashboards, Metrics & Analytics
Dashboards, Metrics & Analytics
Recording & ProvisioningRecording & Provisioning
Education &Best PracticesEducation &Best Practices
APIs &Data Feeds APIs &
Data Feeds Alerting & DiagnosticsAlerting & Diagnostics
Web 2.0, RIA, Streaming, Mobile
World’s Most Comprehensive Testing NetworkWorld’s Most Comprehensive Testing Network168+ countries 2,500+ ISPs168+ countries 2,500+ ISPs
100+ commercial nodes and data centers
100+ commercial nodes and data centers
500+ combos of
browsers & O/S
500+ combos of
browsers & O/S
5,000+ supported
mobile devices
5,000+ supported
mobile devices
100,000+ consumer‐grade
desktops
100,000+ consumer‐grade
desktops
Gomez Customers Enjoy Measurable Benefits
Increased conversions 10%
Reduced homepage load time from 11 3 d t 3 4 d11.3 seconds to 3.4 seconds
Improved page load times 23%
Reduced seven‐step transaction time by 50%
Saved 50%+ in staff and fees
Reduced downtime 45%
y
Achieved under 3 second
Validated decision to consolidate
Achieved under 3 second response time and 99%+ availability
Validated decision to consolidate three data centers
Appendix
Ensuring Performance Of Login Process
Company: Online presence for a popular TV show
• Following episodes of the TV show the web site sees high traffic spikes
• Goal was to achieve 1500 logins per minute• Goal was to achieve 1500 logins per minute
• Load tested DB to improve performance in anticipation of another traffic spike
3rd Party/ Browsers
1Local ISP
y/Cloud Services and devices
Load Balancing
Web
Load Balancing
Web
Users
Internet
Servers
App Servers
DB
Servers
App Servers
DB MajorISP
Storage
Servers
Mobile Components
Storage
Servers
Mobile Components
Content DeliveryNetworks
Mobile Carrier
ComponentsComponents
Application Bottleneck Causes Immediate Response Time Issue
• As users were added the• As users were added, the response time of step 3 (the login) climbed immediately
• The test bottlenecked at 160 logins per minute (Goal 1500)
• But quickly dropped off as users received server errors
• New login query was not optimized and was bottlenecking the database
’ CPUservers’ CPUs