Holiday Readiness: Best Practices for Successful Holiday Readiness Testing
-
Upload
apica-systems -
Category
Technology
-
view
96 -
download
0
Transcript of Holiday Readiness: Best Practices for Successful Holiday Readiness Testing
![Page 1: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/1.jpg)
Best Practices for Successful Holiday Readiness Testing
![Page 2: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/2.jpg)
Why Load Test?
![Page 3: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/3.jpg)
Performance MattersOrganizations with degraded business apps (IDG)
Problems identified for IT by end-users (EMA)- employees – 43%- business stakeholders – 13% - customers – 12% Problems resolved > 1 month, if ever (Forrester)
Engineering time spent in triage (TechValidate)
75%68%
31%40%
![Page 4: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/4.jpg)
![Page 5: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/5.jpg)
Black Friday 2015Forbes: The Slow, The Crashed, The Out Of Stock: #BlackFriday #Fail Twitter Report
26th
27th
![Page 6: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/6.jpg)
Unit and Functional Tests Don’t Find Everything• Concurrency Bugs
– Bugs that only present when same code is run multiple times at the same time
• Compositional Bugs– Bugs caused by the interplay between separate pieces of your
application• DB Indexes and Locks
– Do the indexes and query locks on your database play nicely at load• Application or Web Server Configuration
– Do your application or web servers exceed memory, socket, or other configuration constraints
• Infrastructure Limitations– Bandwidth, Session Tables, Disk IO, Etc.
![Page 7: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/7.jpg)
Application Complexity is Increasing
![Page 8: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/8.jpg)
THEN
![Page 9: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/9.jpg)
Application Complexity is IncreasingNOW
![Page 10: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/10.jpg)
Can You Answer These Basic Questions?• What are your application’s bottlenecks?
• Performance at normal and peak usage levels?
• Most common causes of performance issues or load based failures?
• Is your hardware optimized to meet business objectives?
![Page 11: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/11.jpg)
When to Load Test?
![Page 12: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/12.jpg)
When to Load Test – Early is GOOD
When performing load tests to prepare for holiday traffic, start early so you
have time to fix problems
• Allow enough time for several fix+test iterations• Take your code lock date into account• Understand the complexity of your application• Compensate for external teams
![Page 13: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/13.jpg)
When to Load Test – Always is BETTER
Consistently running load tests ensures that you already know your limits, and aren’t
forced to rush development for peak events
• Integrate load testing as part of your deployment strategy
• Run smaller load tests before or during deployment, and schedule periodic larger load tests to understand your performance, bottlenecks, and breaking points under peak loads
![Page 14: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/14.jpg)
![Page 15: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/15.jpg)
What Kind of Load Tests Should You Use?
![Page 16: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/16.jpg)
Load Test Types
Stress Concurrency
Disaster Recovery
![Page 17: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/17.jpg)
Stress Tests
Continuously increase load to determine absolute theoretical bottlenecks and breaking points
• Start with known good traffic levels• Run successive tests while doubling the traffic each time• When breaking point is found, you can run tests between
the last successful test and this one to find your exact breaking point(s)
![Page 18: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/18.jpg)
Concurrency Tests
Realistic traffic to find practical performance and failure limits
• Determine your actual traffic levels (average and peak), and the ratio of user flows/actions
• Run a combination of load tests that simulate realistic traffic• Perform “what if” tests by either changing the mix of user
flows/actions, or increasing the load of one or more of them• Concurrency Tests against auto-scaling environments can
test scaling response times
![Page 19: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/19.jpg)
Disaster Recovery Tests
Sustained traffic at realistic load levels to test application response to failures
• Start a long-running load test• Simulate various application or infrastructure failures (e.g.
turn off a web server, or unplug a DB)• See if failover processes work as intended, and as fast as
required• DR tests without simulated failures are ENDURANCE tests
![Page 20: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/20.jpg)
Recording Your Load Test
![Page 21: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/21.jpg)
Questions to Ask When Recording Scripts
What do you want to know?• Max concurrent users before failure• Performance at expected peak• 3rd party content impact on performance• Current bottlenecks• Auto-scaling results at various loads• Etc.
![Page 22: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/22.jpg)
Questions to Ask When Recording Scripts
What requests matter?• Do you care about static/CDN served content?• Do you need to include 3rd party content?• Do supporting requests (e.g. auto-populate) matter?
![Page 23: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/23.jpg)
Questions to Ask When Recording Scripts
Do you need unique or dynamic data?• Usernames and passwords• Form data (e.g. dates)• Product or record ID’s• Etc.
![Page 24: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/24.jpg)
Questions to Ask When Recording Scripts
What constitutes a failure?• Content verification• HTTP Status checks• General or request specific time limits• Unusual response sizes
![Page 25: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/25.jpg)
Steps to Recording Scripts1. Determine the user flow2. Record using a proxy recorder3. Remove unnecessary requests4. Handle variables5. Add external data6. Configure failure detection
![Page 26: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/26.jpg)
Running Your Load Tests
![Page 27: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/27.jpg)
Considerations When Running Load Tests•SaaS vs Private•Target Environment (Dev, DR, Prod)•Other performance metrics (APM, etc.)•Database preparation•Team communication methods•Criteria for aborting
![Page 28: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/28.jpg)
Interpreting Results
![Page 29: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/29.jpg)
What Does It Mean? (Load Curves)
USERSSESS
ION
DU
RATI
ON
USERSTH
ROU
GH
PUT
USERSFAIL
URE
S / E
RRO
RS
Ideal Results
![Page 30: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/30.jpg)
What Does It Mean? (Load Curves)
USERSSESS
ION
DU
RATI
ON
USERSTH
ROU
GH
PUT
USERSFAIL
URE
S / E
RRO
RS
Realistic ResultsFinding Your Peak Load
SAFE PEAK UNSTABLE
![Page 31: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/31.jpg)
What Environments?
![Page 32: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/32.jpg)
Load Testing @ Deployment Stages• Dev Environments
– Good for finding problems early, but unreliable for absolute performance metrics– Catch concurrency, composition, or DB problems
• Staging/QA Environments– Catch problems caused by rolled up commits that weren’t visible when testing singular changes– Compare to previous runs to get early warnings for reduced performance or peak capacity– May or may not be reliable indicators of production performance depending on the stability of the
environment, and similarity to the production environment• Pre-Production
– Should be as close to the production environment as possible– Best indication of production performance and load capacity before live deployment
• Production– Useful for absolute verification of readiness for expected traffic spikes (e.g. product launces, sales,
holidays, marketing events)– Can be used for periodic validations (DR, seamless code deployment, performance validation)
![Page 33: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/33.jpg)
Questions?
![Page 34: Holiday Readiness: Best Practices for Successful Holiday Readiness Testing](https://reader033.fdocuments.us/reader033/viewer/2022042907/58cee4a71a28ab333d8b4ea7/html5/thumbnails/34.jpg)
Additional Resources
apicasystems.com: Enterprise cloud and mobile performance solutions- Whitepapers, Webinars, Case Studies- Blog- Download Apica Zebratester Free Edition | 50 Free VUs
- Signup for Apica LoadTest Free Trial | 500 Free VUs- Signup for Apica Monitoring Free Trial
community.zebratester.com: Load test community