Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating...
Transcript of Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating...
![Page 1: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/1.jpg)
© Planit Software Testing
Performance Testing Mobile Devices
Presenter: Jaron Marsh
![Page 2: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/2.jpg)
© Planit Software Testing
Our knowledge is your power
Software Testing
Consulting, Advisory, Delivery,
Managed Services, Training
![Page 3: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/3.jpg)
© Planit Software Testing
Introduction
![Page 4: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/4.jpg)
© Planit Software Testing
Agenda
1. What are we testing
2. What do we need to consider
3. Mobile application case study
4. Different approaches
5. Lessons learned
![Page 5: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/5.jpg)
© Planit Software Testing
Common current perception:
“If you haven’t performance tested a
mobile app before, you wont know how
to do it!”
![Page 6: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/6.jpg)
© Planit Software Testing
What are we testing?
![Page 7: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/7.jpg)
© Planit Software Testing
Firstly – what is Performance Testing?
Simulating user behaviour of application
Testing against Non Functional
Requirements (NFRs )
Analysing all results (test and monitoring results)
to produce a view of performance behaviour
![Page 8: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/8.jpg)
© Planit Software Testing
Different mobile applications
1. Native application (e.g. Facebook, Angry Birds)
2. Mobile Web application (e.g. Financial Times)
3. Hybrid application (e.g. Instagram
mobile app)
![Page 9: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/9.jpg)
© Planit Software Testing
How do they work?
Largely the same as a desktop application!
1. User interacts with the application
2. Application may or may not send a request to the
backend
3. Backend returns a response to the user, back to step 1.
![Page 10: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/10.jpg)
© Planit Software Testing
What do we need to consider?
![Page 11: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/11.jpg)
© Planit Software Testing
What type of app are we testing?
Does it matter?
![Page 12: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/12.jpg)
© Planit Software Testing
Script Creation
Three options:
1. Manually build each request
2. Record traffic and re-build requests
3. Record with performance tool through a proxy to create
the requests
![Page 13: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/13.jpg)
© Planit Software Testing
Load Generation
What do you want to replicate?
Where can you generate the load from?
![Page 14: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/14.jpg)
© Planit Software Testing
Load Generation
Be aware of the impact with different
options of load generation
![Page 15: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/15.jpg)
© Planit Software Testing
Is there anything unique about apps?
• Data/CSS etc. often loaded during app start up
• Not as easy to ‘record and replay’
• User-Agent HTTP header is critical to simulate the
correct device/browser
![Page 16: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/16.jpg)
© Planit Software Testing
Mobile App Case Study
![Page 17: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/17.jpg)
© Planit Software Testing
Project Overview
• Hybrid financial transaction app
• Used across multiple devices/platforms/networks
• Integrated to use an ESB
• All IBM components
• Large amount of backend requests with user interactions
• Financial and legal requirement for up to date information
![Page 18: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/18.jpg)
© Planit Software Testing
Infrastructure
![Page 19: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/19.jpg)
© Planit Software Testing
Challenges
• Lack of end to end transaction knowledge
• Multiple vendors
• Lack of understanding on app usage
• Difficulty recording/building transactions
• Lack of sufficient Non Functional Reqs
• Changes of scope
• No enterprise monitoring solution
![Page 20: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/20.jpg)
© Planit Software Testing
Approach
• Built transactions based off information provided by
developers
• Transactions re-built after recording traffic
• Testing from the HTTP layer
• Load injected from as close to the test environment as
possible
• SNMP monitoring across as many servers as possible
![Page 21: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/21.jpg)
© Planit Software Testing
Issues
Similar issues identified as a normal performance testing
engagement!
![Page 22: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/22.jpg)
© Planit Software Testing
Different Approaches
![Page 23: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/23.jpg)
© Planit Software Testing
What approaches are there?
Three main approaches:
1. UI performance
2. Infrastructure performance
3. Combination of the first two approaches
![Page 24: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/24.jpg)
© Planit Software Testing
What approaches are there?
Three main approaches:
1. UI performance
2. Infrastructure performance
3. Combination of the first two approaches
![Page 25: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/25.jpg)
© Planit Software Testing
UI Performance
![Page 26: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/26.jpg)
© Planit Software Testing
What approaches are there?
Three main approaches:
1. UI performance
2. Infrastructure performance
3. Combination of the first two approaches
![Page 27: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/27.jpg)
© Planit Software Testing
Infrastructure Performance
![Page 28: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/28.jpg)
© Planit Software Testing
Infrastructure Performance
![Page 29: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/29.jpg)
© Planit Software Testing
What approaches are there?
Three main approaches:
1. UI performance
2. Infrastructure performance
3. Combination of the first two approaches
![Page 30: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/30.jpg)
© Planit Software Testing
Blended Approach
Outcome
1. We have tested the infrastructure (main performance risk)
under load
2. We have identified performance of the application on a mobile
device while the infrastructure is under load
![Page 31: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/31.jpg)
© Planit Software Testing
Lessons Learned
![Page 32: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/32.jpg)
© Planit Software Testing
Lessons Learned
• More companies looking at mobile apps
• Higher turnaround getting to market
• Higher turnaround pushing out updates
• Can’t guarantee performance for every single user
• Similar approach to traditional performance testing
• Keep things light, no one wants an app that drains
battery or mobile data
![Page 33: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/33.jpg)
© Planit Software Testing
Tips
• Get in as early as possible (single user/request tests)
• Define exactly what is being tested as early as
possible
• Keep an eye on message sizes while scripting
• Physically use the mobile device
• Education is often the key to success
• Monitor as much as possible
![Page 34: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/34.jpg)
© Planit Software Testing
Summary
Performance testing mobile devices IS similar to traditional
performance testing – with a few tweaks!
![Page 35: Performance Testing Mobile Devices - ANZTB - …Firstly –what is Performance Testing? Simulating user behaviour of application Testing against Non Functional Requirements (NFRs )](https://reader034.fdocuments.us/reader034/viewer/2022042120/5e99831eabbebe5f7771abaf/html5/thumbnails/35.jpg)
© Planit Software Testing
Jaron Marsh
Technical Test Consultant
+64 27 526 9793
Performance Testing Mobile Devices
Contact Planit
www.planittesting.co.nz
+64 4 815 8190