Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile...
Transcript of Presented by Jorge Asensio - ucaat.etsi.org · Advanced Automated Testing 2 Trends The Mobile...
Parallel Automated Testing on Mobile Devices through BDD-Cucumber and Open Source Resources
Presented by Jorge Asensio
© All rights reserved 1
Summary
1. Why Mobile Testing, DevOps and BDD • Background
• The Mobile DevOps challenge
• BDD and Mobile Projects
2. A bit of history • Our needs and how we tackled them
3. The chain of command • Workflow
• Tools, languages and frameworks
• Test run
4. Demo of UAT Automation • Live demo with four different devices
• Case Study example
••
© All rights reserved
User Conference on
Advanced Automated Testing 2
• Trends
• The Mobile DevOps challenge
• BDD and Mobile Projects
2. A bit of history • What we needed
3. The chain of command • Workflow
• Tool
4. Demo of UAT Automation • Live demo with 4 different devices
5. Outlook in the future • App usage will increase
• Pitfalls
• What remains as is and prospective improvements
1- Why Mobile Testing, DevOps and BDD
© All rights reserved
User Conference on
Advanced Automated Testing 3
Background
• Apps are leading the market.
• Many different devices with different OS type/Versions.
• Quick releases: automation solution.
• High demand for mobile testing in DevOps environments.
© All rights reserved
User Conference on
Advanced Automated Testing 4
Source: ComScore Media
Background
• Apps are leading the market.
• Many different devices with different OS type/Versions.
• Quick releases: automation solution.
• High demand for mobile testing in DevOps environments.
© All rights reserved
User Conference on
Advanced Automated Testing 5
Source: ComScore Media
The Mobile DevOps challenge
• Market fragmentation.
• Different tooling than Desktop DevOps.
• Speeding-up DevOps activities: cloud/local/3rd parties.
• No continuous deployments.
• No rollback in releases.
© All rights reserved
User Conference on
Advanced Automated Testing 6
The Mobile DevOps challenge (2)
• Scalable automation.
• Continuous Everything against real environments.
• Poor app instrumentation.
• Back-end changes: services may not serve mobile.
© All rights reserved
User Conference on
Advanced Automated Testing 7
BDD and mobile projects
• Mobile testing is particularly user XP-oriented.
• Gherkin standardizes documentation.
• BDD enhances interdepartmental collaboration.
© All rights reserved
User Conference on
Advanced Automated Testing 8
2- A bit of history
© All rights reserved
User Conference on
Advanced Automated Testing 9
Our needs and how we tackled them
© All rights reserved
User Conference on
Advanced Automated Testing 10
Requirement\Issue Solution
Real environments. Real devices, self-hosted solution.
Time, costs, customizable tools and frameworks. Free open-source tools, self-hosted CI solution.
Privacy and security of the data. Self-hosted solution.
Portability. Wheeled structure.
Improve \ modify framework. Scalable code (e.g. Page Object Model techniques).
Current commercial tools not satisfying. Customisable open-source tools.
Dead batteries. USB-powered on hub machine.
© All rights reserved
User Conference on
Advanced Automated Testing 11
3- The chain of command
© All rights reserved
User Conference on
Advanced Automated Testing 12
Workflow • Test project triggered from CI.
• Test automation run on devices connected to the slave machine.
• Unified report delivered.
© All rights reserved
User Conference on
Advanced Automated Testing 13
Tools, languages and frameworks
© All rights reserved
User Conference on
Advanced Automated Testing 14
Test run
© All rights reserved
User Conference on
Advanced Automated Testing 15
4- Demo of UAT Automation
© All rights reserved
User Conference on
Advanced Automated Testing 16
Live demo with 4 different devices
© All rights reserved
User Conference on
Advanced Automated Testing 17
Samsung A5 under
Android 7.0
iPhone 7 under iOS 11.2.4
Sony Xperia XZ1 under Android
8.0
HTC U11 under Android 7.1.1
• Threads run in parallel.
• Unified reports.
5- Outlook in the future
© All rights reserved
User Conference on
Advanced Automated Testing 18
App usage will increase • App download estimation: over 79% increase in the next 4 years
• Market points towards mobile app direction
© All rights reserved
User Conference on
Advanced Automated Testing 19
Source: TechCrunch
Pitfalls • SDKs and tool versions not aligned with OS.
• Test environment restrictions.
• iOS device driving requirements.
• iOS parallel testing is tricky.
• App code is not test-friendly.
• Powerful hub machines.
• Appium API deprecations.
• Different Android implementations.
• Steep learning curve.
© All rights reserved
User Conference on
Advanced Automated Testing 20
What remains as is and prospective improvements • Solution remains self-hosted and using Gherkin.
• Performance tests: JMeter/Locust: • Load/Stress Testing: web and webservices.
• Measure global performance in terms of time.
• Security tests: OWASP Zed Attack Proxy/ LinkedIn QARK.
• Improved test reports.
© All rights reserved
User Conference on
Advanced Automated Testing 21
Thank you for your attention!
© All rights reserved
User Conference on
Advanced Automated Testing 22