VishalSRE-SET Automation Night #1
Reliable Mobile Test Automation
Software Engineer in Test
Case Study (Mobile Automation)
SWET (past)
Core Member
Vishal Banthia @vbanthia @vbanthia_
Some Previous Talks, Blogs & Tutorials
• Android E2E Testing at Mercari @ Android Test Night
• Android Device Farm at Mercari
• Appium Docker Demo
• STF Appium Example
• Test Automation Life Cycle
• How to achieve Reliable Automation
• Demo
Agenda
Test Automation Life Cycle
• Have you ever wrote any kind of UI Tests? Browser,
Mobile, Desktop, etc
• Do you think reliable mobile test automation is possible?
Questionnaire!
Test Automation Life Cycle ...
Product Manager
QA Team
Engineering Team
➢ Product manager comes up with new features
➢ Developers implement them
➢ QA team test new features and do regression testing
➢ Release
Development flow
Whats up?
Development flow ...
Product ManagerQA Team Engineering Team
Where is the
feature?
Code is done.
Waiting for QAToo many features
Too many test cases
Too many devices
No Time
I am dying ...
Retrospective Meeting ...
Product Manager
QA Team Engineering Team
We Need Automation!
Retrospective Result ...
Creation of Test Automation Team
Engineering Team
Test
AutomationTeamEngineering Team
Test Automation Team
• SET @google, @mercari
• SWET @DeNA
• SDET @microsoft
Journey of Automation Team
Start looking for automation tools
detox
Initial Learning
• Initial learning such as setup, writing simple scenario is not too difficult.
• With some programming experience one can start writing mobile e2e tests.
General Approach
Try to automate tests as much as possible from manual QA test cases list.
Writing tests is the easiest part in whole journey!
Difficult part is to actually use them in RELEASE flow!
What are the difficulties?
• Flaky tests
• Test execution environment
Because of these problems, people start losing
trust in it!
And this happens…
Sad Reality :(
So, what’s next!
Let’s go back to difficulties
• Flaky tests
• Test execution environment
Flaky Tests
● User Interface issues:○ UI is not working as per feature specification
○ Test code is not handling UI automation logic properly for scenarios
involving scroll, swipe etc and test is flaky
● Backend issues:○ Backend is temporary down
○ API is not working as per specification
When can a test fail?
When can a test fail? ...
● 3rd party service issues:○ Service is temporary down and error handling is not being done properly
in test automation logic.
● Hardware issues:○ Test is failing on specific device because of hardware limitation such as
Memory, Processor.
○ Device wifi is unstable.
Problem is not the flaky test!
Problem is we don’t know what is happening
during test execution
So, what’s solution?
Record Everything!
Record ...
• Screenshots• Screenrecord (Video)• ADB logs• Timestamp of each UI action and test assertion• Test script logs• Appium logs
And put them in a nice report!
rspec_html_reporter
• https://github.com/vbanthia/rspec_html_reporter
• Sample Report
• Android Example
http://tech.mercari.com/entry/2017/12/12/103741
rspec_html_reporter ...
• Extension of RSpec HTML Formatter with feature such as:• Pretty report• Can embed metadata such as screenshot and screenrecord
• ADB Logcat & other metadata are WIP.
This report will not solve flaky test itself, but it will help in
distinguishing between a bug and flaky test!
Flaky test solution
Let’s go back to difficulties
• Flaky tests
• Test execution environment
Test Execution Environment
Test Execution Environment
In general, try to use Cloud Services as much as possible. This provides quick feedback and they are easily scalable.
And anyone should be able to execute test cases anytime easily
Android E2E Testing at Mercari
Android Device Farm at Mercari
Mercari Android CI (Tools & Service)
• Bitrise: To build application• Google Cloud Storage: To store built apks• Google Container Registry: To store docker images• OpenSTF: For Android Device Farm• CircleCI 2.0: Build docker image & run tests in parallel containers• CircleCI Artifacts: To store test reports• AWS Lambda: To trigger nightly builds• Slack: Notification & manual triggers
Demo Video
• Mobile test automation is challenging but not impossible.
• With correct approach and planning it is possible to create
reliable test automation and actually improve product
quality and release efficiency
Conclusion
Thank you for coming!
Top Related