VishalSRE-SET Automation Night #1

Reliable Mobile Test Automation

Software Engineer in Test

Case Study (Mobile Automation)

SWET (past)

Core Member

Vishal Banthia @vbanthia @vbanthia_

Some Previous Talks, Blogs & Tutorials

• Android E2E Testing at Mercari @ Android Test Night

• Android Device Farm at Mercari

• Appium Docker Demo

• STF Appium Example

• Test Automation Life Cycle

• How to achieve Reliable Automation

• Demo

Agenda

Test Automation Life Cycle

• Have you ever wrote any kind of UI Tests? Browser,

Mobile, Desktop, etc

• Do you think reliable mobile test automation is possible?

Questionnaire!

Test Automation Life Cycle ...

Product Manager

QA Team

Engineering Team

➢ Product manager comes up with new features

➢ Developers implement them

➢ QA team test new features and do regression testing

➢ Release

Development flow

Whats up?

Development flow ...

Product ManagerQA Team Engineering Team

Where is the

feature?

Code is done.

Waiting for QAToo many features

Too many test cases

Too many devices

No Time

I am dying ...

Retrospective Meeting ...

Product Manager

QA Team Engineering Team

We Need Automation!

Retrospective Result ...

Creation of Test Automation Team

Engineering Team

Test

AutomationTeamEngineering Team

Test Automation Team

• SET @google, @mercari

• SWET @DeNA

• SDET @microsoft

Journey of Automation Team

Start looking for automation tools

detox

Initial Learning

• Initial learning such as setup, writing simple scenario is not too difficult.

• With some programming experience one can start writing mobile e2e tests.

General Approach

Try to automate tests as much as possible from manual QA test cases list.

Writing tests is the easiest part in whole journey!

Difficult part is to actually use them in RELEASE flow!

What are the difficulties?

• Flaky tests

• Test execution environment

Because of these problems, people start losing

trust in it!

And this happens…

Sad Reality :(

So, what’s next!

Let’s go back to difficulties

• Flaky tests

• Test execution environment

Flaky Tests

● User Interface issues:○ UI is not working as per feature specification

○ Test code is not handling UI automation logic properly for scenarios

involving scroll, swipe etc and test is flaky

● Backend issues:○ Backend is temporary down

○ API is not working as per specification

When can a test fail?

When can a test fail? ...

● 3rd party service issues:○ Service is temporary down and error handling is not being done properly

in test automation logic.

● Hardware issues:○ Test is failing on specific device because of hardware limitation such as

Memory, Processor.

○ Device wifi is unstable.

Problem is not the flaky test!

Problem is we don’t know what is happening

during test execution

So, what’s solution?

Record Everything!

Record ...

• Screenshots• Screenrecord (Video)• ADB logs• Timestamp of each UI action and test assertion• Test script logs• Appium logs

And put them in a nice report!

rspec_html_reporter

• https://github.com/vbanthia/rspec_html_reporter

• Sample Report

• Android Example

http://tech.mercari.com/entry/2017/12/12/103741

rspec_html_reporter ...

• Extension of RSpec HTML Formatter with feature such as:• Pretty report• Can embed metadata such as screenshot and screenrecord

• ADB Logcat & other metadata are WIP.

This report will not solve flaky test itself, but it will help in

distinguishing between a bug and flaky test!

Flaky test solution

Let’s go back to difficulties

• Flaky tests

• Test execution environment

Test Execution Environment

Test Execution Environment

In general, try to use Cloud Services as much as possible. This provides quick feedback and they are easily scalable.

And anyone should be able to execute test cases anytime easily

Android E2E Testing at Mercari

Android Device Farm at Mercari

Mercari Android CI (Tools & Service)

• Bitrise: To build application• Google Cloud Storage: To store built apks• Google Container Registry: To store docker images• OpenSTF: For Android Device Farm• CircleCI 2.0: Build docker image & run tests in parallel containers• CircleCI Artifacts: To store test reports• AWS Lambda: To trigger nightly builds• Slack: Notification & manual triggers

Demo Video

• Mobile test automation is challenging but not impossible.

• With correct approach and planning it is possible to create

reliable test automation and actually improve product

quality and release efficiency

Conclusion

Thank you for coming!