Post on 22-Jan-2018
Mobile Testing at SoundCloud
Brigit Lyonsbrigit.lyons@soundcloud.com
MBLTDev 2015
title, date, 01 of 10
● Product engineers write unit tests
● Product engineers write automated UI tests for story’s
acceptance criteria
● Team performs manual QA check for every story before
it’s “Done”
● Automated tests run after every push to master/release
● Manual regression testing done before every production
release
General Approach
title, date, 01 of 10
● Provide tools and services that enable product engineers
to build and release high quality software efficiently
● Understand team’s pain points and provide solutions
Software Engineer in Test
title, date, 01 of 10
Continuous Integration
title, date, 01 of 10
UI Tests
title, date, 01 of 10
Models a page in application that tests interact with in order to:● Reduce duplicated code● Limit code refactoring after UI changes● Ensure tests only care about the “services” they interact with and not
implementation details
The only thing that has a deep knowledge of the structure of the page (or parts of a page)
PageObject
title, date, 01 of 10
Example
title, date, 01 of 10
● Parallelization vs Fragmentation● 34 Nexus 5 + 2 emulators● 22 Mac Minis 44 iOS simulators
UI Tests Are Run on Devices
title, date, 01 of 10
Why Parallelize?
title, date, 01 of 10
1 hour 42 minutes 19 seconds
title, date, 01 of 10
2 hours 1 minute 23 seconds
title, date, 01 of 10
title, date, 01 of 10
● Ruby gems● Android: Rubotium https://github.com/ssmiech/rubotium● iOS: Distributed Cucumber and Cmdr
Custom Test Runners
title, date, 01 of 10
● Given application and test packages, create a list of tests● Determine all available devices● Run a test on an available device, repeat
Custom Test Runners
title, date, 01 of 10
10 minutes 4 seconds
title, date, 01 of 10
8 minutes 11 seconds
title, date, 01 of 10
● Configuration options:○ run on a specific type of device○ run tests with a set of specified annotations○ run tests without a set of specified annotations
● Better debugging:○ HTML reports○ For all failed tests:
■ Stack trace■ Full logs■ Screenshots -> videos
Custom Test Runner Benefits
title, date, 01 of 10
● No built-in scheduling or resource management
○ Must rely on Jenkins
● Only CI jobs can run tests in this environment
● We have a few configurations that run on CI (master,
release, development branches, etc)
● CI jobs that use same resources for UI tests must block
one another, leading to inefficient wait time
Custom Test Runner Problems
title, date, 01 of 10
● Run configurations are set within Jenkins job
● New configurations or one-off configurations require
setting up a new Jenkins job
● Pre-commit testing is not feasible
Custom Test Runner Problems
title, date, 01 of 10
● Test running order cannot be specified
● Test results are only collected from devices after all tests
have been completed
● Failed tests are retried a second time, but this is not
visible in the reports
Custom Test Runner Problems
title, date, 01 of 10
Mobile Test Runner Service!
title, date, 01 of 10
● Android only for now, but iOS support coming soon
● Mostly written in Scala
● Communicates with devices through ADB using ddmlib
● REST API for clients
Mobile Test Runner Service
title, date, 01 of 10
Mobile Test Runner Service
title, date, 01 of 10
● Resource management is controlled by the service itself
○ Don’t have to rely on Jenkins
● Test runs can be executed in parallel
● Resources are used efficiently; device workers keep
executing tests as long as there are tests in the queue
● Developers can test pre-commit changes on same
environment used by CI
Results
title, date, 01 of 10
● Test queue ordering is also controlled by the service
itself, so we can run the most important tests first (test
age, flakiness, etc)
Results
title, date, 01 of 10
● Test results (stack trace, full log, and videos for failed tests) are available as soon as each individual test finishes
● Retries for failed tests clearly indicated in HTML report● Data about devices, tests, and results are stored in a DB,
making statistical analysis easier
Results
title, date, 01 of 10
title, date, 01 of 10
● Test runs are easily configured via REST API
Results
title, date, 01 of 10
● Built a Java client library
● Integrate into Gradle tasks so test configuration for CI is
part of the Android codebase
But API Endpoints can be annoying to remember...
title, date, 01 of 10
● Not easily parameterized
● Names can be annoying to to remember
● Command line is not the most user friendly interface
And Gradle tasks have problems, too...
title, date, 01 of 10
● Developers can run tests for local changes on CI
environment right from Android Studio
● Right click to run specific test method, test class, test
package
● Results are reported natively, including links to videos
and logs for failed tests
Solution: IntelliJ Plugin
title, date, 01 of 10
title, date, 01 of 10
Thanks!
Brigit Lyons
brigit.lyons@soundcloud.com