Post on 08-Apr-2017
Test ButlerReliable Android Testing, at Your Service
Drew Hannay (@drewhannay)
Android Testing @ LinkedIn● ~5000 instrumentation tests
● 2 runs per commit (pre-commit + post-commit)
● ~50 commits per day
● ~100 nightly runs to detect flaky tests
● 5000 * ((2 * 50) + 100) =
~1 million tests run per day
● Emulator flakiness
○ Animations can cause Espresso flakiness
○ CPU can randomly go to sleep
○ WiFi can randomly disconnect
○ Lock screen can appear and block test runs
○ Other apps can crash/ANR in the background and
block test runs
● Most of these happen < 1% of the time
○ But with 1 million tests run per day, that’s not
insignificant
● Tests shouldn’t fail due to issues with the
testing infrastructure
Testing Environment
Controlling the environment from tests● Turn WiFi on/off
● Change locales
● Set screen orientation
● Turn GPS on/off
Extra permissions● Many of these problems are fixable via reflection hacks
● Often requires special Android permissions
○ e.g. SET_ANIMATION_SCALE
● Painful at best, dangerous at worst
○ Add permissions and remember to remove before releasing (bad!)
○ Add permissions only to the debug manifest (better)
○ Either way gives your app different behavior in dev than in production (bad!)
● Some permissions are “development” permissions
○ These need to be granted with adb before they can be used
Introducing Test Butler● Two-part project: App & Library
● Library communicates with Test Butler app via RPC
● Test Butler app has permission to change
device settings
App apk Test apk
Test Butler Library
Test Butler apk Emulator
Test Butler: Permissions● Test Butler requests lots of permissions so your app doesn’t need to
● Signed with the system keystore for the AOSP emulators
● All signature-level permissions are granted automatically
○ No need to grant permissions with adb!
● This also means it can access permissions that can’t normally be used by
non-system apps
Test Butler: Custom IActivityController● Holds signature permission SET_ACTIVITY_WATCHER● Allows it to set a custom implementation of IActivityController
Test Butler: Custom IActivityController● No more system crash or ANR dialogs breaking tests!
● Caveat:
○ isUserAMonkey will always return true while using Test Butler
● Workaround:
○ Write a custom isUserAMonkey method that returns false
while running instrumentation tests
How do I use it?
How do I use it?
What’s next?● Add sample scripts to download & install the Test Butler app on emulators
● Add additional emulator controls (e.g. toggle airplane mode)
● Stabilize the rest of the Android test pipeline (stay tuned!)
Questions
References● Test Butler Github Repo
● Test Butler Announcement Blog
● Going Green: Cleaning up the Toxic Mobile Environment