The Mobile Grid - Getting Started Android & iOS

Kiev, Ukraine - 26 Feb 2016

How will you test all of these?!The Challenge!

And each one of these cases?

• Validate every locale / languages your app supports.

• Validate every OS version your app supports.

• Validate different resolutions.

• Validate different screen sizes (phones, tablets, phablets etc..).

• Validate different device manufacturers.

You could try the single test approach!Not recommend…

Or the multi-threaded approach!Also, not recommend…

Though, this guy doesn’t seem to mind.Still not recommend, however…

Hire an army of zombie testers?!Nah, definitely not recommended!!!

So what options do we have?!

• Test Automation of course!

• But how, why, and where???

Run Tests Single threaded?Still, not recommend…

Parallelization!

Selenium Grid + Appium

BFFS 4EVER

Test On The Cloud• The Pros:

• It’s the present and the future of automated testing.

• Ease of setup and integration.

• Save on costs of buying mobile devices or machines.

• Some test services provide you with excellent analytics and reporting.

• Some test services integrate well with cloud CI services. e.g. Travis CI.

• You expose your app to a much larger range of devices and OS’s.

• The Cons:

• It can be difficult sometimes to narrow down issues.

• Network problems.

• Latency issues.

• Outages can occur.

• Bugs in their software.

• Acquired by another company and then shutdown.

• Costs can become significant when running CI on every commit.

Cloud Test Services• Sauce Labs

• Test Object

• Amazon Test Cloud (was Appthwack)

• Google Test Lab (was Appurify)

• Perfecto Mobile

• Testdroid

• TouchTest

• Keynote (was Device Anywhere)

Just to name a few…

The Mobile Grid• The Pros:

• You can run tests an unlimited amount without worrying about costs.

• You’re not handcuffed to a cloud test services 24/7 availability.

• You have all the critical test data local for debugging.

• Have full access to every connected device via adb or instruments.

• No latency issues.

• You don't expose your app or sensitive data to the cloud.

• The Cons:

• Can be tough finding documentation or good examples specific to your bindings.

• Takes a bit of time to setup and smooth things out.

• Require some degree of ops knowledge.

• You might not be able to expose your app to a large range of devices and os’s.

• Build and maintain the entire setup (software & hardware).

• Devices can be a real pain! (I’ll cover this later.)

What I’ll demo…• Getting all of the connected devices programmatically.

• Launching Grid hub & Appium nodes programmatically.

• Capturing screenshots, logs, video and logcat for reporting.

• Running single threaded.

• Running tests distributed.

• Running tests in parallel.

• Leveraging cloud test services such as Sauce Labs.

• Reduce your runtime costs by only executing selected scenarios.

• Generating an Allure report with all above metadata attached.

• The grid setup at Wunderlist.

• and lets briefly look at the core code…

Get Connected Device Data

• We retrieve connected device data and insert into an environment variable. • The “thread” value will then match our “TEST_NUMBER” env var for

parallelization. TEST_NUMBER is created from the parallel gem.

• Capture device data for every connected device.

• At a minimum, you need the UDID to connect appium to a grid node.

• However, it is important to know the OS, and SDK versions too.

Node Config Method

• Programmatically create your node based on the connected devices.

Generated Node Config

Appium Server Setup

• Method to setup appium server and pass in any arguments. • The wonderful Appium developers baked a grid node into the

appium server.

Launch Hub & Appium Server Methods

• We set a “THREAD” variable. This will tell our parallel library how many processes it should run.

• Launch the nodes in parallel so all start at the same time.

ADB Methods (Video and Logcat)

• Video records up to 180 seconds for real devices. • Capture logcat data.

Spec Helper Methods

• Parse the DEVICES env variable and match it’s thread to the running process, then initialize appium.

Test Setup and Teardown

• Set the UDID environment variable for your tests.

• Before each test, start video and logcat.

• After each test, stop video and logcat.

• Only attach output to report on failures!

Test Example

• Here is where the UDID environment variable is used. This will let us know which device ran a specific test.

Android Setup

• Download every SDK version your app supports. • Install Intel HAXM Accelerator!

Create Android Emulators

• Create an emulator for each sdk you installed.

Single Thread Test Example

• As you can see, this example was pretty slow. 48 seconds in total.

• Distributed means we spread each test to an available node. e.g. device.

• This example only has two tests (specs). So in theory we only need two devices, with each running a test.

• In this example we have one emulator (left) and one real device (right).

• Our runtime goes from 48 seconds single-threaded to 27 seconds multi-threaded.

Distributed Test Example

Allure Report Distributed

Parallel Test Example• Parallel in this case means we

run same tests on every available node. e.g. device.

• This example has two tests (specs). All tests will run on all devices/emulators.

• We have five emulators, each running a different SDK version, and one real device (bottom right).

• By testing on all SDK versions, this gives us a warm fuzzy feeling our app works correctly on all versions.

• As you saw, emulator-5554 (GRID5) crashed. Lets look at the report!

Allure Report Parallel

Run On The Cloud

• In the example tests theres only one spec tagged with “sauce”. If you we had more, we could run in parallel for faster test execution in Sauce Labs. e.g rake android[dist,sauce].

Allure Report Sauce Labs

• If needed, you can also download the test assets and attach to the allure. Example: https://github.com/isonic1/appium-mobile-grid/blob/master/common/dowload_sauce_assets.rb

iOS Setup

• Enable Developer Mode on your devices.

• Pair your devices to your machine.

• Enable UI Automation in the Developer menu.

• Ideally, connect a device running each os version your app supports. iOS is generally pretty good at backwards compatibility of os versions.

Real devices…

Parallel iOS

iOS Report

Wunderlist Mobile Grid

• Mac Pro - 3.5 GHz 6-Core - 32 GB Ram.

• OSX Server with performance mode enabled.

• Two Anker 9 Port USB Hubs.

• Energenie programmable power strip. (Reboots USB hubs, router, Mac)

• Blink(1) programmable usb light to display CI status.

• Jenkins - builds and runs smoke tests on every commit.

CI Machine Specs

That is great but reporting is key!

Challenges• Adb disconnects.

• I programmatically rebooting USB hubs before every test run. This reconnects adb.

• WiFi issues.

• I toggle airplane mode on/off prior to every test run.

• Other Possible Solutions:

• Reverse USB tethering. (requires rooting.)

• Using multiple routers on different broadcasting channels.

• Alternatives to real devices:

• Android Studio emulators.

• Cannot video record. *I will open a source a solution to this.

• A big plus IMO is that you have access your apps files (database) without rooting.

• Can use up a significant amount of memory depending on specs and the number running.

• Using Genymotion emulators. It’s the closest vm software to a real device.

• Has video recording but no API or CLI to start the recording. :/

• Some devices just behave erratically. (random reboots, adb connection issues, wonky text output, battery issues).

Things to be excited about!• A replacement for Apple's xcodebuild that makes it easier to build and test iOS or OSX apps.

• https://github.com/facebook/xctool

• WebDriverAgent is a WebDriver server for iOS that runs inside the Simulator and is written entirely in Objective-C. Can run multiple iOS simulators in parallel.

• https://github.com/facebook/webdriveragent

• Run iOS tests on multiple simulators in parallel at the same time.

• https://github.com/plu/parallel_ios_tests

• Appium iOS driver, backed by Apple XCUITest.

• https://github.com/appium/appium-xcuitest-driver

• Remote control all your Smartphone devices From the comfort of your browser.

• http://openstf.io/

Lastly, some thank you’s

• Thank you to the Selenium open source developers and the community whom make WebDriver happen.

• Also Dan Cuellar, the creator of Appium, whom is here at the conference with us.

• If you see Dan, please give him a big bearhug! He’ll love it!

• Without Dan, people like him, and all the open source developers we would never have these amazing tools to help do our jobs more efficiently, and for free.

Thank You!

• email: justin.ison@gmail.com

• twitter: @isonic1

• https://github.com/isonic1/appium-mobile-grid

• Android Distributed

• https://www.dropbox.com/s/eidrtd71dbmv7uh/android-distributed.mov?dl=0

• https://www.dropbox.com/s/gildgq9oossdp4k/android-dist-report.mov?dl=0

• Android Parallel

• https://www.dropbox.com/s/fjla64pggg3cbr6/android-parallel.mov?dl=0

• https://www.dropbox.com/s/14ow5q2joikne1s/android-parallel-report.mov?dl=0

• Android Sauce Labs

• https://www.dropbox.com/s/n8gx9lsoxrn77te/android-saucelabs.mov?dl=0

• https://www.dropbox.com/s/4umdim23cvheevj/android-saucelabs-report.mov?dl=0

• iOS Parallel

• https://www.dropbox.com/s/161azld3lq0lc2f/ios_parallel.mov?dl=0

• https://www.dropbox.com/s/pxlzxx2ljl6q1wd/ios_allure_report.mov?dl=0

• Wunderlist Mobile Grid

• https://www.dropbox.com/s/y5xsmnxfldguzr0/grid-trimmed.mov?dl=0

• https://www.dropbox.com/s/675nonhixzq0ct7/emulators.mov?dl=0

• https://www.dropbox.com/s/a8g0erkabpa5zny/signup_failure.mov?dl=0

Slide Videos