Post on 16-Jul-2015
We make mobile developers’ lives easier.
Ville-Veikko Helppi Technical Product Manager
ville-veikko.helppi@bitbar.com +358400949001
16 April 2015
Public Device Cloud on-demand devices
(multitenant)
Mobile app testing on thousands of real
Android and iOS devices hosted by
Bitbar
Private Device Cloud Reserved Devices
Hosted by Bitbar in the US and/or Europe
Devices chosen by
and reserved only for the Customer
On-Premise Device Cloud
Automated mobile app testing devices
hosted by the customer, usually
30-500 devices
1 Product – 3 Deployment Options
Testdroid Cloud Testdroid Enterprise Testdroid PrivateCloud
iOS 8.0
iOS 8.0.2
iOS 8.1
iOS 8.1.1
iOS 8.2
iOS 8 ALL KitKat 4.4
KitKat 4.4.2
KitKat 4.4.3
KitKat 4.4.4 KitKat ALL Lollipop 5.0
Lollipop 5.0.1
Lollipop 5.0.2
Lollipop ALL
0 5 10 15 20 25 30 35 Failed test runs. Percentage (%).
How Many Devices is Enough?
~90% market coverage can
be achieved with
128 devices
~20% market coverage can
be achieved with
12 devices
US Market
25 Android devices
= ~2/3 market
Global Market
60 Android devices = ~1/2 market
OS versions
Chipsets CPU + GPU
Tens of OEMs
Memory Displays
(resolutions, physical hw)
OEM mods
Other hardware (connectivitycalibration)
Relation to other software
Where Test Automation Can Help
Correct behaviour across platforms and browsers Integration with web back-ends
Typically need to fully utilize HW (CPU+GPU) Resource (e.g. battery) consumption OpenGL ES 2/3
Functionality and usability
Screen
orientations, connectivity, user profiles
Robustness
Robustness and security!
Brand
Compliances,
verification with back-ends and data
Different Mobile 'App Verticals'
Why Real Devices Are Must-to-Have • Emulators cannot help you to test...
• User Experience • Usability • Hardware • Software • Infrastructure
0 % = the percentage of your app users that use emulator to run your app!
Manual vs. Automation
Smaller coverage, More money burnt & time wasted, Error-prone
Manual Automation Large
coverage, quickly
completed, Less money & time wasted, Exact results.
Different Ways to Automate Testing Automatic test exercisers Record and Playback Hand written test scripts
Benefits: Accurate, specific to your testing needs, plenty of options with frameworks, tools
Fast to create, accurate, not as sensitive to human-errors as hand-written tests, tools avail’ty
Fastest & extremely automated, excellent for smoke testing/quick testing, availability
Tradeoffs: Takes a lot of time, ties resources to write test cases/scripts, error-prone (humans)
Compelling Recorder+Playback tools available for only few test automation frameworks
Not accurate as real test cases
Categorizing the Testing • Functionality of the "game-play", features Functional Testing • Done always when new features /
regressions are included Regression Testing
• How game runs on different configurations Compatibility Testing
• Different languages, geo-focused materials Localization Testing • Endurance test to determine if system can
handle the load Soak Testing
• Measures the capacity of the system Stress Testing • Simplest form of performance testing,
measures how system handles loads Load Testing • Isolation of the environment (e.g. from
network) to see how game works Hermetic Testing
Feature-based testing
Performance testing
End-user testing
Family Tree of Android Test Automation Frameworks
JUnit
Android Instrumentation Framework
Robotium Espresso and Espresso v2
uiautomator
Appium
ExtSolo
Calabash
What Framework Works You The Best? Robotium uiautomator Espresso Appium Calabash
Android Yes Yes Yes Yes Yes
iOS No No No Yes Yes
Mobile web Yes (Android)
Limited to x.y clicks
No Yes (Android &
iOS)
Yes (Android)
Scripting Language
Java Java Java Almost any Ruby
Test creation tools
Testdroid Recorder
UI Automator viewer
Hierarchy Viewer
Appium.app CLI
Supported API levels
All 16 => 8, 10, 15- All All
Community Contributors Google Google Active Pretty quiet
Client Side Appium at Testdroid Cloud
Test Script
Test Case
Desired Capabilities
{ “device”: “Android”, “app”: “/Users/user/ApiDemos.apk” “app-package”: “com.example.android.apis” “app-activity”: “.ApiDemos” }
Test Script
Test Case
Desired Capabilities
WebDriver http://localhost:4723/wd/hub
Appium Server
4723
Device
Localhost
(DesiredCaps)
http://localhost:4723/wd/hub
*Testdroid Caps
http://appium.testdroid.com/wd/hub
(DesiredCaps)
Test Script
Test Case
Desired Capabilities
WebDriver
From Localhost to Testdroid
Client Side Execution Add Testdroid Desired
Caps to test script
{ “testdroid_username”: “user@domain.com”, “testdroid_password”: “p4s$w0rd”, “testdroid_project”: “My First Project”, “testdroid_testrun”: “Test 1”, “testdroid_device”: “iPad Mini 7.0.4 A1432”, “testdroid_app”: “http://domain.com/app_v1.ipa” . . “app”: “com.bitbar.testdroid.BitbarIOSSample” }
Get a Device Name
Go to cloud.testdroid.com
Client Side Execution
driver = webdriver.Remote("http://appium.testdroid.com/wd/hub", desired_caps);
Point the Webdriver to http://appium.testdroid.com/wd/hub
Add Testdroid Desired Caps to test script
Get a Device Name
Go to cloud.testdroid.com
Client Side Execution
Run the Test Script Get Results from Testdroid Cloud
Point the Webdriver to http://appium.testdroid.com/wd/hub
Add Testdroid Desired Caps to test script
Get a Device Name
Go to cloud.testdroid.com
Client Side Execution
Pull the Results from the Result URL
driver = webdriver.Remote("http://appium.testdroid.com/wd/hub", desired_caps);
Run the Test Script Get Results from Testdroid Cloud
Point the Webdriver to http://appium.testdroid.com/wd/hub
Add Testdroid Desired Caps to test script
Get a Device Name
Go to cloud.testdroid.com
Multiple Devices – Client Side python testscript.py –device <devicename> 1
python testscript.py –device <devicename> 2
python testscript.py –device <devicename> n
Test Script
Test Cases
Instigator Script
deviceArray= [ “iPad 4 6.0.1 A1458”, “iPad mini 7.0.4 A1432”, . . . “iPhone 4S 6.1.3 A1387”, ]
2 1
3
4
5
6
7
WebDriver Session Request @http://appium.testdroid.com/wd/hub
WebDriver Session Response
Testrun
Configure Project
Appium Ready
Wait for Device to Become Available
Device 1 Device 2 Device 3
Session Map
Proxy Device Cluster
Start Appium
Desired Capabilities, .apk / .ipa
SessionID
SessionID
Sessionid
Test Script
Appium Broker
Client Side – Behind the Scenes
Setup • Using real Android devices at Testdroid Cloud • Parallel test runs without a need to configure desired
capabilities • Device groups (= set of devices used for runs) can be
manually created and configured
File Structure • pom.xml (maven) • testdroid.properties (overwritten after submitted to Cloud) • run-test.sh (shell script for execution) • image files
Image Recognition • Resolution agnostic implementation • Identifies stretched and rotated images as well