Increase Your Mobile Test Automation ROI - AFourTech · Increase Your Mobile Test Automation ROI...

Increase Your Mobile Test Automation ROI

Make your Mobile Test Automation more mature and dependable. Take your framework to an all new level of robustness by executing your tests on real devices and at the same time increase your ROI

This white paper should help you in understanding the nuances of mobile test automation, its challenges and how to derive more from your test automation effort without having to compromise on cost or quality

About the Author Aniket Deshpande

Pune, India Master in Business Administration (Information Technology)

Aniket Deshpande is a Test Architect at AFour Technologies. He has around 10 years of experience in testing varied applications across different platforms including mobile applications and has formulated testing strategies of various mobile applications. As a Test Architect, he has led teams in diverse domains like Social Engineering, Health & Fitness, E-Commerce, etc. His primary interests lie in conceptualizing & implementing mobile automation frameworks and environments and setting up mobile device labs.

2

Introduction

Mobile Application testing can be a tricky proposition. With the plethora of devices,

operating systems and their versions, further compound the problem. As gatekeepers

of quality, we QAs cannot be satisfied with testing only on simulators and emulators. It

is always risky not to test on real devices for various reasons, above all being, end

users would not be using the application on simulators but on real world devices.

Hence, your mobile test automation should also sign-off the build on real devices.

Should you choose to run your automation on real devices, which devices would you

narrow down on, from the ever-growing list of mobile devices?

Using your short-listed devices, you would need to manage your automation testing

cycles across these devices cleverly to be able to derive more from your investment.

For planned automation initiatives, choosing the right tool is very important for

maximizing returns on your automation effort. You do not want your tool to throw you

ugly surprises later in your framework development efforts.

Integrating your automation runs with the build cycles gives you fantastic ROI as you

may have already realized with your normal web and desktop automation frameworks.

Setting up such a process on your mobile automation will not only fetch optimal

returns, but also make your automation a truly hands-off process.

This White Paper will give you insights into how we achieved increased ROI at minimal

cost while implementing a mobile automation framework for one of our clients. The

solution includes choosing a tool, automating the tests, setting up of a mobile device

lab for test execution and integrating the automation runs with the build process.

Introduction | Trends | Problem | Solution | Benefits | Enhancements | Conclusion | References

“It is always risky not to test on real devices”

“Which devices would you

choose?”

“Need to manage your automation testing cycles across these devices”

“Choosing the right tool is

very important”

“Integrating your automation runs with the build cycles”

3

Trends

More mobile apps are being downloaded than ever before and the number is

increasing yearly. The year 2013 has seen an increase of more than 50% in apps

being downloaded from the respective Stores1. By the end of year 2013, there

were more smartphone owners in the world than PC owners. One in every 5

people now owns some kind of a smartphone2.

This further goes on to show the demand for mobile apps is ever increasing and

there are more smartphone users than PC users. This would mean enterprises

would want to quickly reach out to users on the mobile platforms than traditional

websites approach and would want faster time-to-market, which in turn would

demand process innovations in QA cycles.

Businesses with mobile apps have a customer conversion rate of more than 5% as

against less than 1% for businesses with no mobile interface3.

Gartner says that, by 2015, around 80% of all application development activities

will be for smartphones & tablets. And having a mobile application, systems &

devices are “one of three top CIO priorities”4.

All this points to a huge upsurge in mobile application development.

Many of our customers have release cycles as small as 1 week. The reason being

the ease of submitting the app to the Store and application upgrade on the user’s

device.

As QAs we need to be ahead of the curve and make sure we are prepared to

handle such demanding timelines. More than technological challenges, there is a

need for upheaval in the QA process itself, which can help in making the process

sleeker and agile.


“One in every 5 people now own some kind of a smart phone”

“Enterprises want faster time-to-market”

Gartner: “By 2015, 80% of all application development activities will be for smartphones & tablets”

4


About the App:

The mobile application under test is a

Health & Fitness application, which is

freely downloadable from App Store (iOS)

and Play Store (Android). The application

has implemented WebViews for user

interactions and REST API calls for server

communication.

Problem

1. Tool Selection:

This, by far, was the most vital of all our

activities for implementing the solution.

We had to find a tool that we could use

across both iOS and Android platforms,

because using 2 different tools is just not

the right way to go. Among other things,

maintaining 2 separate code bases and

technologies would be an overhead later

in the automation cycles.

Also, we wanted a tool that would

seamlessly work with a CI server so that

we could integrate it with the

development build cycles, which would

ensure a truly hands-off automation

execution.

The tool should also have rich reporting.

We could trade-off this feature by

implementing our own reporting module,

but an in-built reporting engine is always

more preferable than a custom one.

And importantly, the tool should work

smoothly on real devices and not only on

emulators & simulators.

Reasons for running on real devices:

The end-user downloading the

application from the Store, will be

using the application on an actual

device and not on a simulator.

Features like GPS, for location

services, can be tested reliably on a

real device than on a simulator.

Simulators & emulators use the host

machine’s hardware and processor

architecture, which will be different

than the actual device.

RAM, Disc space, Battery usage, etc.

can cause performance issues on a

device but not on simulators.

We have had many instances on

other projects where we found a bug

on the device but not on the

simulator or vice-versa, which made

the test results unreliable.

Using a device, we could test

applications on cellular data

bandwidth as against wireless/LAN

in case of simulators, which is

obviously always reachable and

faster than cellular data.

Most importantly, for Android

devices, the operating systems

released by OEMs contain many

customizations on middleware and

drivers, which may impact the

application.

You cannot install other applications

on simulators, like browsers, or if

your application uses social media,

then dependency on the social app

needs to be tested. This cannot be

done on a simulator.

2. Mobile Device Lab:

The reasons mentioned above justified

setting up an in-house Mobile Device

Lab. We could have used the services of

a third-party device lab like

DeviceAnywhere, but it would mean

sharing data with third-party vendors.

Also, with such device labs, the devices

are not exclusive and are shared. You

always run the risk of a rogue application

sitting on the device listening to your

activities.

Since these shared devices are cleaned &

reset after use, running automation on it

would require you to install and create all

pre-requisites every time you want to

trigger an automation run.

Another issue was, if you find a bug on

that device and if later, the developer

wants to reproduce it, then the device

may no longer be available or would have

been reset.

3. Continuous Integration:

We are working with a team in diverse

locations.

Setting up a continuous integration server

would make the automation execution

truly hands-off. You have various options

of scheduling or triggering your

automation runs based on events that

happen in your build cycles. Moreover, it

mandates that your environments are

standardized and encourages the team to

automate the pre and post tasks as well.

This approach also helps in managing your

test cycles & reporting. If the triggers /

schedule is setup cleverly, you do not

have to wait for a QA to fire the runs. It

becomes a 24/7 system.

Stitching It All Together:

The above challenges are inter-related

with each one having an impact on the

other. The UI automation tool selection,

device lab design and CI tooling & strategy

all need to be tied up together to create a

tight harness.

Lets see how the solution was

implemented.

5


Solution

1. Tool Selection:

Looking at the different problems and the importance of choosing the right tool, a detailed PoC

(Proof of Concept) was carried out. The litmus test of the tool was automating 3 end-to-end test

cases. We identified these test cases based on the complexity of the flow and the various controls

encountered on the screen. For the PoC we considered, Calabash, Ranorex, Appium, Robotium &

Sikuli. We evaluated these tools on the following features:

Platform Support: Support for both iOS & Android was mandatory. Maintaining separate code

bases for both platforms will be an overhead in the long run. Since WebViews are used

throughout the application, application flows across both platforms are similar. We could

take advantage of this by writing a common code base for both platforms.

The above criterion allowed us to focus our attention on Calabash & Appium.

Tool Cost: Both Calabash & Appium are Free & Open Source tools. Other tools like EggPlant

were also on the list, but the cost was too high to be considered as an option.

Test Cases Automated: The ability of the tool to automate the 3 identified test cases. We

found that Appium was suspect when it came to automating WebViews, while Calabash was

quite stable. Also, Appium did not fully support iOS7 at the time of the PoC.

Ease of coding: Calabash uses Ruby+Gherkin, while Appium uses Java. It is easier to add new

test cases to Calabash due to Gherkin BDD (Behavior Driven Development) implementation,

but on the other hand, it is easier to find Java resources than Ruby ones. Hence, both scored

average in our rating.

Code reusability: Both support modular framework design hence code becomes highly

reusable. Sikuli and Ranorex have image processing engines for object identification. This

makes the code highly unreliable and unstructured.

Tool Support: Both are open source and have ample support on various forums. Upgrades for

new operating systems were also fairly quickly available.

Source Code Requirement: For iOS, Calabash adds its own library to the application source

code and then builds and deploys the application. This is a huge negative for Calabash if your

client is not comfortable sharing the source code. Appium requires just the UDIDs of the

Apple device be included in the build. Both however, do not require source code for Android

automation.

Reporting: Calabash has its own reporting mechanism while Appium uses TestNG. Both have

powerful reporting and Exception handling. They both conformed to our reporting

requirements (Error logging, screenshots, run summary, etc.)

Execution on real devices: Both Appium & Calabash can run automation on real devices. The

process of device detection & test execution for both was quite simple and easily

configurable. The execution speed was comparable to that of simulators.

CI integration: Both have strong command-line execution processes and can be easily

integrated into a CI server. We tried firing automation runs on a device for both Calabash &

Appium through a Jenkins job and both ran successfully.

Calabash Appium Others

Calabash Appium

Calabash Appium

Calabash Appium

Calabash Appium

Calabash Appium

Calabash Appium

Calabash Appium

Calabash Appium

Calabash Appium

6


Both Appium & Calabash scored relatively equally on our PoC. In the end, the inability of Appium to support WebViews & the client agreeing to share the iOS source code tilted the decision in Calabash’s favor.

You can find out more about the tool at http://calaba.sh

The Framework:

Next, we had to create an automation framework that will leverage the advantages of the Calabash tool, foremost being, able to create and maintain a common code base for both iOS and Android automation. Although, both applications load the same WebViews, the method of referencing objects for both are a little different. We created an object repository where we maintained the object reference of each object for both Android & iOS. It looks something like this:

Android Repository:

emailId=#email

password=#password

loginButton=#login

iOS Repository:

emailId=WebView css:'#email'

password=WebView css:'#password'

loginButton=WebView css:'#login'

In the code, we reference the object name:

def Login(email,passwd)

if elementExists ($ids[‘emailId’],30)

typeText($ids[‘emailId’], email)

end

if elementExists($ids[‘password’],30)

typeText($ids[‘password’], passwd)

end

if elementExists($ids[‘loginButton’],30)

touch($ids[‘loginButton’])

end

end

Exception handling is done in the above code to capture failures and log them.

Before execution begins, the respective Object Repository is automatically loaded based on the execution environment (iOS / Android).

Finally, the test cases are written in Gherkin (BDD):

Scenario: Login with valid credentials

Given I open the App and login with "[email protected]","spiderman"

And I touch Profile tab

Then I see that the screen changes to profile screen

And step definitions are written in Ruby:

Given /^I open the App and login with "([^\"]*)","([^\"]*)"$/ do |email, password|

homePage = Home.new

homePage.checkWelcomeScreen()

homePage.Login(email, password)

end

By designing such a framework, we were able to create & maintain a single code base for both

Android & iOS.

http://calaba.sh/

7


2. Mobile Device Lab:

To support the application testing we had to test on multiple devices. Testing on iPhones is quite

straightforward since their devices are standardized and common throughout the world. But when

it comes to Android there are a wide range of devices, resolutions & OEMs we need to consider.

To finalize the list of devices that we would use for testing, we sought statistics about the active

users of the application on both Stores. We found the top 5 devices & resolutions that had users

actively using the application. Apart from the 4 iPhones (4, 4s, 5 & 5s), we also procured the top 5

Android devices as per the statistics, to be part of our mobile lab. Collectively, they made up

around 80% of the customer base.

We created a separate physical space for these devices in our server room with dedicated power

supply and Wi-Fi support. By using services like VNC, we were able to remotely connect and

control these devices over LAN. We also setup a VPN for this lab for our client to connect to our

network and access these devices over VNC.

Now that the remote access problem was resolved, we had to solve the automation problem,

where we had to trigger automation runs remotely on these devices. Calabash had very good

support for this. We setup a separate Windows machine for the Android devices and a Mac Mini

for the iOS devices. Using Calabash’s command-line arguments, we could trigger automation runs

on specific devices using the device’s UDIDs and IP addresses.

The dedicated Mobile Device Lab

at AFour Technologies

8


3. Continuous Integration:

Having an automated job that will build and deploy applications on the mobile, trigger automation

runs, email notifications, etc. would really help in achieving a 24/7 service. You would not need a

person to trigger off this complicated set of steps and everything would be a hands-off process.

These are just some of the benefits that a CI server offers.

We decided to use Jenkins as our CI Server where we would create jobs, which automate the

entire run execution process.

Common automation execution steps would include:

Picking up the specified “.apk” file (for Android) from a common repository

Or, pickup a specific iOS code base from a repository

Build the code and/or deploy the application on the specified device

Pickup the automation code from the repository

Trigger the automation

Email the execution report to the stakeholders

We created various parameterized jobs in Jenkins and configured all the above steps in it. By

starting the job, all the above steps would be carried out and the final execution report is emailed

to the specified people.

Now, our client has integrated this into their build server. So, whenever new code is checked into

their object repository, the Jenkins jobs are triggered through their integration server using the

very useful Jenkins REST API calls. They can even specify the specific application build number

against which they wish to fire the automation and the rest is taken care by Jenkins which would

also notify them when the automation run finishes by emailing the execution report.

The Windows machine and the Mac PC are configured as Jenkins slaves and the jobs have been

configured to run from these machines.

Jenkins

9


Benefits

1. Common Code Base

By choosing Calabash that can be run

across both platforms (Android & iOS), we

reduced our coding & code maintenance

efforts by nearly 40%. And we will keep

reaping its rewards in the future as we

add more test cases or update existing

application flows.

The same automation engineer can write

scripts for both Android & iOS.

2. Automation Transparency

Now everyone in the chain knows exactly

what test cases are being executed and

the test data being used, because

Calabash mandates you to write your test

cases in Gherkin (Given-When-Then)

format which are easily readable and can

be understood by everyone involved in

the software development chain. Anyone

can easily change the test case and test

data or even add new automated test

cases to the test suite.

We, as QAs, have started writing all our

test cases in a standardized Gherkin

format instead of the old & traditional

test case writing format. Eventually, we

can just pick these tests and plug them

into the framework. Again, saving us

precious time & effort of re-writing

automation scripts.

3. Reporting

With Calabash’s rich HTML reporting

system with screenshots and activity logs,

our automation engineers can spend their

valuable time in writing quality code

rather than worrying about how should

the error be reported. On encountering

an exception, Calabash will automatically

report the error with screenshots and

activity logs.

4. Device Security

Since these devices are physically located

in our secure server room, physical access

is restricted, making the lab secure with

specific access rights to authorized

personnel, thus making the devices

immune to unscrupulous software or

malware installations. Remote access to

the devices is audited and activity log is

maintained.

Data and software being accessed on the

devices are limited to the device making

client data & other collaterals secure.

5. Dedicated Access

All the stakeholders in the software

development cycle have dedicated 24/7

access to the devices making bug

reproduction and testing more

dependable and convenient. We seldom

require resetting the devices since they

are dedicated and we can securely install

third-party application (like Social Apps)

to check compatibility.

6. Testing on Real Devices

Not testing on real devices is always a

risky proposition as already discussed in

this paper. Our testing results are more

reliable since we test on devices. We can

report issues in the software encountered

for low bandwidth, security, performance,

etc. We can also reliably sign-off on builds

with the knowledge that builds are not

misbehaving when installed on specific

OEM operating systems.

7. Continuous Testing

Since the Jenkins environment is

integrated with the client’s build &

integration environment, automation runs

on real devices can be kicked off

immediately without any manual

intervention or waiting for the QA team to

start the process.

Abstracting the mobile environment layer

under Jenkins provided the client with a

more user-friendly access to the mobile

lab.

By automating the entire pre & post steps

required for automation runs, we have

been able to save 1 full resource day

spent in running automation daily on all 9

devices.

8. Scalability & Standardization

The Mobile Device Lab mandates a

particular process for adding devices to

the lab. Such a process standardizes the

devices and their inclusion into the device

lab. It also makes the lab easily scalable.

Common processes can be set for tasks

such as resetting devices, installing /

uninstalling software on the devices,

software or OS upgrade, etc.

We can now quickly scale up our mobile

lab by just duplicating the VMs & Jenkins

jobs and they become immediately

available for automation execution.

Parallel execution of test suites can be

easily configured and fired under Jenkins.

Although parallel execution-ability of the

test suite depends largely on the design of

the framework and test data, if done

correctly, it helps in drastically reducing

the time required to get the test results.

10


Future Planned Enhancements

Code Coverage Statistics

Using tools, we can find the code coverage statistics of an automation run. These statistics can

be a useful indicator for the impact of the automation run. Using tools like Emma for Android

and internal XCode configurations in iOS, we can get a detailed report of the code that was

traversed during an automation run. We can then tweak or add test scenarios to increase this

coverage.

Suite Execution Controller

A controller that will execute customized test suites where the user should be able to provide

details of the tests to run and the Jenkins job should execute only the selected test cases.

Enhanced Reporting Dashboard

Create a dashboard where the user can view current and past execution reports. Implement

live reporting where the user can see the results of the test being executed. Create a

mechanism where the user can select test cases and choose to execute them from the

Reporting Dashboard itself, which will internally call the Jenkins API for job execution.

Page Load Statistics

Creating a simple logging mechanism, which will record the page load time during automation

execution. This should provide us valuable information over various builds about the

degradation of page load times among other things. This should also help us in finding

performance bottlenecks.

11


Conclusion

The customer saw great benefit in the solution that we provided them. They could now integrate

their development builds with testing and get an immediate view on the health of the build to take

a go/no-go decision.

In the end, it’s the entire turnkey solution that counts and that is what we have been successful in

achieving. It was an interesting challenge to execute since there was a merging of multiple

technologies and platforms. Watching the automation get executed on devices, which gets

automatically triggered from a remote build completion was pure pleasure.

There were many small challenges in the way and, for us, to work around those challenges made

this a memorable solution to work on.

The key to realizing the various benefits was to understand the pain-points of the customer and

the QA team and then, design a foolproof solution that would satisfy all the needs. This solution

has been able to address many such problems.

12


References

1 - http://www.gartner.com/newsroom/id/2592315 2 - http://www.businessinsider.in/One-In-Every-5-People-In-The-World-Own-A-Smartphone-One-In-Every-17-Own-A-Tablet-CHART/articleshow/24353354.cms 3 - https://econsultancy.com/blog/62656-45-of-businesses-still-don-t-have-a-mobile-site-or-app-report 4 - http://www.cioinsight.com/c/a/Enterprise-Apps/Mobile-Application-Development-A-Top-CIO-Priority-895960/

http://www.gartner.com/newsroom/id/2592315

http://www.businessinsider.in/One-In-Every-5-People-In-The-World-Own-A-Smartphone-One-In-Every-17-Own-A-Tablet-CHART/articleshow/24353354.cms

https://econsultancy.com/blog/62656-45-of-businesses-still-don-t-have-a-mobile-site-or-app-report

http://www.cioinsight.com/c/a/Enterprise-Apps/Mobile-Application-Development-A-Top-CIO-Priority-895960/

Increase Your Mobile Test Automation ROI - AFourTech · Increase Your Mobile Test Automation ROI...

Documents

Transcript of Increase Your Mobile Test Automation ROI - AFourTech · Increase Your Mobile Test Automation ROI...