Amalgamation of BDD, parallel execution and mobile automation

Amalgamation of Behavior driven development (BDD), parallel execution and mobile automation

1

- Jigesh Shah- Mradul Kapoor

2 Amalgamation of Behavior driven development, parallel execution and mobile automation All rights reserved.

Author’s profile

Mradul Kapoor

Mradul Kapoor is working in one of the leading consulting firm based in Mumbai, India. His experience, knowledge and skills include Business Analysis, Automation and Functional Testing. Mradul has worked on various testing tools like QTP, Selenium, Oracle Oats, Coded UI, SpecFlow, SpecLog, Appium, Microsoft team system and various test management tools. He has over 7.5 years of experience in US Healthcare, Insurance, eLearning, Life Science and Human and Health Services

Jigesh Shah

Jigesh Shah is working in one of the leading consulting firm based in Mumbai, India. His experience, knowledge and skills include Business Automation and Functional Testing. Jigesh a Certified Java Programmer has worked on various testing tools like QTP, Selenium, QC, Rally, JIRA , SpecFlow, SpecLog, Appium, Microsoft team system and various test management tools. He has over 7.5 years of experience in US Healthcare, Insurance, and eLearning


Preface

With the rise of Mobile first approaches, a substantial traffic has now shifted from the desktops to the Mobile phones. To ensure quality in mobile apps, Testing needs to be continuous and inclusive. This makes Mobile Test Automation a formidable business driver. Along with that there needs to be strong collaboration between developers, testers and non-technical or business participants in a software project to maintain application consistency across multiple platforms.

The intent of this presentation will be to provide a comprehensive mobile automation testing strategy using Behavior Driven Development, optimized selection of target devices, and an effective combination of manual and automated testing; use of correct tools to cover appropriate functional testing essential for getting your mobile applications to market on time and within budget.

This presentation will also discuss on how the scenarios which are derived from the requirement can be automated using BDD approach across Web and Mobile. Also, it will highlight on how parallel execution can be achieved using BDD & Mobile Automation.

Target audience : Senior Leadership, Head Testing Practice, Head Quality Assurance, Delivery Head, Test Managers, Test Leads and Test Professionals


Current market analysis and trends

With the fantastic growth of mobile computing platforms there has been a dramatic increase in the value of mobile applications for most companies.One of the biggest challenges that one faces when developing a mobile application is how to test it.

Period Samsung Apple Lenovo Huawei Others

Q1 2015 24.60% 18.30% 5.60% 5.20% 46.30%Q1 2014 30.70% 15.20% 6.60% 4.70% 42.80%Q1 2013 31.50% 16.90% 4.70% 4.20% 42.70%Q1 2012 28.90% 22.90% 5.00% 3.40% 39.80%

Period Android iOSWindows

PhoneBlackBerry

OSOthers

Q1 2015 78.00% 18.30% 2.70% 0.30% 0.70%Q1 2014 81.20% 15.20% 2.50% 0.50% 0.70%Q1 2013 75.50% 16.90% 3.20% 2.90% 1.50%Q1 2012 59.20% 22.90% 2.00% 6.30% 9.50%

Source : IDC May 2015

Issues and challenges


What are the testing challenges faced by today’s organizations while automating mobile and web automation?

Mobile automation challenges

Outcome

Parallel execution

Rapid rollout schedule• Rapid rollouts of bug fixes

• Constant expectation to meet Application quality

Device fragmentation• Issues with multiple Devices and

Browsers

• Object Identification support issues

Mobile lab• Difficult to test in the field

• No VPN on old devices

• How to push updates and test them?

Mob

ile a

utom

atio

nB

ehavior driven development

Communication issues • Issues with understanding

the requirements

• Each team has developed their own understanding mechanism, making communication ineffective

Testing challenges• Identify devices to test on

to support your user base

• Identify device limitations and characteristics

Right tools & approach• Applications supported

• Lack of Frameworks

• Low Automation expertise

Lack of documentation• Documentation is neglected

at times

• Document not inline with the code changes

• Delays in timely decisions

• Lack of proper communication channel

• No correct ROI of automation

• High script maintenance

• Low resource moral

• No correct channel is followed for coding standards

Mitigation strategies


Common approaches/options Description Benefits Challenges

Web automation, manual on mobile

• All manual don’t automates • No additional cost for transition or to build Mobile Automation Framework

• No need for Mobile Lab Setup• No additional

learning required

• Does not address underlying issues• Risk of credibility and brand value• Loss of revenue and business

Web and mobile automation, no parallel execution

• Automate Web and Mobile Application without Continuous Integration and Parallel Execution

• Efficient Testing• Upgradation and Reusability• User Environment Simulation• Reduces Human Error

• May result into code redundancy for mobile and web scripts

• Higher transition cost• mobile technologies changes too fast –

automation need continuous support.• May result into inconsistency

Recommended approach Description Benefits Challenges

Automated customer centric parallel execution

• Automate Web and Mobile Application including Customer Centric Solution, Continuous Integration and Parallel Execution

• Thinking through solutions, before starting to code

• Documentation of Assumptions and Specifications

• Addresses Device Fragmentation issue

• Remote and cloud automation• Test consistently. Each test is

executed in the same way

• Active involvement from Business and Technical Team

• Resistance to change• Mobile technologies changes too fast –

automation need continuous support

Here are some key options…Mitigation strategies

Approach to overcome the issues and challenges


Architecture and design

Object repository• Captures the unique

identifier to handle the object

Configuration files• Stores the environment

and other global variables to determine the config values

Data repository• Handles the Application

under Test data and dynamic data

Controller scripts

Function Libraries

• Reusable page/module level functions

Reporting module

• Reports and Logs Generation

Recovery scripts

• Contains the logic to handle exceptions

Common utilities

• Reusable functions for Native and Web Apps

Test scripts• Contains the actual Test

Execution Steps

Scenarios• Given Set of Preconditions

• When Set of Test Condition

Mobile and web applications

• Then Set of Expected Results

BDDWeb and mobile automation


Continuous integration for mobile and web automation

Execute Automated

Scripts

Create mobile project for testing :QA creates the scripts and

submits it to the Code Repository

Tester

Stakeholders

Execution on node machinesHub sends the tests to each node and when it is executed the result is sent back to the

Hub machine

Source code repository

CI Servers

Hub machine

Run deployment scripts on hubCI Server compiles and builds scripts

and sends it to the Hub Machine

Continuous reportingThe Automation Test Execution results

are sent to QAs and Stakeholders

Publish

ResultsSend test results

The Hub Machine sends the consolidated result back to the CI

Server

Business case study


These elements can serve as a foundation to build an efficient solutionBusiness case study

Category Impact

Test Efficiency • Ran the Test Suite which comprised of 20 Test Scripts on Mobile and Web Application

concurrently.

• Out of these 80 Test scripts(20* 4) , 96.25 % scripts passed

Test Coverage • Developed automation test suite comprising of 20 test scripts covering 80% of application

screens on both Mobile and Web

• Also covered 80% of the gestures supported by the Mobile Automation Tool

Effort Saving and Flexibility

• Test Execution time for Manual Test Cases is around 20 Test Cases * 4 Devices * 15 min = Approx. 1200 min/Cycle =20Hr/cycle

• Test Execution time for Automation Test Cases is around 20 Test Cases * 4 Devices * 8 min = Approx. 640 min/Cycle =11Hr/cycle.

• Executing parallel on 4 nodes will again reduce the time by 11 / 4 = ~ 3 hours

Enhanced Validation • Defects are detected early whenever a new build is deployed

• Recovery scenarios helps the Test Cases to proceed ahead in case of any unwanted failures

• Details trace of the test is also provided for analysis which can help in concluding an issue.

Reusability and Extensibility

• Using BDD Components, the same scenario can reused across multiple device with minimal effort.

• The Web scripts that were used to automate the Web Application have been extended to incorporate the code for Mobiles

Benefits/value delivered


Benefits/value delivered

• Reduced coding & maintenance efforts as the tests runs on multiple platforms• Reaping the rewards in the future as we can add more test cases or update existing application

flows.

Common CodeBase

• The Mobile Device Lab mandates a particular process for adding devices to the lab. And also makes the lab easily scalable. Common processes can be set for tasks such as resetting installing/uninstalling software on the devices, software or OS upgrade, etc.

• It helps in drastically reducing the time required to get the test results

Framework and Programming scalability

• Everyone in the chain knows exactly what test cases are being executed and the test data being used, as the test cases are written in Gherkin (Given-When-Then) format which are easily readable and can be understood by everyone involved in the SDLC

• QAs can just pick these tests and plug them into the framework saving the time & effort of re-writing automation scripts.

Automation Execution transparency to Customer

• Testing results are more reliable since we test on devices.• Issues can be reported in the software encountered for low bandwidth, security, performance, etc.

Testing on Real Devices

• Automation runs on real devices can be kicked off immediately without any manual intervention or waiting for the QA team to start the process.

• Parallel execution of test suites can be easily configured and fired under CI tool

Continuous Testing

Implementation approach


There are four phases to implement BDD with mobile automationImplementation approach

Feasibility Study and POC• Benefits of automation in this project? • Identifying parts that can be automated on mobile and integrate

with web

Pilot• Implement the POC along with BDD• Identify challenges and scope of optimization

Implementation • Implement the framework along with parallel execution• Integrate mobile and web automation effectively• Add BDD along with required traceability

Rollout and Optimize• Rollout the implementation across project• Optimize the common components based

on learning from different projects• Rollout the updated implementation

Best practices andkey takeaways


Lessons learned and best practices

Shift focus from Tests to Behavior :It helps to describe behaviors in a single notation which is directly accessible to domain experts, testers and developers, so as to improve communication

Invest in Continuous Integration : Having an automated job that will build and deploy applications on the mobile, trigger automation runs, email notifications, etc. would really help in achieving a 24/7 service.

Invest in Mobile Labs : Testing on real devices helps to uncover some critical bugs and failures and also helps to determine the actual behavior of the devices in various conditions

Use Page Object Patterns : It helps the operations and flows in the UI to be separated from verification. This concept makes the code cleaner and easy to understand.

Develop Strong Test Recovery :Develop agents that can restart test components on devices. Enhance test case recovery, capability to restart test case.


For organizations facing the challenges related to merging of multiple technologies and platforms, the customer centric solution amalgamated with parallel execution and Mobile Automation will help to address these needs.

If implanted properly it can help Organizations to achieve• Collaboration between Business stakeholders, Business Analysts, QA team and Developers• Multilingual Test Scenarios (35+ Languages) • Right tools and the right approach• Remote and Cloud Execution• Parallel Executions• Framework and Programming scalability• Continuous Integration

Key takeaways


• IDC : IDC May 2015• Jigesh Shah & Mradul Kapoor : “Traceability and Automation using BDD”• Cucumber : https ://cucumber.io/• Parallel and Web Automation : http ://www.seleniumhq.org/• Mobile Automation : http ://appium.io/

References

Thank You!

Amalgamation of BDD, parallel execution and mobile automation

Technology

Transcript of Amalgamation of BDD, parallel execution and mobile automation