Experitest & Wipro Co-Webinar

June 2015

Wipro & Experitest Co-webinar:

Perfect Your Mobile App with Load Testing and Test Automation

Guy Arieli

CTO

Experitest

Sudheer Mohan

Director - Mobility

Certification &

Automation

Wipro Technologies

Speakers

June 2015

Agenda

1

2

3

4

Non Functional Testing for Mobile

Demo- See Test Integration with LoadRunner

Case Studies- Major Bank and US Retailer

Challenges in Mobile Load Testing

6

5

Mobility Automation

Wipro’s Automation Framework

© 2015 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL4

Non Functional Testing

Performance issues

( Slow , freeze, crash)

Heavy mobile resource utilization

Uses heavy network bandwidth

Accessibility compliance

Low application ratings

Unsatisfied customers &

Potential loss of Business

Enterprise challenges Platforms &

platform versions

Device

Manufactures

Real world

conditionsService

Providers


Wipro Non-Functional test solution

Mobile

AppsMobile

Platform

Mobile

network

App profiling

Interruptions

App response time

Exploratory testing

Accessibility testing

Network condition

Handover situations

Data transfer

Sensitive data validation

Simulate mobile user load*

Load testing*

Stress testing*

Security testing

Load testing *

Stress testing *

Security testing

Enterprise

Services

Focused non-functional testing* Covered as part of SeeTest load testing solutions

3rd Party

Integration

Social media

Crash analytic

tools

Analytic tools

Mobile generate

more load


Mobile users uses your application

more extensively

Mobile generate more load

The number of mobile users

already extended the number of

desktop user.

The Market shifts from

web to Mobile

Mobile generate

more load


Thin-Thick Client

Mobile

applications are

replacing Web

Application

“The mobile

enterprise business

application market is

projected to double

by 2018”

Gartner

Mobile App is relatively

more complex


web to Mobile

Mobile generate

more load


MobileWeb

JSon over http, JSon over TCP,

XML-RPC, Soap, REST, RMI …Single Protocol- HTTPProtocol

Json, XML, Free Text …HTMLData

Both synchronized and non-

synchronized

Synchronized –

request / responseCommunication

Multiple threadsSingle threaded appConcurrency

Unlimited methodsUnified security –

browser basedSecurity

multiple requests

Single request / single

response -

Submit / next page

Transaction

Thin-Thick Client

Mobile apps are relatively complex

Mobile load create bigger

effect on your servers


more complex

The Market shifts

from web to Mobile

Mobile generate

more load


Mobile load create bigger effect on your servers

One of the main

cause to load on your

server is number of

concurrent session

Mobile session takes

longer time to be

processed due to

mobile network

conditions- result in

over load on your

servers

Same number of user

mobile compare to

desktop can result in

over 20% more load

on your servers.

Traditional Load

Testing is Synthetic




more complex

The Market shifts

from web to Mobile

Mobile generate

more load


Traditional Load Testing is Synthetic

In traditional load testing the application under test is being removed and a synthetic network traffic is being generated

Typical request can have dozen or even hundreds of fields. One of the big challenges of load engineer is to

understand the nature of all of those fields

How does your application traffic

look like?

Your application open TCP connection to

the server (SYN-> SYN,ACK<- ACK->)

Your application send a request.

The server send a response.

Your application is closing the connection.

What would you find in your

application request?

Key value property data structure include

information like:

Request information

Request parameters

Session ID

Transaction ID

syntactic load is Risky

Traditional Load





more complex


web to Mobile

Mobile generate

more load



2 type

of correlations:

Why

do I need to

modify the

request

fields?

User parameters to request fields

Server response to user request

To simulate different user inputs i.e different user name / password or different UI

form population

To support internal server protocol - a very common server protocol scenario is that after

the login request, the server will return a session ID. The client need to use this session ID

in any request that is been perform.


It will create no effect on the server

The server will ignore this field

The server will ignore the entire request

The request will cause a server internal error

With Synthetic load you don't really know what will happen, it is a trial and error process

Main

Risk

What will

happen if I will

modify the

request field?

Hard to validate

syntactic load

syntactic

load is Risky

Traditional Load





more complex

The Market shifts

from web to Mobile

Mobile generate

more load


validate syntactic load

Performance teams need to adopt methodologies to validate that the generated load is valid.

Few approaches can be:

Database / UI validation to verify that the transaction generated the expected

effect

Looking at the server log files and find transaction completion log messages.

Monitor the server vital to verify the load effect the server CPU, memory, number

of threads…

Hard to validate

syntactic load

syntactic

load is Risky

Traditional Load





more complex

The Market shifts

from web to Mobile

Mobile generate

more load


As you generating the synthetic load you would like to generate few interaction with the servers based on real

devices running your real application.

Monitoring your load using real device transaction

Provide real

transaction

response time.

Provide validation

to the syntactic

load effect.

BenefitsProvide response

time reference

(when executed

without the

syntactic load)

End-to-End Product Suite - Manual, Automation, Monitoring, Load, Network Virtualization, Manager

• Productivity tool for Manual testers

• Report with screenshots of each step

• Used in conjunction with SeeTestCloud

• Allow remote debugging

• Device Lab located onsite (in the customer VPN)

• Devices can be reserved and accessed by any remote user

• Used in conjunction with SeeTestAutomation/Manual for testing of remote devices

SeeTestManual

SeeTestCloud

• Automation tool

• Same test runs on different devices

• Plugs into all testing & CI environments:

SeeTestAutomation

SeeTestNetworkVirtualization

• Device performance under varying network conditions

• Simulating mobile network bandwidth latency and reception while on the move

SeeTestLoad

• Real device generated mobile load

• Full integration to LoadRunner, Jmeter, etc.

Mobile Add-On for UFT• Full & complete integration into HP UFT:

• Record a test, edit it, run it, and view results

• Integration of object repository and more…

SeeTestManager

Free

Tool!

N

L

G

U

Mobile Add-On for LoadRunner• Full integration to LoadRunner through

SeeTestAutomation


Mobility Automation


More releases in a year

Parallel Tracks- Phone/ Tablet/ Web

Less time to test

Numerous Devices/ Platforms/ Versions

Not just UI Testing- Services/ Backend

End to End Testing

Why is automation a must?

In Mobility- Automation is the Key!!


1. Early Automation Support Agile & BDD projects

Faster Testing by in- sprint automation

Optimize testing cost by expanding

automation coverage

2. Platform Agnostic Multi platform and device support

Reusable libraries

Jump start automated testing in new

projects

Reduce testing costs3. End to End Automation Services & Back end validation

Test in various environments by

pointing to different systems

behind the scenes

High quality apps

4. Unattended Testing

Parallel Execution

DevOps adaptation

Fully Automated

Integration with Test Management

& CI solutions

Faster test execution

Shorter test cycles & more release support

Wipro Automation

Framework

Wipro Automation Solutions Overview

5.


5

Early Automation Approach

64

2

Application Flow

Objects in each screen

Automatable scenarios

Analyze

Wireframes

Develop Framework

Build reusable functions/ pseudo scripts

Object repository mapping table ( Key->Value pair)

Develop

Automation scripts

Map actual Objects to

Object repository

Execution Ready

Dry Run

Ensure modified scripts

are running fine

Re Engineer

Do modifications to the

scripts to rectify the issues

found in dry run and make it

stable

Execution

Start test case execution on

test devices

Dev-ops or CI/CD

In-sprint automation

Accelerated Testing

1 3


Automated

Build

Checks

in CodeAutomated

Deployment

Automated

Reporting

Automated

Testing

Un-Attended Testing

DevOps support

Quick testing

Integration with existing test management solutions


Demo

Build on Java, leveraging SeeTest

Supports keyword driven and data driven approach

Supports parallel execution for faster testing

Easy to configure and use even by a manual tester

Supports early automation approach

Supports integration with 3rd party test management tools

Supports un-attended testing

Framework Highlights

• Configure the environments via environment sheet

• Write the test cases using Excel based template

using keywords

• Demonstrate un-attended test execution using

Jenkins by automatic deployment of build

• Demonstrate parallel execution using SeeTest on

one Android and iOS device

• Integration with HP ALM – Updating results

• Demonstrate standalone report

Demo – In brief


Case in Point: Complete Mobile Automation- Leading Bank in US

Testing of Bank’s mobile application on

iOS and Android devices

A flexible and highly reusable

automation framework which supports

unattended testing and integration with

existing test management tool (HP

ALM)

Backend mainframe validation for

transactions from mobile devices

Reduce testing cycles and improved

ROI for mobile automation

Up to 40%* effort savings with automation in

comparison with manual testing

Continuous integration using Jenkins and

enhanced reporting capability by automated results

being updated in HP ALM

Reduce testing cycles using early automation and

parallel execution

Shift-left automation script development by

improving ROI for mobile automation

Automated backend validation for each transaction

Easily extendable to other OS platforms if needed

Leveraged and customized Wipro mobile

automation framework which supports:

Platform agonistic scripting

Data driven

Highly reusable

Parallel automation execution

Leveraged SeeTest partnership to build a

Mainframe validation component and

incorporated in to Wipro framework

Integrated Wipro automation framework with

Jenkins and HP ALM for un attended testing with

SeeTest as the automation tool

Implemented in-sprint automation using Wipro

automation framework

Implemented Parallel automation test execution

using SeeTest for faster testing cycles

Problem StatementWipro Solution

Benefits!

* Based on 60% automation and early automation approach


Case in Point: Automation Productivity Enhancement- US Retailer

Testing on iOS and Android devices

High automation script reusability across

customer mobile applications

Build an automation framework which is

easy to use and allows non-technical

testers to do automation

Faster testing cycles and improved ROI

for mobile automation

Enhanced automated test reports based

on customer requirement

Reduce overall testing cost by reduce as

much as commercial tools

Flexibility to run automation across

various test environments

Jump start automation by enabling non technical

testers to do automation using a simple excel

spreadsheet

Improved automation development productivity

using excel based framework

Flexible and highly reusable automation framework

which is completely configurable via excel sheets

Shift-left automation script development by

improving ROI for mobile automation

Developed a Java based automation framework

which allows writing of test cases using Excel

sheets which enables even non-technical testers

to do automation

Core technical team to write new keyword

functions

Framework supports parallel execution by

configuring the devices in the configuration file

Custom made reports for holistic view of test

execution across all devices in a single report

Integrated with Jenkins for Continuous Integration

support

Implemented Parallel automation test execution

using SeeTest for faster testing cycles


Benefits!


Case in Point: Enhancing Customer Experience for a Leading US

Retailer

Reduce production defects reported by

end users ( Performance issues, app

freezing , no-product found)

Improve app rating of existing mobile

applications

Ensure seamless customer satisfaction

Ensure latest/upcoming features &

technologies like iBeacon, barcode

scanning , iwatch, etc.

Improved app ratings from 3 to 4.5 (iOS) & 1

to 4.3 (Android)

Rated as one of the fastest growing retail

mobile apps

Stable application under real world scenarios

Tested app under various network conditions

and identified the real world issues faced by

real user

Conducted usability testing to analyze the

application from an end user perspective and

identified the usability issues

Conducted app profiling to find out the

resource utilization of mobile application

Conducted network monitoring to ensure

optimum network data transfer

Conducted field/in-store testing to ensure

seamless end user experience

Simulate network load and backend server

load and identified the issues faced by user

in peak load times

Conducted app comparisons to identify the

issues and benchmark the app against other

competitors


Benefits!


Thank you

Sudheer Mohan

[email protected]

Experitest & Wipro Co-Webinar

Software

Transcript of Experitest & Wipro Co-Webinar