Performance Testing v1 - · PDF fileKPI Data in seconds Data in seconds Data in seconds Data...

PerformanceTesting of Mobile Applications

A White Paper

Architect Testing Practice

By Amit Mohanty

An end-to-end approach to MobileApplication Testing

2Point of View Paper on Performance Testing of Mobile Applications Mphasis

Contents

Introduction........................................................ 3Why Performance Testing?................................ 3The Importance of Performance Testing........................ 3How Performance Testing Helps.................................... 3

Performance Testing Challenges ofMobile Applications............................................ 3Key Parameters - Performance Testing ofMobile Applications............................................ 4Key Focus Areas - Performance Testing of Mobile Applications............................................ 4Performance Testing of Mobile Applications- Process Flow.................................................... 6End-User Perspective (FunctionalPerformance Testing)......................................... 6Overview......................................................................... 6

Benefits.......................................................................... 7Topology......................................................................... 7Device Perspective............................................ 7Overview........................................................................ 7Benefits.......................................................................... 7Topology......................................................................... 8Network Perspective......................................... 8Overview........................................................................ 8Benefits.......................................................................... 10Server Perspective............................................ 10Overview........................................................................ 10Topology......................................................................... 10Benefits.......................................................................... 11The Conclusion.................................................. 11

Introduction

Why Performance Testing?

Performance Testing Challenges of Mobile Applications

There are diverse conditions in which a mobile application is expected to perform at its best:

• A wide diversity in devices and platforms• The need for the best user experience• A variety of network connectivity options and • The ever-growing user base of smartphones

It’s no wonder that Performance Testing of mobile applications is supremely important.

At Mphasis, our Mobility Testing Center of Excellence has developeda specific approach to test various aspects of mobile application performance that covers end user, device network and server parameters.

The Importance of Performance Testing

Performance testing of mobile applications is very important because they are installed on mobile devices based on various mobile platforms/OSs like Android, iOS, Blackberry, Windows, etc. all of which have various hardware configurations like memory, processor, screen-resolution and having varying network connectivity. Yet these applications are expected to work well in all circumstances, no matter how much load there is on the server; whatever the strength of the network may be; and what the device configurations are.

How Performance Testing Helps

Performance testing helps to ensure that the system/application responds consistently within acceptable levels (as per baseline or SLA) and functions correctly under the current load condition or expected load conditions. Performance testing of mobile applications helps in bridging the gap between testing of a mobile application in a test environment with test data and a real world scenario, thereby contributing to business success. It gives confidence to launch an application in this rapidly changing mobile space with users always expecting the best features and a great user experience.

The challenges all deal with how the application performs in all these conditions:• Varying hardware configurations like battery, processor, memory, etc• Varying amounts of disk space• When the same action is performed multiple times• When the same action is performed for long period of time• On varying network conditions like bandwidth, network latency, packet

loss, etc.• When there is heavy load on the backend application server

3Point of View Paper on Performance Testing of Mobile Applications

Performance Testing helps to ensure that the system/application responds consistently within acceptable levels (per baseline or SLAs) and functions correctly under the current load condition or expected load conditions.

Mphasis

Per

form

ance

Tes

ting

of

Mo

bile

Ap

plic

atio

ns

End-UserPerspective

Device Perspective

NetworkPerspective

ServerPerspective

Functional Performance Testing or Testing of the Performance of the application

KPIs - Application Launch time, Navigation Response Time, Screen Transition time, Transaction time measured on the application etc.

Testing the Resource Utilization by the application

KPIs - CPU, Battery, Memory, Data Throughput

Testing the mobile application under varying network conditions

KPIs - Application features tested under varying network conditions like bandwidth, latency and packet loss and 2G and 3G network profiles

Testing the mobile application by loading the backend application server using device emulator or real device

KPIs- Transaction Time of the server by varying the bandwidth, latency and packet loss of the TCP/IP network

Key Parameters - Performance Testing of Mobile ApplicationsThe key performance parameters of mobile applications are as follows:

Figure 1: Key performance parameters of mobile applications

Key Focus Areas - Performance Testing of Mobile ApplicationsThe four main perspectives of Performance Testing of mobile applications and the key focus areas of each can be categorized as below:

Figure 2: Four main perspectives of Performance Testing of mobile applications and the focus areas


Application Response Time

Reliability

Configuration Sizing

Scalability

Transaction Response Time

Bottleneck Identification

Power Management

Resource Utilizations

Does the application respond quickly enough for the intended users?

How stable is the system under a heavy work load?

Which configuration provides the best performance level?

Is the application stable under expected and unexpected user loads?

How long does it take for the transaction to complete?

What is the cause of degradation in performance?

Does the application causes faster battery drain out?

How much CPU and Memory is consumed by theapplication?

Mphasis

Client

Android

ios

Server

End-User Perspective•• Navigation response time• Screen transition time• Transaction time measured

on the application

Application launch timeNetwork Perspective• Application features tested

under varying network conditions like bandwidth, latency and packet loss and 2G, 3G, network profiles.

Server Perspective• Transaction Time of the

server by varying the bandwidth, latency and packet loss of the TCL/IP network.

PerformanceTest Area

Transition Time

ApplicationLaunch Time

Load Time

PageComponentAnalysis

Response Time

KPI

Data inseconds

Data inseconds

Data inseconds

Data inseconds

Data inseconds


KPI PerformanceTest Area

KPI

Transition Time

ApplicationLaunch Time

Load Time

Pagecomponentanalysis

Response time

Data inseconds under2G/3G networkconditions





BackendServerLoad

Bandwidth,Latency

TransactionResponseTime

Device Perspective• CPU• Memory• Battery• Data throughput


KPI

PowerManagement

CPU

Performance

Memory

Battey Usage

CPU Kernel

CPU Total

CPU User

Memory Cache

Memory Free

Memory Used

Performance Testing of Mobile Applications- End-To-End Approach


Figure 3: End-To-End Approach For Performance Testing Of Mobile Applications

Mphasis

Understand Performance testRequirements ,system Architecture,Workload & Performance Goals

Develop Performance Test Assets(Test Plan, Test Scripts,Monitoring Strategy etc…)

Execute Performance Tests

Analyze Results and MonitorMetric

Complete testing

PerformanceTuning

Collect Monitoringmetrics

Restartafter tuning

Criterianot met

PerformanceCriteria are met

Performance Testing of Mobile Applications - Process Flow

Figure 4: Process flow of performance testing of mobile applications

End-User Perspective (Functional Performance Testing)Overview

End-user perspective or functional Performance Testing deals with the performance of the application on a real device connected to a real network, when the backend application server is in a loaded state. All this can be achieved by the tester sitting at some geographic location and accessing mobile devices which are remotely located over the cloud. This helps in reducing the total cost of ownership (TCO) as the testing team does not have the overhead of maintaining the devices over the cloud or the server connecting to the device cloud. Since the behavior of the mobile application on a real device is checked when the server is in a loaded condition, similar to a real world environment, it gives a lot of confidence about the user experience in terms of performance before an application is launched for large scale usage.

To check the end-user perspective of the performance of a mobile application, the backend application server of the mobile application is loaded by a number of virtual users hitting the backend server in parallel by using HP Loadrunner. While the load on the backend server is being maintained at a specific level, a functional testing script with various transactions defined to capture the page load time, application launch time, screen transition time, etc. written in HP Unified Functional Testing is run simultaneously from within the HP Loadrunner Controller. Then, the result is observed in the HP Loadrunner Analyzer in terms of how many transactions passed a specific load condition, what time was taken for these transactions to complete, etc. These values are then analyzed against the expected values or the SLA.


End-user perspective or functional performance testing deals with the performance of the application on a real device connected to a real network, when the backend application server is in a loaded state.

Mphasis

Remote Mobile CloudWeb Interface

SoftwaresRequired

Test Teams

Virtual User Load Applied To Server

Remote Mobile Cloud

Central NodeServer

Remote Mobile Cloud DeviceRacks

Devices with Operator SIMs onnetwork

Test Management

Global Delivery Centers Enterprise Client Location

UFT UFT Mobile

LoadrunnerQuality

Center/ ALM

Enterprise Server

Benefits

It simulates a network condition or scenario very closely resembling the real world scenario.

It provides greater confidence to launch the application for large scale usage due to the close resemblance of the test environment with a real world environment.

Since the load generation script and the functional script with various transactions run simultaneously, it helps the in-depth analysis of the reports and uncovers potential performance issues.

It reduces the total cost of ownership for the testing team due to the usage of cloud based device access.

Topology

Figure 5: End-user perspective

Device PerspectiveOverview

The device perspective of Performance Testing of mobile applications helps in gauging the device vitals while an application is. This kind of Performance Testing helps in checking how the mobile application performs under the limited hardware resources of a mobile device and check CPU usage, memory consumption, battery consumption, data throughput, etc. when the application is in use. This also helps in testing a mobile application under stressed memory conditions like low memory and low battery, thus ensuring a great user application.

Benefits

Using any standard web browser, a user can remotely activate a handset located anywhere worldwide.

The solution allows collecting all vital information about the device like memory usage, battery level, battery voltage, CPU usage, etc.

Certain tools installed on mobile devices help accurately collect information about the battery consumption and radio interface.


This kind ofPerformance Testing helps in checking how the mobile application performsunder the limited hardware resources of a mobile device and check CPUusage, memory consumption, battery consumption, data throughput, etc.when the application is in use.

Mphasis

Remote Test Lab

Internet

Single user performance tuning can be done using this analysis.

Topology

Figure 6: Device perspective

Network PerspectiveOverview

The network perspective of Performance Testing of mobile applications deals with testing the application performance on a real mobile device under varying network conditions like bandwidth, latency, packet-loss, etc. which is otherwise called the network profile. Since network profiles similar to 2G, 3G and 3.5G networks can be simulated using different approaches, it is closest to checking the performance of the mobile application in a real world scenario. It enables testers to check the application on various mobile devices in the simulated network condition while they sit remotely at some other location, with the help of mobile device cloud. Also, it gives a lot of confidence to mobile application developers to launch the application for large scale use due to its close resemblance to a real world scenario.

There are two approaches to the network perspective of Performance Testing of mobile applications:

Testing of single user mobile application performance using various network profiles remotely on a real device in the mobile cloud using the network virtualization appliance from Shunra

Testing of mobile application performance remotely on a real device connected to a real network on the mobile cloud while the load is being


The network perspective of Performance Testing of mobile applicationsdeals with testing the application performance on a real mobile deviceunder varying network conditions like bandwidth, latency, packet-loss, etc.which is otherwise called the network profile.

Mphasis


Test Teams

Remote Mobile Cloud

Central NodeServer



Test Management

Global Delivery Centres Enterprise Client Location

UFT/NativeScripting

QualityCenter/ ALM

Enterprise Server

NetworkVirtualization

Shunra

applied to the backend application server using specific network profiles which simulate a real world load on the server

Topology of single user network perspective of mobile application Performance Testing

Figure 7: Network perspective- single user

In the case of single user Performance Testing of a mobile application by varying the network profiles, the Shunra appliance is connected to a single device on the cloud via Wi-Fi. So, the network connection of the mobile device can be configured by configuring the Shunra appliance with the help of an application available on the mobile device. While the mobile device is accessing the network with a specific network profile, the performance of the mobile application running on the devices is assessed.

Topology of network perspective of mobile application Performance Testing with loaded backend server

Figure 8: Network perspective- loaded backend server



SoftwaresRequired

Test Teams

Virtual User Load Applied To Server

Remote Mobile Cloud

Central NodeServer



Test Management

Global Delivery Centers Enterprise Client Location

UFT UFT Mobile

LoadrunnerQuality

Center/ ALM

Enterprise Server

Shunra NV

Mphasis

Performance TestController

Real DevicesThe Mobile Cloud

Load Generators

Performance Diagnostics

Virtual Mobile Users- Multi user/multi location simulations

MobileApplication

Service

L0

L1

L2

L3

L4

L5

V1

V2

V3

V4

V5

In this kind of Performance Testing, virtual user load is applied on the backend application server using specified network profiles similar to a real world scenario using the Shunra NV Add-In on Loadrunner. While the load is being applied to the backend server, the application’s performance on a real device connected to a real network on the mobile device cloud is assessed with the help of UFT and UFT Add-In from the tester’s computer.

Benefits

Using Network Virtualization solution for real-world load and capacity testing

Virtualized network conditions – unlimited points of presence

Support for accurate simulation of users from any geographical locations/regions

Offer of bandwidth utilization per transaction summary

Server PerspectiveOverview

The server perspective of Performance Testing of mobile application deals with testing the backend server of the mobile application by applying varying the virtual user load and then extrapolating the data for the production server capacity. It gives confidence about the performance of the application at all times during a day/week/month whenever the load on the server varies. It also incorporates the capability through which bandwidth and latency can be varied to create a more life-like scenario. It is the traditional load testing of mobile applications backend server capabilities using Loadrunner.

Topology

Figure 9: Server perspective


Testing the backend server of the mobile application by applying varying the virtual user load.

Mphasis

Benefits

Emulates conditions of a controlled load on the server

Simulates real-world mobile network characteristics

Measures application and server performance under load

Checks where performance delays occur: network or application related delays; CPU performance; I/O delays; and monitors the network and server resources under load

Supports generating load through real devices with QTP integration

The ConclusionA growing number of mobile users are using their mobile devices for their day to day tasks including important transactions like money transfers, paying credit card bills, etc. Due to this, today’s mobile applications are very important and are expected to perform at their best at all times. The surging growth in the number of mobile users using high end mobile devices and relying on them for various kinds of transactions necessitates that all mobile applications are thoroughly tested for performance glitches before their launch. Failing to prepare for these highly variable conditions can lead to customer dissatisfaction and cause damage to the brand reputation.

This whitepaper aims at combining the various solutions and perspectives for Performance Testing of mobile applications and provides a guide to the overall process of Performance Testing.


EFFICIENCY/PERFORMANCE(INCLUDING NETWORK - RELATED)

FUNCTIONALITY

PORTABILITY

USER INTERFACE/EASE OF USE

COMPATIBILITY/ REGRESSION TESTINGSECURITY (PROTECTION OF SENSITIVEDATA OVER THE PHONE OR AIR)

CERTIFICATION OF APPLICATION

64%

48%46%

36%31%

18%14%

Mphasis

About AuthorAmit MohantyArchitect (Testing Practice)

Amit has more than 11 years of experience in Application Testing, especially in the Mobility space and spanning across multiple companies. He has played various roles in Testing, Automation, Application Branding, Requirement Management, Test Management and has led teams located across geographies. He has experience in Performance and Load Testing of mobile applications using leading tools in the market.

With an MS in Quality Management from BITS, Pilani and BE in Computer Science from Utkal University, Bhubaneswar, Amit is also Six Sigma Green Belt certified and ISTQB certified.

12Point of View Paper on Performance Testing of Mobile Applications Mphasis

About Mphasis

For more information, contact: [email protected]

To find out more about Mphasis’s ability to help your organization in thePerformance Testing of Mobile Applications space, please contact:

Amit MohantyArchitect - Testing [email protected]

USA460 Park Avenue SouthSuite #1101, New YorkNY 10016, USATel.: +1 212 686 6655

UK88 Wood StreetLondon EC2V 7RS, UKTel.: +44 20 85281000

UNITED STATES

CANADA

GREAT BRITAIN

FRANCE

IRELAND

NETHERLANDS

BELGIUM

GERMANY

SWITZERLAND

INDIA

JAPAN

CHINA

PHILIPPINES

SRI LANKA

AUSTRALIA

NEW ZEALAND

SINGAPORE

INDONESIA

POLAND

Delivery Footprint Client Footprint India Centre (BPO) Global Service Centre (ITO & APPS) Training AcademyOUR GLOBAL FOOTPRINT

Vadodara

Indore

Raipur

Chennai

Pondicherry

Pune

Mphasis (an HP Company) enables chosen customers to meet the demands of an evolving market place. Mphasis fuels this by combining superior human capital with cutting edge solutions in hyper-specialized areas. Contact with us on www.mphasis.com

INDIABagmane World Technology CentreMarathahalli Outer Ring RoadDoddanakhundi Village, MahadevapuraBangalore 560 048, India.Tel.: +91 080 3352 5000

Copyright Information

Mphasis and the Mphasis logo are registered trademarks of Mphasis Corporation. All other brand or product names are trademarks or registered marks of their respective owners.

Copyright © Mphasis Corporation. All rights reserved.

Bhubaneshwar

Bangalore

Mangalore

Mumbai

Performance Testing v1 - · PDF fileKPI Data in seconds Data in seconds Data in seconds Data...

Documents

Transcript of Performance Testing v1 - · PDF fileKPI Data in seconds Data in seconds Data in seconds Data...