T10 Mobile Testing

5/2/2013 11:15:00 AM

Mobile Testing Methodologies: Trends,

Successes, and Pitfalls

Presented by:

Eran Kinsbruner

Perfecto Mobile

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Eran Kinsbruner

Eran Kinsbruner is director of product marketing at PerfectoMobile, one of the leading mobile cloud and automation companies. Formerly CTO for mobile testing and Texas Instruments project manager at Matrix, Eran has been in testing since 1999 with experience that includes managing teams at Qulicke & Soffa, Sun Microsystems, General Electric, and NeuStar. The co-inventor of a test exclusion automated mechanism for mobile J2ME testing at Sun Microsystems, Eran has experience in the mobile testing world. You can find Eran on Facebook, Twitter @ek121268, LinkedIn, and his professional mobile testing blog at ek121268.wordpress.com.

4/16/2013

Perfecto Mobile 1

Mobile Mobile Testing Testing Center of Excellence GroupCenter of Excellence GroupPerfectomobilePerfectomobile

Mobile Testing Methodologies Trends,

Successes, and Pitfalls

StarEast - 2013

Enabling Enterprise mobility

Extending Software Quality to Mobile

About me and my company

Mobile Market Trends

Mobile Quality Trends

Mobile Quality Challenges

Mobile Risk Management

Mobile Testing Best Practices

AgendaAgenda

4/16/2013

Perfecto Mobile 2

∙ Founded 2006

∙ 80+ employees

∙ Office in Boston, London, Israel

∙ Over 600 customers

∙ Fortune 500 enterprises, Telcos, SMBs

∙ SaaS Platform + Licenses

∙ Recurring revenue model

Mobile Application QualityMobile Application Qualityacross the mobile application lifecycle

MonitoringMonitoring TestingTesting AutomationAutomation

ALM Integration

ALM Integration

OffshoringOffshoringContinuous IntegrationContinuous Integration

Public Cloud Private Cloud

Mobile Market Trends

4/16/2013

Perfecto Mobile 3

Global Global Mobile Market Mobile Market SStatstats

Source: nielsen, 2012

New Mobile

OS’s

FireFox OS

Tizen

Samsung/Intel

Ubuntu

SailFish

By Jolla

RIM’s BlackBerry OS

10

Windows Phone

PEAK KEON

Z10 & Q10

Mobile OS market expected Mobile OS market expected

to grow (even more)to grow (even more)

4/16/2013

Perfecto Mobile 4

Cost Of Mobile Quality RisesCost Of Mobile Quality Rises

Mobile accounts for 11% of U.S e-Commerce

Existing Mobile OS’s Existing Mobile OS’s –– End of 2012End of 2012

Samsung accounted for 63 Million out of the 159 Million in Q42012

Source:IDC

4/16/2013

Perfecto Mobile 5

Worldwide smartphone vendors Market share Worldwide smartphone vendors Market share

Source: IDC Worldwide Mobile Phone Tracker, January 24, 2013

Market insights Market insights –– Forecasts and trends Forecasts and trends

Mobile based shopping expected to grow

by 24% in 2013 (U.S)

4/16/2013

Perfecto Mobile 6

Mobile OS battle continues

•New open source competitors trying to influence the eco system

•Windows Phone and BlackBerry OS 10 – critical year for their positioning in the market

Mobile apps become more complex – impact testing and automation

HTML5 and Mobile web expected to grow significantly

EMM establishment – Processes, Apps, Tools, Roles (CMOO)

Mini-tablets and “Phablets” will successfully blur the lines between device types

Market trends Market trends -- 20132013

Mobile Quality Trends

4/16/2013

Perfecto Mobile 7

World Quality report 2012World Quality report 2012--20132013

Mobile Testing is Behind the CurveMobile Testing is Behind the Curve

Source: CapGemini

FragmentedFragmented

DynamicDynamic

ComplexComplex

Business criticalBusiness critical

Mobile EnvironmentMobile Environment

Source: Google, 2012

4/16/2013

Perfecto Mobile 8

Countless screen Countless screen resolutionsresolutions

Source: opensignalmaps, 2012 (Android only)

Consumer Reaction Consumer Reaction –– Less Tolerant than In Desktop Less Tolerant than In Desktop

4/16/2013

Perfecto Mobile 9

HTML5 Trend Involves RisksHTML5 Trend Involves RisksTo help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.

Mobile Quality Challenges

4/16/2013

Perfecto Mobile 10

Mobile Introduces various challengesMobile Introduces various challenges

Various OS

Types and Versions

Various Device/Tablets

OEM’s

Sample Screen Sizes

(Only Smartphone)

Application Types

On various browsers

AGILE ALM

Native Hybrid Web

Functional

Automation

Performance

2.3.3, 4.0.3

4.1.2, 4.2.2

4.3.3, 5.1

6.1, 6.1.1

QVGA: quarter

VGA (240×320

pixels)

HVGA: half VGA

(320×480 pixels)

WVGA: wide VGA

(480×800 pixels)

FWVGA: full wide

VGA (480×854

pixels)

nHD: one-ninth

high definition

(360×640 pixels)

qHD: one-quarter

high definition

(540×960 pixels)

MMobile obile SSecurity ecurity TThreatshreats

Source: Techcrunch

4/16/2013

Perfecto Mobile 11

Mobile OS Regression bugs Mobile OS Regression bugs –– What does it have to do What does it have to do

with my Mobile Testing?with my Mobile Testing?

Source: LINK

Android defect database

Apple Bug Reporter

Pre-release testing (iOS6.1.1 Beta)

Per Device version

Mobile Web Objects Mobile Web Objects –– Automation ChallengesAutomation Challenges

4/16/2013

Perfecto Mobile 12

Apple iOS Apple iOS –– UI UI GuidelinesGuidelines

SourceSource: :

httphttp://developer.apple.com/library/ios/#://developer.apple.com/library/ios/#documentation/UserExperience/Conceptual/Mobildocumentation/UserExperience/Conceptual/Mobil

eHIG/Introduction/Introduction.htmleHIG/Introduction/Introduction.html

Android OS Android OS –– UI Guidelines UI Guidelines

Source:

http://developer.android.com/design/index.html

4/16/2013

Perfecto Mobile 13

Risks Management

Cost of quality in mobile grows and impacted by

• Availability

• TTM

• Functionality and UX

• New platform support (Mobile OS, Mobile devices)

• Previous app version compatibility

Mobile Quality Risk AnalysisMobile Quality Risk Analysis

4/16/2013

Perfecto Mobile 14

Mobile endMobile end--user Less Tolerantuser Less Tolerant

Obviously Obviously someone forgot someone forgot

to do their regression to do their regression

testing on a Galaxy S3 testing on a Galaxy S3

before releasing this before releasing this

version to market…version to market…

�Listen to the Listen to the

marketmarket

4/16/2013

Perfecto Mobile 15

A clear process as part of the “mobile project” lifecycleA clear process as part of the “mobile project” lifecycle

Define platforms and categoriesDefine platforms and categories

Define coverageDefine coverage

Acceptable not to test - You can’t test everything

Define change frequencyDefine change frequency

Adapt to the market trends as they happen

Define SLA for changes

Define coverage as part of the ALM phaseDefine coverage as part of the ALM phase

Coverage visibility reports to project managers and executivesCoverage visibility reports to project managers and executives

Device governance Device governance

Device Governance is a risk management for mobile projects

You can’t cover everythingYou can’t cover everything

But you should define a target, which is the acceptable

risk

iOS is not as simple as you thinkiOS is not as simple as you think

At least 3 HW platforms (3GS, 4, 4S, 5)

(iPADs, iPODs)

At least 3 SW platforms

Device Governance Device Governance –– Define coverageDefine coverage

Latest

55%

prev

minor

25%

prev

major

15%

Other

5%

4/16/2013

Perfecto Mobile 16

Mobile Malware Mobile Malware -- Trojans, viruses and rootkits migrating from traditional online banking and Trojans, viruses and rootkits migrating from traditional online banking and

designed specifically for the mobile marketplace. Researchers see an increase in mobile designed specifically for the mobile marketplace. Researchers see an increase in mobile

malware development malware development -- in pace with market growth.in pace with market growth.

ThirdThird--Party Apps Party Apps -- Consumers love their smart phone and tablet applications, but often these Consumers love their smart phone and tablet applications, but often these

apps come from third parties with questionable security practices. Or worse, the apps are apps come from third parties with questionable security practices. Or worse, the apps are

created by fraudsters and loaded with malware.created by fraudsters and loaded with malware.

Unsecured WiUnsecured Wi--Fi Fi -- The unsecured wireless network is a tollThe unsecured wireless network is a toll--free highway for free highway for fraudstersfraudsters to gain to gain

access to access to mobile devicesmobile devices, either to seize control of or , either to seize control of or gaingain access to account informationaccess to account information..

User Behavior User Behavior -- Consumers are prone to download Consumers are prone to download thirdthird--party appsparty apps, use unsecured wireless , use unsecured wireless

networks, open and click links in networks, open and click links in SMS text SMS text messages and emessages and e--mails, and mails, and lose their mobile deviceslose their mobile devices..

Mobile testing and Hardware Mobile testing and Hardware (Payment card readers, healthcare units and more) (Payment card readers, healthcare units and more)

Enterprises OWNS the risk for Mobile qualityEnterprises OWNS the risk for Mobile quality

Uncompromised necessity for testing mobile apps

Should adhere to Should adhere to regulationsregulations ((GLBAGLBA//FFIECFFIEC, , RDCRDC, , FCRAFCRA, , PCI DSSPCI DSS) around security, ) around security,

authentications, payment and moreauthentications, payment and more

Usually support complex business work flows (deposit check via camera)Usually support complex business work flows (deposit check via camera)

Multi tier interactions (Databases, trading accounts, other services) Multi tier interactions (Databases, trading accounts, other services)

High Availability and MonitoringHigh Availability and Monitoring

Must address mobile end user actionsMust address mobile end user actions

Using Home and Back button during application run

Network lose during transaction

Roaming during user transaction

Application behavior in various networks Application behavior in various networks

3G, 4G, WiFi

DSS Compliance test case example

4/16/2013

Perfecto Mobile 17

Mobile Mobile IIs s MMandated andated TTo o SStrict trict RRegulationsegulations

Source: LinkSource: LINK

Mobile Testing Best Practices

4/16/2013

Perfecto Mobile 18

This is not a mobile test lab!This is not a mobile test lab!

One Chair

10 Devices

3 Extra Devices Many Cables

3 Workstations

Choose the right tools and platformChoose the right tools and platform

Understand your users device and platform choices

Optimize device and platform testingOptimize device and platform testing

Automate as much as possible using device agnostic scripting

Use mobile cloud based solutions (Collaborative, Governed, Scalable,

Cost Effective)

Use CD/CI solutions as part of your mobile ALM

Do not reDo not re--invent the wheelinvent the wheel

Extend existing ALM tools to Mobile.

Building Building Blocks Blocks OOf f MMobile obile TTesting esting SStrategytrategy

4/16/2013

Perfecto Mobile 19

World Quality report 2012World Quality report 2012--20132013

Source: CapGemini

Consider Consider outsourceoutsource to reduce to reduce mobile manual testing mobile manual testing costscosts

Test for Test for performanceperformance before releasing to the marketbefore releasing to the market

Make sure you do Make sure you do regressionregression on older supported platforms on older supported platforms

Do not have your end users do your QA work

Perform pro-active testing on new devices and mobile OS’s

MonitorMonitor in production to assure end user experience and in production to assure end user experience and

availabilityavailability

ListenListen to your end users and act upon their feedback (Markets to your end users and act upon their feedback (Markets

have significant impact on your business and branding)have significant impact on your business and branding)

Building Building Blocks Blocks OOf f MMobile obile TTesting esting SStrategytrategy

4/16/2013

Perfecto Mobile 20

(True) Mobile (True) Mobile CloudCloud

Real Devices Real Devices + Emulators+ Emulators

Manual + Manual + AutomationAutomation

ALM ALM IntegrationIntegration

EndEnd--toto--End End Mobile ALMMobile ALM

Avoid device management

Secure, Centralized

Real smartphones

Real Tablets

Emulators

Device agnostic

Interactive

Core level extension

Short learning curve

Development

Functional

Performance

Monitoring

Erank@perfectomobile.com

4/16/2013

Perfecto Mobile 21

END