Designing a Mobile Device Lab - XBOSoft Webinar

with Costa Avradopoulos

Designing a Mobile Device Lab

#MobileDeviceLab

XBOSoft Dedicated to software quality improvement

Founded in 2006

We speed products to market with our expert:

• Software QA consulting

• Software testing

Global team with offices in San Francisco, Beijing and Amsterdam

“Thorough, accurate and fast”

XBOSoft, Inc. All Rights Reserved. 2

House Rules Participants other than the speakers are muted

Questions via the GoToWebinar control on the right side of your screen or through Twitter @XBOSoft

Questions may be asked throughout the webinar - we’ll try to answer them at the end

You will receive info on recording after the webinar


Webinar Hashtag: #MobileDeviceLab

Meet Our Speakers

Philip Lew CEO and Founder, XBOSoft • Relevant specialties and

passions o Software quality process,

evaluation, measurement and improvement

o Software quality in use / UX design

o Mobile User Experience and usability

o Cycling and travel


Karla Smith Marketing Manager, XBOSoft • Emails you about our

industry hot topic webinars

• Invites you to download our latest trends in software testing whitepapers

XBOSoft Team

Guest Presenter Costa Avradopoulos


• Over 25 years’ experience in development, from

requirements, to delivery, and process engineering

• Roles in Development, QA, Automation, Product Management, VP, and CTO. Management of systems up to 15 million users in verticals such as Telecommunications, Financial, Wireless, Transportation, and Retail.

• Much of career dedicated to the mobility space, culminating with a breakthrough

patented mobile technology

• Frequent speaker. Writing book due to be published soon: Winning Mobile Strategies – Bridging the Quality Gap

• Principal for Avracom. Advises organizations in the creation of winning mobile strategies to develop best of breed applications.

• Certified Six Sigma and TPI

[email protected] www.avracom.com

mailto:[email protected]

http://www.avracom.com/

Why We’re Here


• Mobile applications – disruption in the market and in the SDLC

• Mobile testing requires a different approach • A good strategy starts with designing a good

mobile device lab • With thousands of variations, traditional

approaches are insufficient

Agenda


• Mobile Testing Challenges • Defining Lab Requirements • Building a World Class Device Lab

Out of Scope for Today: • Discussion on specific tools • Automation


How Do You Define Mobile?

1 ABLE TO MOVE OR BE MOVED FREELY OR EASILY. 2 OF OR RELATING TO CELLULAR PHONES,

HANDHELD COMPUTERS, AND SIMILAR TECHNOLOGY.

Smartphones, Feature phones, tablets, laptops, ruggedized devices, medical devices, wearables, kiosks, ATMs…


World Quality Report 2014

KEY FINDINGS:

The QA function is adapting to business demands by streamlining and centralizing its structure, to achieve efficiency and cost optimization

Significant investment in Test Environments may also be undermined by a lack of specialist expertise

Testing’s late engagement in the application delivery lifecycle and its reliance on basic IT metrics is possibly applying a negative brake on this increasing maturity

Mobile testing has increased in importance as a key discipline within the function, but lacks specialized methods, expertise and environment


World Quality Report 2014

KEY FINDINGS (Cont.):

Cloud adoption and cloud-based testing has slowed as organizations continue to manage concerns regarding data security and performance

A higher proportion of overall IT budget is being invested in testing and focused on transformation projects

Agile development is now widely adopted but still gives rise to problems for testing, particularly in relation to specific methodologies and expertise

NOTE: Mobile Testing is a dedicated section in this report To obtain a copy of this report, email [email protected]

Mobile Testing Areas of Focus - 2014

Source: World Quality Report, 6th edition

11

XBOSoft, Inc. All Rights Reserved.

The Cost of Poor Quality Escalates

65% OVER of all apps have a 1 star rating or less

Top quality issues in last 12 months exceed $2 Billion in losses:

• Security: breach at top 10 major retailer ($570M+ loss) • Performance/Outages: Top 3 social media company – 5 major

outages in 2014, and 2 outages in 2015 • Security: breach at top 10 major retailer ($500M+ loss) • Functional: Top 3 bank, double payments issue

12


Impact of Failure

Cost of Poor Quality is Accelerated with Mobile

Visibility is far greater, exposing your brand & reputation:

• Social Media makes failures exponentially more visible, with potential to go viral

• App store is a public report card - you cannot hide or cover up poor ratings

A plethora of choices for customers = competitive risks

• Near zero switching costs for customers • 60% of users will abandon your app/site if does not load in 3 seconds • Less than half will return to give a slow app/site a 2nd chance

13 XBOSoft, Inc. All Rights Reserved.

Mission Critical

Low Criticality

Mobile Testing Challenges

Security

Usability

Performance

Networks

Fragmentation

Due to involvement of GSM/ CDMA carriers identification of threats, risk and determining its business impact is crucial

Thorough understanding of Visual, Accessibility and certification testing is a must

Due to the limited hardware architecture of mobile devices, success and failure of application in market depends on performance of the app

Testing to be performed on different networks like 3G, 4G, Wi-Fi and Edge, etc

When you factor in different OS versions, unique hardware models, and carrier versions, the permutations are in the thousands

Tools Specialized knowledge of tools specific to mobile testing

Lack of Awareness There are many mobile tools and test approaches available in market, but all are not mature


Internal Cloud

Definitions of Cloud What do we mean by:

PUBLIC CLOUD

A mobile lab hosted by the testing tool provider. All cloud clients have access to the devices in the lab. A specific tester reserves the device he needs and uses it when its available. Accessible through the internet.

PRIVATE CLOUD

A mobile lab hosted by the testing tool provider dedicated to a specific client. The tester can reserve a device without having to share the devices with other clients. Accessible through the internet.

ON-PREMISE CLOUD

A secure mobile lab set up by the client itself. All the devices are connected to a server inside the company network. The company has complete control over what devices are procured and who uses them.

External Cloud


Shared Advantages

• Enables better collaboration between Dev/QA

• Both provide very robust lab/device

management, supporting provisioning and

maintenance

• Resource management. Both provide a

mechanism for scheduling/reserving devices

• Improve efficiency of test execution

• Enable continuous integration for DevOps


Using an External Cloud

Pros:

• Scalability - easy access to hundreds of devices on demand

• No need to maintain devices (provider does it)

• Newer devices added to the pool at no extra cost

• Flexible for more advanced solutions (ex. GPS)

Cons:

• Tester session performance tends to be slower, especially with higher

bandwidth apps like video

• Some security concerns –Testing done outside the firewall, while data

is transferred back and forth through the internet

• Different companies use the same devices with

your app and potentially your private test data


Using an On-premise Cloud

Types:

• Commercial administration via cart/hub

• DIY lab

Pros:

• Higher performance real-time user sessions

• Increased security over devices and test data

• Allows users to pull devices from lab for hands-on tests

• DIY is least expensive

Cons:

• Limited scalability

• Maintaining devices requires a lab manager

• Procuring/retiring devices can be challenging

• DIY is most difficult to maintain and physically secure


Emulators Benefits:

• Most cost effective

• Least infrastructure setup

• Flexible and Scalable

• Support for most test plans

Constraints:

• Cannot execute Usability, Performance, & Security test cases on emulators

• Network and hardware interrupt testing cannot be done on emulators

Mobile Testing Comparison

Physical Devices in Lab Benefits: Covers all test types Safest option Should be used for most

common device types Best for UAT / Usability Constraints: Very costly to support across

all OS, device, and version combinations

Procurement and setup of new devices can delay testing

Physical Devices in Cloud Benefits: Cost effective Fastest start up Flexible and Scalable Can cover approx 90% of test

plan Constraints: Latency during usage and

longer test execution time Lower level of security

excludes this as an option for some clients


Defining Lab Requirements


Use Real Mobile Devices the degree to which a test system matches a production system, including hardware, firmware, devices, software, appliances, networks, etc.

TEST INTEGRITY

Emulators/Simulators are tempting due to ease of use and low cost, however:

Proven to cause significant defect leakage (stats & peers)

Lowest level of test integrity Certain functions cannot be replicated


Other Device Lab Considerations

LAB SETUP WILL DEPEND ON:

Test objectives on coverage

Manual vs. automation Testing types Security Budget Scale Location of

Teams


POLL Approximately how many unique models of mobile

devices exist today?

A. 5,000 B. 10,000 C. 15,000 D. 20,000 E. 35,000

Assumptions: • We are actually looking for approximate total # of permutations • Therefore a device with the same model number, but on a different carrier,

counts as more than 1 “model” or instance • Total active devices in use today globally, including feature phones, tablets,

wearables, but excluding IoT/embedded (cars, kiosks, appliances)


Test Coverage

Device type (handheld, tablet, kiosk)

Screen size & resolution – dozens just in the top 80%

Device models – 20,000+ unique variations exist!

Operating system – 27 variations in the top 85%

Connection type (USB, WiFi, Bluetooth, Cellular)

Mobile Web vs. App

Variance in carrier installations – 750 carriers globally

Even using 80/20 rule, you are left with thousands of permutations! So what do we do?


Use Analytics to Understand Your Users

Implicit Analytics: Platform usage and user behavior

Omniture Google Analytics

Flurry TestFlight (monitor behavior)

Mixpanel (native vs. mobile web)

Localytics


Use Analytics to Understand Your Users

Xyo

Applause (external, sentiment)

Appsfire

Meltwater HootSuite

Explicit Analytics: What your users say


Applause Sample Dashboard

27

Customer Feedback: Great place to find those hidden test scenarios! 1. Social Media (Facebook, LinkedIn Groups, Twitter, etc.) 2. App reviews 3. App ratings / comments 4. Comments, forum posts and articles on the web 5. Complaints and support requests 6. Trending analytics (ex via Applause)

Ensuring Proper Test Coverage – Rounding it Out


Lab Definition Exercise


First step – review analytics The data says our customers are: • 55% on iOS

– 62% iPhone 6, 15% iPhone 6+, 23% iPad

• 43% on Android – 45% on Samsung S5, 20% on S6, 20% Note 5, 15%

Tab S (tablet)

• 2% on Other (Windows/Blackberry)



Additional considerations: • 8 screen sizes/resolutions in top 80% • 12 OS versions in top 80% • 4 major U.S. carriers • 1,000 functional test cases



Tier iOS Android Other Carriers Screens Tests Execution

Total 6 6 2 4 8 1,000 N/A

Tier 1 4 4 2 3 5 1,000 All tests

Tier 2 3 3 0 2 3 750 Mission Critical

Tier 3 2 2 0 1 2 400 Light regression

Tier 4 1 1 1 1 1 100 Exploratory/ Smoke

Test Coverage Sample By Phase


Source: Perfecto Mobile

Test Coverage Sample By Index



Test Coverage Sample By OS



Building a World Class Device Lab


Overview of Device Lab Components

Desktop Users U.S. or India

Device Management Interface (View device inventory, reserve, etc.)

Real-time Device Viewer (launch sessions)

Remote Control protocol (access to launch apps, perform gestures, text, etc.)

Device interface (USB hub, Desktop/Server, Proprietary Appliance)

Real Devices, On-premise or Cloud


U.S. Test Team

Firewall

Example On-premise Cloud setup

India Test Team

Mobile Devices on-premise Cloud

Device access via desktop tool

Wi-Fi Access Point

Carrier Network


Mobile Devices on-premise Cloud

Wi-Fi Access Point

Carrier Network

Firewall


U.S. Test Team

Firewall

Example Public Cloud setup

India Test Team

Mobile Devices Off-site Cloud B Device access

via desktop tool Wi-Fi Access Point

Carrier Network


Mobile Devices Off-site Cloud A

Wi-Fi Access Point

Carrier Network

Firewall


Mobile Device Lab Recap


• Define mobile for your organization • Create thorough test objectives • Weigh all factors when defining lab

requirements • Decide on Cloud vs. On-premise • Define test coverage by leveraging analytics • Monitor channels for continuous feedback • Use tier-based approach to optimize coverage

Post your questions on Twitter and we'll answer them @XBOSoft Join us to keep updated on all our webinars, reports and whitepapers: facebook.com/xbosoft +xbosoft linkedin.com/company/xbosoft We post regularly on our blog – check us out! http://xbosoft.com/software-quality-blog/ Download our free Whitepapers, available here: http://xbosoft.com/knowledge-center/ You will receive an email with information on today’s webinar slides and recording. Please email us with any further queries regarding our services or ideas for future webinars! [email protected]

Q+A

www.xbosoft.com XBOSoft, Inc. All Rights Reserved.

http://xbosoft.com/software-quality-blog/






http://xbosoft.com/knowledge-center/software-testing-white-paper/








http://www.xbosoft.com

Designing a Mobile Device Lab - XBOSoft Webinar

Technology

Transcript of Designing a Mobile Device Lab - XBOSoft Webinar