LAB CONFIGURATIONIncrease deal size AND provide exact solution to the customer

QUALITYVELOCITY

USEREXPERIENCE RELEASE

TIMELINE

INTRODUCTION

ETERNAL CONFLICT

QUALITYVELOCITY

USEREXPERIENCE RELEASE

TIMELINE

INTRODUCTION

NOT SO MUCH WITH THE RIGHT SIZED LAB

DIGGING IN

GATHER THE DATA TO RIGHT SIZE THE LAB TO THE CUSTOMER OBJECTIVE

Platform Coverage

VMs Platforms

Managed HowCoverage strategy

Velocity

SprintsRelease to Production

Manual & Automated

Testing CoverageBuild Tests E2E Tests

How LongHow FrequentWhat PriorityHow Many

What testsBy who

% Low% Medium% Critical

Test Case

Priority

Automation %

Real User Conditions

Build Frequency

QUALITY: SCENARIO COVERAGE

EACH CELL = MARKET SEGMENT

Q: What scenario are you willing to risk going uncovered?

1630%

2550%

3280%1 USE PERFECTO COVERAGE INDEX AS A STARTING POINT

For instance, customer believes that 16 PLATFORMS (30% COVERAGE) are sufficient

QUALITY: PLATFORM COVERAGE

3 ASK THE FOLLOWING QUESTIONSQ: Do you only test top OS versions?A: Majority of organizations test on n, n-1, (n-2 for Android) plus beta releases

Q: Should we include non-revenue generating device/OS combinations?A: Yes, that’s where users typically have highest number of issues!

Q: Should we take into account your future users’ needs?A: Yes, additional Platforms may be necessary in addition to those already identified

1 DOES THE CUSTOMER HAVE TRAFFIC ANALYTICS?2 NO? USE PERFECTO COVERAGE INDEX

For instance, customer believes that 16 PLATFORMS (30% COVERAGE) are sufficient

4 ARRIVE AT RECOMMENDED NUMBER OF PLATFORMS REQUIRED FOR QUALITY COVERAGELet’s assume 20 PLATFORM COMBINATIONS are sufficient for coverage

QUALITY

VELOCITY

Duration of release cycle in waterfall model in months

12

IN THE PAST…releases were tightly controlled and businesses could complete test-related release activities with a small set of platforms.

They would also only need a small number of manual testers, who could finish their job on time.

Quality was acceptable, and customers rarely complained.

VELOCITY

Typical agile release cycle

duration in weeks

3

TODAY… as development teams are moving to agile, testing needs to execute faster.

Regardless of whether it is manual or automated, compressed testing timelines require parallel execution –

• Executing all platforms in parallel• Duplicating platforms to split test groups

Lack of sufficient parallel capacity to finish testing on time means that business has to compromise

VELOCITY1. Adopt automation• Benefit is often a factor of three

2. Execute tests in parallel • Implement grid strategy

• Was: Average test duration (mins) * test cases * platforms• Now: Average test duration (mins) * test cases

• (Create opportunity to move some in-cycle)

3. Use business logic to tune coverage to match desired feedback window• Prioritize test cases (H, M, L) & platforms (Primary, Secondary) into

groups• Execute High & Medium priority test cases on both platform groups• Execute low priority test cases on primary platforms only

QUALITYVELOCITY16 PLATFORMS

REQUIRED COVERAGE

3 WEEK SPRINT

VELOCITY

ARE 16 PLATFORMS ENOUGH TO COMPLETEFULL REGRESSION IN 3 DAYS?

1,000 TEST CASES X 2 PERSONAS = 2,000

3 DAY REGRESSION WINDOW

VELOCITY

COVERAGECAPACITY

EACH PLATFORM EXECUTING 2,000 TEST CASES

3 DAYSDESIRED

REGRESSION TIME

10 MIN * 2,000 TEST CASES * 16 PLATFORMS = 333 HOURS ( +2 WEEKS)

2 WEEKSACTUAL REGRESSION

TIME

20 X

Simplifying assumption – Manual test duration = Automated test duration2 WEEKS > 3 DAYS = NOT ENOUGH PLATFORMS!

VELOCITY

COVERAGECAPACITY

EACH DEVICE EXECUTING 2000 TEST CASES

3 DAYSDESIRED

REGRESSION TIME

Q: HOW MANY ADDITIONAL RESOURCES ARE NEEDED TO SHRINK TESTING TIMELINE FROM 2

WEEKS DOWN TO DESIRED 3 DAYS?

2 WEEKSACTUAL REGRESSION

TIME

16 X PARALLELCAPACITY? X ADDITIONAL

PLATFORMS

VELOCITY

REQUIREDCOVERAGE

EACH DEVICE EXECUTING 2000 TEST CASES

SOLUTION: 333 HOURS / 72 HOURS (3 DAYS)= 4X ADDITIONAL PLATFORMS (16 X 4 = 64)

2 WEEKSDESIRED = ACTUAL REGRESSION TIME

16 X ADD PARALLELCAPACITY64 X ADDITIONAL

PLATFORMS

VELOCITY

COVERAGECAPACITY

EACH DEVICE EXECUTING 2000 TEST CASES

NOTE: RECOMMENDED IS A MULTIPLE OF THE COVERAGE SET: 16 * 3 = 48 (ROUNDING UP FROM 72).

EXTRA PLATFORMS CAN BE USED FOR MANUAL TESTING AND SCRIPTING

3 DAYSDESIRED = ACTUAL REGRESSION TIME

16 X PARALLELCAPACITY64 X ADDITIONAL

PLATFORMS

Use business logic to tune coverage to match desired feedback window• Prioritize test cases (H, M, L) & platforms (Primary, Secondary) into

groups• Execute High & Medium priority test cases on both platform groups• Execute low priority test cases on primary platforms only

100 Critical Path Tests700 High Priority Tests1,200 Low Priority Tests

ALL TESTS (2,000) WILL RUN ON PRIMARY

DEVICES

PRIORITY TESTS (800) WILL RUN ON PRIMARY AND SECONDARY DEVICES{ }

VELOCITY

PRIMARYDEVICES

48 + 8 = 56 TOTAL DEVICES FOR OPTIMIZED COVERAGE15% RIGHT SIZING BENEFIT TO THE CUSTOMER

SECONDARYDEVICES

48 X

8 X

12 DEVICE TYPESALL TESTS

4 DEVICE TYPESPRIORITY TESTS10 MIN * 800 TEST CASES = 34 HOURS (185% OF 72 HOURS)2 DEVICES OF EACH MODEL ARE SUFFICIENT FOR SECONDARY

10 MIN * 2,000 TEST CASES = 67 HOURS (463% OF 72 HOURS)4 DEVICES OF EACH MODEL STILL NEEDED FOR PRIMARY

VELOCITY

VELOCITY

TO MEET THE CUSTOMERS OBJECTIVE

COVERAGE =16 UNIQUE PLATFORMS

VELOCITY =56 PLATFORMS

IN AN OPTIMIZED

LAB

RINSE & REPEAT

By ProjectKeeping in mind

THEORETICAL?

SALES STRATEGY• Get agreement from different group heads

on required number of Platforms before EB G/NG

MINIMUM DEVICE SET• 12 Tier 1 Platforms (35% customer’s

coverage)

MULTIPLIERS• SDLC stages: DRR, Build• Manual testers• Additional Tier 2 Platforms for spot checks

EXAMPLE: USAA

SALES STRATEGY• Align to company-wide BT2020

initiative• Present different sizing options and a

ramp-up strategy

MINIMUM DEVICE SET• 27 Tier 1 Platforms (customer

marketing data)

MULTIPLIERS• Test types: full regression, nightly

smoke• Persona-based testing

EXAMPLE: DISCOVER

VELOCITY

Q: The customer doesn’t know the average test case execution time. What do I do now?

You can calculate average test case execution time by using the following current state metrics: # of testers, # of Platforms under test, duration of test cycle, # of test cases, tester productivity (% of manual tester’s time actually spent testing)

Today, it takes 5 manual FTEs 2.5 weeks (100 hours) to execute a partial regression (400 test cases) on 10 Platforms. On average, testers spend 70% of their time testing.

A:

EX:

VELOCITY

TEAMS ENVIRONMENTSLOCALIZATIONTEST TYPES PEOPLE

DON’T FORGET THE MULTIPLIERS!

VELOCITY

APPLICATIONSENVIRONMENTSLOCALIZATIONTEST TYPES PEOPLE

GATHER RELEVANT METRICS

# ENVIRONME

NTS# BACKENDS

# APPLICATIO

NS# APP

VERSIONS% TC

GROWTH

# REGIONS#

LANGUAGES

SMOKEREGRESSIONPERFORMAN

CEPERSONA

MULTIPLE TEAMS

MANUAL TESTERS

OFFSHORE