Increasing Cost Predictability in Performance Testing ...

Business Assurance & Testing

WHITEPAPER July 2015

www.hcltech.com

Increasing Cost Predictability in Performance Testing Services via unit-Based Pricing Model

Across the spectrum of IT projects, whether it is custom application development, systems integration, package implementation, infrastructure services or technology consulting, the engagement model and monetization of IT services have evolved over the last two decades. From Time & Material (T&M) to Fixed Price to Risk Reward to Outcome/ Value Based Pricing Model, there has been a drastic shift. Of course the ‘right’ pricing strategy for either party depends on various factors including scope of work, division of labor between client and vendor, relationship history, track record, trust factor etc. Performance Testing activities are a minor part of SDLC in terms of effort, but also is a very critical quality gateway before release to production. By and large, Performance Testing projects have been operated in T&M mode or Fixed Price, if it is a finite term engagement with definite scope and deliverables. However, the market demand is for Performance Testing engagement models to shift to the next gear of maturity and operate on a more predictable model with increased accountability of cost.

AuthOr:Vijayanand ChelliahdhasSr. Solutions Architect at HCL America Inc. currently responsible for solutions and business development for specialized testing services including Performance Engineering and Mobility Testing across the US geography

QEx WHITEPAPER

INCREASING COST PREDICTABILITY IN PERFORMANCE TESTING SERVICES VIA UNIT-BASED PRICING MODEL | JULY 2015

© 2015, HCl TECHNOlOGIES. REPRODuCTION PROHIBITED. THIS DOCuMENT IS PROTECTED uNDER COPyRIGHT By THE AuTHOR, All RIGHTS RESERVED. 2

TABLE OF CONTENT

1. INTRODUCTION 3

2. WHAT IS UPM IN PERFORMANCE TESTING SERVICES? 4

3. HOW THE UPM WORKS? 5

4. MONTHLY BILLING CYCLE – PROCESS FLOW 7

5. PRE-REQUISITES FOR IMPLEMENTING UPM 7

6. ADVANTAGES & BENEFITS 8

7. CONCLUSION 8



1. INTRODUCTION Performance Testing as a discipline has evolved significantly over the last decade, to a point where a large number of clients have integrated Performance Testing as a standard quality gateway in their SDLC. Performance Testing projects come in different sizes and durations. For instance, a long running program of 12 to 18 months duration may require Performance Testing for an elapsed duration of 2 to 6 months only depending on the scope and complexity of work.

The scope of work typically involves performance requirements gathering, workload modeling and characterization to identify the performance critical use cases and workload scenarios, developing test assets (scripts/ data/ environment), executing various tests such as Load, Capacity, Endurance etc., conducting results analysis, identifying performance bottlenecks with a significant overhead for coordination and reviews with various stakeholder teams.

These set of activities could be executed for a given individual project or could be executed for multiple projects by a group of performance test engineers operating as a Centralized Team/ Shared Services Team/ CoE Model. In the former case, estimation and pricing model is straightforward by opting either Time & Material or Fixed Price Model. However, Shared Services Operating Model brings in certain inherent administrative and operational challenges, which are amplified by the nature of Performance Testing as a discipline.

Briefly explaining, not all projects scheduled in the Enterprise Release Management calendar undergo Performance Testing. Hence some of them might be exempted by virtue of low criticality of the enhancement or the high maturity of the application. The ones that enter the performance test environment might run for varied durations, depending on the test preparation required or the performance issues identified, leading to multiple iterations of Performance Testing and tuning. Due to this, there is a high potential for effort overruns or under-utilization of resources depending on the situation.

Given such challenges in a Shared Services Model, Fixed Price may not be a good fit since the scope of work cannot be defined certain enough upfront to determine the fixed effort and cost. Time & Material is a better choice, however it brings in tremendous administrative overhead for the client. Constantly estimating the effort for each Performance Testing request, planning the resources and adjusting the team size according to the workload is even more challenging in short Release Cycles such as monthly. Not to mention the knowledge management challenges in such a fluid operating model.

While any pricing model can possibly be applied for the individual project engagements, it makes more operational and economic sense to leverage a higher maturity pricing mechanism. For example it is relevant to choose Unit-Based Pricing Model (UPM) for a Shared Services Operating Model due to the fluctuating rate of arrival of project requests, option of distributed locations and close-to-factory like program delivery.

UPM breaks down the varying size and complexity of the projects into a uniform granular unit of work. The number of units of work per Monthly Release Cycle will be measured and tracked via a defined process. A pre-determined cost for each unit of work will be agreed upon between the client and vendor. This ensures the cost per Release Cycle is predictable for the client, at the same time contractually obligating the vendor to be accountable for the cost in terms of successful project delivery.



This white paper explains an actual implementation of UPM for a Shared Services Performance Testing Delivery Engagement, for a large US based Asset Management and Insurance client. The following sections explain:

y Constructs of the model

y How the model works

y What are the tracking and measurement mechanisms to be in place for the model to work

y Benefits of implementing such a model for Performance Testing services.

2. WHAT IS UPM IN PERFORMANCE TESTING SERVICES?UPM is a mechanism by which the vendor charges the client for the amount of work done in a stipulated period of time (say 1 month), basis the agreed price for one unit of work successfully completed.

In this particular client illustration, the unit of work in Performance Testing services was represented as the number of Performance Testing scripts successfully executed, inclusive of all the performance test cycle activities for that script such as requirements gathering, test planning, test design, test data setup, test script development, test execution and results analysis. The payment would be made only for successfully completed units of work i.e. scripts that were developed and tested successfully.

One Unit of Work = One Performance Test Script successfully executed

[including requirements gathering, test planning, test design, data setup, script development, test execution and results analysis]

The Performance Testing scripts were classified as Simple, Medium and Complex on basis of the following criteria, as agreed with the client. Each type of script complexity was charged at a different price.

Script Complexity Technology

Number of Parameterization

Elements

Number of Transactions/

Operations Involved in Script

Price

Low Web Services < = 7 < 10 $ X

Medium Web HTTP/ HTML 8 to 15 10 to 20 $ Y

High Other than above > 15 > 20 $ Z

The cost for a Performance Testing project was calculated as a product of the number of scripts in scope of the project and the cost per script based on complexity type.

Cost for one Performance Testing request/ project = Number of scripts * Cost per script based on complexity type



3. HOW DOES THE UPM WORKS?At this client site, a monthly Release Cycle was in place and the average workload every month used to be anywhere between 10 and 15 projects. Each project had to undergo the complete performance test lifecycle from reviewing the requirements and changes to the application to test execution and closure on time for the release to production. Broadly, the following 5 activities needed to be performed in every Release Cycle to operate and track the UPM.

Sizing the Projects

Though the Release Cycle was monthly, the performance test requests used to be raised to the Shared Services Team about 6 to 8 weeks in advance. The Performance Testing requests were raised in a pre-defined ‘PT Request Form’ which consisted of all necessary inputs to arrive at the estimates and develop the test strategy. The assigned performance test engineer would review the PT Request Form and conduct a requirements review meeting with the project team, to confirm the number of business scenarios to be scripted, the number of tests to be done, clarify any understanding gaps and agree on the next steps.

Cost Estimates – Iteration1

Based on the information gathered on each of the in-scope scenarios, the performance test engineer would classify the scenarios into Simple, Medium and Complex in accordance with the script complexity criteria (defined in the previous section).

An automated pricing calculator called ‘Unit Price Tracker’ was developed which also served as the master of record for all key details about each Performance Testing script. The initial information gathered during requirements review would be input into the ‘Unit Price Tracker’ to determine the initial cost forecast, which will be published to the project team. Following is a snippet of the ‘Project Forecast’ sheet of the Unit Price Tracker.

Release Month Project/ Application Name Area

Projected Total No. of Scripts

Projected # of Low

Complexity Scripts

Projected # of Medium Complexity

Scripts

Projected # of High

Complexity Scripts

Estimated cost of Low Complexity

Scripts

Estimated cost of Medium

Complexity Scripts

Estimated cost of High Complexity

Scripts

Mar Customer Protection RIT 41 22 14 3 XX YY ZZ

Mar Express IRA RIT 2 4 2 3 XX YY ZZ

Mar2013 HAWC –

March ReleaseRIT 46 25 16 4 XX YY ZZ

Workload Measurement and Tracking

In the Unit Price Tracker, a detailed spreadsheet called ‘Scripts Detailed Report’ was maintained to collect the attributes of each script for all the projects in each monthly release. The individual performance test engineers were responsible for updating it on a daily basis which would be reviewed by the vendor Technical Manager with the client Manager on a weekly basis. Following is a snippet of the ‘Scripts Detailed Report’ from the Unit Price Tracker.

Release Month

Project/ Application Name

Project ID

Script Information

Script Name (as in PT Dashboard)

Script Type

Number of Transactions

# Parameterization

Elements

New/ Modified/ Re-used

Script Complexity

Mar Customer Protection SAS0928004 User Security RSV1_logins WS 1 7 New Low

Mar Customer Protection SAS0928004 User Security RSV1_logins_

site2user-image-phrase WS 1 6 New Low

Mar Customer Protection SAS0928004 User Security RSV1_logins_

authenticate WS 1 8 New Low



Cost Estimates – Iteration2

As and when the script development was completed, the performance test engineers would update the ‘Scripts Detailed Report’ with the actual values for the various parameters such as number of transactions, number of parameterization elements, etc. based on which the Unit Price Tracker would calculate the revised project pricing automatically in the ‘Project Wise Cost’ worksheet. Following is a snippet of this worksheet

Release Month Project/ Application Name Project ID

Number of Scripts Used Scripts Complexity Classification Project Cost

New Modified Re-Used Total Low Medium High Total Low Medium High Total

Mar Customer Protection SAS0928004 35 0 0 35 30 4 1 35 XX YY ZZ TTMar Express IRA SPA1008002 2 0 0 2 0 2 0 2 XX YY ZZ TT

Mar 2013 HAWC – March Release STO1019020 26 31 12 69 51 18 0 69 XX YY ZZ TT

Monthly Cost Reconciliation

Once the monthly release was completed and all the projects signed-off by the respective stakeholders, the monthly Invoice Amount derived automatically by the ‘Unit Price Tracker’, based on the number of Units/Scripts successfully executed (grouped by projects) was published to the Enterprise Release Management team for review. Post review, the monthly cost details were shared with the individual projects for reconciliation with the two sets of estimates provided earlier. Once this was approved, the vendor would raise the invoice for the agreed amount in accordance with the work successfully delivered. Following is the snippet of the ‘Monthly Project Cost Report’ sheet from the Unit Price Tracker.

January February March April

Project Completed 0 0 7 3

Test Run Statistics 0 0

Environment Readiness (Smoke) 0 0 15 15

Load 0 0 6 6

Stress/ Scalability 0 0 0 0

Load/ Longevity 0 0 3 1

Volume 0 0 0 0

Failover 0 0 0 0

Total No. of Test Runs Executed 0 0 24 22

Scripts Complexity Classification

# of Low Complexity Scripts 0 0 83 56

# of Medium Complexity Scripts 0 0 58 44

# of High Complexity Scripts 0 0 3 12

Total No. of Test Scripts Used 0 0 144 112

Scripts Cost

Cost for Low Complexity Scripts 0 0 XX XX

Cost for Medium Complexity Scripts 0 0 YY YY

Cost for High Complexity Scripts 0 0 XX XX

Total Cost 0 0 TT TT

Monthly Project Cost Report - Actuals



4. MONTHLY BILLING CYCLE – PROCESS FLOWFollowing is the process flow for the Monthly Billing Cycle as per the Unit-Based Pricing mechanism, adapted for Performance Testing services.

T T + 5 days Sign-off date + 1 day Sign-off date + 3 days

Proj

ect

Team

Clie

nt P

T Le

adVe

ndor

PT

Lead

Vend

or P

T En

gine

erVe

ndor

PT

Team

Send the PT Request Form to Client PT Lead

Review the PT Request Form and setup

Requirements Review Meeting with the Project

Team

Obtain the number of Scenarios to be

Scripted

Project the estimated project cost in “Projects Forecast’ (in Unit

Price Tracker)

Provide the Cost Estimate to the Project Team

Estimation Completed

Develop the Performance

Test Plan

Daily update of script details in

‘Scripts Detailed Rep’ in Unit Price

Tracker

Develop the Performance Test Scripts

Execute Performance

Tests

Secure Performance Test Results

Sign-offs

Review scripts details in ‘Scripts Detailed Report’

Obtain the Actual Project Cost from

‘Project Wise Cost’ sheet

Update Test Run details based

on data fromMetrics DB

Obtain monthly total cost from ‘Monthly

Project Cost Report’

Review and acknowledge the

total monthly project cost for

Invoicing

Provide the Actual Cost to

the Project Team

Monthly Costing Completed

Actual Cost Reconciliated

Acknowledge the PT Request and assign

to Vendor

Nominate the PT resource for the

project

Sign

-off

date

+

3 da

ys

Timelines

Key Inference

The Risk Assessment exercise illustrated here was conducted by 2 non-functional consultants over a 6 weeks period at the client site. Given the effort it takes to conduct such a comprehensive exercise, it may not be a feasible solution for short-term projects, low complex applications or highly matured systems. However for software product vendors and applications with large scale internet usage with unanticipated workload patterns and business growth scenarios, the qualitative and the quantitative benefits of conducting a Non-Functional Risk Assessment exercise clearly outweigh this modest effort and investment.

5. PRE-REQUISITES FOR IMPLEMENTING UPMThough UPM is applicable in any customer context, it is recommended to ensure the following five key pre-requisites before embarking on such an initiative.

It is immensely critical to gain adequate understanding of the customer’s ways of working, in terms of the types of performance test requests received, Release Cycle followed, complexity of the applications, performance test methodology adapted, environment and data dependencies, various stakeholders involved, etc., before commencing the implementation of UPM.

A comprehensive Performance Testing process should be in place which is repeatable and aligned with the UPM. Unless the customer process is robust and matured, implementing an efficient pricing mechanism such as UPM would be an overhead and may result as counter-productive.



Even though the working model exists per se, a certain degree of customization is required to ensure a successful implementation. Script complexity criteria should be baselined for any given customer context in accordance with the technology protocol, size of the projects and the tools used or any other factor deemed fit.

The billing process flow should be tailored to the client’s release management process, including the process on the client side to chargeback their internal customers which should be thought through and developed upfront.

The Operations Manager on the client side should be knowledgeable on how UPM functions and how the Unit Price Tracker works. Multiple detailed orientation sessions should be conducted for all the key stakeholders. This is essential to ensure the cost reconciliation process is smooth between the client and the vendor and also between the client’s Performance Testing team and their internal customers and stakeholders.

6. BENEFITSFollowing were the key benefits reaped by the client by engaging the vendor in the UPM for Performance Testing Services:

y Complete control over OPEX enabled by the pay per use nature of the UPM

y Near accurate cost predictability thus ensuring easy and fast budget approvals

y Minimal administrative effort in invoicing thus increasing bandwidth for other governance tasks

y Shared accountability of the Unit Price Tracker enhancing the transparency and trust in the vendor

7. CONCLUSIONThe operating and pricing model is one of the key indicators of the maturity of the client-vendor relationship. It is representative of the deep domain and operational knowledge of the vendor, and the trust developed with the client.

Any Performance Testing team providing enterprise-wide Performance Testing services across multiple consecutive releases should definitely explore implementing the UPM. This eliminates significant amount of administrative overhead for the client allowing bandwidth to focus on the technical tasks. At the same time, the vendor management team is able to clearly map the vendor output to the expense incurred. Such predictability and accountability brings confidence to both the client and vendor management that the operating model is scalable.

Hello there! I am an Ideapreneur. I believe that sustainable business outcomes are driven by relationships nurtured through values like trust, transparency and flexibility. I respect the contract, but believe in going beyond through collaboration, applied innovation and new generation partnership models that put your interest above everything else. Right now 105,000 Ideapreneurs are in a Relationship Beyond the Contract™ with 500 customers in 31 countries. How can I help you?

ABOUT HCL

About HCL Technologies

HCL Technologies is a leading global IT services company working with clients in the areas that impact and redefine the core of their businesses. Since its emergence on the global landscape, and after its IPO in 1999, HCL has focused on ‘transformational outsourcing’, underlined by innovation and value creation, offering an integrated portfolio of services including software-led IT solutions, remote infrastructure management, engineering and R&D services and business services. HCL leverages its extensive global offshore infrastructure and network of offices in 31 countries to provide holistic, multi-service delivery in key industry verticals including Financial Services, Manufacturing, Consumer Services, Public Services and Healthcare & Life sciences. HCL takes pride in its philosophy of ‘Employees First, Customers Second’ which empowers its 104,184 transformers to create real value for customers. HCL Technologies, along with its subsidiaries, had consolidated revenues of US$ 5.8 billion, for the Financial Year ended as on 31st March 2015 (on LTM basis). For more information, please visit www.hcltech.com

About HCL Enterprise

HCL is a $6.8 billion leading global technology and IT enterprise comprising two companies listed in India – HCL Technologies and HCL Infosystems. Founded in 1976, HCL is one of India’s original IT garage start-ups. A pioneer of modern computing, HCL is a global transformational enterprise today. Its range of offerings includes product engineering, custom & package applications, BPO, IT infrastructure services, IT hardware, systems integration, and distribution of information and communications technology (ICT) products across a wide range of focused industry verticals. The HCL team consists of over 109,643 professionals of diverse nationalities, who operate from 31 countries including over 505 points of presence in India. HCL has partnerships with several leading global 1000 firms, including leading IT and technology firms. For more information, please visit www.hcl.com

Increasing Cost Predictability in Performance Testing ...

Documents

Transcript of Increasing Cost Predictability in Performance Testing ...