Challenges and Best Practices in Enterprise Test Process Assessments

1. Abstract

Test Process Assessment can achieve various strategic and tactical objectives of an IT Organization. Assessments provide a methodical way of gathering the pain areas of a QA organization, and developing a practical road map to address these areas. An assessment model provides the framework against which the maturity or the level of capability of an organization or an entity can be measured. A methodology guides the organization in conducting

an assessment using the chosen assessment model. Tools can be

used to improve the efficiency of Test Process Assessment.

Assessment models have their own capabilities and limitations. The

skill and experience of the assessing consultants are the key

elements in addressing the areas that Assessment models cannot.

2. Introduction

The paper – Challenges and Best Practices in Test Process Assessment - provides an insight to the key aspects of Test Process Assessment such as customers’ expectations, assistance from models and caveats to keep in mind during an assessment.Four case studies of Test Process Assessment are presented in this whitepaper, which had different expectations and needs in each of them.

It is critical to choose the right level of information based on the

needs and expectations of the customers. The model that is chosen

to assess any Quality Assurance QA organization should be the

closest to the expectations. With only a few models present in the

literature that are of real help, it is up to the Assessment consultants

to decide how to leverage the structure of the models to achieve the

intended result of finding a solution to the customer’s problem.

Various models to assess the maturity of a Testing Process are listed

by T.C.Staab in “Assessing your testing process maturity” [1]. According to him the two models that can be used in their current format are Software Testing Maturity Model (SW-TMM) and Test Process Improvement ® (TPI) (www.sogeti.com). In his article, he further states that SW-TMM is found most suitable for his clientele. However, I have found TPI also to be of immense help.

2

3

3. Why do enterprises need Test Process Assessment?

A formal Test Process Assessment provides an enterprise with a

picture of where they stand with respect to industry best practices.

At times the benchmarking is done against an in-house baseline as

well.

An assessment also provides an insight to the strengths and

weaknesses of the QA Organization. It also provides insight to the efficiency of its interface with other communities like Business and Development. Assessments provide a methodical way of gathering the pain areas of a QA organization, and developing a practical road map to address these areas. Some of the common (but not comprehensive) pain areas are as follows:?Large number of defects found in User Acceptance Testing

(UAT) and/or in Production?Long testing windows – implying longer time to market ?Inconsistencies in test processes across QA organization

?Inefficient resource utilization – lack of flexibility and

productivity

?Immature automation frameworks and so on

Assessments also provide an opportunity for the IT Management to

gather feedback on the processes and methodologies that are

currently being followed.

Some IT organizations look at assessment as an opportunity to revisit their direction and their Operational Process Framework.There is a growing need from IT organizations for formal Test Process Assessments with one or more of the aforementioned objectives in mind.

Usually the senior management would want flexibility in viewing the facts and results from any initiative. A 20 KPA view might not be appropriate for all the assessments.

For example, one of the IT organizations wanted to improve their

Predictability, Flexibility, Efficiency and Effectiveness, and

Learning and Growth of their QA organization. A view with 20

KPAs of TPI did not help them in establishing their position and in

visualizing where they wanted to go with respect to the

aforementioned factors.

6. Model and Methodology

An assessment model provides the framework against which the

maturity or the level of capability of an organization or an entity can

be measured. It will have a standard set of information to be sought

which will provide an insight to the strengths and opportunities of

the assessed.

A methodology guides the organization in conducting an assessment using the chosen assessment model. It will assist in classifying a Testing Community into various horizontals and verticals for sampling purposes. The methodology will also assist in interacting with various stakeholders during the planning and assessment execution phases.

For example, using the same TPI® model one can simply send out the TPI questionnaire to the stakeholders to seek the information

For this customer a dashboard was created as depicted in Figure 1 with the data represented against the four focus areas. As the customer could easily correlate the facts and milestones with how they wanted to perceive and measure their QA organization, the

customer satisfaction was naturally very high. The Test Managers

and QA Directors also wanted to have granular details in the

dashboard, which would help them identify the minutest detail of

improvement. We changed the level of granularity and also mapped

the terminology to match their IT nomenclature.

4

Figure 1. Customer Specific Dashboard mapping back to KPAs

5

(which is very ineffective) or one can follow a detailed methodology, wherein the key stakeholders are more interactively engaged in interviews and discussions

Work experience

I have successfully conducted Test Process Assessments for IT

organizations having different sets of objectives.

Case Study A

The new management of a US based Insurance Company wanted

to assess the maturity of their QA organization with respect to the

Industry best practices in Testing and explore the various areas of

improvement.

Case Study BA Canada based major banking company with multiple Business Units, each one having multiple Portfolios, wanted to integrate its QA community and reduce the cost of testing operations by streamlining the Processes, Methodologies and Operational Framework.

Case Study CA US based insurance company wanted to introduce more

flexibility, consistency and efficiency across its QA organization

spanning their three (3) main Lines of Business.

Case Study D

A US based Credit Cards company with its QA Organization split

across multiple locations within the US wanted to reduce their End-

to-End testing cycle time, which was their main concern.

Methodology applied for the case studiesThe assessment methodology was different for each of the customers. The type of resources that were met or interviewed, the type and duration of interaction with each of them and so on had to be tuned to meet the required level of intricate information for each of these customers.

At a high-level the methodology comprises of the following four

phases:

6

Figure 2.Assessment Methodology - High level view

1) Preparation

?Getting and setting the expectations from the assessment – Current state against the End State

?Identifying the scope, and the Champions (from the customer’s QA Organization) for the engagement

?Categorizing the stakeholders, and selecting the interviewees

?Scheduling interviews

?Identifying artifacts – Process documents, project reports, guidelines, templates and so on

?Tailoring questionnaires

?Fine tuning, if any, of assessment tool kit to suit the specific objectives of the engagement

2) Assessment

?Discussing and following-up with the stakeholders – Business, Development, Testing, Environment Support and PMO

?Reviewing artifacts

?Documenting the observations from interviews and artifact reviews

?Documenting the findings (corroborated observations)

?Providing the Causal analysis of the findings

Preparation

Assessment

Recommendation

Road map

Figure 3.Causal analysis of Findings

?Playing back the findings to the interviewees for their approval

and buy-in

3) Recommendation?Preparing recommendations or remedies?Presenting the recommendations or remedies to the Champions?Internal sharing of the aforementioned recommendations for

the community’s buy-in

4) Implementation Roadmap?Developing a high level implementation road map for

implementing the necessary recommendations?Reviewing and approving of the road map

?Deployment of the recommendations according to the road

map is usually rolled out as set of multiple small and large

projects over a period of time.

?These phases are a natural sequence of activities to bridge the

gaps between the current state and the more desired end state.

However the rigor that goes in each of the above steps decides

the success or failure of the assessment. These details are not

covered in this paper.

Capabilities of Assessment models

QA organizations have issues with respect to high testing window, higher defects in UAT/Production, high lead time for testing team set up, turnover, high operating costs and so on.

There is no single, straightforward answer for the aforementioned issues. However, models can still help in channeling the thought

process and arriving at a feasible solution to address the

aforementioned issues. People usually are aware of the most of the

problems. Hence collection of information is not the primary

challenge in an assessment. The primary challenge lies in isolating

the ‘causes’ from the ‘effects’ and developing a solution to address

the causes. A format-free list of issues is not always sufficient to

form an efficient implementation plan to address the issues.

Assessment models put together a structure to capture the issues

and strengths, thus providing the means to produce a better structured solution out of it. A typical assessment model would provide high level information on the aspects of the domain that

7

8

one can consider. For example, TPI provides 20 dimensions such as

Test Strategy, Life Cycle, Estimation and planning and so on, while SWTMM provides 5 levels of maturity to the SWCMM maturity levels.

Assessment models also help consolidate information about a QA Organization in a unified fashion, thus marking the beginning of the streamlining of processes across various teams. They provide a platform to uniformly synthesize the information gathered from multiple sources. They also provide pointers to the key areas to focus on. Some models such as TPI® also provide suggestions for

improvement to achieve a particular level of maturity within a key

process area.

Limitations of Assessment models

When to have an assessment?

Assessment models cannot tell when to conduct an assessment.

The models do not usually guide the execution of an assessment, that is, the methodology part of an assessment. The methodology has to evolve out of the experience of a person or a team or a company.

How to handle assessment?Assessment models usually do not guide the handling of delicate issues such as ‘Resistance’ and ‘Culture’ during the assessment phase. These factors play a significant role in the success or failure of an assessment.

How to tailor the assessment model?

Assessment models usually have a standard set of definitions (like

the 20 KPAs of TPI or the 5 maturity levels and questionnaires of

SW-TMM). However, the models would not specify how to express

these definitions to suit the customer’s specific needs. They also

wouldn’t specify how to tailor these definitions when the

assessment has to be restricted to some specific areas (For example

– when assessing maturity of test automation).The skill and experience of the assessors determines how to modify the capabilities of the models and assess an organization against these modified models.

9

How to estimate the benefits?Not all models provide ‘suggestions for improvement’ to reach a particular level of maturity specified. Even the models like TPI® that provide suggestions for improvement do not support the estimation of the benefits of implementing the recommendations

suggested by them. In such cases, one has to heavily rely on the

experience of the consultants and the stakeholders (Test Managers

and so on) of the QA organization to arrive at the estimated benefits

specific to that organization.

How to prioritize and roll-out?

Assessment models do not provide guidance on prioritizing the

recommendations and rolling them out in the real world to bridge

the gaps between the current state and the desired state. Here, one has to entirely rely on project management and the execution experience of the assessors and the Champions from within the IT organization.

Bridging the gaps

The skill and experience of the assessors are the key elements that can bridge these gaps. My assessment experience has enabled us to

constantly improve our assessment methodology and supplement it

with various templates and tools. The methodology, the templates

and the tools have assisted us in filling these gaps, thereby providing

end-to-end consulting service capability. The updates to our

methodology and the refinements to the questionnaire are not

provided in this whitepaper. However two of the many tools that

were developed are discussed in this whitepaper.

Interview FacilitatorDuring assessment phase statements from interviewees and observations from artifact reviews would be noted down. These would be translated into ‘Findings’ when the same observation is obtained from at least one more source (individual statement or an artifact evidence). It is important that all the statements are corroborated to ensure they all make it to the ‘Findings’. This implies that during the interviews the interviewers have to focus primarily on all those statements that are yet to be corroborated before exploring the areas that are already well-corroborated.

To facilitate the same, a tool is created which takes input from each

interview and prioritizes the areas (and the statements within) that need to be focused in the subsequent interview. This provides a big advantage in terms of coverage of all areas of assessment and the

stakeholders’ statements against each.

Prioritization Streamliner

An assessment exercise often produces a big list of

recommendations (50-60 high level recommendations, sometimes

with up to 2-3 sublevels within each recommendation).

Each recommendation provides a value of its own. For example,

some could be very high in benefits; some could have very low implementation costs, while some others could be implemented in a short time frame. It will be difficult to arrive at an implementation plan unless the priorities of the organization are set upfront. It is also not practically possible to implement all the recommendations at the same time. It is necessary to understand the accommodating capacity of the organization in terms of the ‘magnitude’ and ‘nature’ of changes.

A tool was created to provide a means of streamlining the priorities

for the recommendations based on the importance of factors such

as benefits, cost and so on. The tool also provides guidance on

factors to be considered for prioritization and on the ratings for

each recommendation against each factor. This tool is well

supported by templates and examples from prior Assessment

exercises. The tool has enabled the sponsors and the Champions

from the QA organizations to arrive at the best possible

implementation road map for their organization.

Key Success Factors for a Test Process Assessment

Right ExpectationsAssessments are initiated by a sponsor from the Senior Management. It is very common to get the requirements from customers in the form of high level statements. Though most of the initial statements from sponsors were sounding very similar,

significant differences were observed in what they wanted to

achieve and how they wanted to view it.

It is very critical to get their expectations right. Hence detailed and

10

objective discussions with the customers are a must to ensure that their requirements are captured in a clear and measurable fashion. Our best practice is to share the deliverables from earlier

engagements and provide an indicative quality or format of

contents during the planning and preparation level.

Right Estimation

A complete cycle of Assessment is more complex than it initially

looks. The effort and time required from an organization should not

be underestimated. The effort has to be estimated not only for the

primary discussions but also for the follow-ups to get clarifications,

if any, on their artifacts and statements.

No underestimation of Existing ProcessesDuring the initial stages it is common to hear the words such as ‘We do not have any processes’. However, one should not underestimate the test processes in place. Even when the sponsors say they don’t have ‘any’ processes, there could still be ‘some’ processes practiced in remote spots of the organization and there could be a few people who find these processes indispensable. It is critical to take into consideration all these factors during the formulation of

recommendations and the implementation road map.

Effective Communication

Success of an Assessment depends mainly on how it is

communicated within the customer’s organization. An observation

from our case studies is that some stakeholders (usually at some

lower level of the organization or belonging to a different

department such as Development or Business) were not aware of

either the expectations and/or the progress of the assessment. It is necessary that the expectations as well as the progress of the assessment have to be communicated not only to the grass roots of the QA organization, but also to other relevant departments such as Development and Business.

Clarity of Objectives Members of QA organization might have some questions or concerns about motive of the assessment. A common observation is that people are concerned that the findings of the assessment

might negatively affect their performance appraisals. During

assessment it has to be made clear that the intention is to identify

problems with the process, and that the individuals will not be

targeted. To overcome this, the findings against any individual or a

11

group of individual should not be tagged. Confidentiality of the interviewees is a key factor in successfully extracting the required information.

Justification

Not all problems need a formal process assessment. A process

assessment involves lot of time and energy of a QA organization.

Some of the problems are better fixed locally than by going through

an assessment of entire QA organization. It is important to

understand the problem statement and its underlying factors/needs

and make a judicious decision on the need for a formal Test Process

Assessment.

Conclusion

The common challenges that the Testing organizations face are as follows:?Large number of defects in Production or UAT?Long testing window?High operational costs

?Inconsistent and immature testing practices within the

enterprise

There is a growing need for IT organizations to seek a formal Test

Process Assessment. Some of the key objectives behind the

assessments are as follows:

?Addressing the aforementioned challenges

?Benchmarking against industry

?Highlighting the strengths and weaknesses of the QA

community?Revisiting their directionAssessments provide a systematic way of analyzing the current state and to develop a roadmap to reach a more mature state. Assessment methodology has to be fine-tuned to suit the IT organizations’ specific needs.

In the paper we have elaborated on the capabilities and limitations of the Assessment models. The skill and experience of the assessors play an important role in bridging the gaps.

This whitepaper has also brought out some of the key success

factors for a Test Process Assessment. This whitepaper also covers

the usage of tools in improving the efficiency of an assessment.

12

Hello there. I am from HCL Technologies. We work behind the scenes, helping our customers to shift paradigms and start revolutions. We use digital engineering to build superhuman capabilities. We make sure that the rate of progress far exceeds the price. And right now, 59000 of us bright sparks are busy developing solutions for 500 customers in 20 countries across the world.

How can I help you?transform@hcl.in

13