Making the Case for Test Process

8/6/2019 Making the Case for Test Process

http://slidepdf.com/reader/full/making-the-case-for-test-process 1/13

Making the Case for Test Process

Testing engenders trust

mercuryinteractive.co.uk missiontesting.comrational.com/

worldwide/uk/ simgroup.comisintegration.com

white paper number 3 of a series

The leading organisations in the testing field

are supporting the formation of an independent

testing group within Intellect, the UK IT industry’s

representative body.

The aim is to promote best practice for the

benefit of customers and the whole IT industry.

We believe that working together will promotea sense of trust in the testing profession and help

leading organisations and customers to improve

the effectiveness of their critical IT projects.

This is the third in a series of White Papers

that will demonstrate the importance of testing

to senior management and will form the basis

of a best practice framework.

© 2004This document is for information purposes only. The authors make no warranties expressed or implied in this paper.

intellectuk.org

Non-SponsoringContributors:

IBM (UK) LtdQBIT Ltd

Satyam ComputerServices

Xansa Plc.



Executive Summary

Testing is increasingly being seen as a vital aspect of softwaredevelopment and delivery. However, ineffective testing can be

just as damaging as no testing, costing significant project time,

effort and money. Ultimately poor testing may fail to address

software quality and reliability, with the unacceptable result that

your customers are the ones who find and report the defects

in your software.

If testing is the right thing to do, how can you ensure that

testing is done right?

This white paper, the third in a series of white papers from theIntellect Testing Group, will explain how your organisation can

leverage the vast body of testing best practice that has been

accumulated and used by those organisations that do testing

right, helping to ensure you consistently reduce testing time,

effort and cost, and deliver higher quality software more quickly.

The white paper is structured as follows:

•Introduction - which reviews the challenges facing

organisations engaged in testing and discusses the

role of process in addressing them

• Making the Case for Test Process - provides a number

of success stories of companies adopting test process

• Process Implementation and Improvement - explains the key

elements of a test process with guidance on how to define

and implement the appropriate test process for your organisation

• Ensuring the Successful Adoption of Test Process - lists thekey do’s and don’ts of successful test process adoption

• Where can I go for Further Information - provides guidance

on where you can go to find further information on test process.

1

Table of Contents

white paper number 3 of a series Making the Case for Test Process

Executive Summary Page 1

Introduction

Making the case for Test Process Page 3

Process Implementation & Improvement Page 5

Ensuring the Successful

Adoption of Test Process Page 8

Where can I go for Further Information? Page 10

Conclusion Page 11

References Page 11

A brief definition of Test Process Page 12



Introduction

2

When you ask senior managers in organisations

developing software if they follow industry bestpractices in their testing activities, the vast

majority of them will confidently assure you

that they do. The reality is often far less clear,

and even where a formal scheme to follow

such best practice has been introduced in an

organisation, there are likely to be a variety of

different approaches being used that result in

varying levels of quality within the organisation.

This thought raises more questions: do you know

how much time, effort and money this testingcosts your organisation? Can you estimate just

how much risk is carried in terms of late delivery,

with poor testing resulting in the release of poor

quality software? Can you afford to run the risk

of customers refusing to accept such software,

or of serious defects being found during live

operation with costly consequences? Can you be

sure software is being tested thoroughly where

required and not being over tested? You may

recall from a previous white paper in this series

(Ref: 1) that the US National Institute of Standards

and Technology (NIST) reported that for every

£1Million spent on software implementations,

businesses incur more than £210K of additional

cost due to problems associated with the impact

of post-implementation faults (Ref: 2).

Furthermore, in today’s highly competitive

IT business, there are massive pressures on

companies to be as efficient as possible in

developing and delivering software solutions.If you don’t find strategies to reduce the cost

of software development, your competitors

will, allowing them to undercut your prices,

offer to develop and deliver products faster,

and ultimately, to take business from you.

Typically, companies put up with this situation

because they take a blinkered short-term view of

the projects they run; much better to just get on

with it and "make progress" than to take a more

enlightened, but longer-term view to actually

address the problems and fix them. Every project

has things that were done well and things

that were done poorly. An organisation shouldestablish a self-improving process where they

learn from mistakes and remember what made

a success.

Numerous organisations are now implementing

repeatable processes as the solution to these

problems. In this context, a process (such as

a testing process) provides a means of

disseminating industry best practice in software

quality and testing to all of the staff in your

organisation. Processes define who shoulddo what and when, with standard roles and

responsibilities for project staff, and guidance

on the correct way of completing their tasks.

Processes also provide standard re-usable

templates for things like test plans, test scripts

and testing summary reports, and even address

issues of process improvement (if you are

interested in delving deeper, a more

comprehensive definition of process can be

found later in this paper).

But talk is cheap; how many silver IT bullets have

we seen over the years that have turned out to

be blanks? And just because large numbers of

organisations have adopted, or are in the course

of adopting, a test process, this does not

necessarily mean that they are realising genuine

tangible benefits.

Just what hard evidence is there that such

organisations are actually succeeding withimplementing test processes?




3

Even where senior managers in organisations

appreciate that adopting a test process is theright thing to do, they will still need to convince

their colleagues that there is commercial benefit

to be gained; bottom line – what evidence is

there that other organisations are succeeding

with a test process, how much will a test process

save your organisation, how will it help manage

risk, and what return will you accrue from your

investment in adopting a test process?

This section presents a series of success stories

describing prominent organisations that havebenefited from the introduction and use of a

test process. These (and other success stories

available via the Intellect web site) can be used

as supporting material to make a case for the

benefits of test process.

Making the case for Test Process


Business Problem: Cap Gemini Ernst & Young(CGE&Y) recognised the need for a standard testprocess supported by integrated lifecycle tools andtest automation to ensure consistent global deliveryof software development services in order to reduce

risk and ensure quality.

Solution: The adoption of a proprietary softwaredevelopment and test process used in conjunctionwith an integrated set of software development andtesting tools delivered a series of tangible benefits:

• CGE&Y were able to meet client demands forconsistently-delivered global services

• Adoption of test process enabled the differentiationof CGE&Y’s services and improved their win-rate

• Adoption of test process enabled betterinformation sharing and development efficiency,resulting in reduced risk and costs

• CGE&Y successfully tested and delivered a majorsoftware solution to meet their client’s preciserequirements with no defects even after 5 monthsof live use

• Adoption of test process ensured more robustand flexible system architectures to be developedand reused with concomitant benefits in termsof reduced timescales and cost

Please read more about these andother test process success stories at:www.intellectuk.org/groups/testing.asp

Success Story 1Cap Gemini Ernst & Young achieve 50-75% reduction in test time and deliver zero defect system



Making the case for Test Process

Business Problem: Commercial pressures meantthat it was essential for a major high street retailer torelocate to new office premises. The need for a smoothand seamless move was identified as being key to thesuccess of the relocation project, with the staff beingable to continue using the company’s mission criticalsoftware without disruption following the move.As part of the move, it was also decided that thecustomer would update their core infrastructure toa Windows 2000 solution, both from a client andserver perspective. This situation was furthercomplicated by the need to migrate a large numberof software applications (some 300 in total), and toensure their successful integration and operation inthe new infrastructure.

Solution: Avanade employed testing industry bestpractice within the relocation project to ensure that themigrated applications would both function correctly inthe new infrastructure and interoperate successfully

with the other migrated applications. To this end arigorous test process was defined and documented,covering for example such items as checklists toensure that the migration success criteria were met.The documented test process was founded on arisk-based approach whereby applications formigration were graded on:

• The technical complexity of the changes applied• The business criticality of the software• The number of users affected

Other aspects of the test process included the use ofautomated testing tools to ensure that the base buildof the Desktop was not affected by the introduction ofenhanced/upgraded applications. Furthermore the testprocess verified that the applications could still co-existon the target infrastructure. The customer continuesto benefit from the successful application of the testprocess.

Success Story 2

Test Process Enables Avanade to Manage Successful Relocation of Major High Street Retailer

4


Business Problem : In the face of growing commercialcompetition and demand from their customers forinternet based private banking, LTPB were keen toinvestigate strategies for delivering high quality andhigh reliability web based software as quickly aspossible with as little disruption to their business aspossible; conservative estimates suggested deliveryof poor software could potentially cost LTPB millionsof pounds in lost business, possible litigation, as wellas the more intangible (but no less damaging) loss ofpublic confidence. Adoption of thorough and effectivetesting for the new technology solution was identifiedas being imperative.

Solution: A test process was designed that articulatedthe business-needs that initially were poorly defined.A pragmatic and disciplined approach to identify andeliminate software defects was then implemented. Keysupport in terms of test automation tools and resourcemanagement was incorporated into the process.A risk assessment strategy was also completed at thisstage. Adopting test process enabled LTPB to:

• Launch the web based offering with only threelow impact problems being discovered post delivery

• Achieve a high level of transfer of competencies andskills throughout their staff, ensuring all staff wereable to employ industry best practice in their testingtasks on all projects they were engaged on

Success Story 3 – Lloyds TSB Private Banking (LTPB) deliver mission critical privatebanking software with just three minor problems being observed post launch



5

The first thing organisations often find when they

try to understand a common test process is thatmany projects they have are quite different from

each other. Finding a common way to perform

testing seems impossible at an initial analysis.

This is to be expected as a project is made up

of a unique combination of technical platforms,

application systems and user organisations.

These factors play a large role in determining

the right process and test strategy. So how can

a common test process be established?

Policies and Principles

The place to start is to establish the goals for

testing and quality within the organisation overall.

The extent to which testing will be part of an

IT organisation or a business organisation is an

important executive decision. The quality goals

the organisation believes in are very important

as is the visibility the testing groups will have

throughout project life cycles. These core beliefs

and goals can be distilled into a short set of easy

to understand policies and principles for an

organisation. This common foundation for

quality and testing will apply to all projects.

Development and Implementation methods

There are various ways to develop and implement

systems. The primary ones involve strategic

decisions between building systems and buying

packages. It would not be uncommon to see

both types in an organisation but consciouslydecided for different systems. Another strategic

decision is the extent to which development and

implementation is done internally or outsourced.

This plays a dramatic role in deciding optimum

internal processes. For example the quality and

testing processes required for an in house

development of a sales system differ significantly

from the processes required for the outsourced

development of a CRM system.

Process Implementation and Improvement


Platform and technical environment

The platforms used by an organisation will also

vary according to technical decisions made over

a period of time and as new technologies are

introduced in the market. Whilst these variations

play a lesser role in determining a testing process,

they do influence how a testing process is

optimised for a given technology.

Project or programme instance

Each project will then be defined to do a specificset of developments or changes to application

systems that impact all or part of an organisation.

At this level the testing process should be more

prescriptive in determining the best practices to

be used by a project but must still be adaptive

to the needs of the project.

The common process

Finding the common process through all of these

variables involves starting with policies, then

making the policies clearer with development

strategies (buy verses build and in source verses

out source). This determines the need for quality

management and hands on testing processes.

The platform and the project determine what the

processes should look like.

Having recognised that your organisation will

benefit from the introduction and use of a test

process, you will still need to decide on a strategyfor adopting and delivering your process.

The main options can be summarised as follows:

1. Develop your own test process

from first principles

2. Get a specialist testing consultancy

to implement your process

3. Buy an off-the-shelf process

and customise it to meet your needs



6

Option 1

This option has the advantage that you shouldend up with a process that closely matches

your organisational requirements for developing

and testing software. However, it does have a

significant disadvantage; you will need to have

staff with suitable experience plus their time and

availability to define the process. You will also

need someone to maintain the process to ensure

it keeps up-to-date with changing needs and

technologies, and to ensure that you are at least

aware of developments in current industry

standards and best practice.

With the best intentions in the world, this strategy

frequently fails because staff are simply pulled off

the task of developing the process to "fire fight"

on higher priority projects (ironically, these

problem projects are often precisely the ones

that could benefit from process).

Option 2

This option involves engaging a specialist-testing

consultancy to develop and deliver a test process

tailored to your organisations particular testing

requirements.

Putting the development of your test process

in the hands of a reputable testing consultancy

means that there is a high probability that the

process will end up following industry standards

and best practice. Furthermore, the delivered

product should provide a very close match

to your organisational requirements for suchprocess. This strategy will still require

commitment and investment, but it has the

major advantage that it is unlikely to tie up

valuable internal resources.

Typically, such specialist companies will seek

to review and understand your testing

requirements, identify key areas for

improvements, and plan an incremental delivery

of elements of the test process. The goal of

this approach is to quickly deliver realisable

testing benefit, ensuring rapid return

on investment.

To ensure long-term benefits and continuing

return on investment, the process should be

designed to be easy to maintain in the event of

changes in technology and/or the development

of new methods or approaches to testing.

Advice on where to go to find further information

on specialist testing consultancies is provided

later in this white paper.

Option 3

This option involves the purchase of a

commercial-off-the-shelf test process.

While such products provide comprehensive

guidance on industry best practice, to gain

maximum benefit, they should be customised

to more closely match the specific testing

requirements of your own organisation (such tools

typically include guidance on customisation and

the facilities to adapt and maintain the process).

Assistance can also be obtained from the

process vendor or from specialist testing

consultancies, who are able to provide guidance

and assistance on the rollout and customisation

of such products.

In addition to guidance on testing best practice,

commercial test process products should also

include the means of delivering the process

(such as via an intranet browser interface),as well as industry standard downloadable

artefacts (document templates, pro-formas,

blank test scripts, etc). These features are

intended to help those organisations using such

products to realise the return on investment as

quickly as possible.

Advice on where to go to find further information

on off-the-shelf test process products is provided

later in this white paper.





7

Process Delivery

The most important success factor of deliveringa process is to get the right balance betweenintroducing change and commitment from thepeople that will use the process.

During the development of your process,you will need to consider how the informationit contains will be delivered to your staff,how they can be encouraged to make useof the process, and how the process will bemaintained and improved.

Delivery can be as simple as providing paperbased manuals, and there are numerousexamples of organisations successfullyimplementing such a strategy. One issue toconsider in adopting such an approach is thatthere is likely to be a significant maintenance costassociated with updating a paper based process,which relies on staff updating their own copies

with the inevitable risk of missed or lostupdates; the challenge of adopting paper basedprocesses is to prevent them ending up onshelves collecting dust rather than being usedas a day-to-day reference.

Another strategy is to provide access to theprocess electronically, often by means of yourorganisations own intranet system.

In this approach, the process is held on a centralserver and delivered to the staff on their ownworkstations using a browser interface. Updatescan be managed centrally with no danger of staffusing out of date information. Another powerfulfeature of such an approach is that electroniccopies of standard document templates canbe downloaded from a central repository andcustomised by staff for use on a project-by-project basis.



Process Improvement

No test process is ever static; technologychanges, new testing techniques are introduced,and your organisations test requirements willchange. Maintenance of your test process is anecessary part of ensuring it remains effective,efficient and relevant to your testing needs. Inparallel to the maintenance of the test process,you should also consider how to improve theprocess, looking at how defect detection ratescan be increased and cost of testing reduced, forexample.

Process improvement (Ref 8) is a topic that couldeasily justify a white paper in its own right;however, in the context of this paper we candefine process improvement as a strategy thatincludes both the means of introducing changesto the test process and the means of assessingthe success of those changes.

One example of a change to the test processcould be the introduction of automated testing

tools. Assessing the success of changes istypically achieved by the collection and analysis ofmetrics (Ref 7 provides guidance on theintroduction and use of metrics in a testingcontext plus a suggested metrics starter set).

Examples of typical metrics that are collectedinclude: how much effort was expended incompleting the testing project, how many defectswere found during testing, and how many defectswere found by the client following delivery of thetested system. Such information can be used todetermine whether the test process is improvingover time, by for example, comparing the currenteffectiveness in finding defects against previouslyobserved results, or for investigating the impact ofa specific change to the process.

Where your test process has been implementedby a specialist-testing consultancy, you shouldexpect the process to incorporate a scheme forprocess improvement. Similarly, for thoseorganisations adopting an off-the-shelf process

product, the tool should include guidance onprocess improvement.



8

You may believe you have implemented the best

test process in the world, but it may all have beenfor nothing if nobody in your organisation actually

uses the process. And worse, you will have

wasted the valuable time, effort and money spent

in researching, developing, documenting and

rolling out the process.

It is just as important to consider strategies for

successful adoption and use of the process, as it

is to "get the process right". This section

provides useful advice in the form of the most

common "do’s and don’ts" of process adoption.

• Do ensure commitment of all key decision

makers in your organisation ; too often

process implementations have started-up with

the highest ideals and goals, and then

foundered when senior management take

their eyes off the ball and are distracted by

a new endeavour, or are promoted, retire or

move sideways in the organisation.

• Don’t try to rollout your test process

across the whole of your organisation

in one go ; identify a suitable testing project

and consider piloting your process within that

project. Get it right on the pilot first, and then,

using the lessons learnt, plan a wider rollout.

• Do identify a Process Champion ;

such an individual, typically with a senior

management or technical profile and good

enthusiastic communication skills can beinvaluable in promoting the cause of test

process within your organisation. Such an

individual should be able to communicate at

all levels within the organisation and should

be sufficiently technically aware to be able

to make the case and argue for the adoption

and use of process.

• Don’t forget to plan for process training ;

ensure you budget for training and mentoring

of staff, and incorporate this training into the

delivery plan. Remember, delivering the training

too early will be as ineffective as delivering thetraining too late; ensure that staff are given

instruction in a timely manner so that they

can start to use the information as quickly

as possible on real projects.

• Do ensure good access to the information

contained within your process ; if you are

delivering a paper based process ensure it

is easily available to all the relevant staff.

Where appropriate, consider delivery of your

process using your intranet system to improveaccess and to simplify maintenance and update.

• Don’t allow the "not invented here"

syndrome to block process adoption ;

there are likely be other groups or individuals

in your organisation who already have an

interest in process and who may be unwilling

to support a new approach. Such groups or

individuals should be identified early during the

requirements phase of the development of

your process, and their views addressed within

the process; approached properly such

individuals can become powerful advocates

for process. Approached badly or ignored,

they can work passively or actively against

successful adoption of your process.

• Do promote success ; nothing succeeds

like success. Consider setting up a test

process special interest group that can

organise periodic meetings to discuss processissues and present successes. Newsletters,

your company intranet, and your process

champion can all be used to promote success,

reiterate the benefits of, and encourage

adoption and use of your process.

Ensuring the Successful Adoption of Test Process




9

• Don’t give up too soon ; although benefits

such as improved quality of delivered softwaremay be seen relatively quickly, the overall

monetary return on investment will take longer

to accrue. Many staff, particularly those with

an accountant mentality, may push for process

to be dropped in the absence of instant fiscal

results. Be careful to manage the expectations

of such colleagues; don’t present them with

an over optimistic view of how soon they will

see a return on investment, but instead

emphasise the other benefits that can be more

quickly realised.

• Do implement a process improvement

scheme ; ensure you collect metrics describing

the state of your testing before implementing

your test process (otherwise you will have no

basis to measure the improvements process

will bring). Continue to record metrics in order

to understand how continued use of the test

process or introduction of new features has

impacted your overall testing effectivenessand efficiency.

Ensuring the Successful Adoption of Test Process




10


If this white paper has caught your interest in test

process and you would like to find additionalinformation on the subject, this section contains a

number of useful sources of further information on

test process.

The following web sites are recommended as

useful sources of test process information:

• You can obtain general testing guidance,

further details of test process success stories,

and access to the other white papers in this

series at the Intellect Testing Group pages -www.intellectuk.org/groups/it/groups

/testing.asp

• The British Computer Society Special Interest

Group in Software Testing (BCS SIGiST)

provides a vigorous forum for test process

discussion - www.sigist.org.uk

• The Software Process Engineering Metamodel

(SPEM) is a proposal for a standard software

development and test process, which is

sponsored by the Object Management Group

(OMG). Ref 6 provides a specification

for SPEM. Also see

www.omg.org/technology/documents

/formal/spem.htm

Where can I go for Further Information?

There are a number of books that can be referred

to for further information on test process:

• Ref 7 provides a comprehensive view of the

test process, includes case studies showing

how organisations have adopted test process

tailored to their own test requirements,

as well as a comprehensive set of standard

testing templates.

• Ref 8 documents the Capability Maturity

Model (CMM) – an example of a scheme that

can be applied to test process improvement

• References 4 and 5 provide a good source

of information on testing fundamentals.

Details of organisations who can provide

specialist test process advice and guidance,

and suppliers of off-the-shelf testing products

can be found at

www.intellectuk.org/groups/it/groups

/testing.asp




11

Both this white paper and the previous one in this

series (Ref: 1), have made the case for testing asan activity that adds real and quantifiable value to

the software development life cycle.

As we said in the introduction, ineffective

testing can be as damaging as not testing at all,

and worse, you will have expended valuable

time and money on wasted effort. Test process

provides a means to ensure that your testing time

and money are spent effectively and efficiently.

A test process can encapsulate industry bestpractice in a form that can be used by all of your

staff, giving all members of your organisation the

means to perform their testing tasks to the levels

achieved by the most effective practitioners

in the testing field.

At the organisational level, test process provides

your company with the means to achieve savings

in time, effort and cost of testing, and improve

the quality and reliability of your deliveredsoftware. As a consequence, increasing numbers

of organisations are adopting test process in

order to accelerate delivery of software and

to provide themselves with a commercial

advantage over their competitors.

This white paper provides an objective view

on the benefits that adoption of test process

can deliver, as well as the means for you to

quantify the value of process in the context

of your own company.

The longest journey, as they say, starts with

but a single step; take that step by using

the information provided in this white paper

to investigate the benefits that test process

can bring to your own organisation.

In conclusion

1 Intellect Testing Group: "Realising the True Value of Testing" , Intellect - 2003.Available from the Intellect web site.

2 National Institute of Standards & Technology(NIST), US Dept. of Commerce: The EconomicImpacts of Inadequate Infrastructure forSoftware Testing – May 2002

3 IEEE 829-1983 Standard for Software TestDocumentation. Software Engineering

Standards Committee of the IEEE ComputerSociety, New York

4 "The Complete Guide to Software Testing - Second Edition" , Hetzel, B., QED InformationSciences Inc, Massachusetts, 1988.

5 "The Art of Software Testing" , Myers, G. J.,John Wiley, New York, 1979.

6 "Software Process Engineering Meta Model Specification" , the Object ManagementGroup Inc., November 2002.

7 "Testing IT: An Off-the-Shelf Software Test process" , Watkins, J., Cambridge UniversityPress, 2000.

8 "Key Practices of the Capability Maturity Model - Version 1.1" , Paulk, M. C., et al,Software Engineering Institute - CarnegieMellon University.

References



12


A software test process is a documented set of

best (or fit for purpose) practices describing howto achieve effective and efficient testing of

software applications.

A test process should not be a static document ,

but should be regularly reviewed and maintained

in order to incorporate new best practice or to

accommodate changes in technology and/or

approaches to software development and testing.

A test process should incorporate the means

to facilitate improvements to the process so thatchanges to the process can be measured and

analysed to determine the benefit, or otherwise,

of those changes.

A test process should also consider, address

and integrate with other aspects of software

development, such as requirements

management, configuration management,

change management and defect tracking.

Specifically, a software test process should

document (in no significant order):

• The tasks that need to be completed and

the order in which they are undertaken

(such as the need to develop a test plan)

• The inputs and outputs to and from tasks

(such as the software to be tested, the date

it will be available for testing, and the delivery

date for the tested software)

• Details of the testing phases(such as their names (Unit/Component test,

Integration/link/module testing, System testing,

etc), the order they take place in, and the

staff responsible for conducting them)

A Brief Definition of Test Process

• A glossary of relevant terms

• The roles and responsibilities of the testingstaff as well as their, reporting and liaison

(such as Test Manager, Test Designer,

and Tester).

• Any artefacts employed in the process,

such as:

• test plan documents

• test cases

• test scripts

• test logs

• test summary reports

• Specific guidance on best practice

(such as how to create realistic and

representative test data)

• The relationships/interfaces to other disciplines

(such as the need for access by the Test

Manager and Test Designer to the current

requirements for the software under test)

• Guidance on the use of specific techniques

and tools (such as the use of Boundary

Analysis in creating test data, or the role

and use of automated testing tools).

A test process should also demonstrate

awareness of other sources of testing information

(such as referencing test documentation

standards (such as Ref: 3)), definitive texts

(such as those by Bill Hetzel (Ref: 4) and/or

Glenford Myers (Ref: 5)), and process

improvement schemes (such as the Capability

Maturity Model scheme (Ref: 8)) for example).

1. Where document can include paper orelectronic means of delivering a test process.

Making the Case for Test Process

Documents

Transcript of Making the Case for Test Process