Isabel Evans eBook

Isabel [email protected]

Working ourselves out of a job a passion for improvement

This paper asks what is the purpose of a tester? Are we really needed? Should we be needed? If everything went well, if no-one made mistakes, if nothing needed improving, then testing and testers would not be needed. We only need testers because people make mistakes. What if we concentrated on preventing the mistakes instead of finding them after they have been made?

Presenting the Winner of the Best Conference Paper Award

at EuroSTAR 2010


1

Contents

Introduction 2

A challenge to our comfort from Capers Jones 3

What do we mean by test and tester? 4

Focusing on stakeholder quality goals 8

Making the case for preventative testing 9

Summary 12

References 12

PAG

E

www. e u r o s t a r c o n f e r e n c e s . c om

Isabel was presented with the EuroSTAR 2010 Best Paper Award on the evening of Wednesday December 1st at Copenhagen City Hall, Denmark. This accolade

was presented by Oracle, who sponsored the 2010 award. We would like to congratulate Isabel on her achievement and we look forward to the EuroSTAR

2011 awards in Manchester.


2

PAG

E

Introduction In October 2010, shortly after completing this paper, I become Quality Manager at Dolphin Computer Access. Dolphins range of computer software delivers independence to computer users with vision impairments and learning disabilities. (web: www.YourDolphin.com). Dolphins vision is to deliver independence to people with vision and print impairments all over the World I was excited to take this role for many reasons not least because it provided an opportunity to help an organization improve all its activities, not just to improve the testing of the products and services.

DOLPHINS VISION It is our vision to deliver independence to people with vision and print impairments all over

the World. With over 160 million visually impaired people and 600 million dyslexic

or learning disabled people, thats no small challenge. But were determined

to make a difference and have been working hard on that goal since Dolphin

started in 1986.

This vision, which is so broad and significant, provides a challenge to any test or quality practitioner working with the company. The paper reflects my personal move from test consultant back to being a quality manager after nearly 20 years. It is based on my own experiences as a tester, quality manager, consultant and trainer. It is also based on my research and reading across the IT industry and other industries. In this paper, while reflecting on my own experiences and challenges, I will discuss lessons that I believe apply to the whole IT and testing industry,

focusing on our need to improve. This need to improve is true whatever domain we are working in (Evans, EuroSTAR 2007). The three main lessons I want to discuss are:

Focus on improving the service to the customer, not on improving testing. Focus on enabling others to improve, not on increasing the need for our involvement. Focus our passion on the big picture while helping others to focus on the details.

In a previous company, in the 1980s I joined as the only specialist tester with about 200 developers and business analysts. As part of improving the companys products and services, I trained everyone in reviews and testing; quality became everyones responsibility, and the reviews and testing were carried out in an exemplary fashion. I became the company quality manager with a remit to advise everyone as required, coach and mentor across the company. Why then would a specialist test team be required? In writing this paper, before taking up my post at Dolphin, I am reflecting on those experiences and considering what approach is best to take to the quality activities in my new company.

TWEETABLE

Focus on improving the service to the customer, not on improving testing


3

PAG

E

A challenge to our comfort from Capers Jones In his 2009 book, Software Engineering Best Practices, Capers Jones make a bold and unsettling statement:

Poor software engineering, which gave rise to seriously flawed economic models, helped cause the recession. As the recession deepens, it is urgent that those concerned with software engineering take a hard look at fundamental issues: quality, security, measurement of results, and development best practices. Capers Jones, Software Engineering Best Practice

He suggests that we - the IT industry - must take our share of the blame for the recession. I suggest that, as quality managers or testers, we should consider what part our actions or inactions may have contributed, or the recession will lead to cutbacks which will compound our problems (Figure 1).

It is easy to point to poor software development as a source of the problem, but I believe that we should also consider what part poor software testing played in the delivery of flawed models. This will be in two parts: badly executed or mis-focused test efforts and bad measurement, metrics, information and reporting (figure 2). If the quality and testing department do not alert


4

PAG

E

our customer stakeholders to the risks of their actions including reliance on flawed models who will do that?

This failure that Capers Jones identifies the failure to provide a testing service that informs our stakeholders correctly of the risk includes, I suggest, both the failure to identify flaws in the models and failure to identify flaws in the software implementation of the models. Additionally I suggest that the testing activity has failed to provide a good service to stakeholders if the information it provides via measurement, metrics and identification of indicators, fails to convince stakeholders of the risks that have been identified. As it is a result of the flawed economic models that we find ourselves in recession, we need to consider what effect that recession will have on future modeling. Will the lessons from the recession include a willingness to fund improvement in the modeling and the IT systems? Will the

recession result in IT cuts which may further damage the ability of the IT teams to improve the models? In figure 4, I suggest that the failure to provide evidence for stakeholders, both about risk and about the worth of the test effort, may possibly result in the test effort being perceived as of no value and cut.

Indeed, if testing is fundamentally flawed in an organization and severe defects are resulting in live failure, it could be argued that the organization may as well cease to support a test activity with little ill effect. More dangerous is the situation where the test department actually performs useful testing, but is not perceived as providing any value. In that case, the test activities may be cut, resulting in additional defects in the live systems. However, there are solutions to these problems. In the rest of this paper I will discuss those solutions.

In the industry we commonly use the words Test, Tester and Testing in several ways.

For example Test and Testing can refer to activities, objects or to teams. The table on

What do we mean by test and tester?

Stakeholders

Customers

Managers

Builders

Supporters

Measurers

Society

People who (could) measure qualityCustomers e.g. Reqs Reviews, Task -based testing

Managerse.g. QP and metrics

Builders e.g. Design ReviewsTechnical test

Supporters e.g. Reqs ReviewsTech Test, OAT

Society e.g. Beta, PIR

DO WE NEED A

SPECIALIST ROLE HERE ANY MORE?

Quality ViewpointsCustomers USER , VALUE, TRANSCENDENT

Managers VALUE,TRANSCENDENT

Builders MANUFACTURING, PRODUCT,TRANSCENDENT

Supporters PRODUCT, USER, MANUFACTURING,TRANSCENDENT

MeasurersUnderstanding quality viewpoints?

Society USER,VALUE,TRANSCENDENT

Figure 3: Stakeholders for quality , people who can measure quality and quality viewpoints


5

PAG

E

the next page captures some usages I have employed or heard. I have not completely populated the table simply given a few examples showing that the same word can be a person, team, department, activity, description, identifier, concept, object or other qualifier. It is reasonable for people to use language in many ways. However, it is worth considering whether the industry and its customers best served by the usages which emphasise organizational teams and job titles. Alternatively, given the rise in agile cross functional teams, and the wise scope

of test activities carried out by teams and individuals without test in their titles, the words may best be used to describe activities and objects (tests, test environment, test plan, testing).

Each stakeholder typically has a different view of what constitutes success for a project, as demonstrated in the quotations from (Evans 2004) following the table. Figure 3 shows the stakeholders for testing each of which holds a different quality viewpoint as discussed in (Evans 2004).

Person, Team or Department An activity A description or identier A thing, object or concept

Test Test are holding a review

meeting I will test the input validation

Is the test environment ready?

Has the test completed successfully?

Tester Who is the tester for the

input validation? The tester mindset A type of bed

Testing Have Testing nished all their

work? Has the testing completed? This is a very testing activity All the testing is complete

Quality Quality are running the PIR Weve done the quality

stu A poor quality product Quality can be measured

QA QA will check the process Use QA to check the process The QA report has been

agreed

QC Jane is joining Quality Control The team have completed QC The QC tool set needs

installing

QM Quality management will

send someone to the meeting

We need to QM this programme

My QM plan includes root cause analysis

Table 1: Typical usages of the words test, quality and related words and phrase s

Customers: Why do so many IT people think the world was started with a requirements catalogue? Dont they understand theres more to my business than that... I want them

to understand what makes my business tick And why are they so arrogant and rude? Personally Id take on the guy who has fewer technical skills, if I thought he understood my

world, and he treated me like a human being. Two Customers for IT services compare notes on just why they hate

their IT departments

Managers: As a business, were crippled by our system! My Development Manager has just told me that removing the annual management charge from our Premium Account

is going to take 50 days effort. Our competitors do this sort of thing in five minutes. Unbelievable!

Awkward Chief Operating Officer (COO), lamenting those of us who dont think beyond implementation!

Measurer: Im going to buy a magic wand, and then when the Development Manager says to me Weve finished the build, now can you do the quality stuff I can just wave the

wand and make it happen Quality Assurance Manager complaining about the way quality activities are

regarded in projects.


6

PAG

E

When we look at what quality activities including measurement of quality - could be undertaken and indeed often are undertaken by each of the groups, we can see that we could consider not having a specific test/measurement role, provided all the required quality measurement is undertaken by other stakeholder groups. Each of the quality viewpoints (manufacturing, product, user, value and transcendent) can be covered by a different stakeholder group and each of the test phases or types could be owned by the other stakeholder groups. What value then does a specific, dedicated test role either an individual or a team bring to the organization? The reasoning for and against testers and test teams includes:

Bringing independence to the testing activity - Pro: it is easier to find someone elses mistakes - Con: may promulgate separation and confrontation rather than cooperation and team work; independence is a state of mind rather than organizational structure

Bringing specialist knowledge to the testing activity - Pro: specialist testers have particular

aptitudes and skills which mean that they will perform testing more efficiently and effectively than others - Con: the specialist knowledge and skills can be taught to others, and some specialist testers are not particularly skilled

Specific test phases and types may not obviously fit with any other group - Pro: in particular system test is sometimes seen as a half way house or quality gate between the more technical unit and integration test carried out by the developers and the task-based testing carried out in user and operational acceptance test - Con: the cost of maintaining a separate specialist group when the system test could be carried out by for example systems analysts trained in test methods.

Testing being seen as fundamentally an activity carried out on delivered code (dynamic testing) - Pro: exercises and finds defects in the delivered product - Con: late and expensive way to find defects.

Builder: So I spent 2 months designing and building the new interface and its underlying software. Its got everything we discussed with the users, and I added some extras that

looked useful, after Id talked to the customer last week. You know I was here all night last Thursday getting it finished for delivery into test on Friday. And then at 10:00am on Friday

morning the tester comes up to me and starts moaning about the interface being too complex. How long could she have spent looking at it? I ask you!.

Software engineer having a dramatic moment when the tester complained (rather rudely) about the complexity of the software interface

Supporter: My team is landed with picking up after this latest wow project. We need to run several manual batch jobs each day and Im getting beaten up because the support cost has gone up by 30% since implementation. I think the project team got a bonus for

this mess!Cynical Application Maintenance Manager wondering why no one involves

Supporters until after delivery!

Quotes from Evans 2004


7

PAG

E

As we see from the quotes above, quality specialists have long regarded the need for early static testing as essential to providing improved products as efficiently and effectively as possible. But my own observation from over 20 years in the industry is that the presence of a specialist team - especially if it is designated as a system test or acceptance test team, or a QA/QC - group is seems to reinforce an emphasis on late dynamic testing, by an isolated rather than independent team, who may not be very highly skilled, and whose work effectiveness is constricted by late deliveries. Many efforts have been made by test specialists to improve the way dynamic testing is done, but is that enough if it is a fundamentally expensive and ineffective way to find defects? It is worth considering a mental experiment in which we change the English language to see what happens if we restrict the way that the words test, tester, testing can be used. Suppose the following restrictions: Testing means an activity NOT a department or team

Testing is an activity undertaken by many people as part of their roles Tester is not a job title nor is it part of a job title.

The role of the quality/test specialist now becomes that of an advocate, advisor, coach and mentor. Job titles that have been suggested by various speakers and writers in the last few years include: Quality Engineer; Quality Manager; Quality Facilitator; Excellence Advocate and Customer Advocate (for example Tom McCoy at STARWest 2010 suggested Customer Quality Facilitator). We now have a number of activities which can be carried out by various stakeholders (see table 2 and 3 below).

Definition Person, Team or

Department An activity A description or

identifier A thing, object or

concept

Test A type of QC

I will test the input validation

Is the test environment ready?

Has the test completed successfully?

Audit A type of QA

John has audited the ABC

project The audit report has

been agrees The audit starts

tomorrow Tester A type of bed

Testing QC activities on products and

their specications to measurequality and report on risks

Has the testing completed?

This is a very testing activity

All the testing is complete

Quality

A set of attributes of a product or service, measured as

manufacturing, product, user, value or transcendent

properties

A poor quality product Quality can be measured

Quality Assurance

(QA)

Activities to identify whether a process or method is suitable in a particular circumstance

and whether it has been conformed to

Use QA to check the process

The QA report has been agreed

Quality Control (QC)

Activities to identify whether a product is suitable for use and

whether risks or defects are present

The team have completed

QC The QC tool set needs

installing

Quality Management

(QM)

Activities to plan for, carry out, check and report on QA and

QC, and to select appropriate We need to QM this programme of work

My QM plan includes root cause analysis

Table 2: Proposed usages of the words test, quality and related words and phrases

TWEETABLE

Over time, if the industry collects data about successful and unsuccessful projects our ability to identify useful indicators will improve


8

PAG

E

Activity Possibly part of the job of With help of Quality Specialist: Quality Assurance Audit Any SME can audit a peer group Excellence Advocate

Process review Any SME can review a peer group Excellence Advocate Process improvement projects Any stakeholder group Excellence Advocate

Quality Control Checking acceptance criteria All stakeholders represented Customer Advocate Requirements review Customer

IT Infrastructure Customer Advocate

Design review Development IT Infrastructure

Quality facilitator

Code review Development Quality facilitator unit test and TDD Development Quality facilitator Integration test Development Quality facilitator System test Development

IT Infrastructure Customer

Quality facilitator

System integration test Development IT Infrastructure Customer

Customer Advocate

User acceptance test Customer Customer Advocate Operational acceptance test IT Infrastructure Customer Advocate Test automation Development Quality facilitator Inspection. / review leader All stakeholders Customer Advocate

Quality Management (also includes QA and QC)

Quality planning Project managers/Scrum masters Customer Advocate Metrics Project managers/Scrum masters Customer Advocate Reporting Project managers/Scrum masters Customer Advocate

Advocacy Quality specialist Advising Quality specialist Coaching Quality specialist Mentoring Quality specialist

Table 3: Possible test and quality responsibilities within jobs

Focusing on stakeholder quality goalsFocusing our passion on the big picture and encouraging the whole team to deliver to the correct quality goals might include the quality specialist: Coaching and mentoring Business Analysts, Developers and Customers to do better testing. Involving ourselves in requirements/user story definition and review. Pair work with Business Analysts, Developers and Customers. Support for releasing early. Advising against a release. Root cause analysis of problems.

To do this well we need to focus on the

excellence goals for the organization as a whole, and ensure that IT is aligned to those goals. The organization might be using a balanced business score card (BBSC) as described in Kaplan and Norton or an excellence model such as the EFQM Model. These models can be used to model the IT response to the business goals (Evans 2005).

I have done this in a previous role: in 1987 I joined K3 Group Ltd as their first specialist tester. Clearly I was not going to do all the testing (there were about 150 developers and business analysts over three sites). Instead I helped everyone in the organization take responsibility for quality. We focused on document release using tailored review methods, entry and exit criteria for all activities, and root cause analysis for problems (solve dont blame). These processes had a number of important characteristics:


9

PAG

E

They applied to everyone in the company, including the Board, Sales and Marketing Each area of the business developed its own processes champions from each area worked on principles with me and then took the message back to their groups Project managers were given the responsibility for implementing quality practices in their teams, including selection and tailoring of processes from the central quality management system They included frequent re-planning points They were shared with the customers, who also were trained in the review process.

As a result of these actions, failures in live dropped, customer complaints dropped, revenue increased, market share increased, reputation increased and the type of customer and project became more prestigious. Test improvement alone would not have achieved this.

Making the case for preventative quality and testingIt can be difficult to make the case quantitatively for quality and early testing in a way that engages with the stakeholders for a project. However, the publication of Capers Jones book at the end of 2009 provides useful metrics based on 30 years of evidence gathered from IT projects, which allow us to make an approximation of numbers of defects in a product and the cost of finding those defects, Here is an example from Capers Jones book, based on C code. Note that figures are in the books for other programming languages. Table 4 below illustrates a project with typical project

manager questions, test metrics, and data from Capers Jones to provide estimated costs. The project managers goal is to reduce costs. Suppose we are running a C project and the predicted size for this project is 15000 lines of code (this is not necessarily the best size measure, but this example is simply to illustrate using published figures for modeling costs). We can use the tables of data in Capers Jones, plus any metrics we have about our past projects to calculate: the likely number of code and non-code defects; the cost of finding these defects by static and dynamic test; and the likely impact on project timescale of the static and dynamic tests. We can report this as a range. Tables 5 to 8 show this calculation. Note that I am using lines of code (LOC) as the estimated size of the project for ease, but Capers Jones also show function points, and it is possible to model with other early project sizing estimates.

I have supposed also for this model that we have some data from previous projects. Let us imagine that this data shows that we are finding more defects in live than Capers Jones ranges. The message to the project manager is clear: the best way to save time and money on the project is by carrying out early static testing. Get started early and we can spend 1.5 months and 48,000 dollars before release for a total cost of 69,000 dollars, with 40-80 defects reported post live or get started late and we can spend 2 months and 93,000 dollars before release for a total cost of 207,000 dollars, with 200-500 defects reported post live. The key point then is who carries out the static testing? Does the current test team have the required skill set? Do we train them, or do we train the requirements engineers, analysts and developers to carry out the static test, with quality specialists leading the review processes?

TWEETABLE

Focus our passion on the big picture while helping others to focus on the details


10

PAG

E

PM questions Useful metrics Test metrics that contribute Cost and time of last test project:

How much did it cost and how long did it take?

Why was it so expensive last time?

Why did it take so long? For the next test project:

Can I have it cheaper and better?

How much eort will it take? When will you nish? What resources do you need? And what will those cost?

Cost of testing: preparation / execution / managing; supporting activities; early preventative activities

Cost of failure to test eciently: e.g. man-days lost waiting for environment

Cost of failure to test eectively: e.g. time wasted doing or inappropriate wrong tests

Cost of failure: cost of defects/failures in live systems

Money saved by doing testing Money that could be saved by

improving

Number of signicant defects found during previous release testing, cost to repair those defects

Number of failures in live and cost to repair the damage from those defects

Time wasted Estimated cost of tests that would have

found those problems Size of previous release and size of next

release Predicted # defects in next release (see

CJ) Predicted cost to nd those defects

(see CJ) Predicted cost of not nding those

defects (see CJ) (CJ = Capers Jones: Software Engineering Best Practice)

Table 4: Questions and metrics regarding the cost of testing

Language Code size Non-code defects

Code defects Predicted total defects Defects/kloc

Example C 125,000 3000 2000 5000 5-25 (Capers Jones)

Our C project Est. 15,000 360 40 (our history)

Table 5: Capers Jones and our proposed project compared for size and potential defects

Capers Jones Example A

cost per LOC

Cost of our project with static test

Capers Jones Example B

cost per LOC

Cost of our project without static test

inspections 168,750.00 1.35 20,250.00

static analysis 81,250.00 0.65 9,750.00

dynamic test stages 150,000.00 1.20 18,000.00 775,000.00 6.20 93,000.00

total pre release 400,000.00 3.20 48,000.00 775,000.00 6.20 93,000.00

total post release 175,000.00 1.40 21,000.00 950,000.00 7.60 114,000.00

total cost 575,000.00 4.60 69,000.00 1,725,000.00 13.80 207,000.00

Table 6: Capers Jones and our proposed project compared by cost and test approach


11

PAG

E

Of course, these are just models; they will be inaccurate, but gathering data and analyzing it over time should improve our ability to predict indicators. It is similar to the indicators we have about the effect of smoking on health. Smoking increases ones likelihood of illness, but does not make illness the certain outcome. Not smoking decreases ones chance of illness but is not an absolute protection against illness. Over time, if the industry collects data about successful and unsuccessful projects our ability to identify useful indicators will improve.

Figure 4 below shows that the cost of failure for an organization is reduced as the cost of quality activities is increased, with a typical lowest cost point. An organization may be focused on time to market, regardless of defects, or on reducing costs, or on minimizing failures in live systems. Capers Jones figures may be used to inform decisions about testing activities that will help meet the organizational goals.

TIMESCALES CJ example A Our project +ST CJ example B Our project no ST

development schedule (months) 12 1.44 16 1.92

Table 7: Capers Jones and our proposed project compared by likely timescales, given test approach

LIVE FAILURES CJ example A Our project +ST CJ example B Our project no ST Total defects 5000 600 5000 600

found by static test 4150 498 0 0 found by dynamic test 500 60 1600 192

found post release 350 42 1900 228

Table 8: Capers Jones and our proposed project compared by likely live failures and test approach

TWEETABLE

Test and testing should be activities not job or team titles


12

PAG

E

SummaryThe evidence is now available to organizations to cost out what the likely impact of early static versus late dynamic test is for many types of project. The organizations goals in terms of total cost of quality, desire for early release of products and desire to protect from failure will indicate what value will be placed on the figures available to us and will inform the decision about what approach to quality is most worthwhile for organizations. We should consider whether we would serve our customers better by increasing the skill sets of all stakeholders in IT projects to enable them to carry out static testing more efficiently and effectively, rather than continuing the emphasis on large teams of testers. Test and testing should be activities not job or team titles. A smaller number of quality specialists could improve the IT delivery by: Focusing on improving the service to the customer, not on improving testing. Focusing on enabling others to improve, not on increasing the need for our involvement. Focusing our passion on the big picture while helping others to focus on the details.

References Evans 07 EuroSTAR papers 2007 and 2008 on development of the profession Evans 04 Achieving Software Quality Through Teamwork Capers Jones Software Engineering Best Practices Evans 05 STAREast 2005 Balanced Business Scorecard for Testing McCoy STARWest 2010 People Side of Testing Kaplan and Norton - Balanced Business Scorecard Kaplan and Norton - Alignment EFQM Model - see EFQM website www.efqm.org

TWEETABLE

Many efforts have been made by test specialists to improve the way dynamic testing is done, but is that enough if it is a fundamentally expensive and ineffective way to find defects?

TWEETABLE

Focus on enabling others to improve, not on increasing the need for our involvement

BiographyIsabel Evans has more than twenty years of experience in the IT industry, in quality management and testing, working in the financial, communications, and software sectors. Since the

mid-1980s, her quality management work has focused on encouraging IT teams and customers to work together, delivering results via flexible, customer-focused, risk- and test-driven processes designed and tailored by the teams that will use them. Isabel is a popular speaker at software quality conferences worldwide and has been a member of several working groups for industry improvement. Her publications include Achieving Software Quality Through Teamwork and chapters in Agile Testing: How to Succeed in an eXtreme Testing Environment; The Testing Practitioner and Foundations of Software Testing. Isabel is a Chartered IT Professional, a Fellow of the British Computer Society, and Quality Manager at Dolphin Computer Access.


13

PAG

E

The next EuroSTAR Conference will take place in Manchester from 21-24 November, 2011. The EuroSTAR 2011 Call for Submissions is now open and

if you wish to speak at the conference you can find out more by viewing the Call for Submissions details at www.eurostarconferences.com. The

deadline for the Call for Submissions is 25th February.

Join the conversation... If youve enjoyed this eBook, then come and interact with the

EuroSTAR Community! Youll find great Software Testing content, and great minds that are passionate about software testing

Follow us on Twitter @esconfsRemember to use our hash tag #esconfs when tweeting about EuroSTAR 2011!

Become a fan of EuroSTAR on Facebook

Join our LinkedIn Group

Keep up to date with the EuroSTAR Blog

Subscribe to our Newsletter - STARtester

Check out our free Webinar Archive

w w w. e u r o s t a r c o n f e r e n c e s . c o m

The EuroSTAR BlogAll the latest testing News

Newsletter

Isabel Evans eBook

Documents

Transcript of Isabel Evans eBook