© 2011 Codenomicon Ltd. 1

Testing and the Bottom Line

A New Method to Estimate the Value of Good Software Testing

Juha-Matti Tirilä

Codenomicon Ltd.

Outline

• What are the problems in estimating the cost of testing

• What happens when code is tested well: the effect on bugfix price of early discovery

• Our method for estimating the cost impacts of software testing

This is why a functional cost model matters

Why develop a new cost model

• Problem: the cost structure of testing is typically misunderstood

• The difference in nature between testing and development– Cost of testing cannot be analyzed with the same

tools as the cost of development

Where do bugs come from

Cost of development

• Direct cost: we have to pay this to actually produce the desired outcome

Cost of testing

• Indirect cost: if we do not test properly, how much is it going to cost us

• Direct cost: the price of performing the testing – Plus the cost of fixing the issues? It depends.

Why does "it depend"?

• You may have found the bug otherwise just as well, only now you find it earlier, so there’s no extra cost

• Even though it costs you money, testing will have positive consequences beyond the particular bug– (That are very difficult to measure)

Testing acts as an enabler

Cost-benefits of early bug discovery

• Especially in a security context: vulnerabilities detected by third parties tend to be expensive due to – Negative reputation– Downtime– Increased need for customer support– Etc...

Cost-benefits of early defect discovery

• Thinking of security critical bugs: – Post-release vs. pre-release– Impact on accountability – Flexibility in resource allocation; no need to fix

straight away

Cost-benefits of early defect discovery

– The person who wrote the code in the first place still working

– Maybe even still remembers the logic of the code in question

– Organization-level practices in maintaining good quality, improved performance in the long run

Cost-benefits of early defect discovery

– Especially in security testing, it is a rather limited set of bugs that appear again and again so that testing: • Improves your knowhow• Accelerates production speed• With the best tools:

– very to-the-point reports on what types of bugs were discovered

– easy to fix bugs– educate developers to avoid similar mistakes in the future

Our method, graphically, phase 1

The model explained

• Product release at T = 6 months• The price of bugs increases until T, then drops

a little, starts to rapidly rise immediately afterwards

• Some bugs cost nothing• The effect of testing: ”all” bugs discovered

earlier so that the distribution tends to the left, and down

Comparison to traditional incident probability calculations

– One approach: try to determine the probability and cost of impact, and evaluate your testing budget against this expected loss

– Our approach: more statistically oriented, more geared towards estimating the average savings irrespective of whether issues ever surface or not

But you never have the data before it’s too late, right?

• Remark: the dual presentation done above is something you in reality never get. You have a certain level of testing, and you don't know what the other alternatives would have cost you. How to deal with this problem?

• Solution: use statistical distributions and models, and tune the parameters of the model to reflect your development environment.

A peek at the parameters

– Agree upon a trend for how the cost of a bug fix will develop towards release, and after release

– Agree upon the way testing is going to affect discovery times of issues

– Agree upon "variance", i.e. how widely the cost per delivery time is distributed

The example revisited, with statistical distributions

The example, continued

Benefits of using distributions

• Very little immediate effect on the estimated price of fixing bugs

BUT• Provides lots of additional information• The mechanism is easy to understand and

tune to match a particular development environment

In the long run, better estimates to rely on

Benefits of using distributions

• You can readily calculate various statistics, such as: – The expected cost of fixing an average bug, with

and without testing– The expected cost of fixing bugs detected in

pre/post -development phases– Etc... , to your liking, any statistics computable

from a statistical distribution – So for example also estimate risks

Challenges

• Of course, the quality of all the estimates depends on how well you tune the parameters

But:

• You can use real-life data to estimate the parameters

Thank you!