Outsourcing, subcontracting and use of COTS

Torbjørn Skramstad

Contents

We will cover the following topics• Testing as a confidence building activity• Testing and outsourcing• Testing COTS components• Sequential testing• Simple Bayesian methods

Responsibility

It is important to bear in mind that• The company that brings the product to the

marketplace carries full responsibility for the product’s quality.

• It is only possible to seek redress from the company we outsourced to if we can show that they did not fulfill their contract

Testing and confidence

The role of testing during:• Development – find and remove defects.• Acceptance – build confidence in the component

• When we use testing for COTS or components where the development has been outsourced or developed by a subcontractor, we want to build confidence.

A product trustworthiness pattern

Product is trustworthy

Trustworthinessdefinition

Product related

Process related

People related

Environment definition

System definition

Means to create product trustBased on the product trust pattern, we see that

we build trust based on • The product itself – e.g. a COTS component• The process – how it was developed and

tested• People – the personnel that developed and

tested the component

A process trustworthiness pattern

Activity is trustworthy

Argument byconsideringprocess

Trustworthinessdefinition

Process definition

Team is competent

Method addressproblem

Process istraceable

Means to create process trust

• If we apply the pattern on the previous slide we see that trust in the process stems from three sources:– Who does it – “Team is competent”– How is it done – “Method addresses problem”– We can check that the process is used correctly –

“Process is traceable”

Testing and outsourcing • If we outsource development, testing need to

be an integrated part of the development process. Testing is thus a contract question.

• If we apply the trustworthiness pattern, we need to include requirements for– The component - what– The competence of the personnel – who– The process – how

Outsourcing requirements - 1

When drawing up an outsourcing contract we should include:

• Personnel requirements – the right persons for the job. We need to assess the CV for each person.

• Development process – including testing. The trust can come from– A certificate – e.g. ISO 9001 or a CMMI

assessment (3rd party assessment)– Our own process audits (2nd party assessment)

Outsourcing requirements - 2

Last but not least, we need to see and inspect some important artifacts:

• Project plan – when shall they do what?– Activities, Milestones, Toll gates

• Test strategy – how will they test our component requirements?

• Test plan – how and when will the tests be run?• Test log – what were the results of the tests?

Trust in the component

• The trust we have in the component will depend on how satisfied we are with the answers to the questions on the previous slide.

• We can, however, also build our trust on previous experience with the company. – The more we trust the company based on earlier

experiences, the less rigor we will need in the contract.

Testing COTS• We can test COTS by using e.g. black box

testing or domain partition testing.• Experience has shown that we will get the

greatest benefit from our effort by focusing on tests for– Internal robustness– External robustness

Robustness – 1

• There are several ways to categorize these two robustness modes. We will use the following definitions:– Internal robustness – the ability to handle faults in

the component or its environment. Here we will need wrappers, fault injection etc.

– External robustness – the ability to handle faulty input. Here we will only need the component “as is”

Robustness – 2

• The importance of the two types of robustness will vary over component types.– Internal robustness - components that are only

visible inside the system border– External robustness – components that are part of

the user interface.

Internal robustness testing

Internal robustness is the ability to• Survive all erroneous situations, e.g.

– Memory faults – both code and data– Failing function calls, including calls to OS

functions• Go to a defined, safe state after having given

the error message• Continued after the erroneous situation with a

minimum loss of information.

Why do we need a wrapperBy using a wrapper, we obtain some important

effects:• We control the component’s input, even

though the component is inserted into the real system.

• We can collect and report input and output from the component.

• We can manipulate the exception handling and effect this component only.

What is a wrapper – 1 A wrapper has two essential characteristics • An implementation that defines the functionality

that we wish to access. This may, or may not be an object (one example of a non-object implementation would be a DLL whose functions we need to access).

• The “wrapper” class that provides an object interface to access the implementation and methods to manage the implementation. The client calls a method on the wrapper which access the implementation as needed to fulfill the request.

What is a wrapper – 2 A wrapper provides interface for, and services to, behavior that is defined elsewhere

Fault injection – 1 • On order to test robustness, we need to be

able to modify the component’s code – usually through fault injection.

• A fault is an abnormal condition or defect which may lead to a failure.

• Fault injection involves the deliberate insertion of faults or errors into a computer system in order to determine its response. The goal is not to recreate the conditions that produced the fault

Fault injection – 2 There are two steps to Fault Injection:• Identify the set of faults that can occur

within an application, module, class, method. E.g. if the application does not use the network then there’s no point in injecting network faults

• Exercise those faults to evaluate how the application responds. Does testing the application detect the fault, is it isolated and does the application recover?

• Typically used in robustness testing

Software implemented fault injection

• Compile-time injection: Source code is modified to inject simulated faults into the system– Changes existing lines of code– Adds code. Use of perturbation functions. Simple functions

which take an existing value and perturb it

• Run-time injection– Corruption of memory– Intercepting operating system calls made by user-level

software and injecting faults into them– Network level fault injection. Corruption, loss or reordering

of network packets

Change / add code exampleChange code: a = a + 1 to a = a – 1

Add code:int pFunc(int value) { return value + 20; } int main(int argc, char * argv[]) { int a = pFunc(aFunction(atoi(argv[1]))); if (a > 20) { /* do something */ } else { /* do something else */ } }

External robustness testing – 1

Error handling must be tested to show that• wrong input gives an error message• the error message is understandable for the

intended users• application continues after the error with a

minimum loss of information.

External robustness testing – 2

External robustness is the ability to• Survive the input of faulty data – no crash• Give an easy-to-understand error message

that helps the user to correct the error in the input

• Go to a defined state• Continue after the erroneous situation with a

minimum loss of information.

Easy-to-understand message – 1

• While all the other characteristics of the external robustness are easy to test, the error message requirement can only be tested by involving the users.

• We need to know which info the user needs in order to:– Correct the faulty input– Carry on with his work from the component’s

current state

Easy-to-understand message – 2

• The simple way to test the error messages is to have a user to– start working on a real task – insert an error in the input at some point during

this task• We can then observe how the user tries to get

out of the situation and how satisfied he is with the assistance he get from the component (the error message).

Sequential testing*

In order to use sequential testing we need:• Target failure rate p1

• Unacceptable failure rate p2 and p2 > p1

• The acceptable probability of doing a type I (false positive) or type II decision error (false negative) – and hese two values are used to compute a and b, given as

1

lna

1

lnb

*Sequential testing is statistical testing where the sample size is not fixed in advance

Type I and Type II error

• A type I error occurs when the null hypothesis (H0) is true, but is rejected

• A type II error occurs when the null hypothesis is false, but erroneously fails to be rejected

Background - 1

We will assume that the probability of failure is Binomially distributed *. We have:

The probability of observing the number-of-defects sequence x1, x2,…xn can be written as

xnx ppx

nnpxf

)1(),,(

N

ii

N

ii xNnx

N pCpnpxxxf 11 )1(),,,...,( 21

*The binomial distribution is the discrete probability distribution of the number of successes in a sequence of n independent yes/no experiments

Background - 2

We will base our test on the log likelihood ratio, which is defined as:

For the sake of simplicity, we introduce

2

1

12

1

12

1

1

1lnln

),,(

),,(lnln

p

pxNn

p

px

npxf

npxf N

ii

N

ii

i

i

2

1

2

1

1

1ln,ln

p

pv

p

pu

The test statistics Using the notation from the previous slide, we

find that

We have p1, p2 << 1 and can thus use the approximations ln(1-p) = -p, v = (p2 – p1) and further that (u – v) = u

NnvaxvuNnvbabn

ii

1

)(ln

vu

Mvax

vu

Mvb N

ii

1

Sequential test – example

• We will use = 0.05 and = 0.20. This will give us a = -1.6 and b = 2.8.

• We want a failure rate p1 = 10-3 and will not accept a component with a failure rate p2 higher than 2*10-3. Thus we have u = - 0.7 and v = 10-3.

• The lines for the “no decision” area are– xi(reject) = - 4.0 + M*10-3

– xi(accept) = 2.3 + M*10-3

Sequential test – example

M

2.3

-4.0

x

4*103

Accept

Sequential testing - summary• In statistics, sequential analysis is a statistical analysis where the

sample size is not fixed in advance. Data are evaluated as they are collected. Further sampling is stopped in accordance with a pre-defined stopping rule as soon as significant results are observed.

• Testing software – e.g. p < 10-3:The method needs a large number of tests. It should thus only be used for testing robustness based on automatically generated random input.

• Inspecting documents – e.g. p < 10-1:The method will give useful results even when inspecting a reasonable number of documents

Simple Bayesian methods

• Instead of building our trust on only test results, contractual obligations or past experience, we can combine these three factors.

• One way to do this is to use Bayesian statistics.

• We will give a short intro to Bayesian statistics and show one example of how it can be applied to software testing

Bayes theorem

In a simplified version., Bayes’ theorem says that

When we want to estimate B, we will use the likelihood of our observations as our P(B|A) and use P(B) to model our prior knowledge.

)()|()|( BPBAPABP

A Bayes model for reliability

For reliability it is common to use a Beta distribution for the reliability and a Binomial distribution for the number of observed failures. This gives us the following results:

11 )1()1()|( ppppobsXP xnx

11 )1()|( xnx ppobsXP

Estimates

A priori we have that

If x is the number of successes and n is the total number of tests, we have posteriori, that

^

R

n

xR

^

Some Beta probabilities

Testing for reliability We use a Beta distribution to model our prior

knowledge. The knowledge is related to the company that developed the component or system, e.g.

• How competent are the developers• How good is their process, e.g.

– Are they ISO 9001 certified or CMMI assessed– Have we done a quality audit

• What is our previous experience with this company – Are there any changes?

Modeling our confidence

Several handbooks on Bayesian analysis contain tables where we specify two out of three values:

• R1: our mean expected reliability• R2: our upper 5% limit. P(R > R2) = 0.05• R3: our lower 5% limit. P(R < R3) = 0.05When we know our R-values, we can read the

two parameters n0 and x0 out of a table.

The result

We can now find the two parameters for the prior Beta distribution as:

• = x0

• = n0 – x0

if we run N tests and observe x successes then the Bayesian estimate for the reliability is:

R = (x + x0) / (N + n0)

Sequential test with Bayes• We can combine the info supplied by the

Bayesian model with a standard sequential test chart by starting at (n0 - x0, n0) instead of starting at origo as shown in the example on the next slide.

• We have the same number of tests necessary, but n0 of them are virtual and stems from our confidence in the company.

Sequential test with Bayes – example

M

2.3

-4.0

x

4*103

Accept

n0