Testing software

Team Software Development Project

20/04/23 2

Why testing ? Would you trust a completely automated nuclear power plant ?

Would you trust a completely automated pilot whose software was written by yourself ? What if it was written by your class ?

Would you dare to write an expert system to diagnose cancer. What if you are personally held liable in cases where a patient dies because of a malfunction of the software?

All software has faults!

20/04/23 3

Poor testing can cost lives

1. The Mars Climate Orbiter crashed in September 1999 because of a "silly mistake": wrong units in a program.

2. The 1988 shooting down of the Airbus 320 by the USS Vincennes was attributed to the cryptic and misleading output displayed by the tracking software.

3. Several 1985-7 deaths of cancer patients were due to overdoses of radiation resulting from a race condition between concurrent tasks in the Therac-25 software.

4. A 2008 Quantas Flight suddenly has to make an emergency landing due to a bug in the autopilot system – resulting in 115 injuries

(See http://www5.in.tum.de/~huckle/bugse.html for more)

20/04/23 4

Why testing ? “Perfect" software doesn't exist

All released software is imperfect

Software is complex & software engineering is an immature field

60% of errors introduced during design, 40% during implementation

Cost of error-correction goes up with life cycle stage 2/3 of design errors are discovered when the software is

operational.

Testing identifies failures and faults in the software

Some testing principles A programmer should not test his/her own program.

One should test not only that the program does what it is supposed to do, but that it does not do what it is not supposed to.

No amount of testing can guarantee error-free program.

Parts of programs where a lot of errors have already been found are a good place to look for more errors.

The goal is not to embarrass the programmer!

Before testing starts:Inspections and Walkthroughs Inspections and walkthroughs should be carried out

throughout the development process. Done by a team of people. Performed at a meeting

The programmer explains the program line by line. Questions are raised. Can find 30%-70% of errors.

NB: Extreme programming uses pair programming to build this into the core development process

Checklist for inspections

Data declaration

All variables declared?

Default values understood?

Arrays and strings initialized?

Variables with similar names?

Correct initialization?

Control flow

Each loop terminates?

Are errors handled?

Input/output

Format specification correct?

Are invalid inputs handled?

20/04/23 8

Types of testsrequirements

Analysis

Design

Implementation

Acceptance Test

System Test

Integration Test

Unit Test

Each phase of the development life-cycle must be tested

Mostly requires small test programmes to be written

More likely to be GUI based

20/04/23 9

Test strategies Testing only shows the presence or errors, not the absence.

Exhaustive testing is impossible with most software test all valid inputs test all invalid inputs test all edited inputs test all variations on input timing

Proving program correctness is almost impossible with most software

There need to balance the level of certainty of correctness against cost and impact of faults.

20/04/23 10

Classification of test techniques

According to the criterion to measure the adequacy of the set of test cases: Coverage based testing Fault-based testing Error-based testing

Based on the source of information used to derive test cases Black Box Testing (functional, specification-based) White Box Testing (structural, program-based)

All tests require a clear description of what the test does and what result is expected. i.e. Enter numbers 12345 in name field and press okay. Expect a ‘please

type in your name again’ i.e. pass value -1 for length parameter into GetNewAccount function.

Expect error ABC to be thrown.

20/04/23 11

Test Objectives Tests identify failures or faults in the software.

Failures are ways in which the software fails to run according to requirements. Note that some requirements will be implicit: Safety

requirements may not be stated!

Failures are caused by faults which are caused by errors in the developed software. 1 error may lead to many faults A failure may occur a "long way" from the original fault One fault may lead to many failures

Black-Box testing The testing is not based on the structure of the program

(which is hidden/unknown).

Theoretically, in order to ensure correctness, every possible input needs to be tested this is impossible.

The goal: to maximize the number of errors found for the available effort available for testing.

White-Box testing Is based on the internal structure of the program.

There are several alternative criteria for checking “enough” paths in the program.

Even checking all paths within a component does not guarantee finding all errors. Does not cover prior and post implications of errors.

Selection of test cases (for white-box testing)

Select a good coverage criterion. Completeness would require: Cover all paths of the program. Execute every statement at least once. Each decision has a true or false value at least once. Each condition is taking each truth value at least once. Check all possible combinations of conditions in each decision.

For this project, Select key components of the software for white-box testing

Select key paths for detailed testing. Focus on black-box testing for the rest.

Identifying black-box test cases

Partition the program into pieces of code with a single entry/exit point.

For each piece find which variables are set/used/tested.

Create test cases to check that the piece works according to expectation. E.g. handles unexpected

values passed in, functions called out of order etc.

20/04/23 16

Analysing test results

Once testing has discovered failures and faults, they need to be classified by impact on the software. e.g. Type A: System failure – a significant component of the software

does not work causing the whole system to fail. Type B: Module failure – a sub-set of the software does not work,

but does not impact the overall operation of the software Type C: Minor failure – some non-core functionality does not

work according to specification. Type D: Cosmetic issue: e.g. User Interface issues

20/04/23 17

Release criteria Before testing commences release criteria are set by the QA

manager which need to be met before the software can be released: Software should never be released with type A failures/faults.

Where faults/failures exist in the released software, either They need to documented in release notes The functionality associated needs to be removed from the

release

20/04/23 18

Test plan/results document

For this module, the test plan and test results documents have been combined.

All projects must complete the plan and results sections as part of their completed project.

Note: This is very simplified version of what is actually be used.