310414TESTING1 TESTINGTESTING 310414 SOFTWARE ENGINEERING 310414 SOFTWARE ENGINEERING.

59
310414 310414 TESTING TESTING 1 TESTING TESTING 310414 310414 SOFTWARE ENGINEERING SOFTWARE ENGINEERING

Transcript of 310414TESTING1 TESTINGTESTING 310414 SOFTWARE ENGINEERING 310414 SOFTWARE ENGINEERING.

310414310414 TESTINGTESTING1

TESTINGTESTINGTESTINGTESTING

310414310414SOFTWARE ENGINEERINGSOFTWARE ENGINEERING

310414310414SOFTWARE ENGINEERINGSOFTWARE ENGINEERING

310414310414 TESTINGTESTING2

TESTING OUTLINETESTING OUTLINE

The Unified Process and Testing — Overview

– Testing — Life Cycle Role

– Testing Overview

– Testing — Verification & Validation

– Testing Truths

The Unified Development Process — Testing Activities– Plan Tests

– Design Tests: White Box; Black Box; Regression

– Implement Tests

– Perform Tests: Unit; Integration; System; Acceptance

– Evaluate Tests

9

310414310414 TESTINGTESTING3

Inception Elaboration Construction Transition

TESTING LIFE CYCLE ROLETESTING LIFE CYCLE ROLE

PhasesCore Workflows

Requirements

Analysis

Design

Implementation

Testing

iter.#1

iter.#2

— — — — —iter.#n-1

iter.#n

Increments

Iter

atio

n

310414310414 TESTINGTESTING5

TESTING ARTIFACTS & WORKERSTESTING ARTIFACTS & WORKERS

responsible for

TestEngineer

ComponentEngineer

responsible for

TestCase

XTest

ModelTest

EvaluationTest

Procedure

XTestPlan

XDefect

TestComponent

IntegrationTester

SystemTester

responsible for

310414310414 TESTINGTESTING8

TESTING OVERVIEWTESTING OVERVIEW

The process ofThe process of finding differencesfinding differences between thebetween the specifiedspecified (expected) and the(expected) and the observedobserved (existing)(existing) system behaviorsystem behavior..

The process ofThe process of finding differencesfinding differences between thebetween the specifiedspecified (expected) and the(expected) and the observedobserved (existing)(existing) system behaviorsystem behavior..

usually done by developers that were not involved in system implementation

to test a system effectively, must have a detailed understanding of the whole system

not a job for novices

Goal:Goal: design tests that will systematically find defects

Our aim is to break the system (make it fail)

9.1

310414310414 TESTINGTESTING9

TESTING — VERIFICATION & VALIDATIONTESTING — VERIFICATION & VALIDATION

verification is the process of making sure that we have built the product right (i.e., it meets its stated requirements) most of the testing workflow is targeted at doing verification

validation is the process of making sure that we have built the right product (i.e., it is fit for its purpose) acceptance tests deal mainly with validation

testing verifies the results of implementation by testing each build as well as final versions of the system by– planning the tests required for each iteration

– designing and implementing the tests by creating test cases that specify what to test test procedures that specify how to test test components to automate the testing, if possible

310414310414 TESTINGTESTING10

TESTING TRUTHSTESTING TRUTHS

Testing cannot show the absence of software errors;it can only show that software errors are present!

It is impossible to completely test a nontrivial system.(Most systems will have bugs in them.)

Testing is a destructive activity.(We try to make the software fail)

Hence, good testing has the opposite natureof good software engineering

(i.e., good software engineering software does not failgood testing makes the software fail).

310414310414 TESTINGTESTING11

UP — TESTING ACTVITIESUP — TESTING ACTVITIES

IntegrationTester

TestEngineer

SystemTester

PlanTests

PerformIntegration

Tests

ImplementTests

PerformSystemTests

EvaluateTests

ComponentEngineer

DesignTests

PerformUnit Test

310414310414 TESTINGTESTING12

GoalGoal: design a set of tests that has thedesign a set of tests that has the highest highest likelihoodlikelihood of uncovering defects with theof uncovering defects with the minimum amountminimum amount ofof time and efforttime and effort..

GoalGoal: design a set of tests that has thedesign a set of tests that has the highest highest likelihoodlikelihood of uncovering defects with theof uncovering defects with the minimum amountminimum amount ofof time and efforttime and effort..

PLAN TESTSPLAN TESTS

WHY?WHY? Tests cost money and take time to develop, perform and evaluate (up to 40% of project effort often devoted to testing).

Testing strategy: specifies the criteria and goals of testing– what kinds of tests to perform and how to perform them

– the required level of test and code coverage

– the percentage of tests that should execute with a specific result

Estimate of resources required: human/system

Schedule for the testing: when to run which tests

9.5.1

310414310414 TESTINGTESTING13

DESIGN TESTSDESIGN TESTS

Test caseTest case:: one way of testing the system; one way of testing the system; specifies specifies whatwhat, , conditionsconditions, , howhow..

Test caseTest case:: one way of testing the system; one way of testing the system; specifies specifies whatwhat, , conditionsconditions, , howhow..

White Box Tests: “testing-in-the-small”Derive test cases to verify component logic based on data/control structures Availability of source code is required.

(Tests based on knowledge of internal workings of a component.)

Black Box Tests: “testing-in-the-large”Derive test cases to verify component functionality based on the inputs and outputs Availability of source code is not required!

(Tests based only on knowledge of specified functionality of a component.)

Regression Tests: selective White Box and Black Box re-testing to ensure that no new defects are introduced after a change

9.3.2

310414310414 TESTINGTESTING14

WHITE BOX TESTINGWHITE BOX TESTING

Goal:Goal: ensure that we have executed/exercised:ensure that we have executed/exercised:– all independent pathsall independent paths in the codein the code at least onceat least once– all logical decisionsall logical decisions on theiron their truetrue andand false sidesfalse sides– all loopsall loops at their boundaries and within their boundsat their boundaries and within their bounds– internal data structuresinternal data structures to ensure their validityto ensure their validity

Goal:Goal: ensure that we have executed/exercised:ensure that we have executed/exercised:– all independent pathsall independent paths in the codein the code at least onceat least once– all logical decisionsall logical decisions on theiron their truetrue andand false sidesfalse sides– all loopsall loops at their boundaries and within their boundsat their boundaries and within their bounds– internal data structuresinternal data structures to ensure their validityto ensure their validity

Why is this important?

logic errors and incorrect assumptions are inversely proportional to a path’s execution probability

errors more likely in less used parts of the code

we often believe that a path is not likely to be executed reality is often counter intuitive

typos are random its likely that untested paths will contain some

310414310414 TESTINGTESTING15

There are 1014 possible paths!It takes 3,170 years to exhaustively test the program if each test requires 1 ms!!

WHY NOT EXHAUSTIVE CODE TESTING?WHY NOT EXHAUSTIVE CODE TESTING?

loop < 20 timesHow long would it take to How long would it take to exhaustively test this program?exhaustively test this program?

310414310414 TESTINGTESTING16

Even though the false Even though the false branch executes no branch executes no additional statements, additional statements, it needs to be tested!it needs to be tested!

WHY NOT JUST STATEMENT TESTING?WHY NOT JUST STATEMENT TESTING?

i = j + 1if (A && B {

x = y + z;}n = p + 1;

i = j + 1

A && B

x = y + z

n = p + 1

TrueFalse

suppose we modify the program to:

i = 0if (A && B {

i = 1;}x = y / i;

310414310414 TESTINGTESTING17

WHITE BOX TESTINGWHITE BOX TESTING

Using White Box Testing we can do:

1. Basis Path Testing exercise each independent path at least once

2. Condition Testing exercise all logical conditions on their true and false sides

3. Loop Testing execute all loops at their boundaries and within their bounds

4. Data Flow Testing exercise all data structures to ensure their validity

310414310414 TESTINGTESTING18

BASIS PATH TESTINGBASIS PATH TESTING

9.4.2

1. Using the code, draw a corresponding flow graph.

(First drawing the activity diagram for the code may help, but is not required.)

2. Determine the cyclomatic complexity of the flow graph.

3. Determine a basis set of linearly independent paths.

4. Prepare test cases that force the execution of each path in the basis set.

Goal:Goal: exercise each exercise each independent pathindependent path at least once. at least once.Goal:Goal: exercise each exercise each independent pathindependent path at least once. at least once.

310414310414 TESTINGTESTING19

1. From the code, draw a corresponding flow graph.

BASIS PATH TESTINGBASIS PATH TESTING

Sequence

If-then-else

...

Case

Do-untilDo-while

9.4.2

310414310414 TESTINGTESTING20

BASIS PATH TESTING EXAMPLEBASIS PATH TESTING EXAMPLE

procedure: process records1. do while records remain2. read record;3. if record field 1 = 04. then store in buffer;5. increment counter;6. elseif record field 2 = 07. then reset counter;8. else store in file;9. endif10. endif11. enddo

end

310414310414 TESTINGTESTING23

BASIS PATH TESTINGBASIS PATH TESTING

2. Determine the cyclomatic complexity of the flow graph.

cyclomatic complexity: a quantitative measure of thelogical complexity of code

Ways to compute cyclomatic complexity V(G):

V(G) = the number of regions (areas bounded by nodes and edges—area outside the graph is also a region)

V(G) = the number of edges - the number of nodes + 2

V(G) = the number of (simple) predicate nodes + 1

provides an upper bound on the number of pathsthat need to be tested in the code

310414310414 TESTINGTESTING25

BASIS PATH TESTINGBASIS PATH TESTING

3. Determine a basis set of linearly independent paths.

independent path any path that introduces at least one new set of processing statements or a new condition

basis set set of independent paths through the code

test cases derived from a basis set are guaranteed to execute every statement at least one time during testing

basis set is not unique

310414310414 TESTINGTESTING26

1-2-3-6-7-9-10-1-11

BASIS PATH TESTING — EXAMPLEBASIS PATH TESTING — EXAMPLE

1-2-3-6-8-9-10-1-11 1-2-3-4-5-10-1-11 1-11

1

91011

2

4

587

3

6

310414310414 TESTINGTESTING27

BASIS PATH TESTINGBASIS PATH TESTING

4. Prepare test cases that force the execution of each path in the basis set.

Notes:

you do not need an activity diagram, but the picture will help when you trace component paths

count each logical test—compound tests count as the number of Boolean operators + 1 (i.e., count each simple predicate)

basis path testing should be applied to all components, if possible, and to critical components always

Basis path testing does not test all possible combinations of all paths through the code; it just tests every path at least once.

310414310414 TESTINGTESTING28

ASU — BASIS PATH TESTINGASU — BASIS PATH TESTING

checkRegistration Procedure

A student can register for a course if he has taken the prerequisites. However, he may not register for a course if he has already taken a course that is an exclusion. A student who does not have the prerequisites can register for a course if he is currently registered in the prerequisite course and has the permission of the instructor. Notwithstanding the preceding, the instructor may waive the prerequisite for a student.

checkRegistration

If (enrollment is open) AND (student does not have exclusions) AND ((student has prerequisites) OR (student has permission))Registerreturn(success)

Elsereturn(failure)

Endif

310414310414 TESTINGTESTING29

IS BASIS PATH TESTING ENOUGH?IS BASIS PATH TESTING ENOUGH?

int Q;. . .i = j + 1;if ( A() && B() ) {

x = y + z;}n = p + 1;. . .

Boolean A() { Boolean B() {if (Flag1) { if (Flag2) {

Q = 0; return TRUE;return TRUE; } else {

} else { int x = 10/Q;return FALSE; return FALSE;

} }} }

i = j + 1

A()&&B()

x = y + z

n = p + 1

TrueFalse

Flag1=FALSE Flag1=TRUEFlag 2=TRUE

What happens when Flag1=TRUE and Flag2=FALSE?

This isThis issufficientsufficientfor basisfor basispath testingpath testing

310414310414 TESTINGTESTING30

CONDITION TESTINGCONDITION TESTING

Goal:Goal: further exercises the further exercises the true and falsetrue and false value of value of each each simple logical conditionsimple logical condition in a component. in a component.

Goal:Goal: further exercises the further exercises the true and falsetrue and false value of value of each each simple logical conditionsimple logical condition in a component. in a component.

1. Logical conditions

simple condition: (a rel-op b) where rel-op={<, ≤, =, ≠, ≥, >} (may be negated with NOT), e.g., a≤b; NOT(a≤b)

compound condition: two or more simple conditions connected with AND, OR, e.g., (a>b) AND (c<d)

relational expression: (E1 rel-op E2) where E1 and E2 are arithmetic expressions, e.g., ((a*b+c)>(a+b+c))

errors to test for include (incorrect/missing/extra):– Boolean operator – relational operator– Boolean variable – arithmetic expression– Boolean parenthesis

310414310414 TESTINGTESTING31

2. Branch testing

for a compound condition C, test true and false branches of C and every simple condition of C

e.g., for C = (a>b) AND (c<d) we test for: C TRUE and FALSEa>b TRUE and FALSEc<d TRUE and FALSE

CONDITION TESTING (cont’d)CONDITION TESTING (cont’d)

3. Domain testing

for an expression E1 rel-op E2, test for E1 greater than, equal to, or less than E2

– guarantees detection of rel-op error if E1 and E2 are correct

– to detect errors in E1 and E2, the difference between E1 and E2 for the tests E1 greater or less than E2 should be as small as possible

– for an expression with n variables, 2n tests are required

310414310414 TESTINGTESTING32

Goal:Goal: execute execute loopsloops at their boundaries and within their bounds. at their boundaries and within their bounds.Goal:Goal: execute execute loopsloops at their boundaries and within their bounds. at their boundaries and within their bounds.

2. Nested Loops1 conduct simple loop tests for the innermost loop

while holding the outer loops at their minimumiteration

2 work outward, conducting tests for the nextinnermost loop

3 continue until all the loops have been tested

tests grow geometrically as the level of nesting increases

LOOP TESTINGLOOP TESTING

1. Simple Loops (n iterations)1 skip the loop entirely2 only one pass through the loop3 two passes through the loop4 m passes through the loop where m < n5 n-1, n, n+1 passes through the loop

310414310414 TESTINGTESTING33

LOOP TESTING (cont’d)LOOP TESTING (cont’d)

3. Concatenated Loops independent loops simple loop testing dependent loops nested loop testing

4. Unstructured Loops redesign!!!

310414310414 TESTINGTESTING34

DATA FLOW TESTINGDATA FLOW TESTING

Goal:Goal: ensure that the ensure that the value of a variablevalue of a variable is correct at is correct at certain points of execution in the code.certain points of execution in the code.

Goal:Goal: ensure that the ensure that the value of a variablevalue of a variable is correct at is correct at certain points of execution in the code.certain points of execution in the code.

select test paths according to the locations ofdefinitions and uses of a variable

DEF(S) = {X | S contains a definition of X} locations of definition of X

USE(S´) = {X | S´ contains a use of X} locations of use of X

Data Use (DU) Chain (X) is the set of [X, S, S´] where X is “live”(i.e., not redefined between S and S´)

One testing strategy: every data use chain must be covered once for every variable, do a test along the path from where the

variable is defined to the statement(s) where the variable is used (can be combined with basis path testing)

310414310414 TESTINGTESTING35

BLACK BOX TESTINGBLACK BOX TESTING

Black box techniques attempt to find:– incorrect or missing functions– interface incompatibility errors– data structure or external database access errors– performance errors– initialization and termination errors

Codeinput

actual ? predictedoutput = output

Test cases should:– reduce by more than 1 the number of additional test cases required (i.e.,

a test case should cover a range of input or output values)– tell us something about presence or absence of a class of errors (e.g., all

character data is correctly/incorrectly processed)

310414310414 TESTINGTESTING36

all possible input/output values

EQUIVALENCE PARTITIONINGEQUIVALENCE PARTITIONING

For each partition we design test cases which are based on valid and invalid inputs/outputs.

userqueries

numericaldata output

formatrequests

commandkey input

mousepicks onmenu

responsesto prompts

PartitionPartition inputs and outputs to create inputs and outputs to create thorough test coverage of a thorough test coverage of a class of errors.class of errors.

PartitionPartition inputs and outputs to create inputs and outputs to create thorough test coverage of a thorough test coverage of a class of errors.class of errors.

9.4.2

310414310414 TESTINGTESTING37

not member of set

SELECTING EQUIVALENCE CLASSESSELECTING EQUIVALENCE CLASSES

1. If input is a range one valid and two invalid equivalence classes:

2. If input is a specific value one valid and two invalid equivalence classes:

3. If input is a set of related values one valid and one invalid class:

4. If input is Boolean one valid and one invalid class:

in range greater than rangeless than range

value greater than valueless than value

member of set

BooleannonBoolean

310414310414 TESTINGTESTING38

userqueries

SELECTING TEST VALUES — BOUNDARY VALUESSELECTING TEST VALUES — BOUNDARY VALUES

As well as selecting test data “inside” an equivalence class, we also select data at the “edges” of the equivalence class.

numericaldata output

formatrequests

responsesto prompts

commandkey input

mousepicks onmenu

More errors occur at the More errors occur at the boundariesboundaries of an of an input/output domain than in the “input/output domain than in the “centercenter”.”.

More errors occur at the More errors occur at the boundariesboundaries of an of an input/output domain than in the “input/output domain than in the “centercenter”.”.

9.4.2

310414310414 TESTINGTESTING39

not member of set

BOUNDARY VALUE SELECTION — GUIDELINESBOUNDARY VALUE SELECTION — GUIDELINES

1. If the input is a range and is bounded by a and b, then use a, b, and values just above and just below a and b, respectively.

2. If the input is a number of discrete values, use the minimum and the maximum of the values and values just above and just below them, respectively. (Can also be applied to a single specific input value.)

3. For inputs that are a set of values, test all values in the set (if possible) and one value outside the set.

a b

min max

member of set

310414310414 TESTINGTESTING40

BOUNDARY VALUE SELECTION — GUIDELINES BOUNDARY VALUE SELECTION — GUIDELINES (cont’d)(cont’d)

5. Apply guidelines 1 and 2 to create output values at the minimum and maximum expected values.

e.g., if the output is a table, create a minimum size table (1 row) and a maximum size table

6. If data structures have boundaries, test these boundary values and values just above and just below them, respectively.

e.g., for an array of bounds 1 to 10 — test array index equal to 0, 1,2, 9, 10, 11

1 100 2 9 11

4. For inputs that are Boolean, test for both Boolean values (T, F) and for a non-Boolean value.

nonBoolean Boolean

310414310414 TESTINGTESTING41

ASU — BOUNDARY VALUE SELECTIONASU — BOUNDARY VALUE SELECTION

Student Record Search Function

In the ASU System, the Maintain student information use case uses a function that searches the database for the record of a given student. The input to the search function is a student ID. The output is either a success indication and the student record containing the given student ID or a failure indication and a message indicating the nature of the failure. The valid range of student IDs is from 1000000 to 9999999.

What test cases and test values would you use to test whether the search function works correctly and accepts only valid student IDs?

310414310414 TESTINGTESTING42

THREAD TESTINGTHREAD TESTING

event-based approach where tests are based on events which trigger system actions particularly appropriate for object oriented systems

used after classes have been unit tested and integrated into subsystems

need to identify and execute each possible processing thread from use cases

may not be possible to do complete thread testing too many input/output combinations

focus on most commonly exercised threads basic paths through flow of events

310414310414 TESTINGTESTING43

THREAD TESTINGTHREAD TESTING

single thread: obj3obj2obj4

obj1

obj3

obj2

obj4

obj5

I1(obj1)

I2(obj1)

I3(obj1)

I1(obj3)

I1(obj2)

O1(obj4)

O2(obj4)

O1(obj5)

multiple thread: obj1obj2{obj5, obj4} multi-input thread: into obj1

310414310414 TESTINGTESTING44

STATE-BASED TESTINGSTATE-BASED TESTING

focuses on comparing the resulting state of a class with the expected state

derive test cases using statechart diagram for a class

derive a representative set of “stimuli” (events) for each transition

check the attributes of the class after applying a “stimuli” to determine of the specified state has been reached

first need to put the class in the desired state before applying the “stimuli”

9.4.2

310414310414 TESTINGTESTING45

Comparison Testing– when reliability is absolutely critical, write several versions of the

software — perhaps to be used as backups– run same test data on all versions; cross-check results for

consistency Caution: if the error is from a bad specification and all

versions are built from the same specification, then the same bug will be present in all copies!

Cause-Effect Graphing1. causes (input conditions) and effects (actions) are identified for a

subsystem2. a cause-effect graph is developed3. the graph is converted to a decision table4. decision table rules are converted into test cases

OTHER BLACK BOX TECHNIQUESOTHER BLACK BOX TECHNIQUES

Error Guessing– use application domain experience or knowledge to select test values

9.4.2

310414310414 TESTINGTESTING47

IMPLEMENT TESTSIMPLEMENT TESTS

Goal:Goal: automate test procedures as much as possible. automate test procedures as much as possible.Goal:Goal: automate test procedures as much as possible. automate test procedures as much as possible.

running test cases can be very tedious and time consuming– many possible input values and system states to test

test component: a program that automates one or several test procedures or parts of them

there are tools available to help write test components:– perform actions for a test case and the tool “records” the actions

– parameterize the recorded script to accept a variety of input values

spreadsheets and/or database applications can be used to store the required input data and the results of each test

310414310414 TESTINGTESTING48

The time uncertainty in testing is the debug part!

PERFORM TESTSPERFORM TESTS

Softwareconfiguration

Testconfiguration

Testresults

Expectedresults

Errors

Errorratedata

Corrections

PerformTest

Evaluate

Updatereliability& quality

model

Debug

Done

No errors

9.2

310414310414 TESTINGTESTING49

PERFORM TESTS — TESTING STRATEGIESPERFORM TESTS — TESTING STRATEGIES

We need to integrate test cases into a well-planned series of steps that are taken to test a component/subsystem test procedure– what are the steps that need to be conducted to test a component?– when are the steps planned and undertaken?– how much effort, time and resources will be required to do the steps?

A testing strategy indicates which testing techniques (white box, black box, thread) are appropriate at which point in time

We need to balance:flexibility & creativity with planning & management

Testing often occurs when deadline pressures are most severe.(Progress must be measurable and problems identified as early as possible.)

310414310414 TESTINGTESTING50

A TESTING STRATEGYA TESTING STRATEGY

Requirements capture R

U Unit testing

AAnalysis

We developWe developthe systemthe systemfrom thefrom theoutside inoutside in

We testWe testthe systemthe systemfrom thefrom theinside outinside outDDesign

CImplementation(coding)

I Integration testing

SdSystem testing(developer)

ScSystem testing(customer)

310414310414 TESTINGTESTING52

Test cases

UNIT TESTINGUNIT TESTING

Emphasis is on White Box techniques.

interfacelocal data structuresboundary conditionsindependent pathserror-handling paths

Component

9.4.2

310414310414 TESTINGTESTING54

Test cases

UNIT TESTING — TESTING PROCEDURESUNIT TESTING — TESTING PROCEDURES

driver and/or stubs must be developed for each unit test

interfacelocal data structuresboundary conditionsindependent pathserror-handling paths

Driver

Stub StubResults

driver: component that calls component to be testedstub: component called by component to be tested

Componentto be tested

310414310414 TESTINGTESTING55

INTEGRATION TESTINGINTEGRATION TESTING

If components all work individually, why more testing?If components all work individually, why more testing?If components all work individually, why more testing?If components all work individually, why more testing?

Big Bang!! Incremental“builds”

Incrementalconstruction

strategy

Regressiontesting

Interaction errors cannot be uncovered by unit testing!(e.g., interface misuse, interface misunderstanding, timing errors)

Integration approaches

versus

9.4.3

310414310414 TESTINGTESTING56

S2S2

S5S4

S3

S6 S7S6

TOP DOWN INTEGRATION TESTINGTOP DOWN INTEGRATION TESTING

pro: early testing and error detection of user interface components; can demonstrate a complete function of the system early

con: cannot do significant low-level processing until late in the testing; need to write and test stubs

top subsystem is tested with stubs

stubs are replaced one at a time“depth-first” or “breadth-first”

as new subsystems are integrated, some subset of previous tests is re-run (regression testing)

S1

S3

S4

Usually not appropriate for OO development.(Except to test operations of classes.)

S7

S5

310414310414 TESTINGTESTING57

D2

BOTTOM UP INTEGRATION TESTINGBOTTOM UP INTEGRATION TESTING

pro: interaction faults are more easily found; easier test case design and no need for stubs

con: user interface components are tested last

drivers are replaced one ata time “depth-first”

subsystems are groupedinto builds and integrated

Appropriate for OO development.(Test classes use their own test driver.)

S2 S6

S1

S7

S3

S4 S5

310414310414 TESTINGTESTING58

S3

D2

pro: parallel testing activities

con: need many drivers and stubs

SANDWICH INTEGRATION TESTINGSANDWICH INTEGRATION TESTING

lower-level subsystems are groupedinto builds and tested with drivers

top-level subsystemsare tested with stubs

S2 S6 S7

S1

S3

S4 S5 Can significantly shortentotal testing time.

310414310414 TESTINGTESTING59

INTEGRATION TESTING — CRITICAL SUBSYSTEMSINTEGRATION TESTING — CRITICAL SUBSYSTEMS

Critical subsystems should be tested as early as possible!Critical subsystems should be tested as early as possible!Critical subsystems should be tested as early as possible!Critical subsystems should be tested as early as possible!

Critical subsystems are those that:

1. have high risk

2. address several software requirements (i.e., several use cases)

3. have a high level of control

4. are complex or error prone high cyclomatic complexity

5. have specific performance requirements

Regression testing is a must for critical subsystems!

310414310414 TESTINGTESTING60

SYSTEM TESTINGSYSTEM TESTING

We need to We need to test the entire systemtest the entire system to be sure the to be sure thesystem functions properly when integrated.system functions properly when integrated.

We need to We need to test the entire systemtest the entire system to be sure the to be sure thesystem functions properly when integrated.system functions properly when integrated.

Some specific types of system tests:

Functional developers verify that all user functions work as specified in the system requirements

specification

Performance developers verify that the nonfunctional requirements are met

Pilot a selected group of end users verifies common functionality in the target environment

Acceptance customer verifies usability, functional and nonfunctional requirements against system requirements specification

Installation customer verifies usability, functional and nonfunctional requirements in real use

9.4.4

310414310414 TESTINGTESTING61

PERFORMANCE TESTINGPERFORMANCE TESTING

stress tests verify that the system can continue functioning when confronted with many simultaneous requests How high can we go? Do we fail-soft or collapse?     

volume tests verify that the system can handle large amounts of data, high complexity algorithms, or high disk fragmentation

security tests verify that access protection mechanisms work make penetration cost more than value of entry     

timing tests verify that the system meets timing constraints usually for real-time and embedded systems             

recovery tests verify that the system can recover when forced to fail in various ways

database recovery is particularly important          

310414310414 TESTINGTESTING62

PILOT TESTINGPILOT TESTING

Alpha test: a test in a controlled environment so thatdevelopers can observe users

software

Developer site Customer site

customer tests

Beta test: a test in a real environment so that bugs areuncovered from regular usage patterns

Developer site

developerreviews

bug reportssoftware

Customer site

customertests

310414310414 TESTINGTESTING63

ACCEPTANCE TESTINGACCEPTANCE TESTING

The process of ensuring that the software The process of ensuring that the software meets the user’s meets the user’s reasonablereasonable expectations. expectations.

The process of ensuring that the software The process of ensuring that the software meets the user’s meets the user’s reasonablereasonable expectations. expectations.

Can be very subjective good requirements specification from the start is important!

The method of defect resolution should be negotiated before the transition phase!

acceptance tests (black box) cover classes of tests such as:– functionality – documentation– performance – reliability, etc.

configuration review (audit)– ensures that all elements of the software delivered to the customer

(the software configuration) have been properly developed and documented

310414310414 TESTINGTESTING64

EVALUATE TESTSEVALUATE TESTS

We need to– evaluate the results of our testing

– compare the results with the goals outlined in the test plan

– prepare metrics that let the test engineers determine the current quality of the software

How do we know when to stop testing?

We can consider:

– testing completeness: % of test cases run and % of code tested

– reliability: based on testing error rate of previous projects

310414310414 TESTINGTESTING65

TESTING ERROR RATETESTING ERROR RATE

Past history can be used to plot expected failure rate.We compare this with actual failure rate for this project.

Testing time

Fai

lure

s p

er t

est

hou

r

This project

Expected failure rate(based on previous projects)

Possible outcomes:• perform additional tests to find more defects• relax the test criteria, if set too high• deliver acceptable parts of the system;

continue testing unacceptable parts

310414310414 TESTINGTESTING66

SPECIAL CONSIDERATIONS FOR OO TESTINGSPECIAL CONSIDERATIONS FOR OO TESTING

What to unit test?– unit test has to be at least a class, but object state makes testing

difficult a class must be tested in every state it can ever enter

(i.e., use state-based testing)

Encapsulation– encapsulation hides what is inside an object hard to know its state

need to provide a method, for testing only, that reports all of an object’s state

Inheritance and polymorphism– If a subclass overrides methods of an already tested superclass, what

needs to be tested — only the overridden methods? No! All of a subclass’s methods need to be tested again

due to dynamic binding and substitutability

310414310414 TESTINGTESTING67

TESTING UNDER PRESSURETESTING UNDER PRESSURE

All test are importantAll test are important, but if you need to restrict testing … , but if you need to restrict testing … All test are importantAll test are important, but if you need to restrict testing … , but if you need to restrict testing …

testing a system’s capabilities (functionality) is more important than testing its components– Identify things that will stop users doing their jobs.

testing old capabilities is more important than testing new capabilities– Users expect existing features to keep working!

testing typical situations is more important than testing boundary cases– Normal usage patterns are more important than atypical ones.

310414310414 TESTINGTESTING68

TESTING SUMMARYTESTING SUMMARY

Testing requires good planning. know what you are trying to test for

Use the right test in the right situation. white box, black box, regression

Be systematic and thorough! unit, integration, system, acceptance

Decide beforehand how much testing will be enough. have clear criterion for when to stop testing