Testing Throughout the Software Lifecycle - Chap 2

7/28/2019 Testing Throughout the Software Lifecycle - Chap 2

1/77

Slide 1 Internal

Testing Throughout the Software

Life-cycle


2/77

Testing Throughout the Software

Lifecycle

Software Development Models

Test Levels

Test Types the Targets of Testing

Maintenance Testing

Summary


3/77

Software Development Models

The V-Model

Iterative Development Models

Testing within a Lifecycle Model


4/77

V-Model

User/Business

Requirements

System

Requirements

Technical

Specification

Program

Specification

Coding

Unit

Test

Integration

Test

System

Test

Acceptance

Test

Development

Levels

Test

Levels

Acceptance Test

Plan

System Test

Plan

Integration

Test Plan

Unit Test

Plan


5/77

V-Model

The benefits of the V Model include:

The testing phases are given the same level of management attention and

commitment as the corresponding development phases

The outputs from the development phases are reviewed by the testing team toensure their testability

Verification and validation (and early test design) can be carried out during the

development of the software work products

The early planning and preliminary design of tests provides additional review

comments on the outputs from the development phase


6/77

V-Model

The levels of development and testing shown in the model vary from project to

project

For example, there may additional test levels, such as System Integration Testing,

sitting between System Testing and Acceptance Testing (more on these test levels

later)

The work products coming out from any one development level may be utilised in

one or more test levels

For example, whilst the prime source for Acceptance testing is the Business

Requirement, the System Requirements (e.g. Use Cases) may also be needed to

support detailed test design


7/77


Iterative Development Establish Requirements Design the System Build the System Test the System

Achieved with small developments Iterations and Increment withinIterations

As Increments are developed and tested the System grows andgrows. Need for more testing with Regression Testing paramount

E.g. RAD, RUP and Agile development models

Agile development aim is to deliver software early and often

Rapid production and time to market Can handle (and anticipates) changing requirements throughout alldevelopment and test phases


8/77


Rapid Application Development

User

Requirements

Code

Acceptance

Test


9/77

Testing within a Lifecycle Model

Characteristics of good testing in any life cyclemodel:

A Test Level exists for every development Level Each Test Level has specific objectives

Test analysis and design for each Test Level beginsduring corresponding development Level

Early and proactive involvement of testers inreviewing development deliverables benefitsboth parties

Test Levels should be adapted depending onProject nature. May be better to combine TestLevels, e.g. with COTS testing.


10/77

Test Levels

Component Testing

Integration Testing

System testing

Acceptance Testing


11/77

Component Testing

User/Business

Requirements

System

Requirements

Technical

Specification

Program

Specification

Coding

Unit/Component

Test

Integration

Test

System

Test

Acceptance

Test

Development

Levels

Test

Levels

Acceptance Test

Plan

System Test

Plan

Integration

Test Plan

Unit Test

Plan


12/77

Component Testing

ComponentA minimal software item that can be tested in isolation.

Component TestingThe testing of individual software components.

Sometimes known as Unit Testing, Module Testing or Program Testing

Component can be tested in isolation stubs/drivers may be employed

Test cases derived from component specification (module/program spec)

Functional and Non-Functional testing

Usually performed by the developer, with debugging tool

Quick and informal defect fixing

Definition


13/77

Component Testing

Test-First/Test-Drivenapproach create the tests to drive the design and codeconstruction!

Instead of creating a design to tell you how to structure your code, you create atest that defines how a small part of the system should function.

Three steps:

1. Design test that defines how you think a small part of the software should behave(Incremental development).

2. Make the test run as easily and quickly as you can. Don't be concerned about the designof code, just get it to work!

3. Clean up the code. Now that the code is working correctly, take a step back and re-factorto remove any duplication or any other problems that were introduced to get the test to

run.

Russell Gold, Thomas Hammell and Tom Snyder - 2005

Definition


14/77

Integration Testing

Definition

Component Integration Testing

System Integration Testing


15/77

Integration Testing

Definition

Integration Testing - Testing performed to expose defects in the

interfaces and in the interactions between integrated components or

systems

Components may be code modules, operating systems, hardware and

even complete systems

There are 2 levels of Integration Testing




16/77


User/Business

Requirements

System

Requirements

Technical

Specification

Program

Specification

Coding

Unit/ComponentTest

Integration

Test

System

Test

Acceptance

Test

Development

Levels

Test

Levels

Acceptance Test

Plan

System Test

Plan

Integration

Test Plan

Unit Test

Plan


17/77


Definition

component integrationtesting Testing performed to expose defects in the

interfaces and interaction between integrated components

performed by the test team

usually formal (records of test design and execution are kept)

all individual components should be integration tested prior to system testing


18/77


To consider - should the integration testing approach:

Start from top level components and work down? Start from bottom level components and work up? Use the big bang method? Be based on functional groups? Start on critical components first? Be based on business sequencing? Maybe suit System

Test ne.

Knowledge of the system architecture is important

The greater the scope of the integration approach themore difficult it is to isolate defects Non-Functional requirements testing may start here

e.g. early performance measures

Test Planning


19/77


Top-down testing

Stubs

Q R

S T U V

Component under

test

P
http://www.trainboxesplus.com/images/CONSTRUCTION%20WORKER%2014.gifhttp://www.trainboxesplus.com/images/CONSTRUCTION%20WORKER%2014.gifhttp://www.trainboxesplus.com/images/CONSTRUCTION%20WORKER%2014.gifhttp://www.trainboxesplus.com/images/CONSTRUCTION%20WORKER%2014.gif


20/77

Component Integration TestingTop-down testing

Pros

provides a limited working system early in thedesign process

depth first integration demonstrates end-to-endfunctions early in the development process

early detection of design errors through earlyimplementation of the design structure

early testing of major control or decision points

Cons

stubs only provide limited simulations of

lower level components and could

influence spurious results

breadth first means that higher levels of

the system must be artificially forced togenerate output for test observations


21/77

Component Integration TestingBottom-up testing

Q R

S T U

P is the driver for

components Q and R

V

Component under test

P

Same for Q and R

driving their

components
http://www.loti.com/driver.gifhttp://www.loti.com/driver.gifhttp://www.loti.com/driver.gif


22/77

Component Integration TestingBottom-up testing

using drivers instead of upper levelmodules to simulate the environment forlower level modules

necessary for critical, low level systemcomponents

testing can be observed on thecomponents under test from an earlystage

unavailability of a demonstrable system

until late in the development process

late detection of system structure errors

Pros Cons


23/77


Big Bang Approach

Main Menu Function 1 Function 2 Function 3

Component 1 Component 2 Component 3 Component 4 Component 5 Component 6

Not usually the preferred approach


24/77


The following testing techniques are appropriate for IntegrationTesting:

Functional Testing using Black Box Testing techniquesagainst the interfacing requirements for the component

under test Non-functional Testing (where appropriate, forperformance

or reliability testing of the component interfaces, forexample)

Suggested Integration Testing Methodology


25/77


Well talk about System Integration Testing

later.

For now, we should stick to the sequence of

the Test lifecycle.

Which means System Testing next.


26/77

System Testing

Context

Definition

Functional Systems testing

Non-Functional Systems Testing

Good Practices for System Testing


27/77

System Testing

User/Business

Requirements

System

Requirements

Technical

Specification

Program

Specification

Coding

Unit/Component

Test

Integration

Test

System

Test

Acceptance

Test

Development

Levels

Test

Levels

Acceptance Test

Plan

System Test

Plan

Integration

Test Plan

Unit Test

Plan


28/77

System Testing

System Testing - process of testing anintegrated system to verify that it meetsspecified requirements

Concerned with the behaviour of the wholesystem, not with the workings of individualcomponents.

Carried out by the Test Team

Definition


29/77

Functional System Testing

Definition

Requirements-based functional testing

Business process functional testing


30/77


A Functional requirement is (per IEEE):

A requirement that specifies a function that a system or system

component must perform

A Requirement may exist as a text document and/or a model

F i l S T i


31/77


Requirements-based functional testing - Functionality

Accuracy Provision of right or agreed results or effects

Interoperability Ability to interact with specified systems

ComplianceAdhere to applicable standards, conventions, regulationsor laws

Auditability Ability to provide adequate and accurate audit data

SuitabilityPresence and appropriateness of functions for specifiedtasks


32/77


testing against requirements and

specifications

test procedures and cases derived from:

detailed user requirements

system requirements functional specification

User documentation/instructions

high level System design

Requirements based testing

F i l S T i


33/77


Requirements-based functional testing - techniques

Starts by using the most appropriate black-box testing techniques

May support this with white-box techniques (e.g. menu structures, web page

navigation)

Risk based approach important


34/77


test procedures and cases derived from:

expected user profiles

Business scenarios use cases

testing should reflect the businessenvironment and processes in which the

system will operate. therefore, test cases should be based on real

business processes.

Business Process based testing


35/77

Non-functional System Testing

Definition

Non-functional requirements

Non-functional test types

N f ti l S t T ti


36/77


testing of those requirements that do not

relate to functionality

Definition


37/77



Emphasis on non-functional requirements:

Performance

Load

Data volumes

Storage Recovery

Usability

Stress

Security*

* Note that ISTQB treats this as a Functional test. From the syllabus:

Security Testing A type of functional testing, security testing, investigates the functions (e.g. a

firewall) relating to detection of threats, such as viruses, from malicious outsiders.


38/77



The non-functional aspects of a system are all the attributes other than business

functionality, and are as important as the functional aspects. These include:

the look and feel and ease of use of the system

how quickly the system performs

how much the system can do for the user

It is also about:

how easy and quick the system is to install

how robust it is

how quickly the system can recover from a crash

N f ti l S t T ti


39/77

Non-functional test types

Reliability - The capability of software to maintain its level of performance under statedconditions for a stated period of time. Is the software product reliable?

Usability - Is the software product easy to use, learn and understand from the users

perspective? Maintainability: The effort needed to make specified modifications. Is the software product

easy to maintain?

Efficiency: The relationship between the level of performance of the software and the amount

of resources used, under stated conditions. Does the software product use the hardware,

system software and other resources efficiently? Is the number of resources required by the

software product during use affordable and acceptable? Portability: The ability of software to be transferred from one environment to another. Is the

software product portable?


S stem Testing


40/77

System TestingGood Practices for System testing

implement documented procedures for requirements analysis, control andtraceability

review deliverables to ensure feasible, testable requirements and associatedacceptance criteria

trace requirements to the design and tests which prove the requirement has beenmet

test data, facilities and documentation must be sufficient to demonstrateconformance with requirements

Test environment must closely mirror the target production system


41/77


Context

Definition

Objectives

Interfaces to External Systems


42/77

System Integration TestingUser/Business

Requirements

System

Requirements

Technical

Specification

Program

Specification

Coding

Unit/Component

Test

Integration

Test

System

Test

Acceptance

Test

Development

Levels

Test

Levels

Acceptance Test

Plan

System Test

Plan

Integration

Test Plan

Unit Test

Plan

System

Integration

Testing


43/77


System Integration Testing is testing

between the System and Acceptance

phases.

The System has already proven to be

functionally correct, what remains to be

tested is how the system reacts to othersystems and/or organisations.

Definition


44/77


The objective of System Integration Testing is to provideconfidence that the system or application is able to interoperatesuccessfully with other specified software systems and does nothave an adverse affect on other systems that may also be presentin the live environment, or vice versa

It is possible that the testing tasks performed during SystemIntegration Testing may be combined with System Testing,particularly if the system or application has little or no requirementto interoperate with other systems

In terms of the V Model, Systems Integration Testing correspondsto the Functional and Technical Specification phases of thesoftware development lifecycle

Objectives of Systems Integration Testing


45/77


Having completed Component integration

testing and Systems testing, one must execute

the plan for system-to-system integration

Infrastructure may need to be transformed in

order to feed to an external system

Black Box testing techniques used

Testing Interfaces to External Systems


46/77

Acceptance Testing

Context

Definition

User Acceptance Testing

Operational Acceptance Testing

Contract/Regulation acceptance testing

Alpha and Beta testing

Other Acceptance Test terms


47/77

Acceptance TestingUser/Business

Requirements

System

Requirements

Technical

Specification

Program

Specification

Coding

Unit/Component

Test

Integration

Test

System

Test

AcceptanceTest

Development

Levels

Test

Levels

Acceptance Test

Plan

System Test

Plan

Integration

Test Plan

Unit Test

Plan

Acceptance Testing


48/77

Acceptance Testing

Acceptance testing: Formal testing

with respect to user ne, requirements,

and business processes conducted to

determine whether or not a systemsatisfies the acceptance criteria and to

enable the user, customers or other

authorized entity to determine whetheror not to accept the system.

Definition

Acceptance Testing


49/77

Acceptance Testing

Usually the responsibility of the Customer/End user, thoughother stakeholders may be involved. Customer may sub-contract the Acceptance test to a third party

Goal is to establish confidence in the system/part-system orspecific non-functional characteristics (e.g. performance)

Usually for ensuring the system is ready for deployment intoproduction

May also occur at other stages, e.g. Acceptance testing of a COTS product before System Testing

commences

Acceptance testing a components usability during Component testing

Acceptance testing a new significant functionalenhancement/middleware release prior to deployment into SystemTest environment.

Definition

Acceptance Testing
http://images.google.com/imgres?imgurl=http://www.medicalacupuncture.org/aama_marf/newsletter/images/approve.gif&imgrefurl=http://forum.free-games.com.au/forum/General_C2/Humor_&_Jokes_F62/Funiest_insult_ever_used_on_you_P505727/&h=324&w=193&sz=4&tbnid=kE_lzsmqYW3eRM:&tbnh=114&tbnw=67&hl=en&start=3&prev=/images?q=approve&svnum=10&hl=en&lr=&rls=RNWE,RNWE:2004-21,RNWE:en&sa=G


50/77

Acceptance Testing

Usually the final stage of validation

conducted by or visible to the end user and customer

testing is based on the defined user requirements

Often uses the Thread Testing approach: A testing technique used to test the business functionality or business logic of the application in an end-to-end manner, in much

the same way a User or an operator might interact with the system during its normal use .

- Watkins 2001

This approach is also often used for Functional SystemsTest - The same Threadsserve both test activities

User Acceptance Testing (UAT)


51/77

Often use a big bang approach

black box testing techniques most commonlyused

Regression testing to ensure changes have not

regressed other areas of the system

Acceptance Testing

User Acceptance Testing


52/77

The Acceptance of the system by those who have toadminister it.

Features covered include: testing of backup/restore disaster recovery user management maintenance tasks periodic checks of security vulnerabilities

The objective of OAT is to confirm that the ApplicationUnder Test (AUT) meets its operational requirements,and to provide confidence that the system workscorrectly and is usable before it is formally "handedover" to the operation user. OAT is conducted by oneor more Operations Representatives with theassistance of the Test Team 1Watkins 2001

Acceptance Testing

Operational Acceptance Testing (OAT)


53/77

Employs a Black Box Approach for some activities

Also employs a Thread Testing approachOperations representatives performing typical

tasks that they would perform during their

normal usage of the system

Also addresses testing of System Documentation,

such as Operations manuals

Acceptance Testing

Operational Acceptance Testing (OAT)


54/77

Acceptance Testing

Contract Acceptance Testing - testing against theacceptance criteria defined in the contract

final payment to the developer depends on contractacceptance testing being successfully completed

acceptance criteria defined at contract time are oftenimprecise, poorly defined, incomplete and out-of-step withsubsequent changes to the application

Regulation Acceptance testing is performed against anyregulations which must be adhered to, such asgovernmental, legal or safety regulations

Contract/Regulation Acceptance Testing


55/77

Acceptance Testing

early testing of stable product bycustomers/users

feedback provided by alpha and beta testers alpha tests performed at developers site by

customer

beta tests conducted at the customer site by end

user/customer published reviews of beta release test results can

make or break a product (e.g. PC games)

Alpha & Beta Testing


56/77

Acceptance Testing

Factory Acceptance Testing (FAT)

Site Acceptance Testing (SAT)

Both address acceptance testing for systemsthat are tested before and after being movedto a customers site

Other Acceptance test terms

Test Types The Targets of Testing


57/77


Definitions

Functional Testing

Non-Functional Testing

Structural Testing

Confirmation & Regression Testing



58/77


Target For Testing - A group of test activities aimed at verifyingthe software system (or a part of a system) based on a specificreason.

Test type - A group of test activities aimed at testing acomponent or system regarding one or more interrelatedquality attributes. A test type is focused on a specific testobjective, e.g. reliability test

usability test

Structure or architecture of the system/software

regression test

and may take place on one or more test levels or test phases.

Definitions



59/77


A model of the software may be developed and/orused in structural and functional testing

For example, in functional testing

a process flow model a state transition model

a plain language specification

and for structural testing a control flow model

a menu structure model

Definitions

Functional Testing


60/77

Functional Testing

functional testing: Testing based on an analysis of thespecification of the functionality of a component or system.

Specification E.g. Requirements specification, Use Cases,Functional specification or maybe undocumented.

Function what the system does

Functional test based on the Functions and features may beapplied at all Test levels (e.g. Component Test, System Test etc)

Considers the external (not internal) behaviour of thesoftware. Black- Box testing. What it does rather than how itdoes it. More on this later!

Non-Functional Testing


61/77

Non Functional Testing

non-functional testing: Testing the attributes of acomponent or system that do not relate tofunctionality, e.g. reliability, efficiency, usability,interoperability, maintainability and portability

May be performed at all Test levels (not just NonFunctional Systems Testing)

Measuring the characteristics of thesystem/software that can be quantified on avarying scale- e.g. performance test scaling

Structural Testing


62/77

Structural Testing

Structural testing: Testing based on an analysis of the internalstructure of the component or system

Also known as White Box Testing or Glass Box Testing

May be performed at all Test levels but more commonly during

Component Test and Component Integration Test Coverage measured as a % of items tested i.e. how much the

structure has been tested

May be based on the system Architecture e.g. a callinghierarchy

Need for use of Test Tools e.g. for testing coverage ofStatements and Decision in the code

More on White Box testing and Coverage later

Confirmation (Re-Testing) and Regression


63/77

Confirmation (Re Testing) and Regression

Testing

Re-testing: Testing that runs test cases that failedthe last time they were run, in order to verify thesuccess of corrective actions

Regression Testing:Testing of a previously testedprogram following modification to ensure thatdefects have not been introduced or uncovered inunchanged areas of the software, as a result ofthe changes made. It is performed when thesoftware or its environment is changed.


64/77

Whenever a fault is detected and fixed thenthe software should be re-tested to showthat the original fault has been fixed. This isknown as Re-Testing.

It is important that the test case isrepeatable.

In order to support this the test identifiershould be included on the fault report.

It is important that the environment and testdata used are as close as possible as thoseused during the original test.

Confirmation Testing (Re-Testing)

Regression Testing


65/77

If the test is re-run and passes you cannot necessarily say thefault has been resolved because ..

You also need to ensure that the modifications have notcaused unintended side-effects elsewhere and that themodified system still meets its requirementsRegression Testing

Regression testing should be carried out :-

when the system is stable and the system or the environmentchanges

when testing bug-fix releases as part of the maintenance phase

It should be applied at all Test Levels

It should be considered complete when agreed completion

criteria for regression testing have been met Regression test suites evolve over time and given that they

are run frequently are ideal candidates for automation

g g


66/77

Regression Testing Selecting suitable tests involves :-

knowledge of the bug fixes and how they affect the system

understanding the areas that have frequent faults understanding which areas of the system have undergone themost recent changes

understanding the areas of the system which are most criticalto the user

understanding the core features of the system which mustfunction correctly.

The effectiveness of a regression test suite can diminish overtime for a number of reasons :- tests are added for short term goals but not removed tests become redundant due to functionality changes test suite is not updated when major functionality changes are

implemented

execution time becomes prohibitively high maintenance of the test suite becomes prohibitively high.


67/77

Regression Testing Reduction in effectiveness can be countered

by :- maintaining cross references between system

features and their corresponding tests

monitoring the addition of tests to the suite

Periodic review and removal of redundant tests

review of the test suite when major

enhancements are made to the system

evaluation of the effectiveness of the test suiteusing metrics.


68/77

Regression Testing

The probability of making an incorrect change is morethan 50 %.

Much of this is due to overconfidence and ineffective or

nonexistent software change testing.We change just a couple of statements and believe wehave not affected anything adversely.

We execute one case that tests the path that was changed

and tell ourselves that the change has been tested.IS IT, THEN, ANY WONDER THAT WE EXPERIENCE SO MANY

PROBLEMS?!

Hetzel 1998


69/77

Maintenance Testing

What is Maintenance testing?

Objectives of Maintenance testing

Problems of Maintenance testing

Concerns of Maintenance testing

How can we test changes?

Maintenance Testing


70/77

g

Maintenance testing: Testing the changes to an operational system or the impact of a

changed environment to an operational system

testing changes to a Live System

Triggered by, for example,

Modification

software upgrades

Operating system changes system tuning

emergency fixes

software Retirement (may necessitate data archiving tests)

Migration

System migration (including operational tests of new environment plus changed software)

database migration

What is Maintenance Testing?


71/77

Maintenance Testing

Develop tests to detect problems prior to

placing the change into production

Correct problems identified in the live

environment

Test the completeness of needed training

material

Involve users in the testing of software

changes

Objectives of Maintenance Testing

Maintenance Testing


72/77

g

all that is available is the source code(usually with poor internaldocumentation and no record of testing) poor or missing specifications

program structure, global data

structures, system interfaces andperformance and/or design constraintsare difficult to determine and frequentlymisinterpreted

Baselined test plans and/or regressiontest packs often not updated

Problems of Maintenance testing


73/77

Maintenance Testing

will the testing process be planned?

will testing results be recorded?

will new faults be introduced into the system?

will system problems be detected during

testing?

how much regression testing is feasible?

will training be considered?

Concerns of Maintenance testing


74/77

Maintenance TestingHow can we test changes?

Maintenance testing involves testing what has beenchanged (i.e. Re-Testing)

It also, importantly, utilises Impact Analysis as amethod for determining what Regression testing isrequired for the whole system

Traceability of Testware to source documents essentialfor effective impact analysis (we cover this more in alater topic)

Scope of Maintenance tests based on Risk assessment

including size of change and size of system Maintenance testing may involve one or more test

levels and one or more test types

est ng t roug out t e o twareLif l


75/77

Lifecycle Firstly we looked at Software Development Models:

The V-Model identifying the stages of testing, their relationship to theDevelopment stages and the type of work products involved

Iterative Development Models, as used in RAD, RUP and Agiledevelopments

Also the characteristics that make for good testing in ANY life cycle model And that development models must be adapted to the context of project

and product characteristics

Next we talked about the different Test Levels: Component Testing testing of individual software components by the

development team Integration Testing at Component level looking at different approaches

such as Top-Down, Bottom-up and Big Bang System Testing

Functional System Testing requirements and business process based Non-Functional System Testing testing the non-functional attributes of a system

System (level) Integration Testing - testing that a system interoperateswith other systems or organisations

Summary



76/77

And still under Test Levels . Acceptance Testing, comprising: User Acceptance Testing Operational Acceptance Testing Contract and Regulation testing Alpha and Beta testing

Next we talked about Test Types A group of testing activities aimed at testing one or more

quality attributes of the system, such as: Functional Testing Non-Functional Testing

Structural testing Regression Testing Re-testing (confirmation)

All Test Types can be performed at all test levels

LifecycleSummary



77/77

An finally we talked about Maintenance testing

Testing the changes (software and environmental) to anoperational live system

What the reasons (or triggers) are for Maintenance Testing

We looked at the objectives of Maintenance Testing the problems that can be encountered during Maintenance

testing

and what we should consider for our Maintenance testingapproach such as Impact Analysis and Regression testing

LifecycleSummary

Testing Throughout the Software Lifecycle - Chap 2

Documents

Transcript of Testing Throughout the Software Lifecycle - Chap 2