7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
1/77
Slide 1 Internal
Testing Throughout the Software
Life-cycle
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
2/77
Testing Throughout the Software
Lifecycle
Software Development Models
Test Levels
Test Types the Targets of Testing
Maintenance Testing
Summary
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
3/77
Software Development Models
The V-Model
Iterative Development Models
Testing within a Lifecycle Model
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
4/77
V-Model
User/Business
Requirements
System
Requirements
Technical
Specification
Program
Specification
Coding
Unit
Test
Integration
Test
System
Test
Acceptance
Test
Development
Levels
Test
Levels
Acceptance Test
Plan
System Test
Plan
Integration
Test Plan
Unit Test
Plan
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
5/77
V-Model
The benefits of the V Model include:
The testing phases are given the same level of management attention and
commitment as the corresponding development phases
The outputs from the development phases are reviewed by the testing team toensure their testability
Verification and validation (and early test design) can be carried out during the
development of the software work products
The early planning and preliminary design of tests provides additional review
comments on the outputs from the development phase
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
6/77
V-Model
The levels of development and testing shown in the model vary from project to
project
For example, there may additional test levels, such as System Integration Testing,
sitting between System Testing and Acceptance Testing (more on these test levels
later)
The work products coming out from any one development level may be utilised in
one or more test levels
For example, whilst the prime source for Acceptance testing is the Business
Requirement, the System Requirements (e.g. Use Cases) may also be needed to
support detailed test design
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
7/77
Iterative Development Models
Iterative Development Establish Requirements Design the System Build the System Test the System
Achieved with small developments Iterations and Increment withinIterations
As Increments are developed and tested the System grows andgrows. Need for more testing with Regression Testing paramount
E.g. RAD, RUP and Agile development models
Agile development aim is to deliver software early and often
Rapid production and time to market Can handle (and anticipates) changing requirements throughout alldevelopment and test phases
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
8/77
Iterative Development Models
Rapid Application Development
User
Requirements
Code
Acceptance
Test
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
9/77
Testing within a Lifecycle Model
Characteristics of good testing in any life cyclemodel:
A Test Level exists for every development Level Each Test Level has specific objectives
Test analysis and design for each Test Level beginsduring corresponding development Level
Early and proactive involvement of testers inreviewing development deliverables benefitsboth parties
Test Levels should be adapted depending onProject nature. May be better to combine TestLevels, e.g. with COTS testing.
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
10/77
Test Levels
Component Testing
Integration Testing
System testing
Acceptance Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
11/77
Component Testing
User/Business
Requirements
System
Requirements
Technical
Specification
Program
Specification
Coding
Unit/Component
Test
Integration
Test
System
Test
Acceptance
Test
Development
Levels
Test
Levels
Acceptance Test
Plan
System Test
Plan
Integration
Test Plan
Unit Test
Plan
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
12/77
Component Testing
ComponentA minimal software item that can be tested in isolation.
Component TestingThe testing of individual software components.
Sometimes known as Unit Testing, Module Testing or Program Testing
Component can be tested in isolation stubs/drivers may be employed
Test cases derived from component specification (module/program spec)
Functional and Non-Functional testing
Usually performed by the developer, with debugging tool
Quick and informal defect fixing
Definition
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
13/77
Component Testing
Test-First/Test-Drivenapproach create the tests to drive the design and codeconstruction!
Instead of creating a design to tell you how to structure your code, you create atest that defines how a small part of the system should function.
Three steps:
1. Design test that defines how you think a small part of the software should behave(Incremental development).
2. Make the test run as easily and quickly as you can. Don't be concerned about the designof code, just get it to work!
3. Clean up the code. Now that the code is working correctly, take a step back and re-factorto remove any duplication or any other problems that were introduced to get the test to
run.
Russell Gold, Thomas Hammell and Tom Snyder - 2005
Definition
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
14/77
Integration Testing
Definition
Component Integration Testing
System Integration Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
15/77
Integration Testing
Definition
Integration Testing - Testing performed to expose defects in the
interfaces and in the interactions between integrated components or
systems
Components may be code modules, operating systems, hardware and
even complete systems
There are 2 levels of Integration Testing
Component Integration Testing
System Integration Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
16/77
Component Integration Testing
User/Business
Requirements
System
Requirements
Technical
Specification
Program
Specification
Coding
Unit/ComponentTest
Integration
Test
System
Test
Acceptance
Test
Development
Levels
Test
Levels
Acceptance Test
Plan
System Test
Plan
Integration
Test Plan
Unit Test
Plan
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
17/77
Component Integration Testing
Definition
component integrationtesting Testing performed to expose defects in the
interfaces and interaction between integrated components
performed by the test team
usually formal (records of test design and execution are kept)
all individual components should be integration tested prior to system testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
18/77
Component Integration Testing
To consider - should the integration testing approach:
Start from top level components and work down? Start from bottom level components and work up? Use the big bang method? Be based on functional groups? Start on critical components first? Be based on business sequencing? Maybe suit System
Test ne.
Knowledge of the system architecture is important
The greater the scope of the integration approach themore difficult it is to isolate defects Non-Functional requirements testing may start here
e.g. early performance measures
Test Planning
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
19/77
Component Integration Testing
Top-down testing
Stubs
Q R
S T U V
Component under
test
P
http://www.trainboxesplus.com/images/CONSTRUCTION%20WORKER%2014.gifhttp://www.trainboxesplus.com/images/CONSTRUCTION%20WORKER%2014.gifhttp://www.trainboxesplus.com/images/CONSTRUCTION%20WORKER%2014.gifhttp://www.trainboxesplus.com/images/CONSTRUCTION%20WORKER%2014.gif7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
20/77
Component Integration TestingTop-down testing
Pros
provides a limited working system early in thedesign process
depth first integration demonstrates end-to-endfunctions early in the development process
early detection of design errors through earlyimplementation of the design structure
early testing of major control or decision points
Cons
stubs only provide limited simulations of
lower level components and could
influence spurious results
breadth first means that higher levels of
the system must be artificially forced togenerate output for test observations
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
21/77
Component Integration TestingBottom-up testing
Q R
S T U
P is the driver for
components Q and R
V
Component under test
P
Same for Q and R
driving their
components
http://www.loti.com/driver.gifhttp://www.loti.com/driver.gifhttp://www.loti.com/driver.gif7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
22/77
Component Integration TestingBottom-up testing
using drivers instead of upper levelmodules to simulate the environment forlower level modules
necessary for critical, low level systemcomponents
testing can be observed on thecomponents under test from an earlystage
unavailability of a demonstrable system
until late in the development process
late detection of system structure errors
Pros Cons
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
23/77
Component Integration Testing
Big Bang Approach
Main Menu Function 1 Function 2 Function 3
Component 1 Component 2 Component 3 Component 4 Component 5 Component 6
Not usually the preferred approach
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
24/77
Component Integration Testing
The following testing techniques are appropriate for IntegrationTesting:
Functional Testing using Black Box Testing techniquesagainst the interfacing requirements for the component
under test Non-functional Testing (where appropriate, forperformance
or reliability testing of the component interfaces, forexample)
Suggested Integration Testing Methodology
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
25/77
System Integration Testing
Well talk about System Integration Testing
later.
For now, we should stick to the sequence of
the Test lifecycle.
Which means System Testing next.
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
26/77
System Testing
Context
Definition
Functional Systems testing
Non-Functional Systems Testing
Good Practices for System Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
27/77
System Testing
User/Business
Requirements
System
Requirements
Technical
Specification
Program
Specification
Coding
Unit/Component
Test
Integration
Test
System
Test
Acceptance
Test
Development
Levels
Test
Levels
Acceptance Test
Plan
System Test
Plan
Integration
Test Plan
Unit Test
Plan
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
28/77
System Testing
System Testing - process of testing anintegrated system to verify that it meetsspecified requirements
Concerned with the behaviour of the wholesystem, not with the workings of individualcomponents.
Carried out by the Test Team
Definition
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
29/77
Functional System Testing
Definition
Requirements-based functional testing
Business process functional testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
30/77
Functional System Testing
A Functional requirement is (per IEEE):
A requirement that specifies a function that a system or system
component must perform
A Requirement may exist as a text document and/or a model
F i l S T i
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
31/77
Functional System Testing
Requirements-based functional testing - Functionality
Accuracy Provision of right or agreed results or effects
Interoperability Ability to interact with specified systems
ComplianceAdhere to applicable standards, conventions, regulationsor laws
Auditability Ability to provide adequate and accurate audit data
SuitabilityPresence and appropriateness of functions for specifiedtasks
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
32/77
Functional System Testing
testing against requirements and
specifications
test procedures and cases derived from:
detailed user requirements
system requirements functional specification
User documentation/instructions
high level System design
Requirements based testing
F i l S T i
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
33/77
Functional System Testing
Requirements-based functional testing - techniques
Starts by using the most appropriate black-box testing techniques
May support this with white-box techniques (e.g. menu structures, web page
navigation)
Risk based approach important
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
34/77
Functional System Testing
test procedures and cases derived from:
expected user profiles
Business scenarios use cases
testing should reflect the businessenvironment and processes in which the
system will operate. therefore, test cases should be based on real
business processes.
Business Process based testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
35/77
Non-functional System Testing
Definition
Non-functional requirements
Non-functional test types
N f ti l S t T ti
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
36/77
Non-functional System Testing
testing of those requirements that do not
relate to functionality
Definition
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
37/77
Non-functional System Testing
Non-functional requirements
Emphasis on non-functional requirements:
Performance
Load
Data volumes
Storage Recovery
Usability
Stress
Security*
* Note that ISTQB treats this as a Functional test. From the syllabus:
Security Testing A type of functional testing, security testing, investigates the functions (e.g. a
firewall) relating to detection of threats, such as viruses, from malicious outsiders.
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
38/77
Non-functional System Testing
Non-functional requirements
The non-functional aspects of a system are all the attributes other than business
functionality, and are as important as the functional aspects. These include:
the look and feel and ease of use of the system
how quickly the system performs
how much the system can do for the user
It is also about:
how easy and quick the system is to install
how robust it is
how quickly the system can recover from a crash
N f ti l S t T ti
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
39/77
Non-functional test types
Reliability - The capability of software to maintain its level of performance under statedconditions for a stated period of time. Is the software product reliable?
Usability - Is the software product easy to use, learn and understand from the users
perspective? Maintainability: The effort needed to make specified modifications. Is the software product
easy to maintain?
Efficiency: The relationship between the level of performance of the software and the amount
of resources used, under stated conditions. Does the software product use the hardware,
system software and other resources efficiently? Is the number of resources required by the
software product during use affordable and acceptable? Portability: The ability of software to be transferred from one environment to another. Is the
software product portable?
Non-functional System Testing
S stem Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
40/77
System TestingGood Practices for System testing
implement documented procedures for requirements analysis, control andtraceability
review deliverables to ensure feasible, testable requirements and associatedacceptance criteria
trace requirements to the design and tests which prove the requirement has beenmet
test data, facilities and documentation must be sufficient to demonstrateconformance with requirements
Test environment must closely mirror the target production system
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
41/77
System Integration Testing
Context
Definition
Objectives
Interfaces to External Systems
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
42/77
System Integration TestingUser/Business
Requirements
System
Requirements
Technical
Specification
Program
Specification
Coding
Unit/Component
Test
Integration
Test
System
Test
Acceptance
Test
Development
Levels
Test
Levels
Acceptance Test
Plan
System Test
Plan
Integration
Test Plan
Unit Test
Plan
System
Integration
Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
43/77
System Integration Testing
System Integration Testing is testing
between the System and Acceptance
phases.
The System has already proven to be
functionally correct, what remains to be
tested is how the system reacts to othersystems and/or organisations.
Definition
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
44/77
System Integration Testing
The objective of System Integration Testing is to provideconfidence that the system or application is able to interoperatesuccessfully with other specified software systems and does nothave an adverse affect on other systems that may also be presentin the live environment, or vice versa
It is possible that the testing tasks performed during SystemIntegration Testing may be combined with System Testing,particularly if the system or application has little or no requirementto interoperate with other systems
In terms of the V Model, Systems Integration Testing correspondsto the Functional and Technical Specification phases of thesoftware development lifecycle
Objectives of Systems Integration Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
45/77
System Integration Testing
Having completed Component integration
testing and Systems testing, one must execute
the plan for system-to-system integration
Infrastructure may need to be transformed in
order to feed to an external system
Black Box testing techniques used
Testing Interfaces to External Systems
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
46/77
Acceptance Testing
Context
Definition
User Acceptance Testing
Operational Acceptance Testing
Contract/Regulation acceptance testing
Alpha and Beta testing
Other Acceptance Test terms
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
47/77
Acceptance TestingUser/Business
Requirements
System
Requirements
Technical
Specification
Program
Specification
Coding
Unit/Component
Test
Integration
Test
System
Test
AcceptanceTest
Development
Levels
Test
Levels
Acceptance Test
Plan
System Test
Plan
Integration
Test Plan
Unit Test
Plan
Acceptance Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
48/77
Acceptance Testing
Acceptance testing: Formal testing
with respect to user ne, requirements,
and business processes conducted to
determine whether or not a systemsatisfies the acceptance criteria and to
enable the user, customers or other
authorized entity to determine whetheror not to accept the system.
Definition
Acceptance Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
49/77
Acceptance Testing
Usually the responsibility of the Customer/End user, thoughother stakeholders may be involved. Customer may sub-contract the Acceptance test to a third party
Goal is to establish confidence in the system/part-system orspecific non-functional characteristics (e.g. performance)
Usually for ensuring the system is ready for deployment intoproduction
May also occur at other stages, e.g. Acceptance testing of a COTS product before System Testing
commences
Acceptance testing a components usability during Component testing
Acceptance testing a new significant functionalenhancement/middleware release prior to deployment into SystemTest environment.
Definition
Acceptance Testing
http://images.google.com/imgres?imgurl=http://www.medicalacupuncture.org/aama_marf/newsletter/images/approve.gif&imgrefurl=http://forum.free-games.com.au/forum/General_C2/Humor_&_Jokes_F62/Funiest_insult_ever_used_on_you_P505727/&h=324&w=193&sz=4&tbnid=kE_lzsmqYW3eRM:&tbnh=114&tbnw=67&hl=en&start=3&prev=/images?q=approve&svnum=10&hl=en&lr=&rls=RNWE,RNWE:2004-21,RNWE:en&sa=G7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
50/77
Acceptance Testing
Usually the final stage of validation
conducted by or visible to the end user and customer
testing is based on the defined user requirements
Often uses the Thread Testing approach: A testing technique used to test the business functionality or business logic of the application in an end-to-end manner, in much
the same way a User or an operator might interact with the system during its normal use .
- Watkins 2001
This approach is also often used for Functional SystemsTest - The same Threadsserve both test activities
User Acceptance Testing (UAT)
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
51/77
Often use a big bang approach
black box testing techniques most commonlyused
Regression testing to ensure changes have not
regressed other areas of the system
Acceptance Testing
User Acceptance Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
52/77
The Acceptance of the system by those who have toadminister it.
Features covered include: testing of backup/restore disaster recovery user management maintenance tasks periodic checks of security vulnerabilities
The objective of OAT is to confirm that the ApplicationUnder Test (AUT) meets its operational requirements,and to provide confidence that the system workscorrectly and is usable before it is formally "handedover" to the operation user. OAT is conducted by oneor more Operations Representatives with theassistance of the Test Team 1Watkins 2001
Acceptance Testing
Operational Acceptance Testing (OAT)
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
53/77
Employs a Black Box Approach for some activities
Also employs a Thread Testing approachOperations representatives performing typical
tasks that they would perform during their
normal usage of the system
Also addresses testing of System Documentation,
such as Operations manuals
Acceptance Testing
Operational Acceptance Testing (OAT)
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
54/77
Acceptance Testing
Contract Acceptance Testing - testing against theacceptance criteria defined in the contract
final payment to the developer depends on contractacceptance testing being successfully completed
acceptance criteria defined at contract time are oftenimprecise, poorly defined, incomplete and out-of-step withsubsequent changes to the application
Regulation Acceptance testing is performed against anyregulations which must be adhered to, such asgovernmental, legal or safety regulations
Contract/Regulation Acceptance Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
55/77
Acceptance Testing
early testing of stable product bycustomers/users
feedback provided by alpha and beta testers alpha tests performed at developers site by
customer
beta tests conducted at the customer site by end
user/customer published reviews of beta release test results can
make or break a product (e.g. PC games)
Alpha & Beta Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
56/77
Acceptance Testing
Factory Acceptance Testing (FAT)
Site Acceptance Testing (SAT)
Both address acceptance testing for systemsthat are tested before and after being movedto a customers site
Other Acceptance test terms
Test Types The Targets of Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
57/77
Test Types The Targets of Testing
Definitions
Functional Testing
Non-Functional Testing
Structural Testing
Confirmation & Regression Testing
Test Types The Targets of Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
58/77
Test Types The Targets of Testing
Target For Testing - A group of test activities aimed at verifyingthe software system (or a part of a system) based on a specificreason.
Test type - A group of test activities aimed at testing acomponent or system regarding one or more interrelatedquality attributes. A test type is focused on a specific testobjective, e.g. reliability test
usability test
Structure or architecture of the system/software
regression test
and may take place on one or more test levels or test phases.
Definitions
Test Types The Targets of Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
59/77
Test Types The Targets of Testing
A model of the software may be developed and/orused in structural and functional testing
For example, in functional testing
a process flow model a state transition model
a plain language specification
and for structural testing a control flow model
a menu structure model
Definitions
Functional Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
60/77
Functional Testing
functional testing: Testing based on an analysis of thespecification of the functionality of a component or system.
Specification E.g. Requirements specification, Use Cases,Functional specification or maybe undocumented.
Function what the system does
Functional test based on the Functions and features may beapplied at all Test levels (e.g. Component Test, System Test etc)
Considers the external (not internal) behaviour of thesoftware. Black- Box testing. What it does rather than how itdoes it. More on this later!
Non-Functional Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
61/77
Non Functional Testing
non-functional testing: Testing the attributes of acomponent or system that do not relate tofunctionality, e.g. reliability, efficiency, usability,interoperability, maintainability and portability
May be performed at all Test levels (not just NonFunctional Systems Testing)
Measuring the characteristics of thesystem/software that can be quantified on avarying scale- e.g. performance test scaling
Structural Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
62/77
Structural Testing
Structural testing: Testing based on an analysis of the internalstructure of the component or system
Also known as White Box Testing or Glass Box Testing
May be performed at all Test levels but more commonly during
Component Test and Component Integration Test Coverage measured as a % of items tested i.e. how much the
structure has been tested
May be based on the system Architecture e.g. a callinghierarchy
Need for use of Test Tools e.g. for testing coverage ofStatements and Decision in the code
More on White Box testing and Coverage later
Confirmation (Re-Testing) and Regression
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
63/77
Confirmation (Re Testing) and Regression
Testing
Re-testing: Testing that runs test cases that failedthe last time they were run, in order to verify thesuccess of corrective actions
Regression Testing:Testing of a previously testedprogram following modification to ensure thatdefects have not been introduced or uncovered inunchanged areas of the software, as a result ofthe changes made. It is performed when thesoftware or its environment is changed.
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
64/77
Whenever a fault is detected and fixed thenthe software should be re-tested to showthat the original fault has been fixed. This isknown as Re-Testing.
It is important that the test case isrepeatable.
In order to support this the test identifiershould be included on the fault report.
It is important that the environment and testdata used are as close as possible as thoseused during the original test.
Confirmation Testing (Re-Testing)
Regression Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
65/77
If the test is re-run and passes you cannot necessarily say thefault has been resolved because ..
You also need to ensure that the modifications have notcaused unintended side-effects elsewhere and that themodified system still meets its requirementsRegression Testing
Regression testing should be carried out :-
when the system is stable and the system or the environmentchanges
when testing bug-fix releases as part of the maintenance phase
It should be applied at all Test Levels
It should be considered complete when agreed completion
criteria for regression testing have been met Regression test suites evolve over time and given that they
are run frequently are ideal candidates for automation
g g
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
66/77
Regression Testing Selecting suitable tests involves :-
knowledge of the bug fixes and how they affect the system
understanding the areas that have frequent faults understanding which areas of the system have undergone themost recent changes
understanding the areas of the system which are most criticalto the user
understanding the core features of the system which mustfunction correctly.
The effectiveness of a regression test suite can diminish overtime for a number of reasons :- tests are added for short term goals but not removed tests become redundant due to functionality changes test suite is not updated when major functionality changes are
implemented
execution time becomes prohibitively high maintenance of the test suite becomes prohibitively high.
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
67/77
Regression Testing Reduction in effectiveness can be countered
by :- maintaining cross references between system
features and their corresponding tests
monitoring the addition of tests to the suite
Periodic review and removal of redundant tests
review of the test suite when major
enhancements are made to the system
evaluation of the effectiveness of the test suiteusing metrics.
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
68/77
Regression Testing
The probability of making an incorrect change is morethan 50 %.
Much of this is due to overconfidence and ineffective or
nonexistent software change testing.We change just a couple of statements and believe wehave not affected anything adversely.
We execute one case that tests the path that was changed
and tell ourselves that the change has been tested.IS IT, THEN, ANY WONDER THAT WE EXPERIENCE SO MANY
PROBLEMS?!
Hetzel 1998
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
69/77
Maintenance Testing
What is Maintenance testing?
Objectives of Maintenance testing
Problems of Maintenance testing
Concerns of Maintenance testing
How can we test changes?
Maintenance Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
70/77
g
Maintenance testing: Testing the changes to an operational system or the impact of a
changed environment to an operational system
testing changes to a Live System
Triggered by, for example,
Modification
software upgrades
Operating system changes system tuning
emergency fixes
software Retirement (may necessitate data archiving tests)
Migration
System migration (including operational tests of new environment plus changed software)
database migration
What is Maintenance Testing?
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
71/77
Maintenance Testing
Develop tests to detect problems prior to
placing the change into production
Correct problems identified in the live
environment
Test the completeness of needed training
material
Involve users in the testing of software
changes
Objectives of Maintenance Testing
Maintenance Testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
72/77
g
all that is available is the source code(usually with poor internaldocumentation and no record of testing) poor or missing specifications
program structure, global data
structures, system interfaces andperformance and/or design constraintsare difficult to determine and frequentlymisinterpreted
Baselined test plans and/or regressiontest packs often not updated
Problems of Maintenance testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
73/77
Maintenance Testing
will the testing process be planned?
will testing results be recorded?
will new faults be introduced into the system?
will system problems be detected during
testing?
how much regression testing is feasible?
will training be considered?
Concerns of Maintenance testing
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
74/77
Maintenance TestingHow can we test changes?
Maintenance testing involves testing what has beenchanged (i.e. Re-Testing)
It also, importantly, utilises Impact Analysis as amethod for determining what Regression testing isrequired for the whole system
Traceability of Testware to source documents essentialfor effective impact analysis (we cover this more in alater topic)
Scope of Maintenance tests based on Risk assessment
including size of change and size of system Maintenance testing may involve one or more test
levels and one or more test types
est ng t roug out t e o twareLif l
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
75/77
Lifecycle Firstly we looked at Software Development Models:
The V-Model identifying the stages of testing, their relationship to theDevelopment stages and the type of work products involved
Iterative Development Models, as used in RAD, RUP and Agiledevelopments
Also the characteristics that make for good testing in ANY life cycle model And that development models must be adapted to the context of project
and product characteristics
Next we talked about the different Test Levels: Component Testing testing of individual software components by the
development team Integration Testing at Component level looking at different approaches
such as Top-Down, Bottom-up and Big Bang System Testing
Functional System Testing requirements and business process based Non-Functional System Testing testing the non-functional attributes of a system
System (level) Integration Testing - testing that a system interoperateswith other systems or organisations
Summary
est ng t roug out t e o twareLif l
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
76/77
And still under Test Levels . Acceptance Testing, comprising: User Acceptance Testing Operational Acceptance Testing Contract and Regulation testing Alpha and Beta testing
Next we talked about Test Types A group of testing activities aimed at testing one or more
quality attributes of the system, such as: Functional Testing Non-Functional Testing
Structural testing Regression Testing Re-testing (confirmation)
All Test Types can be performed at all test levels
LifecycleSummary
est ng t roug out t e o twareLif l
7/28/2019 Testing Throughout the Software Lifecycle - Chap 2
77/77
An finally we talked about Maintenance testing
Testing the changes (software and environmental) to anoperational live system
What the reasons (or triggers) are for Maintenance Testing
We looked at the objectives of Maintenance Testing the problems that can be encountered during Maintenance
testing
and what we should consider for our Maintenance testingapproach such as Impact Analysis and Regression testing
LifecycleSummary
Top Related