8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 1/50
Software Testing Copyright © André Murbach, 20051
Software Testing – An Overview
& Some Practical Hints
Dr. André Murbach
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 2/50
Software Testing Copyright © André Murbach, 20052
Overview
What Is Testing?
Definition
Why Testing?
Testing & Defects
Testing & Quality
How To Do Testing?
Test Cases & Co.
Testing Activities
Economics of TestingPsychology of Testing
Defect Management
Types of Tests
Testing Techniques
Test Stages
A Minimal Test Process
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 3/50
Software Testing Copyright © André Murbach, 20053
What Is Testing?
Definition A:
The purpose of testing is to discover errors. Testing is theprocess of trying to discover every conceivable fault orweakness in a work product. (Glenford Myers, The Art of
Software Testing, 1979)
-> A single aspect: find problems
-> Serious effort / really try hard to find them
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 4/50Software Testing Copyright © André Murbach, 20054
What Is Testing?
Definition B:
The purpose of software testing is to give increasedconfidence in those areas of the product that work and todocument issues with those areas of the product that do not
work. (Mark Fewster and Dorothy Graham, Software Test Automation, 1999)
-> Constructive: Confidence that the product is good enough
Know the remaining problems
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 5/50Software Testing Copyright © André Murbach, 20055
What Is Testing?
Tasks:
Try to find problems – fault-directed
Measure / estimate quality – conformance-directed
– Quality factors like correctness, robustness, ...
Results: (Justified) confidence that software satisfies expectations
Problems known and documented
Disputed:
Static / dynamic testing
„Hard“ / „soft“ requirements & problems
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 6/50
Software Testing Copyright © André Murbach, 20056
"Is the Moon an Enemy Rocket?"
A system able to detect and identify flying objects. Objects notidentifiable as own airplanes are enemy targets.
The system identified the moon as an enemy target. Was
there a software defect?
"Soft" / user perspective: yes
"Hard" / developer perspective: no - specifications are
satisfied
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 7/50
Software Testing Copyright © André Murbach, 20057
Why Testing?
Testing & Defects
Testing & Quality
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 8/50
Software Testing Copyright © André Murbach, 20058
Why Does Software Have Defects?
Errare humanum est
Software is man made
Errors are inevitable
Contributing factors:
Time pressure
Misunderstandings, communication problems
Size & complexity of software systems
Defects may remain in the SW
Software defects may be difficult to detect
Buggy software is common / accepted by the market(at present)
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 9/50
Software Testing Copyright © André Murbach, 20059
Cost of Defects
Very expensive examples:
Therac-25 accidents killing several people
Ariane 5 explosion
Typical case:
Increased support & maintenance costs
Customer dissatisfaction
– Costs difficult to predict
– Long term consequences
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 10/50
Software Testing Copyright © André Murbach, 200510
Defect Cost During Development
Costs already during development
The longer a defect remainsundetected, the moreexpensive it becomes
Increased development effort
Schedule slips
Investments in software quality
practices reduce those costs– Detect & correct
problems early
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 11/50
Software Testing Copyright © André Murbach, 200511
The Solution?
The most effective remedies:
Highly qualified developers
Reviews
Testing
But why does tested software still have defects?
Unqualified testing?
Tests may be erroneous themselves?
The testing was not exhaustive?
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 12/50
Software Testing Copyright © André Murbach, 200512
Why Does Tested Software Still Have Defects?
Untested code segments
Customer applies instructions in a different sequence thanexecuted during testing
Customer applies combination of inputs not applied duringtesting
Execution environment (HW, OS, other applications) differsfrom the environment used during testing
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 13/50
Software Testing Copyright © André Murbach, 200513
Exhaustive Testing Is Not Feasible
Example: two lines on a screen
A display with 1024 x 768 = 786'432 pixels
Number of ways to draw a line = 786'432^2 6 x 10^11
Two lines: 36 x 10^22
Age of the universe 5 x 10^16 s
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 14/50
Software Testing Copyright © André Murbach, 200514
Testing & Quality
Relationships between Testing and Software Quality
Measuring quality by testing
-> Ready to ship? (release decision)
Improving quality by testing
Testing-in quality?
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 15/50
Software Testing Copyright © André Murbach, 200515
SW Quality (Product Quality)
External Quality vs. Internal Quality
External quality: Reliability, correctness, usability, ...
-> visible to the user
Internal quality: Understandability, maintainability, ...
-> visible to the developer
Quality factors / quality dimensions
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 16/50
Software Testing Copyright © André Murbach, 200516
Why Testing? – Summary
Find & eliminate defects, improve quality
"Form follows failure"
Document issues
Estimate / measure quality
Ready to ship?
Reduce cost
Contractual obligations, industry standards
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 17/50
Software Testing Copyright © André Murbach, 200517
How Is Testing Done?
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 18/50
Software Testing Copyright © André Murbach, 200518
Specifying Tests – Some Terms
Test condition
Condition to cover by a test case
Example: „illegal value for input A“
Test case
Preconditions
Inputs to apply
Expected outcomes
Test procedure
Instructions to the tester
Leads to the execution of test cases
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 19/50
Software Testing Copyright © André Murbach, 200519
Testing Activities (1)
Planning
Breakdown into topics / areas to test
Define goals / confidentiality levels for each area Think of methods to apply (testing techniques)
Effort estimation & schedule
Infrastructure
Test Design
Select testing techniques Identify the conditions to cover (by applying the techniques)
Find test cases covering the conditions
Create test specifications (test procedures, test scripts, testdata)
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 20/50
Software Testing Copyright © André Murbach, 200520
Testing Activities (2)
Test Execution
Execute the tests
Record the results, report defects
Feedback on test specifications
May need to be repeated many times (test cycles)
Debugging related
Make failures reproduceable
Verify fixes (re-testing)
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 21/50
Software Testing Copyright © André Murbach, 200521
Economics of Testing
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 22/50
Software Testing Copyright © André Murbach, 200522
When To Stop Testing?
Effort optimization
Determining test effort is an economical decision – not atechnical one.
Risk / cost / time considerations
Risk / cost of defects; cost of testing; time to market
Test Effort
Bene fi t
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 23/50
Software Testing Copyright © André Murbach, 200523
Test Prioritization
Optimize test execution by prioritizing tests based on risk
Prioritize tests so that whenever you stop testing youhave done the best testing in the time available.
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 24/50
Software Testing Copyright © André Murbach, 200524
Psychology of Testing
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 25/50
Software Testing Copyright © André Murbach, 200525
The Testing Paradoxon
Paradoxical goals:
Testing tries to find defects
Testing tries to demonstrate the absence of defects
Paradoxical approach to second goal:
Looking for something you don't really want to find
It is sometimes difficult to follow both goals in parallel.
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 26/50
Software Testing Copyright © André Murbach, 200526
Analogy: Scientific Experiments
Test - Experiment
Requirements - Theory
Software under test - Reality
Scientific theories can't be proven. They just gain increasing
confidence as they withstand attempts to falsify them.
A good test is similar to a scientific experiment that tries to
confirm or falsify a hypothesis.
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 27/50
Software Testing Copyright © André Murbach, 200527
A Tester's Mindset
Combination of "destructive creativity" and analytical systematicity
Try to break, try to falsify, be nasty
Be systematic - ensure coverage, objectivity, reproduceability
User perspective
Complementing the developper perspective
See yourself as representative of the user community
Be demanding / critical
Required expertise & knowledge
Business / product / domain knowledge
Testing experience
Testing methodology
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 28/50
Software Testing Copyright © André Murbach, 200528
How To Measure Good Quality?
Measuring poor quality is relatively easy
Measuring good quality is difficult
-> Justified confidence in the quality of the software requiresconfidence in the quality of the testing.
SW Quality
Test Quality
few defects
very few defects few defects
many defects+
+
-
--
--
++ +
+
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 29/50
Software Testing Copyright © André Murbach, 200529
A Constructive "Defect Culture"
Defects are bad news:
Debugging instead of implementing new features
Additional work and time needed
Nobody likes to be accused of having done bad work
Deal with defects in a constructive way:
Accept & expect errors / defects– Errors are natural – plan for them
– Focus on the problem to be solved – not on finding the guilty
– Learn from errors / defects Establish an efficient defect management
– Bug tracking tool
– Enforce fixing & verifying defects
– Triage / change control board
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 30/50
Software Testing Copyright © André Murbach, 200530
Types of Tests
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 31/50
Software Testing Copyright © André Murbach, 200531
Testing Techniques
Behavioral
Functional
StaticDynamic
Non-functional
Structural
DataFlow
ControlFlow
Defs/UsesCriteria
StatementCoverage
BranchCoverage Condition
Coverage
Equivalence Partitioning
Boundary Analysis
State Transition Techniques
Usability
Performance
Reliability
etc.etc.
etc.
etc.
Reviews
Inspections
StaticAnalysis
etc.
FormalVerification
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 32/50
Software Testing Copyright © André Murbach, 200532
What Is a Testing Technique?
A procedure for selecting test cases
Systematic, less dependent on individual test designer
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 33/50
Software Testing Copyright © André Murbach, 200533
Equivalence Partitioning
Partition the inputs (or outputs) into areas which areconsidered equivalent (e.g., same qualitative behavior)
Assumption: if one value works, all will work
Take one value from each equivalence class
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 34/50
Software Testing Copyright © André Murbach, 200534
Domain Testing
„Bugs lurk in corners and congregate at boundaries.“
Determine test points for each partition boundary – ON pointsand OFF points
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 35/50
Software Testing Copyright © André Murbach, 200535
State Transition Testing
Based on a model of the expected behavior
Possible coverage criteria:
Exercise every transition
Exercise every transition pair
Invalid transitions: for each state, apply each invalidevent
...
On / RedButton /
„woom!“
Off /
OFF ON
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 36/50
Software Testing Copyright © André Murbach, 200536
Statement Coverage / Branch Coverage
Statement coverage: Exercise every statement in the sourcecode
Branch coverage: Exercise every branch / decision outcome
Read A, B
If (A < 0) then
Print „A negative“
endIf
If (B < 0) then
Print „B negative“
endIf
Read
A
B
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 37/50
Software Testing Copyright © André Murbach, 200537
A Selection Of Testing Techniques
Functional Tests
Equivalence
partitioning Boundary analysis
Domain analysis
Cause-effect analysis State-based tests
Use case testing
Structural Tests
Statement coverage
Branch coverage
Condition coverage
Data flow based tests
Other Techniques
Error guessing
Exploratory / ad hoctesting
Risk-based testing
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 38/50
Software Testing Copyright © André Murbach, 200538
Testing Throughout The Lifecycle
Test Stages ( Testing Techniques):
Unit test / module test
Integration test
System test
Acceptance test
Alpha test
Beta test / field test
Regression test
Smoke test / sanity check
related to the „V-Model“
tests at system level
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 39/50
Software Testing Copyright © André Murbach, 200539
The V-Model
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 40/50
Software Testing Copyright © André Murbach, 200540
System Test – Indispensable
System test finds defects not detectable at lower levels
The complete system exhibits behaviors not present /
testable at lower levels
Severity of defects not detectable by system test typically low
Defects not easily visible at system level are not soproblematic to users
-> Defects found at system level are those that are really
relevant to the users
Other tests at system level (smoke test, beta test, etc.) areless thorough / systematic
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 41/50
Software Testing Copyright © André Murbach, 200541
A Minimal Test Process
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 42/50
Software Testing Copyright © André Murbach, 200542
Introducing a Minimal Test Process
Suggesting a test process that is simple yet effective. Might beuseful if there isn't already anything better in place.
Overview:
A single test stage: system test
Task: Test planning & test design Fault model-based – enumerate potential problems &
cover them by tests
Task: Test execution Apply test cycles on a succession of release candidates
Defect management; change & version control; release
decision
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 43/50
Software Testing Copyright © André Murbach, 200543
Task: Test Planning & Test Design
Test design based on a fault model
Idea: Enumerate potential faults, write test procedures tocover them
The primary challenge: Cover the most critical faults „What is not tested will not work!“ (Murphy, Murbach &
Küpfer, 2002 ;-)
Identifying everything that should be tested is moreimportant than applying sophisticated testingtechniques
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 44/50
Software Testing Copyright © André Murbach, 200544
Proposal: Mind Map As Fault Model
Explore the application – refine the mind map
Identify types of potential faults – what could go wrong?
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 45/50
Software Testing Copyright © André Murbach, 200545
Test Design Based On Fault Model
1. Define fault model
What faults are conceivable? What could possibly go wrong?
– An item might be as simple as: „feature XY notworking for typical values“
Which faults do I want to test for? Base your choice on risk / criticality.
Goal: Not to miss any important faults
2. Select test models & techniques – or find your own „naive“ approach
Such that they cover the fault model
3. Develop / use test models & apply techniques
-> Test conditions
4. Write test procedures
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 46/50
Software Testing Copyright © André Murbach, 200546
Test Design – Writing Test Procedures
To write the test procedures:
To specify the tests to execute, find a format you feel
comfortable with Be short & precise
Always consider maintainability
Example:
Description Details
Feature X in mode Y.- Goto mode Y, set a=0, - Green light goes on
select b such that ... - ...
T k T E i
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 47/50
Software Testing Copyright © André Murbach, 200547
Task: Test Execution
1st full test cycle near code complete
Goal: Gain overview over the pending problems
Development & testing continues in parallel
When you have a defined version without any known defectsplanned to be fixed – declare it a release candidate (RC)
RC is the software (including the known bugs) to beshipped provided no new high-severity defects are found
Typically, there is a succession of RC's (RC1, RC2, ...)
Each time a non-acceptable defect is found, a newversion is produced -> RC(n+1)
A complete test cycle passed on RCn without any non-acceptable defects -> Declare RCn the release version
P C i & T A i i i
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 48/50
Software Testing Copyright © André Murbach, 200548
Process – Construction & Test Activities
RC2 RC3RC1
full cycle
Req. Analysis – Design – Implementation
Test DesignTest Cycles
& Retesting
Test Planning
Release
T t E ti S Hi t
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 49/50
Software Testing Copyright © André Murbach, 200549
Test Execution – Some Hints
Triage on found defects
Test manager & project manager decide on which
defects to fix – and which not
Be aware of the cost and risks of defect fixes late in the project
New defects may appear (newly introduced or previouslymasked)
-> Additional test cycles may be needed
Fix defects as early as possible
Plan for sufficient test cycles
Summary & Complement:Key Testing Principles & Influences
8/4/2019 Software Testing – An Overview & Some Practical Hints - Dr. André Murbach
http://slidepdf.com/reader/full/software-testing-an-overview-some-practical-hints-dr-andre-murbach 50/50
Software Testing Copyright © André Murbach, 200550
Key Testing Principles & Influences
Six Key Testing Principles
Complete testing is notpossible
Testing is creative and
difficult An important reason for
testing is to prevent errors
Testing is risk-based Testing must be planned
Testing requiresindependence
Key Influences
Individual creativity andinsight
Application knowledge and
understanding Testing experience
Testing methodology
Effort and resources
Source: B. Hetzel, The Complete Guide to Software Testing, 1988
Top Related