02-Software Testing Introduction(Tt)_1
-
Upload
le-dung-huynh -
Category
Documents
-
view
222 -
download
0
Transcript of 02-Software Testing Introduction(Tt)_1
-
8/6/2019 02-Software Testing Introduction(Tt)_1
1/32
Software Testing
Introduction
Software Testing 2
The hardest task Can write a program in this 50 minute class Can also have 50 million possible combinations to
test. Labor involved can cost fortunes. Testing is often the first thing to slip when resources
stretch - but it's the entire organization that sufferswhen a defect causes problems.
Measurements show that a defect discovered duringdesign that costs $1 to rectify will cost $1,000 torepair in production. This is a tremendous (ln) costdifferential and clearly points out the advantage ofearly error detection.
-
8/6/2019 02-Software Testing Introduction(Tt)_1
2/32
Software Testing 3
Outline Introduction to the subject
What is... error/bug/fault/failure/testing? Psychology, goals and testing theory Overview of the testing process concepts and
dimensions Testing principles Standard testing questions
Classifying aspects of testing
Types Levels
Software Testing 4
What is...
error: something wrong in software itselffault: manifestation of an error
(observable in software behavior)
failure: something wrong in softwarebehavior
(deviates from requirements)
requirements:for input i,
give output 2*(i^3)
(so 6 yields 432)
software:i=input(STDIN);
i=double(i);i=power(i,3);
output(STDOUT,i);
output (verbose):input: 6
doubling input..computing power..
output: 1728
bug
error fault
fault + failure
-
8/6/2019 02-Software Testing Introduction(Tt)_1
3/32
Software Testing 5
QA Vs QC Your ideas ???
Software Testing 6
The Psychology of Testing
Testing is the process of demonstrating(chng minh, gii thch) that an errors arenot present
Testing is the process of establishingconfidence that the program does what itintends to do
Testing is the process of executingprograms with the intent of finding errors Successful (positive) test: exposes an error
-
8/6/2019 02-Software Testing Introduction(Tt)_1
4/32
Software Testing 7
Possible Goals of Testing
Find faults Glenford Myers, The Art of Software
Testing
Provide confidence of reliability (s tin cy) of (probable) correctness (tnh ng n,
chnh xc)
of detection (therefore absence) ofparticular faults (pht hin li c th)
Software Testing 8
What if testing does not detect anyerrors
Either the software is high quality the testing process is low quality
Metrics are required on our testingprocess if we are to tell which is theright answer.
-
8/6/2019 02-Software Testing Introduction(Tt)_1
5/32
Software Testing 9
Three themes prevail:
(3 ch ph bin) Quality Assessment (What percentage
of defects are captured by our testingprocess, how many remain?)
Risk Management (What is the riskrelated to remaining defects?)
Test Process Improvement (How longdoes our testing process take?)
Software Testing 10
What is...
testing:by experiment, find errors in software (Myers, 1979) establish quality of software (Hetzel,
1988)a succesful test: finds at least one error gives quality judgment (phn quyt)
with maximum confidence with minimumeffort
-
8/6/2019 02-Software Testing Introduction(Tt)_1
6/32
Software Testing 11
Testing Theory (such as it is) Plenty of negative results Nothing guarantees correctness Statistical confidence is prohibitively
expensive Being systematic may not improve fault
detection as compared to simple random testing
So what did you expect, decisionprocedures for undecidable problems?
Software Testing 12
Whats been said? Dijkstra:
Testing can show the presence of bugs, but not the absence Beizer:
1st law: Every method you use to prevent or find bugs leaves aresidue of subtler bugs (xt li nhng li khng d phthin)
2nd law: Software complexity grows to the limits of our abilityto manage it
Beizer:Testers are not better at test design than programmers are at
code design ... Lets start with a global look at the testing process
-
8/6/2019 02-Software Testing Introduction(Tt)_1
7/32
Software Testing 13
Concept map of the testing
process
Software Testing 14
Dimensions of software testing1. What is tested?
a. Software characteristics (design/embedded?/language/...)b. Requirements (what property/quality do we want from the software)c. Purpose (what development phase, what kind of errors, what next step)
2. How to test?a. Method (adequacy criterion: how to generate how many tests)b. Assumptions (limitations, simplifications, heuristics)c. Organization (documentation, implementation, execution:platform,
interactive, batch)
d. Who tests? (programmer, user, third party)3. How are the results evaluated?
-
8/6/2019 02-Software Testing Introduction(Tt)_1
8/32
Software Testing 15
Dimensions + concept map
1a
1b
1c2a2b
2c
2d
3
Software Testing 16
What information can we exploit(khai thc) ? 1b. Requirements
functional: the behavior of the system is correct precise
non-functional: performance, reliability, compatibility,
robustness (stress/volume/recovery),usability, ...
possibly quantifiable (c th xc nh slng hp l), always vague (lun mh)
-
8/6/2019 02-Software Testing Introduction(Tt)_1
9/32
Software Testing 17
What other information can weexploit?
Specifications (formal or informal) in Oracles for Selection, Generation, Adequacy
Designs
Code for Selection, Generation, Adequacy
Usage (historical or models) Organization experience
Software Testing 18
1c. Test purposeSoftware development phases (V-
model):
implementationcode
detaileddesign
specification
requirements acceptancetest
systemtest
integrationtest
unittest
-
8/6/2019 02-Software Testing Introduction(Tt)_1
10/32
Software Testing 19
1c. Test purposeWhat errors to find? typical typos functional mistakes unimplemented required features
software does not do all it should do implemented non-required features
software does things it should not do
Software Testing 20
2a. Test method dimensions
data-based
structure-based
black box white boxerror seeding
typical errorsefficiency
...
-
8/6/2019 02-Software Testing Introduction(Tt)_1
11/32
Software Testing 21
2b. Assumptions, limitations single/multiple fault:
clustering/dependency of errors testability:
can software be tested? time, resources, accessibility,
perfect repair ...
Software Testing 22
2c. Test organization Documentation
for reuse! Implementation
platform batch? inputs, coordination, ...
Execution
duration, timing, interruptions manual/interactive or automated in parallel on several systems
-
8/6/2019 02-Software Testing Introduction(Tt)_1
12/32
Software Testing 23
Testing Principles
Test case must include the definition ofexpected results
A program should not test her/his owncode
A programming organization should nottest its own programs
Thoroughly inspect (xem xt k lng)the results of each test
Test cases must be also written for invalidinputs
Software Testing 24
Testing Principles Check that programs do not do
unexpected things Test cases should not be thrown a way Do not plan testing assuming that there
are no errors The probably of error in a piece of code is
proportional(tng ng) to the errorsfound so far in this part of the code
-
8/6/2019 02-Software Testing Introduction(Tt)_1
13/32
-
8/6/2019 02-Software Testing Introduction(Tt)_1
14/32
Software Testing 27
Testers Mindset
A good tester is mischievous (tinh qui)
Emphasis on breadth of expertise
Product
Expertise
Software Testing 28
Slogan !
-
8/6/2019 02-Software Testing Introduction(Tt)_1
15/32
Software Testing 29
Testing is in Your Head
Technical
Knowledge
Specifications
Product
Ah! Problem!
Problem
Coverage
ProblemReport
Communication
The important parts of testing dont
take place in the computer or on your desk.
CriticalThinking
DomainKnowledge
Experience
Software Testing 30
A Testers Attitude Cautious (cn thn)
Jump to conjectures (phng on), not conclusions (ktlun).
Practice admitting I dont know. Have someone check your work.
Curious (t m) What would happen if? How does that work? Why did that happen?
Critical (ch trch, ph phn) Proceed by conjecture and refutation (bc b). Actively seek counter-evidence.
Good testersare hard to fool.
(and Courageous)
-
8/6/2019 02-Software Testing Introduction(Tt)_1
16/32
Software Testing 31
Standard Testing Questions
Did this test execution succeed or fail? Oracles
How shall we select test cases? Selection; generation
How do we know when weve testedenough? Adequacy
What do we know when were done? Assessment
Software Testing
Classifying aspects of testing
-
8/6/2019 02-Software Testing Introduction(Tt)_1
17/32
Software Testing 33
Basic Methods of Testing Black Box White Box
Software Testing 34
Black Box Testing
View the program as a black box Exhaustive testing is infeasible
(khng th thc hin) even for tinyprograms Can never guarantee correctness Fundamental question is economics Maximize investment Example: Partition test
-
8/6/2019 02-Software Testing Introduction(Tt)_1
18/32
Software Testing 35
Black Box Tests
Targeted at the apparent simplicity of the software Makes assumptions about implementation
Good for testing component interactions Tests the interfaces and behavior
Input Output
Software Testing 36
White Box Testing
Investigate the internal structure ofthe program
Exhaustive path testing is infeasible Does not even guarantee
correctness Specification is needed Missing paths Data dependent paths
if (a-b < epsilon)
-
8/6/2019 02-Software Testing Introduction(Tt)_1
19/32
Software Testing 37
White Box Tests
Targeted at the underlying complexity of the software Intimate knowledge of implementation
Good for testing individual functions Tests the implementation and design
Input Output
Software Testing 38
Equivalence partitioning
Boundary-value analysis
Cause-effect graphing
Error guessing
State space exploration
Black box White box
Coverage/Adequacy
Data flow analysis
Cleanness
Correctness
Mutation
Slicing
Black and White Box TestingMethods
-
8/6/2019 02-Software Testing Introduction(Tt)_1
20/322
Software Testing 39
Structural Coverage Testing (In)adequacy criteria
If significant parts of program structure are nottested, testing is surely inadequate
Control flow coverage criteria Statement (node, basic block) coverage Branch (edge) and condition coverage Data flow (syntactic dependency) coverage Various control-flow criteria
Attempted compromise (dung ha)betweenthe impossible and the inadequate
Software Testing 40
Testing LevelsDifferent testing levels
Unit test (procedure boundaries) Module test Function test (discrepancies external spec.) System test (discrepancies with original objectives) Facility test Volume test
Usability test Security test Performance test ...
-
8/6/2019 02-Software Testing Introduction(Tt)_1
21/322
Software Testing 41
Process-Based Reliability (s tin
cy) Testing Rather than relying only on properties
of the program, we may use historicalcharacteristics of the developmentprocess
Reliability growth models (Musa,Littlewood, et al) project reliabilitybased on experience with the current
system and previous similar systems
Software Testing 42
Fault-based testing
Given a fault model hypothesized set of deviations from correct
program typically, simple syntactic mutations; relies on
coupling of simple faults with complex faults
Coverage criterion: Test set should beadequate to reveal (all, or x%) faults
generated by the model similar to hardware test coverage
-
8/6/2019 02-Software Testing Introduction(Tt)_1
22/322
Software Testing 43
The Budget Coverage Criterion A common answer to when is testing
done When the money is used up When the deadline is reached
This is sometimes a rational (da trn ltr) approach! Implication 1: Test selection is more important
than stopping criteria per se. Implication 2: Practical comparison of
approaches must consider the cost of test caseselection
Software Testing 44
The Importance of Oracles
Much testing research has concentrated onadequacy, and ignored oracles
Much testing practice has relied on theeyeball oracle Expensive, especially for regression testing makes large numbers of tests infeasible
Not dependable Automated oracles are essential to cost-
effective testing
-
8/6/2019 02-Software Testing Introduction(Tt)_1
23/322
Software Testing 45
Sources of Oracles Specifications
sufficiently formal (e.g., Z spec) but possibly incomplete (e.g., assertions in Anna,
ADL, APP, Nana)
Design models treated as specifications, as in protocol
conformance testing
Prior runs (capture/replay)
especially important for regression testing andGUIs; hard problem is parameterization
Software Testing 46
What can be automated? Oracles
assertions; replay; from some specifications
Selection (Generation) YOU scripting; specification-driven; replay variations selective regression test
Coverage (XATAC)
statement, branch, dependence Management (GATE)
-
8/6/2019 02-Software Testing Introduction(Tt)_1
24/322
Software Testing 47
Design for Test: 3 Principles Observability (c th quan st)
Providing the right interfaces to observe thebehavior of an individual unit or subsystem
Controllability (c kh nng iu khin) Providing interfaces to force behaviors of
interest
Partitioning (chia ct, c lp) Separating control and observation of one
component from details of others
Test Cycle:What You Do With a Build1. Receive the product. Formal builds Informal builds Save old builds.
2. Clean your system. Completely uninstall earlier builds.
3. Verify testability. Smoke testing Suspend (hon) test cycle if the product
is untestable.
-
8/6/2019 02-Software Testing Introduction(Tt)_1
25/322
Test Cycle:
What You Do With a Build4. Determine what is new or changed.
Change log
5. Determine what has been fixed. Bug tracking system
6. Test fixes. Many fixes fail! Also test nearby functionality.
7. Test new or changed areas. Exploratory testing.
Test Cycle:What You Do With a Build
8. Perform regression testing. Not performed for an incremental cycle. Automated vs. manual Important tests first!
9. Report results. Coverage Observations Bug status (new, existing, reopened, closed) Assessment of quality Assessment of testability
-
8/6/2019 02-Software Testing Introduction(Tt)_1
26/322
General Black Box TestingTechniques
Function testing Domain testing Stress testing Flow testing Scenario testing User testing Regression testing Risk testing Claims testing Random Testing
For any one of these techniques,
theres somebody, somewhere,
who believes that is the
only way to test.
Software Testing 52
Smoke Testing
MSDN: smoke testing describes the process of
validating code changes before the changes arechecked into the products source tree.
After code reviews, smoke testing is the mostcost effective method for identifying and fixingdefects in software.
Smoke tests are designed to confirm thatchanges in the code function as expected and donot destabilize an entire build.
-
8/6/2019 02-Software Testing Introduction(Tt)_1
27/322
Software Testing 53
Smoke Testing On What is ? Smoke testing is non-exhaustive
software testing, ascertaining that themost crucial functions of a programwork, but not bothering with finerdetails.
The term comes to software testing froma similarly basic type of hardwaretesting, in which the device passed thetest if it didn't catch fire the first time itwas turned on
Function Testing
March through the functions
- A function is something the product can do.
- Identify each function and sub-function.
- Determine how you would know if they worked.
- Test each function, one at a time.
Key Idea:
Summary:
Good for: assessing capability rather than reliability(kh nng v tin cy)
-
8/6/2019 02-Software Testing Introduction(Tt)_1
28/322
Domain Testing
Divide the data, and conquer
- A domain is a set of test data.
- Identify each domain of input and output.
- Analyze limits and properties of each domain.
- Identify combinations of domains to test.
- Select coverage strategy:
e.g., exhaustive, boundaries, best representative
Key Idea:
Summary:
Good for: all purposes
Stress Testing
Overwhelm the product
- Select test items and functions to stress.
- Identify data and platform elements that relateto them.
- Select or generate challenging data andplatform configurations to test with:
e.g., large or complex data structures,
high loads, long test runs, many test cases
Key Idea:
Summary:
Good for: performance, reliability, andefficiency assessment
-
8/6/2019 02-Software Testing Introduction(Tt)_1
29/322
Flow Testing
Do one thing after another
- Define test procedures or high level cases that
incorporate many events and states.
- The events may be in series, parallel, or somecombination thereof.
- Dont reset the system between events.
Key Idea:
Summary:
Good for: finding problems fast(however, bug analysis is more difficult)
Scenario Testing
Test to a compelling story
- Design tests that involve meaningful and complexinteractions with the product.
- A good scenario test is a plausible story of howsomeone who matters might do something thatmatters with the product.
- Incorporate multiple elements of the product, and
data that is realistically complex.
Key Idea:
Summary:
Good for: Finding problems that seem important
-
8/6/2019 02-Software Testing Introduction(Tt)_1
30/323
User Testing
Involve the users
- Identify categories of users.
- Understand favored and disfavored users.
- Find these users and have them do testing orhelp you design tests.
- User testing is powerful when you involve avariety of users.
Key Idea:
Summary:
Good for: all purposes
Regression Testing
Test the changes
- Identify what product elements changed.
- Identify what elements could have been
impacted by the changes.
- Coverage strategy:
e.g., recent bug fixes, past bug fixes,
likely elements, all elements
Key Idea:
Summary:
Good for: managing risks related to productenhancement
-
8/6/2019 02-Software Testing Introduction(Tt)_1
31/323
Claims Testing
Verify every claim
- Identify specifications (implicit or explicit).
- Analyze individual claims about the product.
- Ask customer to clarify vague claims.
- Verify each claim.
- Expect the specification and product to be
brought into (hng v) alignment.
Key Idea:
Summary:
Good for: Simultaneously (ng thi) testing theproduct and specification, while refiningexpectations
Risk Testing
Imagine a problem, then look for it.
- What kinds of problems could the product have?
- How would you detect them if they were there?
- Make a list of interesting problems and designtests specifically to reveal them.
- It may help to consult experts, designdocumentation, past bug reports, or apply risk
heuristics.
Key Idea:
Summary:
Good for: making best use of testing resources;leveraging experience
-
8/6/2019 02-Software Testing Introduction(Tt)_1
32/32
Software Testing 63
Random TestingRun a million different tests
- Look for an opportunity to automaticallygenerate thousands of slightly different tests.
- Create an automated, high speed evaluationstrategy.
- Write a program to generate, execute, andevaluate all the tests.
Key Idea:
Summary:
Good for: Assessing reliability across input and time.
General White Box TestingTechniques
Coverage Testing Branch Logic Statement Data flow
Unit Testing DotNet Unit
PhP Unit Cpp Unit Junit Ruby Unit .
For any one of these techniques,
theres somebody, somewhere,
who believes that is the
only way to test.