SENG 637 Dependability Reliability & Dependability, Reliability &...
70
SENG 637 SENG 637 Dependability Reliability & Dependability Reliability & Dependability, Reliability & Dependability, Reliability & Testing of Software Testing of Software Systems Systems P i f T t P i f T t Preparing for Test Preparing for Test (Chapter 6) (Chapter 6) Department of Electrical & Computer Engineering, University of Calgary B.H. Far （[email protected]） http://www enel ucalgary ca/People/far/Lectures/SENG637/ [email protected] 1 http://www.enel.ucalgary .ca/People/far/Lectures/SENG637/
Transcript of SENG 637 Dependability Reliability & Dependability, Reliability &...
SENG 637SENG 637Dependability Reliability & Dependability Reliability & Dependability, Reliability & Dependability, Reliability & Testing of Software Testing of Software SystemsSystems
P i f T tP i f T tPreparing for TestPreparing for Test(Chapter 6)(Chapter 6)
Department of Electrical & Computer Engineering, University of Calgary
B.H. Far （[email protected]）http://www enel ucalgary ca/People/far/Lectures/SENG637/
http://www.enel.ucalgary.ca/People/far/Lectures/SENG637/
ContentsContentsContentsContents Direct and indirect variables
O ti fi ld d i t t Operation, field and regression test What is a test case?
How to manage test cases? How to manage test cases? Test procedure Equivalence classes and Equivalence classes and
boundary conditions How to develop test cases? How to develop test cases? Preparing for a successful test
SRE: Process (Review)SRE: Process (Review)SRE: Process (Review)SRE: Process (Review)
5 steps in SRE pprocess: Define necessary Define necessary
Define NecessaryReliability
Develop Operational Profilereliabilityreliability
Develop operational profiles
Operational Profile
Prepare for Test
operational profiles Prepare for test Execute test
Execute Test
Apply Failure Data to Guide Decisions
Apply failure data to guide decisions
to Guide Decisions
Chapter 6 Chapter 6 Section 1Section 1
T t M tT t M tTest Management Test Management
Concepts /1Concepts /1Concepts /1Concepts /1 Run:Run: The smallest division of work that can be
initiated by external intervention Run is associatedinitiated by external intervention. Run is associated with input state (set of input variables) and runs with identical input state are of the same run type.
Operation:Operation: An operation is a group of run types as conceived at the development stage.
ll h Test procedure:Test procedure: A controller that sets up environmental conditions and invokes randomly selected test cases at random timesselected test cases at random times.
Concepts /2Concepts /2Concepts /2Concepts /2Common types of testCommon types of test
Feature Test:Feature Test: A single execution of an operation with interaction between operations minimized.
Load Test:Load Test: Testing with field use data and accounting for interactions.
Regression Test:Regression Test: Feature tests after every build involving significant change, i.e., check whether a bug fix workedbug fix worked.
Concepts /3Concepts /3Concepts /3Concepts /3 Direct input variable:Direct input variable: a variables that controls the
operation directly. Example: arguments, selection menu, entered data field. Important during feature test.
Indirect input variable:Indirect input variable: a variable that only influences the operations or its effects are propagated to the operation. Example: traffic load, environmental variable. Important during load test.
ExampleExampleExampleExample Example of direct and indirect input
variables for an operation
Example from Musa’s Book
What is a Test Case?What is a Test Case?What is a Test Case?What is a Test Case? A test case is a partial specification of a run
th h th i f it di t i tthrough the naming of its direct input variables and their values.T t i i d d t f ti l d Test case is independent of operational mode
The same test case can execute in different ti l d Th t toperational modes. Thus a test case can
generate multiple runs each with different potential failure behaviorpotential failure behavior.
PropertiesPropertiesPropertiesPropertiesA “good” test case satisfies the followings: It has a reasonable probability to catch an
error It is not redundant It is the best of its breed It is the best of its breed It is neither too simple nor too complex It makes program failure obvious
ExampleExampleExampleExample Test case is specified with
it di t i t i blits direct input variables In theory, it is possible (but
may not be practical) tomay not be practical) to record all the input variables needed to initiatevariables needed to initiate the runs that make up the whole execution space of the software
Example from Musa’s Book
Test Case & RunTest Case & RunTest Case & RunTest Case & Run Specification of the indirect input variable gives a
test case the necessary context so it can become a runtest case the necessary context so it can become a run. During feature test and regression test, the influence
of indirect input variables should be kept to p pminimum to ensure that the operation being tested is reliable.
di i i bl ff i d i l d Indirect input variables are effective during load test. The load test is divided into a number of operation modes each derived by a test proceduremodes each derived by a test procedure.
ProcedureProcedureProcedureProcedureThe procedure involves two steps:1) Preparing test cases (Test case
management) Using either field data recording or operational
profiles2) P i t t d2) Preparing test procedures
Prepare one test procedure for each operational mode using operational profile and the operationmode using operational profile and the operation occurrence rate
Test Cases ManagementTest Cases ManagementTest Cases ManagementTest Cases ManagementThe procedure for preparing test cases involves:1 E ti t th b f t t d d f1. Estimate the number of new test cases needed for
the current release2. Allocate the number of new test cases among the g
subsystems to be tested (system level)3. Allocate the number of new test cases for each
subsystem among its new operations (operationsubsystem among its new operations (operation level)
4. Specify the new test cases5. Adding the new test cases to the ones already
available (may be from a previous release)
1 Estimate New Test Cases1 Estimate New Test Cases1. Estimate New Test Cases1. Estimate New Test Cases Affected by two factors: time and cost Method:Method:
Compute the number of test cases for the given Time:(available time available staff)/
(average time to prepare a test case) Cost:(available budget) / (average preparation cost per test case)
Select the minimum number of the two
2 Test Case Allocation: System2 Test Case Allocation: System2. Test Case Allocation: System2. Test Case Allocation: System Allocate the bulk
f th t t I te f e to othe teof the test cases to the developed product itself
Interface to other systems
product itself.Developed
componentsAcquired
components
OS, System software
Hardware
2 Test Case Allocation: System2 Test Case Allocation: System2. Test Case Allocation: System2. Test Case Allocation: System Allocate the bulk of the test cases to the developed
product itself.product itself. Give weight to the operations that have higher
occurrence probabilities. ll i d Do not allocate test cases to acquired components
unless they are newly acquired or substituted for the previous ones. In this case, allocate test cases to p ,acquired components based on their size relative to the product and estimated risk.
If the operating system represents a substantial part If the operating system represents a substantial part of the product and its reliability is unknown, assign 20-30% of the test cases to the operating system.
3 Test Case Allocation: Operations3 Test Case Allocation: Operations3. Test Case Allocation: Operations3. Test Case Allocation: Operations1. Convert graphical representation of the operational profile
to the tabular representation by walking through all theto the tabular representation by walking through all the paths, obtain an occurrence probability for each operation (path) by multiplying the branch probabilities together.
2 Identify rarely occurring critical new operationsrarely occurring critical new operations and2. Identify rarely occurring critical new operationsrarely occurring critical new operations and determine how many test cases to pre-assign to each.
3. Assign one test case to each infrequent new operationinfrequent new operation. 4. Determine the allocation probabilities for the remaining remaining
new operationsnew operations. Assign the remaining test cases to the remaining new operations in accordance with the allocation probabilities.
Test Case Allocation: OperationsTest Case Allocation: OperationsTest Case Allocation: OperationsTest Case Allocation: Operations
New operationsDetermine the threshold occurrence
Infrequent
occurrence probability: 0.5 divided by the number of new test
Critical
cases. Assign one test case to each infrequent new infrequent new
Identify rarely rarely occurring critical occurring critical
operationoperation. new operationsnew operationsand assign 2-4 test cases to each.Assign the remaining test cases to the remaining new
operations in accordance with the occurrence probabilities
operations in accordance with the occurrence probabilities.
3 1 Critical Operations3 1 Critical Operations3.1 Critical Operations3.1 Critical Operations A critical operationcritical operation is one for which successful execution
adds a great deal of extra value and failure causes a great deal g gof impact with respect to human life, cost, or system capability (defined by failure severity classes).
An example of a critical operation is the SCRAM operation in l l h h d h inuclear power plants that shuts down a reactor when it starts
to overheat. The operation that handles this condition is very rarely used, but it is extremely critical. Id tif l th l i iti l ti b Identify only the rarely occurring critical operations because sufficient test cases will be allocated to the frequently occurring critical operations because of their substantial occurrence probabilitiesoccurrence probabilities.
3 2 Infrequent Operations3 2 Infrequent Operations3.2 Infrequent Operations3.2 Infrequent Operations Infrequent operationsInfrequent operations are those that would not normally be
assigned test cases because of their very small occurrenceassigned test cases because of their very small occurrence probabilities.
Set the threshold occurrence probability below which you would not normally assign a test case to an operationwould not normally assign a test case to an operation.
A good choice for the threshold occurrence probability is 0.5 divided by the number of new test cases.
Assign one test case to each other new operation whose occurrence probability falls below the threshold.
3 3 New Operations3 3 New Operations3.3 New Operations3.3 New Operations In the case of a new release, set the allocation probabilities for
the remaining new operationsremaining new operations equal to the occurrencethe remaining new operationsremaining new operations equal to the occurrence probabilities of the system operational profile.
This is a satisfactory approximation; the sum of the occurrence probabilities for the other new operations will beoccurrence probabilities for the other new operations will be very close to 1.
For subsequent releases, adjust the system occurrence b biliti b th ti i t ti f t (if )probabilities by the operations interaction factor (if necessary).
Example /1Example /1Example /1Example /1 Total number of test cases: 500 First release: All test cases are new First release: All test cases are new. Suppose that we have one critical operation. Assign 2 test
cases to it. Threshold occurrence probability: 0.5 / 500 = 0.001 Suppose that the number of infrequent operations with
occurrence probabilities below threshold is 2.p Assign 1 test case to each infrequent operation. Distribute the remaining 500 - (2+2) = 496 test cases
th t f ti b d th iamong the rest of operations based on their occurrence probabilities.
Example contd /2Example contd /2Example contd. /2Example contd. /2 Example: Occurrence probabilities for
normal operation mode.
Infrequent Infrequent operations below operations below thresholdthreshold
Critical operationCritical operation Table from Musa’s Book
Example contd /3Example contd /3Example contd. /3Example contd. /3
Divided based on Divided based on occurrence occurrence probabilitiesprobabilities
Infrequent Infrequent operations belowoperations below
probabilitiesprobabilities
operations below operations below thresholdthreshold
Critical operationCritical operationTable from Musa’s Book
4 Specify Test Cases /14 Specify Test Cases /14. Specify Test Cases /14. Specify Test Cases /1 Definition:Definition: A test case is a partial
ifi ti f th h th i fspecification of a run through the naming of its direct input variables and their values.B tt D fi itiB tt D fi iti A t t i i t Better Definition:Better Definition: A test case is an instance (or scenario) of a use-case composed of a set of test inputs execution conditions andof test inputs, execution conditions and expected results.
Prior to specifying test cases one should Prior to specifying test cases one should document the use-cases.
4 Specify Test Cases /24 Specify Test Cases /24. Specify Test Cases /24. Specify Test Cases /2 A typical use-case definition document
i t fconsists of: Use Case name
A t Actor name Objective
PreconditionsActor Use Case
Preconditions Results (Post-conditions) Detailed description
Use-C
as
Detailed description Exceptions and alternative courses
se Definition
n
UseUse--Case Definition WorksheetCase Definition WorksheetUseUse--Case Definition WorksheetCase Definition Worksheet
Test Case DefinitionTest Case DefinitionTest Case DefinitionTest Case Definition At least two test cases for each use-case definition:
One for success and one for failure
More test cases can be generated for exceptions and lt ti f th d fi itialternative courses of the use-case definition
document A test case definition document usually consists of: A test-case definition document usually consists of:
Test-case ID, parent use-case ID, test objective, test condition, operator action, input specification, output , p , p p , pspecification (expected results) and binary decision of pass or fail
TestTest--Case Definition WorksheetCase Definition WorksheetTestTest--Case Definition WorksheetCase Definition Worksheet
5 Adding to Test Suite5 Adding to Test Suite5. Adding to Test Suite5. Adding to Test Suite A test suite is usually composed of old test cases
(f f l ) d t t (f(from a former release) and new test cases (for new features added).A test operational profile for each operation mode A test operational profile for each operation mode must be developed.
Test operational profile is different from operational Test operational profile is different from operational profile for each operation mode due to: The probability assigned to rarely occurring critical new The probability assigned to rarely occurring critical new
operations Adding new operations
5 Adding to Test Suite (cont’d)5 Adding to Test Suite (cont’d)5. Adding to Test Suite (cont d)5. Adding to Test Suite (cont d) To specify the test operational profile for an
ti l d t t ithoperational mode, start with: Operational mode & operational profile (for the first
release)release) Operational mode and test operational profile (for
subsequent releases)
Modify the occurrence probabilities to account for rarely occurring critical new operations.
Modify the occurrence probabilities for newly added operations using operation interaction factor
Example /1Example /1Example /1Example /1 Unadjusted operational profile Operation mode: Peak hour Operation mode: Peak hour 2 test cases (out of 500) assigned to the rarely occurring
critical operation “recover from hardware failure”
Critical operationCritical operation Table from Musa’s Book
Example (contd ) Example (contd ) Example (contd.) Example (contd.) Occurrence probabilities for the rarely occurring critical
operation: 2/500 = 0 004operation: 2/500 = 0.004 Occurrence probabilities are adjusted to account for this.
Test operational profile for the 1st
release
Table from Musa’s BookOccurrence probability of Occurrence probability of Critical operationCritical operation
Example (contd )Example (contd )Example (contd.)Example (contd.) Occurrence probabilities for the 2nd release after adding 2 new operations Operational interaction factor usually 0 2 < x < 0 8 Operational interaction factor usually 0.2 < x < 0.8 The smaller value: higher effectiveness of new operation
Test operational profile for 0.
2 pthe 2nd
release
0
Table from Musa’s Book
Test ProceduresTest ProceduresTest ProceduresTest Procedures Test procedure is a controller that sets up
environmental conditions and invokes randomly selected test cases at random times.
Prepare one test procedure for each operational mode.
How to Specify Test Cases?How to Specify Test Cases?How to Specify Test Cases?How to Specify Test Cases? For a given operation, quantify the value of direct
input variables based on levels (values) for which theinput variables based on levels (values) for which the same behavior will be expected.
A test case for a given operation (function) can be g p ( )specified using: Combination of values for its direct input variables.
i i f i di i i bl Timing of its direct input variables. If the number of combinations exceed the number of
test cases assigned to that operation only a portion oftest cases assigned to that operation only a portion of combinations must be selected to represent all.
Example /1Example /1Example /1Example /1 ATM Withdraw From Checking account Direct input variables
ATM card : readable unreadable Account : valid invalid PIN : valid invalidN : v d v d Account balance : plus minus Cancel key Cancel key
Example /2Example /2Example /2Example /2 Suppose that we are going to build test cases for an
ATM t Th ti th t ld lik tATM system. The operation that we would like to test is “Withdraw from Checking account”. Suppose that a new input variable named “Withdraw amount”that a new input variable named Withdraw amount with the following specification is given.
Variable name/ definition Specification Name: Withdraw amountDefinition: The amount of
i
3 digit numbers are accepted (withdrawal only between 100$ and 1,000$ cash)
h b i h 0cash that can be withdrawn in a single transaction
The number cannot start with 0The rightmost digit must be 0 (10$, 20$, 50$ and 100$ bills only)4 digit numbers, only 1000 is acceptable
g , y pAny other number of digits is not acceptable
Example /2 (cont’d)Example /2 (cont’d)Example /2 (cont d)Example /2 (cont d) Specify the “minimum” set of test-cases to
test the operation for this variable.
It Value of the variable “Withdraw amount” C it i ( f il)Item Value of the variable “Withdraw amount Criteria (pass, fail)1 i where 0 i 9 fail2 ij where 0 i 9 and 0 j 9 fail3 ijk where i =0 and 0 j 9 and 0 k 9 fail4 ijk where 1 i 9 0 j 9 and k 0 fail
5 ijk where 1 i 9 0 j 9 and k =0 pass5 ijk where 1 i 9 0 j 9 and k 0 pass6 ijkl where i=1 and j = k = l = 0 pass7 ijkl where 0 j, k, l 9 and i 1 fail
Chapter 6 Chapter 6 Section 2Section 2
C ti T t CC ti T t CCreating Test Cases Creating Test Cases and Test Suitand Test Suit
How to Create Test Cases?How to Create Test Cases?How to Create Test Cases?How to Create Test Cases?1. Test case creation from user scenarios, use-case
d fi iti ( d/ ti it di )definition (and/or activity diagram).2. Test case creation based on equivalent classes.
P l f h i l Prepare only one test case for the representative class.
3. Test case creation based on boundary conditions.P th t f il ith b d l f il t th Programs that fail with non-boundary values fail at the boundaries, too.
4 Test case creation based on visible state transitions4. Test case creation based on visible state transitions.
1 Equivalence Classes1 Equivalence Classes1. Equivalence Classes1. Equivalence ClassesA group of tests cases are “equivalent” if:A group of tests cases are “equivalent” if: They all test the same operation. If one test case can catch a bug, the others will
probably do. If one test case does not catch a bug, the
others probably will not do. They involve the same input variables. They all affect the same output variables.
Equivalence Classes /1Equivalence Classes /1Equivalence Classes /1Equivalence Classes /1Hint (1):Hint (1):
D ’t f t i l l f i lid i t Don’t forget equivalence classes for invalid inputs.
Example:Example: For a program that is supposed to accept any number between 1 and 99There are 4 equivalence classes:
Any number between 1 and 99 (valid)Any number between 1 and 99 (valid) Any number less than 1 (including zero and negative numbers)
(invalid) Any number greater that 99 (invalid)y g ( ) Any non-number (invalid)
Equivalence Classes /2Equivalence Classes /2Equivalence Classes /2Equivalence Classes /2Hint (2):Hint (2): Look for extreme range of variables.
Example:Example: For a program that is supposed to accept anyExample:Example: For a program that is supposed to accept any integer between 1 and 99 Try very large values for the number (check if both 16 bit
or 32 bit integers work). Try very small negative values to check if sign can be
handled properly.handled properly.
Equivalence Classes /3Equivalence Classes /3Equivalence Classes /3Equivalence Classes /3Hint (3):Hint (3): Look for maximum size of memory (stackable)
variables.
Example:Example: For a depth-first (breath-first) graph search operation.p Set the length of the path (number of the child nodes) to be
very long (to be very high) and check for memory overflowoverflow.
Equivalence Classes /4Equivalence Classes /4Equivalence Classes /4Equivalence Classes /4Hint (4):Hint (4):
If i t t b l t If an input must belong to a group, one equivalence class must include all members of the group.the group.
Example:Example: For a program that accepts users names as its inputinput All strings of alphabet starting with a capital or lower case
letters are acceptable.Li it th i f th t i t h t t it t b Limit the size of the string to one character or set it to be very long and check if both work.
Check for non-ascii characters, if needed.
Equivalence Classes /5Equivalence Classes /5Equivalence Classes /5Equivalence Classes /5Hint (5):Hint (5): Analyze levels for binary value variables.
Example:Example: For a program that asks “Are you sure? (Y/N) One equivalence class contains “Y” (and may be “y” and
“Yes” and “yes”) The other class contains “N” (and may be “n” and “No” The other class contains N (and may be n and No
and “no”) and anything else other than “Y”.
Equivalence Classes /6Equivalence Classes /6Equivalence Classes /6Equivalence Classes /6Hint (6):Hint (6): Look for dependent variables and
replace them by their equivalentsreplace them by their equivalents.
Example:Example: For a program that accepts threeExample:Example: For a program that accepts three angles of a triangle as its input only two of them are independentthem are independent.
2 Boundary Conditions2 Boundary Conditions2. Boundary Conditions2. Boundary Conditions Check if the boundary conditions for variables
are set correctly. Check if the inequality boundary conditions q y y
can be changed to equality or not. Check if the counter variables allow departureCheck if the counter variables allow departure
from the loop correctly or not. etc etc.
Boundary Conditions /1Boundary Conditions /1Boundary Conditions /1Boundary Conditions /1Example (1):Example (1):
If the program expects an upper case letter A-Z, Test @ (ASCII code just below the code for A) and [ (character just beyond Z)and [ (character just beyond Z).
Example (2):Example (2):Example (2):Example (2): If the program needs a specific number of inputs,
give it that many, one more and one fewer.g y,
Boundary Conditions /2Boundary Conditions /2Boundary Conditions /2Boundary Conditions /2Example (3):Example (3): If a counter is set to have values:
0<i< 10, check if 0 and 10 are correct0 i 10, check if 0 and 10 are correct boundary values.
Example (4):Example (4):Example (4):Example (4): When reading from or writing to a disk file,
h k th fi t d l t h t i th filcheck the first and last character in the file.
3 Visible State Transitions3 Visible State Transitions3. Visible State Transitions3. Visible State Transitions Every interaction with the program, i.e., setting an
i t i bl l ti it t k thinput variable, selecting a menu item, etc., makes the program state move to another state. Test all paths that are likely to be followed by ordinary Test all paths that are likely to be followed by ordinary
users under normal conditions. Test any setting in one place whose effects are suspected
to be propagated to elsewhere in the program.T f h d h Test a few other random paths through the program.
Documenting CrashDocumenting CrashDocumenting CrashDocumenting Crash Document all the input variables and/or their
bi ti th t k t dcombinations that make unexpected program crash.W it t t th t d th h Write a test case that can reproduce the crash.
Find equivalent class cases for the crash test case.
The same test case can be used in the i t t h l tregression test phase later.
Preparation for Successful TestPreparation for Successful TestPreparation for Successful TestPreparation for Successful Test What is necessary for successful testing?
Test planning Test processes
T l Test tools Management support for test
T t t i i Test training Test efficiency
Test quality control Test quality control User satisfaction
Assessment ChartAssessment ChartAssessment ChartAssessment Chart Successful
testing can be assessed based on 8 factors.
Example and chart from Perry’s Book
Assessment: Test PlanningAssessment: Test PlanningAssessment: Test PlanningAssessment: Test Planning Does the tester identify the business and technical risks
associated with implementing and operating software, and are p g p gthose risks addressed in the test plan?
Is the test plan created for each software system? Does the plan identify the test requirements/objectives and does it
bli h i i f h iestablish success criteria for each requirement? Are the users actively involved in the development of the test
plan, and does the plan include user involvement in testing? Are the test plans followed? And are the plans changed as
software requirements change? Does the test plan contain criteria that the software must meet
in order to be released and do the customers agree with this criteria?
Example and chart from Perry’s Book
Assessment: Test ProcessesAssessment: Test ProcessesAssessment: Test ProcessesAssessment: Test Processes Do testers follow processes to plan tests, prepare test
data, execute tests and report test results?data, execute tests and report test results? Can documented test processes be correctly
interpreted by testers so that the test procedures can be followed as intended?be followed as intended?
Do the processes provided for testing cover all the activities that are needed to perform effective ptesting?
Has the plan been developed and put in place to mature the test processes so they become moremature the test processes so they become more effective, efficient and on time?
Do the testers build the processes used for testing?
Example and chart from Perry’s Book
Assessment: Test ToolsAssessment: Test ToolsAssessment: Test ToolsAssessment: Test Tools Do testers use an automated tool to generate and
reuse test data?reuse test data? Are test tools selected in a logical manner? Meaning,
test needs drive the search for acquisition of test tools?tools?
Can testers only use test tools after they have received adequate training in how to use the test q gtools?
Is test tool usage specified in the test plan?Has a process for obtaining assistance in sing test Has a process for obtaining assistance in using test tools been established and does it provide testers with the needed instructional information?
Example and chart from Perry’s Book
Assessment: Management Assessment: Management SupportSupportSupportSupport
Does management provides necessary resources to adequately train, plan, execute and evaluate results for software testing p gassignments?
Are testers involved from the inception through termination of software project to ensure that testing concerns are
i l dd dcontinuously addressed? Does management allocate as many resources to the test
processes and tools as it does to the development process and t l ?tools?
Does management spend as much personal time on test planning and execution as it does on the development l i d ti ?planning and execution?
Is management sufficiently knowledgeable in test theory, processes and tools to understand the test planning and execution?
execution?Example and chart from Perry’s Book
Assessment: Test TrainingAssessment: Test TrainingAssessment: Test TrainingAssessment: Test Training Does a career training plan for testers exist? And is it
in use?in use? Are testers adequately trained in test processes
before using those processes for testing?i d i h h f i h Are testers trained in the theory of testing so that
they understand why they perform certain tasks? Are testers trained in statistics so that they know how Are testers trained in statistics so that they know how
to interpret the test results? Are testers trained in how to measure process
performance and do the se the res lts to impro eperformance and do they use the results to improve the test processes?
Example and chart from Perry’s Book
Assessment: Test EfficiencyAssessment: Test EfficiencyAssessment: Test EfficiencyAssessment: Test Efficiency Has the test planned been developed so that the test
resources will be allocated to validate that the majorresources will be allocated to validate that the major risks are addressed prior to minor risks?
Has a measurement process been installed to measure the efficiency of the test processes?measure the efficiency of the test processes?
Is compliance to the budget and schedule measured and variances addressed effectively?y
Is tool usage measured to assess the contribution received from automated testing?Is the percentage of defects remo ed ers s the total Is the percentage of defects removed versus the total defects eventually attribute to a development phase measured?
Example and chart from Perry’s Book
Assessment: Test Quality ControlAssessment: Test Quality ControlAssessment: Test Quality ControlAssessment: Test Quality Control Are defects made by the testers during testing recorded and
efficiently addressed?efficiently addressed? Is the test plan reviewed/inspected during/after completion by
peers for adequacy and compliance to test standards?D th t t l i l d th d th t ill b d t Does the test plan include the procedures that will be used to verify that the plan is executed in accordance with the plan?
Are regular reports prepared that show the full status of testing individual software systems?
Are the quality control reports summarized periodically to show the efficiency and effectiveness of testing in the entireshow the efficiency and effectiveness of testing in the entire information services organization?
Example and chart from Perry’s Book
Assessment: User SatisfactionAssessment: User SatisfactionAssessment: User SatisfactionAssessment: User Satisfaction Do users get the information they need to track test progress
and assess results?and assess results? Are user surveys conducted to determine user satisfaction
with test planning, execution, results, communications, etc.?D ti i th t t th t d t i h th t th Do users practice in the test that determine whether or not the software is acceptable for use?
Are users presented with a plan for testing? And do they approve that if the plan is followed they consider testing to be satisfactory?
Are the user support activities such as data entry, outputAre the user support activities such as data entry, output usage, terminal usage, manual usage, etc., validated as part of testing?
Example and chart from Perry’s Book
ExampleExampleExampleExample Suppose we are testing a software module of an
editor system that allows a user to enter new fonteditor system that allows a user to enter new font identifiers into a font database. The input specification for the module states that a font identifier shouldidentifier should (i) consist of alphanumeric characters; (ii) the range for the total number of characters is between
3 and 15; and3 and 15; and, (iii) the first two characters must be letters.
When preparing test cases we will focus on selecting p p g g“equivalence classes” and “boundary values” for the inputs.
Example (cont’d)Example (cont’d)Example (cont d)Example (cont d) Identify input equivalence classes for this
example.
Condition 1:C1. Part name is alphanumeric, valid.C2. Part name is not alphanumeric, invalid.Condition 2:Condition 2:C3. Font identifier has between 3 and 15 characters, valid.C4. Font identifier has less than 3 characters, invalid.C5. Font identifier has greater than 15 characters, invalid.C diti 3Condition 3:C6. The first 2 characters are letters, valid.C7. The first 2 characters are not letters, invalid.
Example (cont’d)Example (cont’d)Example (cont d)Example (cont d) Identify boundary values for this example.
BLB : a value just below the lower boundLB : the value on the lower boundaryLB : the value on the lower boundaryALB : a value just above the lower boundaryBUB : a value just below the upper boundUB : the value on the upper boundUB : the value on the upper boundAUB : a value just above the upper bound
For our example module the values for the bounds pgroups are:
BLB—2 BUB—14LB—3 UB—15
ALB—4 AUB—16
Example (cont’d)Example (cont’d)Example (cont d)Example (cont d) Develop a set of test cases for this module based on
th i l l d b d lthe equivalence classes and boundary values.
Input values Relevant Conditions Pass/Fail criteriaInput values Relevant Conditions Pass/Fail criteria
1 abc1 C1; C3; C6; ALB Pass
2 ab1 C1; C3; C6; LB Pass
3 abcdef123456789 C1; C3; C6; UB Pass3 abcdef123456789 C1; C3; C6; UB Pass
4 abcde123456789 C1; C3; C6; BUB Pass
5 12* C7 Fail
6 ¥©* C2 Fail6 ¥©* C2 Fail
7 ab C4 Fail
8 abcdedgh1234567890 C5 Fail
