A Review of Software TestingA Review of Software Testing- P David Coward- P David Coward

Reprinted: Information and Software Reprinted: Information and Software Technology; Vol. 30, No. 3 April 1988Technology; Vol. 30, No. 3 April 1988

Software Engineering: The Software Engineering: The Development Process, Vol 1,Development Process, Vol 1,

Chapter 7Chapter 7

Presented By: Andrew DiemerPresented By: Andrew Diemer

Software Engineering II – EEL 6883Software Engineering II – EEL 6883

Aim of paperAim of paper

No guarantee that software meets No guarantee that software meets functional requirementsfunctional requirements Introduces software testing Introduces software testing techniquestechniques

NeedsNeeds Software is to be correctSoftware is to be correct

– What does this meanWhat does this mean It often means the program It often means the program matches the specifications.matches the specifications.

Problem with specificationProblem with specification– Specification could be wrongSpecification could be wrong

NeedsNeeds If this happens then the If this happens then the correctness is measured by the correctness is measured by the software meeting the user software meeting the user requirementsrequirements

NeedsNeeds TestingTesting

– Why testWhy test Tests may have not been Tests may have not been adequate enoughadequate enough

Asses the performance of the Asses the performance of the taskstasks

TerminologyTerminology Verification –vs- Validation Verification –vs- Validation

VerificationVerification– Ensures correctness from phase Ensures correctness from phase to phase of the software life cycle to phase of the software life cycle processprocess– Formal proofs of correctnessFormal proofs of correctness

TerminologyTerminology ValidationValidation

– Checks software against Checks software against requirementsrequirements

Executes software with test dataExecutes software with test data

Author uses testing and checking Author uses testing and checking instead of verification and validationinstead of verification and validation

Categories of TestingCategories of Testing Two categories of testing:Two categories of testing:

– FunctionalFunctional– Non-functionalNon-functional

Functional TestingFunctional Testing FunctionalFunctional

– Addresses to see if the program Addresses to see if the program obtains the correct outputobtains the correct output

It is normally used when testing a It is normally used when testing a modified or new programmodified or new program

Functional TestingFunctional Testing Regression TestingRegression Testing

– Tests following modificationTests following modification– Tests to see if the unchanging Tests to see if the unchanging functions have indeed changedfunctions have indeed changed

Non-functional RequirementsNon-functional Requirements StyleStyle Documentation standardsDocumentation standards Response timesResponse times Legal obligationsLegal obligations

Situation testingSituation testing Two situations testing can fall Two situations testing can fall under:under:

– Testing which finds faults in the Testing which finds faults in the softwaresoftware– Testing which does NOT find Testing which does NOT find faults in the softwarefaults in the software

Situation testingSituation testing Finding faultsFinding faults

– Destructive processDestructive process more probingmore probing

Not finding faultsNot finding faults– miss inherent faultsmiss inherent faults

too gentletoo gentle

QuestionsQuestions How much testing is needed?How much testing is needed? Confidence in testing?Confidence in testing? Ignore faults?Ignore faults?

– Which ones are important?Which ones are important? Are there more faults?Are there more faults? What is the purpose of this What is the purpose of this testing?testing?

Testing StrategiesTesting Strategies Functional -vs- StructuralFunctional -vs- Structural Static -vs- Dynamic analysisStatic -vs- Dynamic analysis

Strategy starting pointsStrategy starting points SpecificationSpecification

– It makes known the required It makes known the required functionsfunctions– Asses to see if they are providedAsses to see if they are provided– Functional testingFunctional testing

Strategy starting pointsStrategy starting points SoftwareSoftware

– Tests the structure of the systemTests the structure of the system– Structural testingStructural testing– Functions are included into the Functions are included into the system but are NOT requiredsystem but are NOT required– Example: accessing a database Example: accessing a database that has not been asked by the userthat has not been asked by the user

Functional testingFunctional testing Identify the functions which the Identify the functions which the software is expected to performsoftware is expected to perform Creating test data that will check Creating test data that will check to see if these functions are to see if these functions are performed by the softwareperformed by the software Does NOT matter how the Does NOT matter how the program performs these functionsprogram performs these functions

Functional testingFunctional testing Rules may be applied to uncover Rules may be applied to uncover the functionsthe functions Functional testing methods of Functional testing methods of formal documentation that includes formal documentation that includes descriptions of faults that correlate descriptions of faults that correlate to each part of the design and the to each part of the design and the design features themselvesdesign features themselves

Functional testingFunctional testing Isolation of these particular Isolation of these particular properties of each function should properties of each function should take placetake place Fault class associationsFault class associations Black box approachBlack box approach Testers have an understanding of Testers have an understanding of what the output should bewhat the output should be

Functional testingFunctional testing OracleOracle

– An expert on what the outcome of An expert on what the outcome of a program will be for a particular testa program will be for a particular test

When might the oracle When might the oracle approach not work?approach not work?

– Simulation testingSimulation testing Only provides a “range of values”Only provides a “range of values”

Structural testingStructural testing Testing is based on the detailed Testing is based on the detailed design rather than the functions design rather than the functions required by the programrequired by the program

Structural testingStructural testing Two approaches for this testingTwo approaches for this testing

– First and most common is to First and most common is to execute the program with test execute the program with test casescases

– Second is symbolic executionSecond is symbolic execution Functions of the program are Functions of the program are compared to the required functions compared to the required functions for congruencyfor congruency

Structural testingStructural testing May require single path or May require single path or percentage testingpercentage testing Research has been conducted to Research has been conducted to find out what the minimum amount find out what the minimum amount of testing would be to ensure a of testing would be to ensure a degree of reliabilitydegree of reliability

Structural testingStructural testing Measure of reliabilityMeasure of reliability

– All statements should be All statements should be executed at least onceexecuted at least once– All branches should be executed All branches should be executed at least onceat least once– All linear code sequence and All linear code sequence and jumps in the program should be jumps in the program should be executed at least onceexecuted at least once

Structural testingStructural testing Measure of reliability (cont.)Measure of reliability (cont.)

– Best approach would be the Best approach would be the exhaustive approach in which every exhaustive approach in which every path is testedpath is tested

Structural testingStructural testing Problems with the exhaustive Problems with the exhaustive approachapproach

– Extensive number of pathsExtensive number of paths– Multiple combinations constitutes Multiple combinations constitutes multiple conditionsmultiple conditions– Infeasible pathsInfeasible paths

Contradictions of predicates at Contradictions of predicates at conditional statementsconditional statements

Structural testingStructural testing Path issuesPath issues

– There is a path for a loop not There is a path for a loop not executing, executing once, and executing, executing once, and executing multiple of timesexecuting multiple of times– Control loops determine the Control loops determine the number of pathsnumber of paths

Structural testingStructural testing Path issuesPath issues

– Known as the “level-i” path or Known as the “level-i” path or island codeisland code– Island codeIsland code

A series of lines of code, following A series of lines of code, following a program termination, which is not a program termination, which is not the destination of a transfer control the destination of a transfer control from somewhere else in the from somewhere else in the programprogram

Structural testingStructural testing Path issuesPath issues

– When does island code occur?When does island code occur? When failing to delete redundant When failing to delete redundant code after maintenancecode after maintenance

Static analysisStatic analysis Does NOT involve execution of Does NOT involve execution of software with data but involves the software with data but involves the use of constraints on the input and use of constraints on the input and output data sets mathematically on output data sets mathematically on software componentssoftware components Examples of static analysis would Examples of static analysis would be program proving and symbolic be program proving and symbolic executionexecution

Static analysisStatic analysis Symbolic executionSymbolic execution

– Use symbolic values for variables Use symbolic values for variables instead of numeric or string valuesinstead of numeric or string values

Dynamic analysisDynamic analysis Relies on program statements Relies on program statements which make calls to analysis which make calls to analysis routinesroutines They record the frequency of They record the frequency of execution of elements of the execution of elements of the programprogram

Dynamic analysisDynamic analysis Act as a liaison between Act as a liaison between functional and structural testingfunctional and structural testing Test cases monitored Test cases monitored dynamically, then structurally tested dynamically, then structurally tested to see what idle code is left by to see what idle code is left by previous testsprevious tests Shows functions the program Shows functions the program should performshould perform

Classification of TechniquesClassification of Techniques There are three classifications:There are three classifications: Static – StructuralStatic – Structural

– Symbolic executionSymbolic execution– Partition analysisPartition analysis– Program provingProgram proving– Anomaly analysisAnomaly analysis

Classification of TechniquesClassification of Techniques Dynamic - FunctionalDynamic - Functional

– Domain testingDomain testing– Random testingRandom testing– Adaptive perturbationAdaptive perturbation– Cause-effect graphingCause-effect graphing

Classification of TechniquesClassification of Techniques Dynamic - StructuralDynamic - Structural

– Domain and computational Domain and computational testingtesting– Automatic test data generationAutomatic test data generation– Mutation analysisMutation analysis

Classification of TechniquesClassification of Techniques Dynamic - StructuralDynamic - Structural

– Domain and computational Domain and computational testingtesting– Automatic test data generationAutomatic test data generation– Mutation analysisMutation analysis

Symbolic executionSymbolic execution Non traditional approachNon traditional approach

– traditional is the execution traditional is the execution requires that a selection of paths requires that a selection of paths through a program is exercised by a through a program is exercised by a set of test classesset of test classes

Produces a set of expressions, Produces a set of expressions, one per output variableone per output variable

Symbolic executionSymbolic execution Usually a program executes using Usually a program executes using input data values with the output input data values with the output resulting in a set of actual valuesresulting in a set of actual values Use of flow-graphsUse of flow-graphs

– Each branch contains decision Each branch contains decision points (directed graph)points (directed graph)– Branch predictions are producedBranch predictions are produced

Symbolic executionSymbolic execution Use of top down approachUse of top down approach During the top down traversal, the During the top down traversal, the input variable is given a symbol in input variable is given a symbol in place of an actual valueplace of an actual value A problem is in the iterationsA problem is in the iterations As mentioned before, no As mentioned before, no executing loop, executing once, and executing loop, executing once, and then executing multiple times.then executing multiple times.

Partition analysisPartition analysis Uses symbolic execution to find Uses symbolic execution to find sub domains of the input domainsub domains of the input domain Path conditions are used to find Path conditions are used to find themthem Input domains that cannot be Input domains that cannot be allocated to a sub domain infer a allocated to a sub domain infer a fault fault

Partition analysisPartition analysis Specifications need to be written Specifications need to be written at a higher leverat a higher lever

Program provingProgram proving At the beginning and end of the At the beginning and end of the procedure, place a mathematical procedure, place a mathematical method assertionsmethod assertions Similar to symbolic executionSimilar to symbolic execution Neither of them execute actual Neither of them execute actual data and both examine source codedata and both examine source code

Program provingProgram proving Tries to come up with a proof that Tries to come up with a proof that encompasses all possible iterationsencompasses all possible iterations Program proving steps:Program proving steps:

– Construct a programConstruct a program– Examine the program and insert Examine the program and insert mathematical assertions at the mathematical assertions at the beginning and end of proceduresbeginning and end of procedures

Program provingProgram proving Program proving steps (cont):Program proving steps (cont):

– Determine whether the code Determine whether the code between each pair of start and end between each pair of start and end assertions will achieve the end assertions will achieve the end assertion given the start assertionassertion given the start assertion– If the code reaches the end If the code reaches the end assertion then the block has been assertion then the block has been provenproven

Program provingProgram proving DeMillo says that proofs can only DeMillo says that proofs can only be stated as acceptable and not be stated as acceptable and not correctcorrect His acceptance is determined by His acceptance is determined by a gathering of people who cannot a gathering of people who cannot find fault with the prooffind fault with the proof

Program provingProgram proving The larger the audience, the more The larger the audience, the more confidence in the softwareconfidence in the software Total correctness means loops Total correctness means loops will terminatewill terminate

Anomaly analysisAnomaly analysis Two levels of anomalies:Two levels of anomalies:

– made by the compiler (language made by the compiler (language syntax)syntax)– problems that are not necessarily problems that are not necessarily wrong by the programming wrong by the programming languagelanguage

Anomaly analysisAnomaly analysis Some anomalies are:Some anomalies are:

– Unexecutable codeUnexecutable code– Array boundariesArray boundaries– Initializing variables wrongInitializing variables wrong– Unused labels and variablesUnused labels and variables– Traversing in and out of loopsTraversing in and out of loops

Anomaly analysisAnomaly analysis Produce flow-graphsProduce flow-graphs Determine infeasible pathsDetermine infeasible paths Some use data-flow analysisSome use data-flow analysis

– Where the input values turn into Where the input values turn into intermediate, which then turn into intermediate, which then turn into output valuesoutput values

Anomaly analysisAnomaly analysis Some data-flow anomalies:Some data-flow anomalies:

– Assigning values to variables Assigning values to variables which are not usedwhich are not used– Using variables that have not Using variables that have not been assigned previously to a valuebeen assigned previously to a value– Re-assigning of a variable without Re-assigning of a variable without making use of a previous variablemaking use of a previous variable– Indicates possible faultsIndicates possible faults

Domain testingDomain testing Based upon informal Based upon informal classifications of the requirementsclassifications of the requirements Test cases executed and Test cases executed and compared against the expected to compared against the expected to determine whether faults have been determine whether faults have been detecteddetected

Random testingRandom testing Produces data without reference Produces data without reference to code or specificationto code or specification Random number generation is Random number generation is usedused Main problem is there is no Main problem is there is no complete coverage guaranteecomplete coverage guarantee

Random testingRandom testing Key is to examine small subsetsKey is to examine small subsets If followed it will give you a high If followed it will give you a high branch coverage success ratebranch coverage success rate

Adaptive perturbation testingAdaptive perturbation testing Concerns with assessing the Concerns with assessing the effectiveness of sets of test caseseffectiveness of sets of test cases Used to generate further test Used to generate further test cases for effectivenesscases for effectiveness

Adaptive perturbation testingAdaptive perturbation testing Optimization is reached when Optimization is reached when routines find the best value to routines find the best value to replace the discarded value so the replace the discarded value so the number of assertions is maximizednumber of assertions is maximized Process is replaced until the Process is replaced until the violated assertions are maximized to violated assertions are maximized to the limitthe limit

Cause-effect graphingCause-effect graphing Power comes from the input Power comes from the input combinations used by logic (AND, combinations used by logic (AND, OR, NOT, etc)OR, NOT, etc)

Cause-effect graphingCause-effect graphing Five steps:Five steps:

– Divide into workable piecesDivide into workable pieces– Identify cause and effectIdentify cause and effect– Represent a graph to link cause Represent a graph to link cause and effect semanticsand effect semantics

Cause-effect graphingCause-effect graphing Five steps (cont):Five steps (cont):

– Annotate to show impossible Annotate to show impossible combinations and effectscombinations and effects– Convert the graph into a limited-Convert the graph into a limited-entity decision tableentity decision table

It helps identify small test casesIt helps identify small test cases

Domain and computational testingDomain and computational testing Based upon selecting test casesBased upon selecting test cases Assignment statements are used Assignment statements are used for deciding computational pathsfor deciding computational paths

Domain and computational testingDomain and computational testing Paths considered:Paths considered:

– Path computationPath computation Set of algebraic expressions, one Set of algebraic expressions, one for each output variable, in terms of for each output variable, in terms of input variables and constraintsinput variables and constraints

– Path conditionPath condition A joining of constraints of a pathA joining of constraints of a path

Domain and computational testingDomain and computational testing Paths considered (cont):Paths considered (cont):

– Path domainPath domain Set of input values that satisfy the Set of input values that satisfy the path conditionpath condition

– Empty pathEmpty path Infeasible paths and cannot Infeasible paths and cannot executeexecute

A path that follows errors is a A path that follows errors is a computation errorcomputation error

Automatic test data generationAutomatic test data generation Courage comes with covering Courage comes with covering metricsmetrics Contradictory paths are infeasibleContradictory paths are infeasible Needs detailed specification to Needs detailed specification to achieve this testingachieve this testing Formal specifications may Formal specifications may provide fundamental helpprovide fundamental help

Mutation analysisMutation analysis Concerns the quality of sets of Concerns the quality of sets of test datatest data Uses the program to test the test Uses the program to test the test datadata This means the original and This means the original and mutant program are tested using the mutant program are tested using the same test datasame test data

Mutation analysisMutation analysis The two outputs are comparedThe two outputs are compared If the mutant output is different If the mutant output is different from the original output, then the from the original output, then the mutant is of little valuemutant is of little value If the two outputs are the same, If the two outputs are the same, then the problem is there has been then the problem is there has been a changea change

Mutation analysisMutation analysis The mutant is then said to be liveThe mutant is then said to be live Ratios are then taken (dead/alive)Ratios are then taken (dead/alive) High ratio of live mutants equals High ratio of live mutants equals poor test datapoor test data If this happens then more tests If this happens then more tests need to be run until the ratio goes need to be run until the ratio goes downdown

ConclusionConclusion I thought this paper was thoroughI thought this paper was thorough This paper gave good examples This paper gave good examples (compartmentalized)(compartmentalized) I thought this paper was a little I thought this paper was a little out of dateout of date