Best Practices in Software Development: Testingfehnkera/portfolio/bpsd/BPSD-3_T… · It provides...

Best Practices in Software Development:TestingAnsgar FehnkerBased on slides by: Arend Rensink, Christoph Bockisch, Lodewijk Bergmans

Testing Agile changes the role of testing Traditional After full integration resembles final product most...

During integration interfaces and integration are error-prone...

When unit is ready to ensure proper quality is delivered

On each implemented feature test as soon as it’s ready requires small-grained test cases

Agile approach Implement tests before functionality

Best Practices in Software Development 2

However, principles of testing remain

Testing PreliminariesSoftware quality assessment divide into two categories:

Static analysis It examines the code and reasons over all behaviors that might

arise during run time Examples: Code review, inspection, and algorithm analysis

Dynamic analysis Actual program execution to expose possible program failure One observe some representative program behavior, and reach

conclusion about the quality of the system

Best Practices in Software Development 3From Naik

and Tripathy

Testing PreliminariesVerification Evaluation of software system that help in determining whether the

product of a given development phase satisfy the requirements established before the start of that phase Building the product correctly

Validation Evaluation of software system that help in determining whether the

product meets its intended use Building the correct product


and Tripathy

Testing PreliminariesFailure, Error, Fault and Defect Failure A failure is said to occur whenever the external behavior of a

system does not conform to that prescribed in the system specification

Error An error is a state of the system. An error state could lead to a failure in the absence of any

corrective action by the system Fault A fault is the adjudged cause of an error

Defect It is synonymous of fault It a.k.a. bug


and Tripathy

Testing Preliminaries

Input vector An input vector is a collection of all data entities read by the routine

whose values must be fixed prior to entering the routine. Members of an input vector can be as follows. Input arguments to the routine Global variables and constants Files Contents of registers (in Assembly language programming) Network connections Timers

6

Testing PreliminariesWhite-box and Black-box Testing White-box testing a.k.a.

structural testing Examines source code with

focus on: Control flow: refers to flow of

control from one instruction to another

Data flow: refers to propagation of values from one variable or constant to another variable

It is applied to individual units of a program

Software developers perform structural testing on the individual program units

Black-box testing a.k.a. functional testing

Examines the program that is accessible from outside

Applies the input to a program and observe the externally visible outcome

It is applied to both an entire program as well as to individual program units

It is performed at the external interface level of a system

It is conducted by a separate software quality assurance group


and Tripathy

Testing PreliminariesTesting Terminology Unit testing Individual program units, such

as procedure, methods in isolation

Integration testing Modules are assembled to

construct larger subsystem and tested

System testing Includes wide spectrum of

testing such as functionality, and load

Acceptance testing Customer’s expectations from

the system


and Tripathy

Unit Testing

The environment of a unit is emulated and tested in isolation

The caller unit is known as test driver

A test driver is a program that invokes the unit under test (UUT)

It provides input data to unit under test and report the test result

The emulation of the units called by the UUT are called stubs

It is a dummy program

The test driver and the stubs are together called scaffolding

The low-level design document provides guidance for selection of

input test data


Unit Testing Unit Testing Environment


Unit Tests (JUnit)

Beste Practices in Software Development 11

Test suite

• A class annotated with@RunWith(Suite.class)@SuiteClasses({TestClass.class, …})

• Groups tests for one system or component

Test case

• A class (e.g., TestClass)• Groups tests for one test item (unit)• Defines common set-up and tear-down operations• @BeforeClass, @AfterClass: before/after all tests• @Before, @After: before/after each test

Test

• A method annotated with@Test

• Implement usage of test item according to specification• Assert that the outcome is correct• Possibly specify timeout and expected exception for test

Techniques to select test dataControl flow testing Identify paths in the CFG to satisfy given selection criteria Derive the path predicate expression from the selection paths Solving the path predicate expression generate the data Data flow testing Use a data flow graph (DFG) from a program unit and then follow

the procedure described in control flow testing. Domain testing Domain errors are defined and then test data are selected to catch

those faults Functional program testing Input/output domains are defined to compute the input values that will

cause the unit to produce expected output values


Test Selection and Coverage

Path selection criteria

Select paths to achieve complete statement coverage.

Select paths to achieve complete branch coverage.

Select paths to achieve predicate or condition coverage.

Select paths to achieve data flow coverage.

Select all paths.

13

Test Selection and Coverage

1 input(Y)2 if (Y<=0) then3 Y := −Y4 end_if5 while (Y>0) do6 input(X)7 Y := Y-18 end_while


2

5

3

6,7

nodes

edges

Pseudo Code and CFG


2

5

3

6,7One test case:input(y) -1input(x) don’t care

Test Selection and CoverageHow many test cases required for Statement Coverage?


2

5

3

6,7


Test Selection and CoverageHow many test cases required for Branch Coverage?


Two test cases:1. input(Y) -1

input(X) don’t care2. input(Y) 1

input(X) don’t care

Test Selection and CoverageDoes Branch Coverage Imply Statement Coverage? Normally, yes… In the absence of dead code


Dead code is not reachable via any executable program path.

Under the (reasonable) assumption of the absence of dead code, Branch Coverage implies Statement Coverage.

Test Selection and CoverageCondition Coverage A branch predicate may have more than one condition

1 input(Y)2 if (Y<=0) or (X=0) then3 Y := −Y4 end_if5 while (Y>0) and (not EOF) do6 input(X)7 Y := Y-18 end_while

Condition Coverage requires that each condition has been true at least once has been false at least once.


Test Selection and CoverageData Flow A program unit accepts inputs, performs computations, assigns new

values to variables, and returns results. One can visualize of “flow” of data values from one statement to

another. A data value produced in one statement is expected to be used later. Example Obtain a file pointer ……. use it later.

If the later use is never verified, we do not know if the earlier assignment is acceptable.

Two motivations of data flow testing The memory location for a variable is accessed in a “desirable” way. Verify the correctness of data values “defined” (i.e. generated) – observe

that all the “uses” of the value produce the desired results.


Test Selection and CoverageOccurrences of variables Definition: A variable gets a new value. i = x; /* The variable i gets a new value. */

Undefinition or kill: This occurs if the value and the location become unbound.

Use: This occurs when the value is fetched from the memory location of the variable. There are two forms of uses of a variable. Computation use (c-use) Example: x = 2*y; /* y has been used to compute a value of x. */

Predicate use (p-use) Example: if (y > 100) { …} /* y has been used in a condition. */

20

Test Selection and CoverageExample data flow criterion: All du-paths: Include all paths from a definition to a use (p-use or c-

use)


int binsearch(int X, int V[], int n){int low, high, mid;low = 0;high = n - 1;while (low <= high) {

mid = (low + high)/2;if (X < V[mid])

high = mid - 1;else if (X > V[mid])

low = mid + 1;else

return mid;}return -1;

}

123456789101112131415

Test Selection and Coverage Path Coverage requires that all program paths have been traversed at least once

Often described as the “strongest” form of code coverage Usually impossible to achieve when loops are present Other criteria: Loop coverage, Bock coverage

Tool support

Tools exist to determine coverage given a citerion Tools exist to generate input vectors to achieve coverage


Test Selection and CoverageMutation Testing

Used to “test” if the tests cover the code Modify a program by introducing a single small change to the code.

The modified program is called mutant Determine the quality of a set of test, by how many mutants it

discovers A mutant is said to be killed when the execution of test case cause

it to fail. A mutant is an equivalent to the given program if it always produce

the same output as the original program A mutation score for a set of test cases is the percentage of non-

equivalent mutants killed by the test suite New test need to be added to ensure all mutants are detected.


Test Selection and CoverageConsider the following program

main(int argc, char *argv) {

int r = 1;

for(int i = 2;i< 3;i++)

{

if (argv[i] > argv[r])

{

r = i;

}

}

cout<<“Result: ” <<r;

exit(0);

}

Test Case 1: input: 1 2 3 output: Result: 3




Mutant 1: Change line 7 to r = 1;• Programs will not pass the test suite (fail case 1 and 2)• Mutant 1 is killed by current test suite.



int r = 1;

for(int i = 2;i< 3;i++)

{


{

r = i;

}

}


exit(0);

}





Mutant 3: Change line 5 to if (argv[i] >= argv[r])• Programs will pass the test suite• However, mutant 3 is killable. Add one more test (2 2 1)



int r = 1;

for(int i = 2;i< 3;i++)

{


{

r = i;

}

}


exit(0);

}





Mutant 3: Change line 3 to for(int i = 1; …)• Programs will pass the test suite• Mutant 3 is equivalent, i.e., is not killable.

Behavior-Driven Test Implementation Behavioral specification of tests: user story Title Narrative (introduction) Scenario (acceptance criteria)

Documentation uses specific keywords

Narrative: In order to <achieve some business value> As a <stakeholder> I want to <effect to achieve>


Behavior-Driven Development Scenario <description> Given <prerequisite> And <additional prerequisite> When <what triggers the scenario> Then <expected outcome> And <additional expected outcome>

Typical structure of test code Making this structure explicit helps documenting tests Be as descriptive as possible!

Supported by tools, e.g., JBehave, Hamcrest


Unit Testing in eXtreme Programming Three laws of Test Driven development (TDD)

1. One may not write production code unless the first failing unit test is written

2. One may not write more of a unit test than is sufficient to fail 3. One may not write more production code than is sufficient to make

the failing unit test pass

Pair programming: In XP code is being developed by two programmers working side by

side One person develops the code tactically and the other one inspects it

methodically by keeping in mind the story they are implementing


Unit Testing in eXtreme Programming

1. Pick a requirement, i.e., a story 2. Write a test case that will verify

a small part of the story and assign a fail verdict to it

3. Write the code that implement particular part of the story to pass the test

4. Execute all test 5. Rework on the code, and test

the code until all tests pass 6. Repeat step 2 to step 5 until the

story is fully implemented


Integration Testing

A software module is a self-contained element of a system Modules are individually tested commonly known as unit testing Next major task is to put the modules, i.e., pieces together to construct

the complete system Construction of a working system from the pieces is not a

straightforward task because of numerous interface errors

Interface errorsInterface errors are those that are associated with structures existing outside

the local environment of a module, but which the module uses

31

Integration Testing

The objective of system integration testing (SIT) is to build a “working” version of the system Putting modules together in an incremental manner Ensuring that the additional modules work as expected without disturbing

the functionalities of the modules already put together Integration testing is said to be complete when The system is fully integrated together All the test cases have been executed All the severe and moderated defects found have been fixed

32

System Tests

Basic tests provide an evidence that the system can be installed, configured and be brought to an operational state

Functionality tests provide comprehensive testing over the full range of the requirements, within the capabilities of the system

Robustness tests determine how well the system recovers from various input errors and other failure situations

Inter-operability tests determine whether the system can inter-operate with other third party products

Performance tests measure the performance characteristics of the system, e.g., throughput and response time, under various conditions

Scalability tests determine the scaling limits of the system, in terms of user scaling, geographic scaling, and resource scaling

33

System Tests

Stress tests put a system under stress in order to determine the limitations of a system and, when it fails, to determine the manner in which the failure occurs

Load and Stability tests provide evidence that the system remains stable for a long period of time under full load

Reliability tests measure the ability of the system to keep operating for a long time without developing failures

Regression tests determine that the system remains stable as it cycles through the integration of other subsystems and through maintenance tasks

Documentation tests ensure that the system’s user guides are accurate and usable

34

Acceptance Testing

Acceptance testing is a formal testing conducted to determine whether a system satisfies its acceptance criteria

There are two categories of acceptance testing: User Acceptance Testing (UAT) It is conducted by the customer to ensure that system satisfies the

contractual acceptance criteria before being signed-off as meeting user needs.

Business Acceptance Testing (BAT) It is undertaken within the development organization of the supplier to

ensure that the system will eventually pass the user acceptance testing.

35

Acceptance Testing

Three major objectives of acceptance testing:

Confirm that the system meets the agreed upon criteria

Identify and resolve discrepancies, if there is any

Determine the readiness of the system for cut-over to live operations

36

Beta Testing

Beta testing is conducted by the potential buyers prior to the official release of the product

The purpose of beta testing is not to find defects, but to obtain feedback from the field about the usability of the product

Three kinds of beta: Marketing beta Technical beta Acceptance beta

37

Test Design Process

One test case is created for each test objective

Figure 11.6: State-transition diagram of a test case. 38

Test Results

Figure 11.7: State transition diagram of a test case results

39

Modeling Defects

40

Testing PreliminariesThe Objectives of Testing It does work It does not work Reduce the risk of failures Reduce the cost of testing Reduce the cost of development

…


Best Practices in Software Development: Testingfehnkera/portfolio/bpsd/BPSD-3_T… · It provides...

Documents

Transcript of Best Practices in Software Development: Testingfehnkera/portfolio/bpsd/BPSD-3_T… · It provides...