The Design Processpeople.tamu.edu/~rob.light/spring2019/csce315... · TESTING FOCUS •Broadly,...

TESTINGPROGRAMMING STUDIO, SPRING 2019

ROBERT LIGHTFOOT

Reference: Code Complete 2nd Edition Ch. 22, Ch. 8, ..

TESTING – WHAT IT IS AND WHAT IT IS NOT

• Testing helps find that errors exist, does not fix them

• Debugging finds their source and fixes them

• Systematic attempt to break a program that is working

• Unlike all other parts of software development, whose goal is to avoid errors

• Requires a different mindset by the developers

• Can never prove absence of errors

• Can you think of all possible cases?

• Testing alone does not improve quality

• To improve quality, the development has to be made better instead of just running

more test cases

• Testing is less effective than “collaborative construction” (will get to that later)

• Different statistics show that Testing alone can only find between 50-60% errors

present

TESTING FOCUS

• Broadly, there are 2 types of testing:

• Black-box testing

• White-box testing

• Black-box testing is done without knowing the code

• Done by non-developers, customers etc.

• Example: Beta testing of Windows outside Microsoft

• White-box is performed by the developers

• Knowledge of the code is used to form adversarial and tactical

test cases

• This is a.k.a developer testing, which is the main focus here

TYPES OF DEVELOPER TESTING

• Unit testing• Testing of a single class, routine, program• Code comes from a single programmer• Testing in isolation from the system

• Component testing• Testing of a class, package, program• Usually involves code from multiple programmers or teams• Still in isolation from the whole system

• Integration testing• Combined test of two or more classes, packages,

components, or subsystems• Absolutely necessary before finish/release, but can start

very early

TYPES OF DEVELOPER TESTING(CONTINUED)

• Regression testing

• Repetition of previously tested cases to find new errors introduced

• The set of test cases keeps growing

• System testing

• Executing software in final configuration, including integration with all other systems and hardware

• Security, performance, resource loss, timing issues

OTHER TYPES OF TESTING

• Usually by specialized test personnel

• User tests

• Performance tests

• Configuration tests

• Usability tests

• Etc.

• We’re interested now in developer tests (and later

in some usability tests)

WRITING TEST CASES FIRST• Helps identify errors more quickly

• Doesn’t take any more effort than writing tests later• Just requires resequencing work

• Requires thinking about requirements and design before writing code• Shows problems with requirements sooner (can’t write

code without good requirements)

• Foundation of Test Driven Development (we will get to this…)

TESTING AS YOU WRITE CODE• Motivation: Waiting until later means you have to

relearn code• Fixes will be less thorough and more fragile• Fixes further from the source are more costly

• Types of Testing in this category:• Boundary Testing• Pre- and Post-conditions• Assertions• Defensive Programming• Error Returns

Table taken from “Code Complete”

BOUNDARY TESTING

• Most bugs occur at boundaries

• If it works at and near boundaries, it likely works elsewhere

• Check loop and conditional bounds when written

• Check that extreme cases are handled

• e.g. Full array, Empty array, One element array

• Usually should check value and +/- 1

• Mental test better than none at all

• Very effective in many cases

if (a < MAX_VAL){….

}Test cases:

a = {MAX_VAL-1, MAX_VAL, MAX_VAL+1}

for(i=0; i<num; i++){….

}Test cases:

num = {-1, 0, 1}

vector<int> a, b(1);sort (a);sort (b);

PRECONDITIONS AND POSTCONDITIONS

• Verify that routine starts with correct preconditions and produces correct post-conditions

• Check to make sure preconditions met• Handle failures cleanly without surprise• Example: Check the denominator before dividing,

check if the file was actually opened before writing etc.

• Verify that post-conditions are met• No inconsistencies created• The array does not get deleted after sorting

• Need to define pre-/post-conditions clearly• “Provable” software relies on this approach

ASSERTIONS• Available in most languages

• assert.h (C/C++),

• Assert, AssertionError (Java)

• Way of checking for pre-/post-conditions

• Helps identify where problem occurs

• Before the assertion

• e.g. usually in calling routine, not callee

• Problem: causes abort

• So, useful for cleansing errors in the beginning

…int a = 1;assert (a >= 1); // will run w/o any problemassert (a < 1); // will abort the program after ..

// ..reporting assertion violation

DEFENSIVE PROGRAMMING

• Add code to handle the “can’t happen” cases

• Program “protects” itself from bad data

• Idea is similar to “defensive driving” training

• Skeptical about the rest of the world (i.e., gracefully handle bad

input w/o crashing)

• Example: Negative ISBN for the books in the IA

ERROR RETURNS

• Good API and routine design includes error codes

• Need to be checked

• Similar in functionality is Exception handling (via try-

catch) supported in many languages now

• Need to be clear about what exceptions are thrown, though

• Should be maintained when code evolves

ERROR RETURN EXAMPLE

int main (){

int writestatus = writetofile (somedata);if (writestatus == 0){

cout << "Write operation successful" << endl;return 0;

}else if (writestatus == ERR_TOO_MUCH_DATA){

cout << "Cannot write: Data too long" <<endl;

return 1;}else if (writestatus == ERR_CANNOT_OPEN){

cout << "Cannot write: File cannot be opened"<< endl;

return 1;}else{

cout << "Unrecognized response from writetofile() function" << endl;

return 1;}

}

#define MAX_DATA_LEN 1024#define ERR_TOO_MUCH_DATA 1#define ERR_CANNOT_OPEN 2

int writetofile (string filename, string data){

if (data.size () > MAX_DATA_LEN){return ERR_TOO_MUCH_DATA;

}ofstream myfile (filename);if (!myfile.is_open()) {

return ERR_CANNOT_OPEN;}myfile << data;myfile.close();return 0;

}

TRY-CATCH EXAMPLE

int main (){

try{writetofile (somedata);

}catch (int writestatus){

if (writestatus == ERR_TOO_MUCH_DATA){cout << "Cannot write: Data too long" << endl;return 1;

}else if (writestatus == ERR_CANNOT_OPEN){

cout << "Cannot write: Cannot open file" << endl;return 1;

}else{

cout << "Unrecognized exception caught from writetofile() function" << endl;

return 1;}

}cout << "Write operation successful" << endl;return 0;

}

#define MAX_DATA_LEN 1024#define ERR_TOO_MUCH_DATA 1#define ERR_CANNOT_OPEN 2

void writetofile (string filename,string data){

if (data.size () >MAX_DATA_LEN){

throw ERR_TOO_MUCH_DATA;}

ofstream myfile (filename);if (!myfile.is_open()) {

throw ERR_CANNOT_OPEN;}myfile << data;myfile.close();

}

SYSTEMATIC TESTING SUGGESTIONS

• Test incrementally

• Test simple parts first

• Know what output to expect

• Verify conservation properties

• Compare independent implementations

• Measure test coverage

TEST INCREMENTALLY

• Don’t wait until everything is finished before test

• Test components, not just system

• Test components individually before connecting

them

TEST SIMPLE PARTS FIRST

• Test most basic, simplest features

• Learn to doubt your simplest code – it is not easy

• Finds the “easy” bugs (and usually most important)

first

class Record{int num;int price;

};

cout<<sizeof(Record);

class Record{int num;int price;void f();

};

cout<<sizeof(Record);

class Record{int num;int price;virtual void f();

};

cout<< sizeof(Record);

KNOW WHAT OUTPUT TO EXPECT

• Design test cases that you will know the answer to!

• Seeing some answer does not mean your program is right

• Make hand-checks convenient

• Pick test cases that you can hand-calculate

• Example: sort an array {1,3,2}

• Not always easy to do

• e.g. compilers, numerical programs, graphics

VERIFY CONSERVATION PROPERTIES

• Specific results may not be easily verifiable

• Have to write the program to compute the answer to

compare to

• But, often we have known output properties related

to input

• e.g. #Start + #Insert - #Delete = #Final

• Can verify these properties even without verifying

larger result

COMPARE INDEPENDENT IMPLEMENTATIONS

• Multiple implementations to compute same data should agree

• Useful for testing tricky code, e.g. to increase performance• Write a slow, brute-force routine

• Compare the results to the new, “elegant” routine

• If two routines communicate (or are inverses), different people writing them helps find errors• Only errors will be from consistent misinterpretation of

description

MEASURE TEST COVERAGE

• What portion of code base is actually tested?

• Important, because actual coverage can be less than what developers usually think

• Code-structure-dependent and analytically computable

• Specific rules to calculate them

• Tend to work well on only small/moderate code pieces

• For large software, tools help judge coverage

STRUCTURED BASIS TESTING

• Testing every line in a program

• Ensure that every statement gets tested

• Need to test each part of a logical statement

• Easily catch basic and typo-like errors

• Fewer cases than other tests

• But, also not as thorough

• Goal is to minimize total number of test cases

• One test case can test several statements

STRUCTURED BASIS TESTING (CONTINUED)

• Start with base case where all Boolean conditions are true

• Design test case for that situation

• Each branch, loop, case statement increases minimum

number of test cases by 1

• One more test case per variation, to test the code for that

variation

if (a)statement_1

elsestatement_2

statement_3if (b)

statement_4else

statement_5

How many test cases?

Case #1: a=T, b=T

Case #2: a=F, b=F

STRUCTURED BASIS TESTING (CONTINUED)

• Start with base case where all Boolean conditions are true

• Design test case for that situation

• Each branch, loop, case statement increases minimum number

of test cases by 1

• One more test case per variation, to test the code for that

variation

How many test cases now?

Case #1: a=T, b=X, c=T

Case #2: a=F, b=T, c=F

Case #3: a=F, b=F, c=X

“X” means does not matter

if (a)statement_1

else if (b)statement_2

else statement_6

statement_3if (c)

statement_4else

statement_5

LOGIC COVERAGE

• A.k.a. Code Coverage

• Testing every path through the code

• Can grow (nearly) exponentially with number of

choices/branches

• Only suitable for small to medium size codes

• Why? - Dependency between statements

if (a)statement_1

elsestatement_2

statement_3if (b)

statement_4else

statement_5

How many possible paths in this code?

Path #1: Statements 1, 3, 4 (a=T, b=T)

Path #2: Statements 1, 3, 5 (a=T, b=F)

Path #3: Statements 2, 3, 4 (a=F, b=T)

Path #4: Statements 2, 3, 5 (a=F, b=F)

How about code with more if conditions??

# of test cases = # of paths (here, 4)

COVERAGE PROBLEM

• Structured basis testing is:

• More practical since test cases grow linearly with number of branching

• However, does not cover dependency between statements

• Logic coverage on the other hand:

• Covers all possible dependency issues

• But very quickly becomes overwhelming

• Due of exponential growth with branching statements

• We need something in-between:

• Answer is Data Flow Testing covered next

DATA FLOW TESTING

• Examines data rather than control

• Data in one of three states:

• Defined – Initialized but not used (int a =1)

• Used – In computation or as argument (b = a+1)

• Killed – Undefined in some way, e.g.,

• memory deleted (delete x)

• File closed (fclose (fp))

• Variables related to routines

• Entered – Routine starts just before variable is acted upon

• Exited – Routine ends immediately after variable is acted upon

DATA FLOW TESTING (CONTINUED)

• First, check for any anomalous data sequences• Defined-defined

• Defined-exited

• Defined-killed

• Entered-killed

• Entered-used

• Killed-killed

• Killed-used

• Used-defined

• Often can indicate a serious problem in code design

• After that check, write test cases

int a = 1;

a = 5;

…



• Defined-exited

• Defined-killed

• Entered-killed

• Entered-used

• Killed-killed

• Killed-used

• Used-defined



void func ()

{

...

...

int a = 1;

return;

}



• Defined-exited

• Defined-killed

• Entered-killed

• Entered-used

• Killed-killed

• Killed-used

• Used-defined



int* x = new int [10];

delete x;



• Defined-exited

• Defined-killed

• Entered-killed

• Entered-used

• Killed-killed

• Killed-used

• Used-defined



void func ()

{

delete x;

fclose (fp);

...

...

}



• Defined-exited

• Defined-killed

• Entered-killed

• Entered-used

• Killed-killed

• Killed-used

• Used-defined



void func ()

{

int a = x;

...

...

}



• Defined-exited

• Defined-killed

• Entered-killed

• Entered-used

• Killed-killed

• Killed-used

• Used-defined



delete x;

...

delete x;



• Defined-exited

• Defined-killed

• Entered-killed

• Entered-used

• Killed-killed

• Killed-used

• Used-defined



delete x;

...

int a = *x + 1



• Defined-exited

• Defined-killed

• Entered-killed

• Entered-used

• Killed-killed

• Killed-used

• Used-defined



int x;

...

int a = x + 1;

...

x = 100;


• Write test cases to examine all defined-used

paths

• Usually requires:

• More cases than structured basis testing

• Fewer cases than logic coverage

EXAMPLE – LOGIC/CODE COVERAGE

1. Conditions: T T T

2. Conditions: T T F

3. Conditions: T F T

4. Conditions: T F F

5. Conditions: F T T

6. Conditions: F T F

7. Conditions: F F T

8. Conditions: F F F

Tests all possible paths

if (cond1) {

x = a;

} else {

x = b;

}

if (cond2) {

y = x+1;

} else {

y = x+2;

}

if (cond3) {

z = c;

} else {

z = d;

}

EXAMPLE - STRUCTURED BASIS TESTING


2. Conditions: F F F

Tests all lines of code

if (cond1) {

x = a;

} else {

x = b;

}

if (cond2) {

y = x+1;

} else {

y = x+2;

}

if (cond3) {

z = c;

} else {

z = d;

}

EXAMPLE - DATA FLOW TESTING


2. Conditions: T F F

3. Conditions: F T X

4. Conditions: F F X

Tests all defined-used paths

Note: cond3 is independent of

first two

if (cond1) {

x = a;

} else {

x = b;

}

if (cond2) {

y = x+1;

} else {

y = x+2;

}

if (cond3) {

z = c;

} else {

z = d;

}

TEST CASE DESIGN(IF YOU DON’T KNOW THE CODE)

• Boundary analysis still applies

• Equivalence partitioning• Don’t create multiple tests to do the same thing

• Bad data• Too much/little

• Wrong kind/size

• Uninitialized

• Good data• Minimum/maximum normal configuration

• “Middle of the Road” data

• Compatibility with old data

TEST AUTOMATION

• Should do lots of tests, and by-hand is not usually appropriate

• Scripts can automatically run test cases, report on errors in output• But, we need to be able to analyze output

automatically…

• Can’t always simulate good input (e.g. interactive programs)

• People cannot be expected to remain sharp over many tests

• Automation reduces workload on programmer, remains available in the future

REGRESSION TESTING

• Goal: Find anything that got broken by “fixing”

something else

• Save test cases, and correct results

• With any modifications, run new code against all

old test cases

• Add new test cases as appropriate

• Part of Test-Driven Development (TDD) (coming

up)

TEST SUPPORT TOOLS

• Test Scaffold

• Framework to provide just enough support and interface

to test

• Stub Routines and Test Harness

• Test Data Generators

• System Perturber

STUB ROUTINES

• This is the thing (routine/class) under test

• Doesn’t provide full functionality, but pretends to

do something when called

• Return control immediately

• Burn cycles to simulate time spent

• Print diagnostic messages

• Return standard answer

• Get input interactively rather than computed

• Could be “working” but slow or less accurate

TEST HARNESS

• Calls the routine being tested

• Fixed set of inputs

• Interactive inputs to test

• Command line arguments

• File-based input

• Predefined input set

• Can run multiple iterations

TEST DATA GENERATORS

• Can generate far more data than by hand

• Can test far wider range of inputs

• Can detect major errors/crashes easily

• Need to know answer to test correctness

• Useful for “inverse” processes – e.g.

encrypt/decrypt

• Should weight toward realistic cases

• The feasible scope is always limited

SYSTEM PERTURBERS

• Modify system so as to avoid problems that

are difficult to test otherwise

• Reinitialize memory to something other than 0

• Find problems not caught because memory is “usually”

null

• Rearrange memory locations

• Find problems where out-of-range queries go to a

consistent place in other tests

• Memory bounds checking

• Memory/system failure simulation

OTHER TESTING TOOLS

• Diff tools• Compare output files for differences

• Coverage monitors• Determine which parts of code tested

• Data recorder/loggers• Log events to files, save state information

• Error databases• Keep track of what’s been found, and rates of errors

• Symbolic debuggers• Will discuss debugging later, but useful for tests

TEST-DRIVEN DEVELOPMENT

• Generally falls under Agile heading

• We will come back to what is meant by this

• A style of software development, not just a matter of

testing your code

• Enforces testing as part of the development process

TEST DRIVEN DEVELOPMENT OVERVIEW

• Repeat this process:

1. Write a new test

2. Run existing code against all tests; it should generally

fail on the new test

3. Change code as needed

4. Run new code against tests; it should pass all tests

5. Refactor the code

TEST WRITING FIRST

• Idea is to write tests, where each test adds some degree

of functionality

• Passing the tests should indicate working code (to a point)

• The tests will ensure that future changes don’t cause

problems

RUNNING TESTS

• Use a test harness/testing framework of some sort

to run the tests

• A variety of ways to do this, including many existing

frameworks that support unit tests

• JUnit is the most well-known, but there is similar

functionality across a wide range of languages

TEST FRAMEWORK

• Specify a test fixture

• Basically builds a state that can be tested

• Set up before tests, removed afterward

• Test suite run against each fixture

• Set of tests (order should not matter) to verify various aspects of

functionality

• Described as series of assertions

• Runs all tests automatically

• Either passes all, or reports failures

• Better frameworks give values that caused failure

REFACTORING

• As code is built, added on to, it becomes messier

• Need to go back and rewrite/reorganize sections

of the code to make it cleaner

• Do this on a regular basis, or when things seem like

they could use it

• Only refactor after all tests are passing

• Test suite guarantees refactoring doesn’t hurt.

REFACTORINGCOMMON OPERATIONS

• Extract Class

• Extract Interface

• Extract Method

• Replace types with subclasses

• Replace conditional with polymorphic objects

• Form template

• Introduce “explaining” variable

• Replace magic number with symbolic constant

RESOURCES

• Test-Driven Development By Example

• Kent Beck; Addison Wesley, 2003

• Test-Driven Development A Practical Guide

• David Astels; Prentice Hall, 2003

• Software Testing A Craftsman’s Approach (3rd edition)

• Paul Jorgensen; Auerback, 2008

• Many other books on testing, TDD, also

The Design Processpeople.tamu.edu/~rob.light/spring2019/csce315... · TESTING FOCUS •Broadly,...

Documents

Transcript of The Design Processpeople.tamu.edu/~rob.light/spring2019/csce315... · TESTING FOCUS •Broadly,...