Object Oriented Analysis & Design

Testing

2

Contents Testing Unit testing CppTest Regression testing Equivalence testing White & black box testing Test coverage Test plans Test roles

3

Testing Testing seeks to

Determine if the game meets requirements If the game works as specified If the game is bug-free

We test using Unit tests Tests by playing the game

4

Importance of Testing Only high-quality games will sell Players do not like bugs in games It is difficult to get a program correct Only testing can determine if a program is

correct

Testing is an important part of producing a game!

5

Unit Testing Unit Testing

Tests each class Tests every method on every class

Unit testing shows that the classes work before they are put into a complete game

Unit tests are written as the classes are developed

6

Test Harness Unit testing involves

Writing code to perform the tests Checking the results Reporting the results

There are frameworks which can aid with A template to write the tests An easy way to check results Automating the reporting of results

7

CppTest CppTest is

A lightweight unit testing framework for C/C++ programs

It can be downloaded from http://cpptest.sourceforge.net

8

A Test A test is

A void method with no parameters Contains code to calculate a result Uses a test assertion to compare the calculated

result with the expected result If the result is incorrect, the assertion will report

the bug

9

Test Suites A test suite is

A class derived from Test::Suite Contains one or more test methods or other test

suites Adds each of the tests to the suite in the

constructor Can be run by calling the run() method which will

run all tests in the suite

10

Assertions TEST_ASSERT(boolean_expr)

Boolean expression is true to pass test TEST_ASSERT_MSG(bool_expr, msg)

If test fails, msg is printed TEST_ASSERT_DELTA(a, b, delta)

Test fails if a is not equal to B +/- delta TEST_ASSERT_DELTA_MSG(a, b, delta, msg)

If test fails, msg is printed

11

Sample Test Suite#include <cpptest.h>

class MathTestSuite: public Test::Suite {public:

MathTestSuite() {TEST_ADD(MathTestSuite::addTest)}

private:virtual void setup() { /* init code before first test */ }virtual void tear_down() { /* cleanup code after last test */ }void addTest() {TEST_ASSERT_MSG(2 + 2, 4, “this does not add up”);}

};

12

Running a Test Suiteint main(int argc, char **argv) {

MathTestSuite mts;Test::TextOutput output(Test::TextOutput::Verbose);

mts.run(output, true); // true runs rest of tests after a

// test fails}

13

Test Guidelines Each test should test one simple operation Each test should contain one assertion Related tests should be in the same test suite Related test suites can be combined into a

test suite using the add(Test::Suite) method of the test suite

14

Regression Testing Once a test suite is created, it can be easily

run again and again This means we can run the test suite

After every change to the code Before the changes are checked into the

repository Before you ship a release

15

Selecting Test Cases It is impossible to test all possible values

We identify groups of values which test the same thing

Then we test only one value from each equivalence group

16

Equivalence Testing Most bugs in programs occur at the

boundaries of the data or on special values Zero is a special value MAXINT and MININT are boundaries An empty string and the longest permisable string

are boundaries A NULL string is a special value

17

Equivalence Testing Test the boundary values Test values adjacent to the boundary values Test values just outside the boundaries to see

if the code handles these correctly Test a few routine values to make sure they

work Test values that should cause the code to fail

18

Sample Equivalence Tests You need to test a function which takes a single

integer parameter greater that zero Use the following test data

Largest negative value -100 random negative value -1 near boundary 0 boundary +1 near boundary 8 routine even value 9 routine odd value Largest positive value -1 Largest positive value

19

Black Box Testing Treats the software as a black box which you

do not know how it works Tests that the documented features work

correctly The black box tests can be designed as soon

as the requirements are available

20

White Box Testing Tests designed based on a knowledge of the

internals of the game Must be designed by a programmer Creates tests which

Test the path through every if and switch Test unusual error conditions Test infrequently executed code Invoke specific methods

21

Testing Coverage Most testing leaves many parts of the code

unexecuted A coverage analyzer can tell you the number

of times each statement was executed This can identify untested code You can design white box tests to test the

untested code

22

Game Testing When most people speak of game testing,

they mean functionality testing This involves

Performing every operation Performing every combination of operations Executing every menu item Executing every level Ensuring the results are as expected

23

Functionality Testing Functionality testing is performed by

Creating a test plan which Specifies how to set up each test Specifies the steps in conducting each test Specifies the expected results

The game tester is responsible for Executing each of the tests Reporting any bugs found

24

Compliance Testing Special agreements are imposed by

Console manufacturers Software / hardware vendors Governmental agencies Entertainment Software Rating Board

Compliance testing involves checking that the game does not violate any of these agreements

25

Compatibility Testing Games need to test for compatibility with

Different versions of the operating system Different versions of graphics libraries Different versions of third party libraries Different input devices (mice & joysticks) Different graphics cards Processors with different numbers of cores The minimal hardware the game requires

26

Localization Testing Most games are sold in international markets

and have to be correct in many languages Check for

Is the locale automatically detected ? Are all menus translated correctly ? Is printed dialogue translated correctly ? Is audio correct ? Are number, date and moetary formats correct? Are there offensive cultural references?

27

Cultural Considerations What is good in one culture is bad in another

Some numbers are bad luck in some cultures Any hand gesture is obscene somewhere Colours have different emotional connotations in

different cultures Cultures have different attitudes towards

Men and women Religion Authority Short term versus long term thinking

28

Soak Testing This involves leaving the game running for

long periods of time in different modes Can also involve performing the same

operation a large number of times Can detect

Memory leaks Buffer overflows Race conditions Huge output files

29

Load Testing This puts a high load on the system to ensure

it will not fail A large number of players on an MMO A large number of enemy characters A large number of threads Fast input over extended time periods Large amounts of scenery

30

Multiplayer Testing Tests to see that the program works well with

multiple players Supports large numbers of players Works over various network types Has good performance under high network load There is acceptable lag in updating the view of

each player Single player mode works as well as multiplayer

31

Installation Testing Does the program install correctly on all target

systems ? Are the correct resources installed for each

locale ? Is the installer in the right language ? Does the documentation agree with the

version of the game it shipped with ? Can users understand the documentation ?

32

The Test Plan The test plan

Provides details of all the test to conduct For each test it specifies

How to prepare for the test The steps in performing the test How to determine if the test passed or failed

33

The Test Report Produced after conducting a test States

The test conducted & time & date Any special conditions for the test The result(s) of the test

If the test failed All available logs from the game when the test

failed Observations by the tester

34

Roles in Testing Game producer

Sets testing deadlines in conjunction with marketing

Lead tester Manages the testing process Keeps records on the tests performed Tracks bugs reported Understands how to program

35

Roles in Testing Test Designer

Responsible for designing the tests Usually a programmer who knows the type of

things which go wrong Tester

The person who actually conducts the tests Does not require programming skills Must be detail oriented Must like repetitive tasks

36

Beta Testing After testing within the company is complete,

the game is released to select customers They get the game early with the agreement

that they will Play the game a lot Report any bugs they find Send in logs produced at the time the bugs

occurred