+

Software Testing and

Game TestingAn Introduction to Software Testing in Games

+Introduction

This presentation gives an overview introduction to software

testing in general, and how game testing adds additional

complexity to the equation

This is just an introduction, but there are links at the end of the

presentation to facilitate further learning

Game QA adds an additional layer on top of this, but that will

not be covered here

+What is Testing?

“Testing is an empirical, technical investigation conducted to

provide stakeholders with information about the quality of the

product or service under test”

Black Box Software Testing: Foundations course [1]

+What is a tester?

Someone with test competence that performs test activities

Software Developer

Software Developer in

Test

Embedded Tester

Acceptance Tester

System Tester

???

Title does not

matter. If you have

test competence

and you perform

test activities you

are (but you may

have many roles)

a tester

+Quality?

“Quality is value to some person” [12]

Quality means different things to different

people. But in the end, when someone says

they want high quality – what they want is

something that is valuable to them

+Different Layers of Tests [13]

Unit Test

Integration Test

Component Interface Test

System Test

Acceptance Test

Which layers you choose to

separate different types of tests

does not really matter – but it

can be good to have some kind

of separation of different types

of tests to facilitate

understanding and separation

of work

+The Boxes of Testing [13]

Black Box Gray Box White Box

No insight into the code and

underlying architectureSome insight into the code

and underlying architecture

Full understanding of the code

and the software architecture

+Test Techniques [21]

People-based techniques (User, Alpha, Beta, Pair, Bug Bash, Expert)

Who does the testing?

Coverage-based techniques (Boundary testing, Equivalence class analysis)

What gets tested?

Problem-based techniques (Input-, Output-, Data- and Computation constraints)

Why are you testing?

Activity-based techniques (Scripted, Exploratory, Smoke, Regression)

How are you testing?

Evaluation-based techniques (Comparison with specification, Comparison with saved data)

How do you know if a test is passed or failed?

+Risk-based Testing [14]

1. Make a prioritized list of risks.

2. Perform testing that explores each risk.

3. As risks evaporate and new ones emerge, adjust your test

effort to stay focused on the current crop.

+Risk Input

There are many things that could have impact on perceived risk level

Recent fixes

Code coverage in changed components

Code complexity in changed components

Historical data

Known dependencies for changed components

Known bugs in changed components

Etc.

+Exploratory Testing

"a style of software testing that emphasizes the personal

freedom and responsibility of the individual tester to continually

optimize the quality of his/her work by treating test-related

learning, test design, test execution, and test result interpretation

as mutually supportive activities that run in parallel throughout the

project.”

Cem KanerWith few exceptions designing manual test cases

upfront is not an efficient way of testing, but

exploratory testing requires a more mature test

process and organization

+Agile Testing Quadrants [22]

+Test Automation

Understand the

Software SystemPlan your Test Design

Create manual tests

Create automated tests

Ensure Software System

TestabilityDesigning effective and efficient

automated tests is a science in itself –

never underestimate the time and

cost of setting up a large scale,

continuous integration system

It is time, and not cost, that you will

reduce with automation

Never try to automate tests that are

better performed manually

+Testing Checklist: ISO 25010 [11]

Functional suitability Reliability Operability

Performance efficiency Compatibility Maintainability

Security

Transferability

It can be good to have a checklist of different categories of test that

you need to cover. This ISO standard is one example, but there are

many others [20].

+Combined Engineering [23]

+Test-driven Development [18]

Test Driven Development is a software development process,

not a test process

“First the developer writes an (initially failing) automated test

case that defines a desired improvement or new function, then

produces the minimum amount of code to pass that test, and

finally refactors the new code to acceptable standards.”

+Crowdsourced Testing [19]

“Crowdsourced testing is an emerging trend in software testing

which exploits the benefits, effectiveness, and efficiency of

crowdsourcing and the cloud platform. It differs from traditional

testing methods in that the testing is carried out by a number of

different testers from different places, and not by hired

consultants and professionals. The software is put to test under

diverse realistic platforms which makes it more reliable, cost-

effective, fast, and bug-free.”

+Testing: There is so much to learn

Black Box Software

Testing Foundation [1]Test Automation [2]

Context-Driven Testing

[3]

Testing at Google &

Microsoft [4][5]ISO29119[6] Testing & Checking [7]

+Software Testing vs. Game Testing

General Software

Testing

Game Testing

+What makes Game Testing Unique

Fun Factor Testing[8] Balance Testing[8]Game Level/World

Testing[8]

AI Testing[8][10]Multiplayer/Network

Testing[8]Audio Testing[8]

Physics Testing[8] Realism Testing[9] Modification API Testing

+There are many similarities

Software is software whether it is a game or not

Software testing is an engineering discipline whether it is games, applications or other types of software

Functionality, performance, stability, compatibility, localization, etc. are mostly the same, but with different focus depending on context

A game tester should have the same general knowledge base as any software tester, but with a special focus on what makes games unique

+Fun Factor Testing

User experience and usability are valid testing areas for all

types of software

What differs games from other types of software is that it has to

be fun – it is entertainment and it needs to have a fun factor

that other types of software may not require

To be able to work with Fun Factor Testing requires a large

domain knowledge of games and what makes them fun

+Balance Testing

Balancing different options is something that is much more

prevalent in games compared to other software

Balancing different character options such as race, class, and

attribute choices

Balancing weapons, boosters, equipment, and other paraphernalia

Balancing levels

Balancing difficulty

+Game Level/World Testing

Somewhat unique to games is the number of instances in which assets are used

An application may use a feature in one place, while a game might use it in 20 different levels or parts of the world

This means that you need to test that the asset is used correctly in all these instances

Especially 3D worlds require a unique approach

Stuck/Sticky spots

Invisible walls

Map holes

Missing geometry

+AI Testing

Facing a computer controlled opponent is common in games, but less common in other types of software

Testing the different attributes and of the AI, and how it reacts in different situations is critical in many games

How well it mimics human behavior

Survival instinct – such as looking for cover in a fire fight

Hunting state – not waiting to react to the player’s actions, but proactively performing actions

Infighting – how it reacts to other AI

Pathfinding

+Multiplayer / Network Testing

Playing against multiple other opponents is quite common in games, and even though other software also communicates with other users and servers through different mediums, it is often to a lesser extent, and less sensitive

Many things can go wrong in multiplayer

Failed connections

Dropped connections

Lag

Invisible players

Scoring errors

Unaccepted invitations

+Audio Testing

Obviously almost all software has some kind of audio

However in games it is often more detrimental to the user experience if something goes wrong

Often much more complex in games, with many sounds playing simultaneously

Many things can go wrong

Audio drop

Skipping

Distortion

Missing sound effects

Volume level

+Physics Testing

Some games have physics engines, which affect both

gameplay and animations

It requires a specific skill to spot physics bugs

Breakable geometry must be tested to assure that it is

destroyed in a way consistent with the desirable physics

Dynamic behavior such as boxes moving when you walk into

them is another example of physics in action

+Realism Testing

This type of testing is also related to how the game managed to mimic the real world in a desirable way

Is the car handling like a real car?

Does the weapon feel like a real weapon?

Running animations and jumping must have the right look and feel

This type of testing requires a lot of domain knowledge – to know if an airplane is realistic enough, you must know about airplanes

+Modification API Testing

If the game allows for user to create their own mods, and it is a competitive game like an mmo this requires a unique approach

Open APIs are common in software, but competitive games add a dimension of not allowing mod users to gain unfair advantages or being able to exploit the game using their mods

Imagining how the APIs will be used is critical to understanding if there will be future problems or not

+Player Types [9]

You can categorize gamers in a slightly different way than

users of other applications and technology in general

Killer Achiever Explorer Socializer

Casual Gamer

Hardcore Gamer

Button Masher

Customizer

Exploiter

+Prioritization based on Player Types

How you prioritize your tests could be influenced by which

player types you are aiming the game for

If you want to please all types, then making sure you have run

sufficient tests in each category will go a long way when it

comes to reducing critical bugs

+Combinatorial Testing [9]

Of course combinatorial testing is nothing unique to games, but

when looking at a large, sprawling 3D game world, one can

imagine the size of the combinatorial explosion

A larger game world, a large amount of actors, and a large

amount of actions for these actors to perform, all results in a

combinatorial nightmare

+Conclusion

Software testing in general, and game testing in particular, are extremely complex activities

It takes a lot of time and effort

It requires a lot of experience and knowledge

It needs to be handled very delicately to not produce waste

It is necessary if you want to release a game that is valuable to some person(s)

+References[1] BBST

http://www.testingeducation.org/BBST/

[2] The A Word

https://leanpub.com/TheAWord

[3] Context-Driven Testing

http://context-driven-testing.com/

[4] How Google Tests Software

http://www.amazon.com/Google-Tests-Software-James-Whittaker/dp/0321803027

[5] How We Test Software at Microsoft

http://www.amazon.com/How-We-Test-Software-Microsoft/dp/0735624259/

[6] ISO29119

http://www.softwaretestingstandard.org/

[7] Testing & Checking

http://www.satisfice.com/blog/archives/856

[8] Game Development Essentials: Game QA & Testing

http://www.amazon.com/Game-Development-Essentials-QA-Testing/dp/1435439473

[9] Game Testing: All on One

http://www.amazon.com/Game-Testing-Second-Charles-Schultz/dp/1936420163/

[10] Artificial Intelligence (Video Games)

http://en.wikipedia.org/wiki/Artificial_intelligence_(video_games)

[11] ISO 25010

http://www.iso.org/iso/catalogue_detail.htm?csnumber=35733

[12] Gerald Weinberg

http://secretsofconsulting.blogspot.se/

[13] Software Testing

http://en.wikipedia.org/wiki/Software_testing

[14] Heuristic Risk-based Testing

http://www.satisfice.com/articles/hrbt.pdf

[15] Test-driven Development

http://en.wikipedia.org/wiki/Test-driven_development

[19] Crowdsourced Testing

http://en.wikipedia.org/wiki/Crowdsourced_testing

[20] Test Heuristics Cheat Sheet

http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf

[21] Lessons Learned in Software Testing

http://www.testingeducation.org/BBST/testdesign/KanerBachPettichord_Lessons_Learned_in_SW_testingCh3-1.pdf

[22] Agile Testing Quadrants

http://www.developsense.com/presentations/2014-06-Dublin-RSTAgileTesting.pdf

[23] To Combine … or not

http://angryweasel.com/blog/to-combine-or-not/