A Test Design Poster for Smarter Testing · A Test Design Poster for Smarter Testing Slide 4...

1

A Test Design Poster

For Smarter Testing

Peter Zimmerer

Siemens AG

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y

Software &EngineeringDevelopmentTechniques

A Test Design Poster for Smarter Testing © Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005

Slide 2

A Test Design Poster

for Smarter Testing

EuroSTAR 2005Copenhagen, Denmark

Peter Zimmerer

Senior Principal Engineer

Siemens AG

Corporate Technology Software & Engineering 1

D-81730 Munich, Germany

[email protected]

http://www.siemens.com/research-and-development/

http://w4.siemens.com/ct/

mailto:[email protected]






http://w4.siemens.com/ct/

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y

© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005


A Test Design Poster for Smarter Testing Slide 3

Contents

• Introduction

• Test design methods

Here: methods, paradigms, techniques, styles, and ideas to create a test.

• Examples and references

• Problem statement

• Poster Test Design Methods on One Page

• Guidelines and experiences

• Summary

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y




Introduction – Test design methods

• Good test design, i.e. high-quality tests,

is very important.

• There are many many different test

design methods and techniques.

• Static, dynamic

• Black-box, white-box, grey-box

• Based on fault model, experience, exploratory

• Statistical (user profiles), random (monkey)

• The tester„s challenge is to adequately

combine these methods dependent

on the given problem, domain, and

requirements.

• This is art as well!

• Black-box test design methods are

often based on models.

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y




Some systematic methods for test design

• Black-box (interfaces, data, models)

• Traceability matrix

• Use case-based testing, scenario testing

• Equivalence class partitioning

• Boundary value analysis

• State-based testing

• Cause-effect graphing

• Decision tables

• Combinatorial testing

• White-box (internal structure, paths)

• Control flow testing

• Data flow testing

• Selection, usage and

applicability depends on

the

• specific domain

• used software technology

• test requirements:

required test intensity,

quality criteria, risks

• existing test basis:

specifications, documents,

models

• project factors: constraints

and opportunities

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y




Example – There are always too many test cases ...

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y




Examples – Demo

• Microsoft PowerPoint

• Microsoft Word

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y




Some references

• Classical testing books cover test design to some extent.

• Boris Beizer: Software Testing Techniques

• Lee Copeland: A Practitioner's Guide to Software Test Design

• James Whittaker: How to Break Software

• James Whittaker, Herbert Thompson: How to Break Software

Security

• Standard for Software Component Testing by the British

Computer Society Specialist Interest Group in Software Testing

(BCS SIGIST) (see http://www.testingstandards.co.uk/)

• There are many different training opportunities by different

providers.

http://www.testingstandards.co.uk/

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y




Problem statement

• Starting from a risk-based testing strategy an adequate test design

is the key for effective and efficient testing.

• Automation of bad test cases is a waste of time and money!

• There are many different test design methods around for a long

time (perhaps too many?) and a lot of books explain them in detail.

• There are different

• categorizations, classifications, and dimensions

• interpretations and understandings

of test design methods which does not simplify their usage …

• So, when we look into practice we can see that often there is quite

limited usage of these test design methods at all.

• What are the reasons behind that?

• How can we overcome this and improve our testing approaches?

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y




Possible reasons and consequences (1)

• Developers do not have/spend time to read a book on testing.

• Missing the big picture.

• Information is not available at a glance.

• Which test design methods are there at all?

• Which test design method should I use in which context?

• A specific test design method is not “available” when needed.

• A specific test design method is too detailed or too complicated

to be used in practice.

• Often the focus is on depth and on perfectionism.

Remark: But that can be especially required, e.g. in a safety-

critical environment!

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y




Possible reasons and consequences (2)

• Test design tools are typically focused and implement only a few

test design methods, e.g.:

• ATD – Automated Test Designer (AtYourSide Consulting,

http://www.atyoursideconsulting.com/):

Cause-effect graphing

• BenderRBT (Richard Bender, http://www.benderrbt.com/):

Cause-effect graphing, quick design (orthogonal pairs based test engine)

• CaseMaker (Diaz & Hilterscheid, http://www.casemaker.de/):

Equivalence classes, cause-effects and combinations, boundary value

analysis, error guessing, pairwise combinations

• CTE (Razorcat, http://www.ats-software.de/):

Classification tree editor

• Reactis (Reactive Systems, http://www.reactive-systems.com/):

Generation of test data from, and validation of, Simulink and Stateflow

models

http://www.atyoursideconsulting.com/

http://www.benderrbt.com/

http://www.casemaker.de/

http://www.ats-software.de/



http://www.reactive-systems.com/



C O

R P

O R

A T

E

T E

C H

N O

L O

G Y




Poster Test Design Methods on One Page (1)

• Idea:

Systematic, structured and categorized overview about different

test design methods on one page.

• Focus more on using an adequate set of test design methods than

on using only one single test design method in depth/perfection.

• Focus more on concrete usage of test design methods

than on defining a few perfect test design methods in detail

which are not used then in the project.

• Focus more on breadth instead on depth.

• Do not miss breadth because of too much depth.

• Do not miss the exploratory, investigative art of testing.

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y





C O

R P

O R

A T

E

T E

C H

N O

L O

G Y





• Categories of test design methods are orthogonal and independent

in some way but should be combined appropriately.

• The selection of the used test design methods depends on many

factors, e.g.:

• Requirements of the system under test and the required quality

• Requirements for the tests – quality of the tests,

i.e. the required intensity and depth of the tests

• Testing strategy: effort/quality of the tests, distribution of the testing in the

development process

• Existing test basis: specifications, documents, models

• Problem to be tested (domain) or rather the underlying question (use case)

• System under test or component under test

• Test level/Test step

• Used technologies (software, hardware)

• Suitable tool support: for some methods absolutely required

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y





• The effort/difficulty for the test design methods or rather the

resulting test intensity is subdivided into 5 levels:

• 1 very low, simple

• 2 low

• 3 means

• 4 high

• 5 very high, complex

• This division into levels is dependent on the factors given above

for the selection of the test design methods and therefore can

only be used as a hint and guideline.

• A test design method also may be used continuously from

“intuitive use” up to “100% complete use” as required.

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y




Example:

Requirement-based with traceability matrix

• In addition describe every test design method on one page to

explain their basic message and intention.

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y




Example: Equivalence class partitioning and

boundary value analysis with 2 parameters

b

a

b

b

a a

max

min

maxmin

b

a

b

b

a a

max

min

maxminb

a

b

b

a a

max

min

maxmin

# test cases for n parameters and

one valid equivalence class: 4n + 1

# test cases for n parameters and

one valid and invalid equivalence

class: 6n + 1

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y




Guidelines and experiences

• Pick up this poster and

• give it to every developer and tester in your team or

• put it on the wall in your tester office or

• make it the standard screensaver or desktop background for all team

members or …

• Use the poster as a checklist for existing test design methods.

• Selection of test design methods is dependent on the context!

So, you should adapt the poster to your specific needs.

• The poster increases visibility and importance of test design

methods, especially also for developers to improve unit testing.

• The poster results in a closer cooperation of testers and

developers: you have something to talk about ...

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y




Summary

• There exist many different methods for adequate test design.

• When looking into practice often these test design methods are

used only sporadically and in a non-systematical way.

• A test design poster containing a systematic, structured and

categorized overview about test design methods will help you to

really get them used in practice in your projects.

• Do not miss breadth because of too much depth.

• This will result in a better and smarter testing overall which will

pay off very soon.

• Use this poster after this conference as one mean to improve the

usage of test design methods in your projects!!!

A Test Design Poster for Smarter Testing · A Test Design Poster for Smarter Testing Slide 4...

Documents

Transcript of A Test Design Poster for Smarter Testing · A Test Design Poster for Smarter Testing Slide 4...