1

SASQAG $99 day, October 21st

Seattle Area Software Quality Assurance GroupAcceptance Test Driven Development

(ATDD) in PracticeThis one-day class covers the whole ATDD cycle, including Discussing the new feature or story, Distilling acceptance criteria into acceptance tests, Developing the feature and automating the acceptance tests, and Demonstrating the resulting feature to the business stakeholder(s) responsible for accepting it.

Elisabeth Hendrickson has over 20 years software industry experience and has been working with Agile teams since 2004.

2010-10-05

Don’t test too much! [or too little](Lessons learned the hard way)

Keith Stobie

3

Agenda

Lessons Learned in Software Testing Using Simple Oracles Test Case size and Known Errors Never be Gatekeeper – know release

goals Testing Exception handling Alternatives to execution Testing Combinations

2010-10-05

4

Five-fold Testing System Classification

From: Lessons Learned in Software Testing

Who: Developers, testers, internal users, beta users, . . .

What: Coverage – Requirements, scenarios, functions, code, errors, . . .

Why: Potential problems – Risks you are testing for.How: Activities – Regression, exploration,

installation, . . .Evaluation: How to tell whether the test passed or

failed. 

Lesson #283: “Apply diverse half measures”

2010-10-05

5

Use Simple Oracles

Test SquareRoot() function:1.Assert(SquareRoot(X) == MySquareRoot(X))

2.Assert(SquareRoot(X) == StdSquareRoot(X))

3.Y := SquareRoot(X)Assert(X == Y*Y)

2010-10-05

6

Over Testing (transactions)

ACID TransactionAtomicity, Consistency, Isolation, Durability

Over test: For each operation, repeat the same operation on a side “test” implementation.

Test system must be as performant as Real system (to keep same level of concurrency)

2010-10-05

7

Banking Transactions

Transfer $100 from checking to savings1. Add money to savings or Remove

from checking2. Remove money from checking or Add to

savings 2 Steps, both or neither must occur

for consistency.

Checking Savings

$100 $100

Checking Savings

$100 $100

2010-10-05

8

Simple Transaction Oracle

Maintain a consistency invariant outside of transactions (but violated inside)1. Randomly choose first operation and value:

Transfer Savings -> Checking 1,234.572. Choose last operation to keep balances

/1,000A. Add more: Savings -> Checking 765.43

(2,000 total transfer)B. Add less : Checking -> Savings 234.57

(1,000 total transfer)2010-10-05

9

Simple vs Accurate

Pros: Simple fast Check

(can be made post production)Cons: Difficult Failure Diagnosis

(just like with real customers)Related notions: Metamorphic testing Sampling & Consistency Oracles

lesson #125 “Avoid Complex logic in your Scripts.”2010-10-05

Metamorphic Relation (MR) Reuse existing test cases to generate

more tests. (Existing tests as Oracles) MR: Any relation among the inputs

and the outcomes of multiple executions of SUT.

SqRt(X)*SqRt(Y) == SqRt(X*Y) Test 1: SqRt(9) == 3 Gen Test 2: SqRt(9)*SqRt(2) ==

SqrRt(9*2)

Shortest Path examples

• ShortestPath(G, a, b)find the shortest path between vertices a and b in graph G and also output its length, where G is an undirected graph with positive edge weights. (dijkstra’s algorithm)

• Reverse:  The shortest path between B and A should be the reverse of the shortest path between A and B.

• Prefix: For any vertix, V, on shortest path between vertices A and B, the Shortest path between A and V must be the same sequence as the start of the path between A and B.

Test Boyer-Moore example

Boyer-Moore algorithmreturns the index of the first occurrence of a specified pattern within a given text. 

Metamorphic Relations Reverse: if string X exists in string

Y, then the reverse of Y exists in the reverse of X.

13

Agenda

Lessons Learned in Software Testing Using Simple OraclesTest Case size and Known Errors Never be Gatekeeper – know release

goals Testing Exception handling Alternatives to execution Testing Combinations

2010-10-05

14

Pass Rate by 25% jumps

1 3 5 7 9 11 13 15 17 19 21 23 25 27 290

25

50

75

100Keith ProjectA ProjectB

Test Cycle (or week)

% Pass Rate

lesson #280 “How to lie with

test cases”

2010-10-05

15

Test in the Presence of Failures Spec Explorer “long test”

Ideal: Single test for all transitions in a model

Pass/Fail test == Pass/Fail product Simplest way to design test cases is

to fail fast. As soon as an error is detected, note the error and stop any further processing (Assert)

Information loss when many Asserts.Symptom: Many “blocking” failures

lesson #34 “It Works” really means it appears to meet some requirement to some

degree”2010-10-05

16

Test Case Size

“Appropriately complex” [Kaner 2003] Towards smaller tests: Fewer blocking failures (less dependency/commonality) Parallel execution. Test Prioritization for regression (running < whole set) Automatic Failure Analysis tools

Towards larger tests: Reduced redundant setup/cleanup time Chance of inadvertently implementing a test idea that

finds a bug. [Marick 2000]

Systemmedium-large?

Integration

medium

Functional

small-med

Unitsmall

2010-10-05

17

Agenda

Lessons Learned in Software Testing Using Simple Oracles Test Case size and Known ErrorsNever be Gatekeeper – know

release goals Testing Exception handling Alternatives to execution Testing Combinations

2010-10-05

18

Lesson #12, “Never be the gate keeper”

New Java Transaction Api (JTA) [Part of J2EE]

Built on top of existing Transaction engine.

1 Week before Beta: Multi-threaded testViolated Isolation! (I in ACID)

I and Dev mgr think must slip to fix Project Mgr understands meets

release goals!Just document in release notesSet customer expectations

#205: “Don’t sign-off to approve the release of a product”2010-10-05

19

Agenda

Lessons Learned in Software Testing Using Simple Oracles Test Case size and Known Errors Never be Gatekeeper – know release

goalsTesting Exception handling Alternatives to execution Testing Combinations

2010-10-05

20

Old: Returning Error codes

2010-10-05

If (!ret) return ret;

21

Modern: Exception Propagation

2010-10-05

22

Normal/Exception States

Start

Norm

al

retru

n

Previous Return

Exception

Exc

epti

on

Exception

Normal return

Previous Return

Normal

Exception

Normal return

2010-10-05

Lesson #25, “All testing is based on models”

23

Pattern to Exercise Transitions

Start

2. N

orm

al

retru

n

Previous Return

Exception

3.

Exc

epti

on

1. Exception

Previous Return

Normal

4. Exception

5. N

orm

al

retru

n

2010-10-05

24

Agenda

Lessons Learned in Software Testing Using Simple Oracles Test Case size and Known Errors Never be Gatekeeper – know release

goals Testing Exception handlingAlternatives to execution Testing Combinations

2010-10-05

25

Avoiding hard stuff?

2010-10-05

26

Agenda

Lessons Learned in Software Testing Using Simple Oracles Test Case size and Known Errors Never be Gatekeeper – know release

goals Testing Exception handling Alternatives to executionTesting Combinations

2010-10-05

27

Testing Combinations

2010-10-05

In the code,the BypassProxyOnLocal property on the WebProxy object does not reflect the fact that it is checked

28

Pairwise PICT results Visual Basic installation testing:

30% fewer test cases vs. minimal test suite created manually: 60 tests from PICT vs. 80-90 manual out of 200 possibilities (exhaustive). Testers had high confidence in quality

235 tests generated for a domain with over 1M possible combinations. All these tests implemented in less than one week; traditional methods: 15-30 tests per week. An order of magnitude of a difference.

Games testing: Asheron’s Call testing: the team was able to cut test

resources by half due to use of PICT Vanguard testing: 4 man-months to build and

implement test casesvs. 44 man-months during last release. Process more efficient by an order of magnitude

Windows command-line tools testing: (findstr.exe &) Attrib.exe testing: 13 test cases generated by PICT

achieved identical block coverage as 370 exhaustive test cases 

2010-10-05

29

Summary

Testing is a business activity with a cost. Understand the product and release

goals, Deal with common issues like known

errors, Anticipate likely errors Choose the right evaluation methods

2010-10-05

30

Remember

Use Simple Oracles Allow for Known Errors Choose “appropriately complex” Test

Case size Know release quality goals

GateKeeeper Test Exception handling Consider Pairwise Testing of

Combinations Consider Alternatives to Execution2010-10-05

31

Thanks!

2010-10-05