Balancing TestAutomationTechniques

Matt Archer@MattArcherUk

{{{

matt.archer@expresssoftware.co.ukhttp://mattarcherblog.wordpress.com

Agenda

Two distinctly different pieces of test automation, both from the same project.

What makes them different?

Why have both?

Behaviour Driven Development (BDD)

User Centric

Unmanned

Key Scenario Coverage

A Specification and an example (bond naming)

• Html file containing both specification and example

Specification: Bond Naming Conventions

The summary name for a bond should be a combination of the issuer, coupon and maturity date in the following format; issuer + coupon (as %) + maturity date. That said, if the bond is perpetual, the maturity date should be replaced by the word "perpetual".

Some examples are shown below.

An Automated test / a fixture (bond naming)

public class CheckBondNames : ColumnFixture { public string Issuer; public string Coupon; public string MaturityDate; public Boolean isPerpetual;  

public string Name() { Bond bond = new Bond();

bond.Issuer = this.Issuer; bond.Coupon = this.Coupon; bond.MaturityDate = this.MaturityDate; bond.isPerpetual = this.isPerpetual; 

NamingService namingService = new NamingService(); String name = namingService.getName(bond);

return name; } }

• Class file to connect the example to the software to test

Test results (bond naming)• Test Results are added to a copy of the original html file

DEMO

Semi-automated Data Regression Testing

Self-Generated Expected Results

Focused, But Thorough

No Pass / Fail Result

A B C D E F

1 4 7 13 56 34

2 34 88 20 12 54

3 45 29 9 93 40

4 91 45 92 59 10

5 71 60 74 43 22

6 87 2 21 9 0

Release Candidate Current Live Release

A B C D E F

1 4 7 13 56 34

2 34 88 20 12 54

3 45 29 9 93 40

4 91 45 92 59 10

5 71 60 73 43 22

6 87 2 21 9 0

Testing a release for regressions in data validity

Oops… A regression bug!

The “compare” routine

Compare

String Live_Url String Test_Url String Save_Location

Void Compare() Void Highlight()

Highlighted ReleaseRelease Candidate

Current live Release

SeleniumExecuted once…

For every widget (~10)

On every screen (~25)

For every client configuration (~12)

= (approx. 3000 data tables to compare)

Manual analysis of results

What is a bug? What is a deliberate change?

Oracles / reqs are rubbish, get over it

• Picture

Test automation polaritiesTest automation polarities

Checking key scenarios vs. Checking all scenariosScripted expected results vs. Captured expected resultsPredefined actions vs. Random actionsCode for checking vs. Code for exploringRun on a trigger vs. Run on demandFast vs. SlowUser perspective vs. Technical perspectiveThrow away vs. Continual investmentAtomic vs. End-to-endRecorded vs. Written

Inspired by James Bach’s Exploratory Testing Polaritieswww.satisfice.com/articles/et-dynamics.pdf

Intuition

• Picture– (Pulling a loose thread)

Matt Archer@MattArcherUkmatt.archer@expresssoftware.co.ukhttp://mattarcherblog.wordpress.com

Any Questions?