Lightning Strikes The Keynotes

Moderated by

Lee Copelandlee@sqe.com

Featuring

Michael BoltonJon Bach Fiona Charles

Griffin Jones

Keith Klain Gerard Meszaros

Janet Gregory Paul Holland

Nate Oster

Michael Bolton

Attention,Deficit,Disorder.

Michael Bolton

http://www.developsense.com

aircanada.com

aircanada.com

Error MessageVia Rail, between Montreal and Toronto, 2007

But I can’t contact my… oh, never mind.

Error MessageVia Rail, between Montreal and Toronto, 2007

Hyatt Regency, 2008

If you can’t do math, it’s a nickel extra.

Hyatt Regency, 2008

Why you shouldn’t let an unsupervised algorithm choose your sponsored links (1).

Vimeo’s Web PageSpring 2010

Why you shouldn’t let an unsupervised algorithm choose your sponsored links (2).

Vimeo’s Web PageSpring 2010

Why you shouldn’t let an unsupervised algorithm choose your sponsored links (3).

Vimeo’s Web PageSpring 2010

Google Chrome

Google Calendar Update

OK, fine.

Don’t Know Who This Was

Adobe Acrobat

Adobe Acrobat

Microsoft Outlook

Intuit Quicken

Intuit Quicken

Intuit Quicken

Here’s another err.

Intuit Quicken

ibm.com

Windows

I think I’ll shoot my own trouble for now.

Windows

Windows

I don’t know!

Windows

iTunes

Testing is more than checking.

Welcome to the new world.

“The World of Heineken”Heineken Experience, Amsterdam, 2006

Paul Holland

Functional Specification Blinders - My Experiences with Creativity

Paul Holland Test Consultant and Teacher at Testing Thoughts

Test Idea Creation

• A “Typical” method of developing test cases or test ideas:

1. Start by reading functional specification2. Write down test ideas (or create TCs)3. Try to think of other ways to test product4. If you think of any, write those down too5. When product is available, execute test ideas6. While testing try to think of more test ideas

April, 2013 ©2013 Testing Thoughts 34

What I Do on my Teams

• Over the past five years as a test manager at Alcatel-Lucent, I would reverse two steps:

1. Think of all ways you can to test product2. Write down test ideas3. Only then you read the functional specification4. Write any new ideas down5. When product is available, execute test ideas6. While testing try to think of more test ideas

April, 2013 ©2013 Testing Thoughts 35

How To Model Without FS

• Oh no! I can’t read the functional specification. How can I possibly think of test ideas?

1. If the product is available, then play with it2. If not, then talk to designers, architects,

marketing, customers, project managers, etc.3. Find out why the new feature/product is being

developed and how it will be used4. Play with competitive/similar products/features5. Learn as much as you can at a high level

April, 2013 ©2013 Testing Thoughts 36

Justification

• For years I had asked my team to think of their own test ideas before reading the functional specification

• I now have (statistically invalid) proof that reading the specification first can stifle individual creativity and limit the testing ideas that are generated

April, 2013 ©2013 Testing Thoughts 37

Experience Report

• During a modeling exercise of the Rapid Software Testing course November, 2012:

– Students were asked to identify all the dimensions of an object that may be relevant to testing it

April, 2013 ©2013 Testing Thoughts 38

Experience Report

• Students typically start with intrinsic dimensions:– Height, width, material, volume, etc.

• They then progress to relative dimensions:– Durability, dishwasher safe, contents, place of

manufacture, where it is now, where it has been, etc.

April, 2013 ©2013 Testing Thoughts 39

Experience Report

• Some students mention “conformance with standards”

• There is an ISO specification for a specific version of this object– A few have found this ISO specification exists– One student of mine found the spec and read it

April, 2013 ©2013 Testing Thoughts 40

Experience Report

• He found the spec BEFORE he had tried to think of his own dimensions

• After he read the spec, he could not think of ANY other dimensions

• In other exercises, this student showed good levels of creativity

• In this exercise, his creativity had been completely stifled by putting on the “specification blinders”

April, 2013 ©2013 Testing Thoughts 41

Summary

• Try to think of your own test ideas BEFORE reading the functional specification, previously existing test cases, or any other rigid documentation that may compartmentalize your thoughts

• Do not stifle your creativity. The functional specification will be waiting for you when you have done some of your own thinking

April, 2013 ©2013 Testing Thoughts 42

April, 2013 ©2013 Testing Thoughts 43

Griffin Jones

WRESTWorkshop on Regulated Software Testing

Software subject to review by an internal or external regulatory body

Purpose and Format• Share and generate ideas and

techniques• Provide a forum for people

interested in the topic• Participation is free and open to all• LAWST-style rules of engagement

Workshop Structure• Facilitated • Series of experiential

presentations and group discussions

• Atmosphere is collaborative, supportive and constructive

• Focus on the practical and useful

More Information• Next WREST: May 3rd 2013

Hosted by SQE at STAREAST

• Contact:Karen N. Johnson, John McConda, Griffin Jones

• Website: wrestworkshop.com

Gerard Meszaros

Avoiding Unmaintainable Automated Tests

Gerard Meszaros

StarCanada2013@gerardm.com

Premise:

Automated Tests are a Self-Verifying Behavior Specification

Because:

They Tell us What the System Should Continue to Do

After any changes or bug fixes we apply

The Challenge:

How to Avoid Creating

Tests/Checks/Specs

that Need Frequent Maintenance ?

My Belief:

Most Automated Tests Have Too Much Detail

Which is What Makes Them Require Frequent Maintenance

Need to specify at the right detail and scope

Behavior Specification at Right Level

Broad Narrow

Detail

Hig

h

Lo

w

Scope

Specify broad scope at minimum detail Specify most detailed req’ts at narrowest scope

Behavior Specification at Right Level

Rules &Algorithms

Multi-Use CaseWorkflows

Transactions(Use Cases)

IncompleteSpec

Too much detailUnmaintainable

Broad Narrow

Detail

Hig

h

Lo

w

Scope

Configurable Bank TX NotificationsExample:

Configure Notification

Rules

Suspend Notification

Resume Notification

AccountHolder

Process Transaction

TransactionSettlement

Testing Notifications - 1

Given: User and Accounts

Example:

When: Notification

Rule Configured

Then:Notification Rule

Configured and Active

Then:Notification Rule Configured and

Active

Testing Notifications - 2

Then: ExpectedNotifications

Medium Detail; Large ScopeToo much detail given the scope

Example:

When: The Transactions to be processed

Use Case: Process

Transactions

Issues With This Test

• Difficult to understand which TX should notify – because cause (rules) and effect (notification) are far

apart

• Only verifies one simple combination of rules– We will require many more tests to test all the other

combinations– Lots of repetition of workflow & data across test cases

• Tests will take a long time to run– Due to need to configure first, then run transaction

processing

Changing Level of Abstraction/Detail• Too detailed for verifying workflow and

• Too broadly scoped for checking the rules

• Need to Reduce Detail or Reduce Scope

Rules &Algorithms

Multi-Use CaseWorkflows

Transactions(Use Cases)

Too much detail

Unmaintainable

IncompleteSpec

Broad Narrow

Detail

Hig

h

L

ow

Scope

Reduce

Scope a Lot!Reduce

Both a bitRed

uce

Det

ail a

Lot

Specifying Notification WorkflowExample:

Broad Scope; Minimum Detail; Irrelevant Details Omitted!

Given: User &

Thresholds

When: Transactions

Are Processed

Then: Expected

Notifications

The overall workflow should be defined at a very high level

Changing Level of Abstraction/Detail• Need the detail to specify notification rules• Therefore, need to reduce scope

Rules &Algorithms

Multi-Use CaseWorkflows

Transactions(Use Cases)

IncompleteSpec

Too much detail

UnmaintainableBroad Narrow

Detail

Hig

h

Lo

w

Scope

Reduce

Scope a Lot!

Red

uce

Det

ail a

Lot

Reduce

Both a bit

Business Rule Specs

Configuration

Process Transaction

Threshold per Charge Type

High Detail; Narrow ScopeEasy to Understand

Example:

Given these rules

When we ask NotificationRequired

? with this transaction:

Then: The answer should be

NotificationLog

• Encapsulation of Rules in “Rules Component”– which can Expose a Test API

• Accept Rules via “Data Injection”– To Simplify Test Automation

• Making Automation a Dev’t Responsibility– To incent design-for-testability

System Under Test

Business Rules Tests Encourage:

NotificationRequired?

Do

Notification.

ProcessTransaction

ConfigureNotification

ThresholdNotification

Rules

Transaction

Interface

ConfigurationUser

Interface

NotificationRule Test

NotificationRules Fake

Gerard MeszarosStarCanada2013@gerardm.com

http://www.xunitpatterns.comhttp://ScopeVsDetail.gerardm.com

In Summary: Manage Scope vs. Detail

– to Avoid Unmaintainable Tests/Specs Provide the Tests/Specs to Dev’t Before

System Built– to allow Dev’t to automate the test – to ensure Design-for-Testability

Thank You!Jolt Productivity Award

winner - Technical Books

http://testingguidance.codeple

x.com/

Our Thanks To

Michael BoltonJon Bach Fiona Charles

Griffin Jones

Keith Klain Gerard Meszaros

Janet Gregory Paul Holland

Nate Oster