A New Model For Testing
-
Upload
eurostar-conference -
Category
Software
-
view
585 -
download
10
description
Transcript of A New Model For Testing
![Page 1: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/1.jpg)
A New Model for Testing@
paul_gerrard
Paul [email protected]
gerrardconsulting.com
Programme Chair 2014
![Page 2: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/2.jpg)
Helping clients transform their testing through
INNOVATION, COACHING and LEADERSHIP
Our CLIENTS– Want to be agile rather than follow Agile
dogma– Have a pragmatic approach and are focused on
delivery– Want a solution that fits, not a badly fitting
suit.
![Page 3: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/3.jpg)
Typical tester job spec? (May 2014)• Skills & Requirements
– Thorough C# knowledge including Linq– Good .Net experience including charting and winforms
applications.– Experience building and maintaining automated test
systems– Ability to interact with the developers to anticipate future
needs– Ability to influence to ensure good test procedures are
developed and implemented• Desirable Skills
– Some knowledge of Java for mobile testing– An interest in photography. Wearable Computing
(Internet o
f Eve
rything)
![Page 4: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/4.jpg)
Agenda• Pressure to Change• Three Innovations• Where is this Leading?• A New Model for Testing• Consequences• Shift-Left is More Important than
Agile
![Page 5: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/5.jpg)
What's happening?• Stampede to mobile computing• Big Data• Internet of (Every)Thing, Pervasive
Computing• Continuous Delivery, DevOps• (Test) Analytics, Data-Driven
Development• Shift-left, embedded testers, no test
team• Did you really think Agile was the last
word?
![Page 6: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/6.jpg)
Three Innovations That Are Driving
ChangeBeing agileShift LeftAnalytics
![Page 7: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/7.jpg)
#1Agile doesn’t work
but being agile might
Agile is no longer innovative.Challenging Agile and being
agile might beIs continuous delivery for
you?
![Page 8: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/8.jpg)
Agile doesn’t work?• It works for some• But many organisations have difficulties• Mostly a cultural problem– Requirements take months to elicit and agree– Complex stakeholder relationships– Stakeholders with ‘all the knowledge’ can’t be
spared to be the on-site customer/product owner– Lack of confidence => big system/acceptance
tests• We are Agile but ...
![Page 9: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/9.jpg)
Pragmatism rules (again)• Pragmatic companies use the things
that work and discard those that don’t• Let's be honest: Most are doing some
form of hybrid approach:– Chaotic interfaces between Agile teams
and structured businesses– TDD, BDD, ATDD, SBE today…– Continuous Delivery, DevOps… tomorrow?
![Page 10: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/10.jpg)
#2Shift Left
Developers in test?Testers in development?
Testing (testers?) everywhere
![Page 11: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/11.jpg)
TestingBDD/Test-Driven)Trusted
Requirements + Stories
Shift left (redistributed testing)
Functional Requirements
Build (+little testing) but lots of rework
Testing (lots of checking + bugs)
Trusted Requirements +
StoriesTesting
SAVEDTIME
BDD/Test-Driven
Who is going to do all this?BAs, Developers or Testers?
![Page 12: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/12.jpg)
From traditional delivery…
… to Continuous Delivery
![Page 13: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/13.jpg)
Deployment into Production is not
the Same as ‘Going Live’
Release to production and test environments on-demand
Go Live weekly/monthly/quarterly…
![Page 14: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/14.jpg)
#3Test Analytics
Analytics from productionAnalytics from test
An opportunity for testing
![Page 15: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/15.jpg)
Analysis Insight Decision Development and Testing
ProductionTesting
Introducing Test Analytics
“Introducing Test Analytics”, Paul Gerrard:http://gerrardconsulting.com/index.php?q=node/630Julian Harty, "Symbiotic Relationships between Testing and SMAC":http://blog.bettersoftwaretesting.com/2013/07/symbiotic-relationships-between-testing-and-social-mobile-analytics-cloud-computing/
“The capture, integration and analysis of test and production monitoring data to inform business and software development decision-making”.
![Page 16: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/16.jpg)
Data silos for test analytics(an illustration)Stakeholder Requirements Assurance (Production)
ApplicationMonitoring
Production Environment Monitoring
Stakeholders Business Goals
and measures Stakeholder
Involvement/ Engagement
Risk
Requirements Stories/Feature
Descriptions Glossary of
terms and Term Usage
Processes Process Paths
(Workflows)
Manual Tests Generated Test
Code (unit, integration, system level)
Application Instrumentation
Automated Test Execution History
Test Status Feature Code
Change History
Application Process Flows
User Accesses/ Activities
Feature Calls, Response times
Feature use cases (data usage)
Application Alerts/Failures
Database Accesses
Database Content
Production failures
System assets Resource
Usage Log Performance
data System Events System Alerts/
Failures/Incidents
Outages
![Page 17: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/17.jpg)
Where is this leading?
If I knew that, I wouldn't be here...
![Page 18: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/18.jpg)
Three development patterns
Structured
Agile Continuous
Goal-B
ased
Hi-Process
Autonomous
![Page 19: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/19.jpg)
There aren't three patterns
There are many
![Page 20: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/20.jpg)
From high process to DIY process• In the past, high process was seen to be
the technical solution to the software problem
• But technical solutions cannot solve human problems
• Agile promised flexibility, but a lot of Agile is really 'high process-in-the-small'
• True agility means a team can adapt or select its process on-the-fly.
![Page 21: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/21.jpg)
The old ways won't work in the future
We need a New Model of Testing (free from logistics)
![Page 22: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/22.jpg)
A New Model for Testing
Don't throw stones. Yet.I'm working on a refined version (with
others).Want to help/review?
![Page 23: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/23.jpg)
Forget Logistics(for the time being)
Document or not?Agile v waterfall?
Planned v exploratory?This technology v that
technology?
![Page 24: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/24.jpg)
ALL Testing is Exploratory
We explore sources of knowledge ...
... to build test models ...... that inform our testing.
![Page 25: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/25.jpg)
All Testing is Based on Models
![Page 26: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/26.jpg)
Examples of models
![Page 27: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/27.jpg)
Models are innate, essential, human
![Page 28: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/28.jpg)
Judgement, exploring and testing
Testing(the system)
Our model(s) are adequate
Our model(s) are not adequate
Exploring (sources) Judgement
Creates testmodels
Uses testmodels
We explore sources of knowledge to build test models that inform our testing
BTW – Do Developers explore the same way? I think so.
![Page 29: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/29.jpg)
Exploration process
ExplorationDefinitions specs/stories
Oracles
Sources
DesignModels
TestModels
Enquiring
Challenging
Sources:People, documents,experience, system under test
Modelling
Test Models:Can be documented
or mental models
Predicting
System under test
![Page 30: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/30.jpg)
Testing process
TestingSystem
Under Test
Refining
Informing Applying
Interpreting
TestModels
Revise the System
Logging Revis
ing
More exploring Reporting
![Page 31: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/31.jpg)
New Model Testing
29 page paper: http://dev.sp.qa/download/newModel
![Page 32: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/32.jpg)
Some Consequences
![Page 33: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/33.jpg)
Do developers and testers think the
same way?Similar processes, not thinking
the sameEasier to embed testers in
developmentDeveloper testing will improve
![Page 34: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/34.jpg)
Covers functional and non-functional
testing?Ignoring logistics means we
can have the same foundational process for ALL
testing(I think)
![Page 35: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/35.jpg)
Relation toTDD and BDD?
TDD is not testing BDD is modelling using stories
![Page 36: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/36.jpg)
Testing v Checking?
I submit that a Check is "a test that can be applied by a
human or tool"But only humans can interpret
![Page 37: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/37.jpg)
Test automation from a different
perspectiveAutomation uses different test
modelsIs that why automation efforts fail so
often?
![Page 38: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/38.jpg)
Developer Testing
The model works (for me)
![Page 39: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/39.jpg)
Capabilities
Enquiring, Modelling, Predicting, Challenging
Informing, Applying, Interpreting, Refining
Reporting and Logging
![Page 40: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/40.jpg)
• Analysis, enquiry and elicitation• Modelling• Creation of custom models, using
heuristics, guesses, brainstorming, ideation, creative thinking
• Custom test design techniques• Comparison of models, value,
advantages, disadvantages, compromises
• Identification, validation and use of oracles
• Predicate logic and proof• Hypothesis and inference• Socratic method• Rapid Review and Inspection
techniques• Test case design• Test models and the meaning of
coverage• Testing as controlled experiment• Observation, Note taking,
recording
A very different skillset• Basic data analysis and statistics• Decision-making with
incomplete data• Computer forensics• Fault tree analysis• Failure diagnosis• Bug advocacy, triage processes
and negotiation• Meaningful software and test
metrics• Visual presentation of data• Reporting and presentation skills• Understanding stakeholders• Test analytics• Risk management, risk-based
testing and decision-making• Critical Thinking• Interpersonal skills• Dealing with
uncertainty/fallibility
![Page 41: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/41.jpg)
Shift-Left is More Important than
Agile
![Page 42: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/42.jpg)
Developers write code; testers test
because developers can’t
No they don’t
![Page 43: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/43.jpg)
Goal: End Manual Feature Checking
Let the tools do it andfree people to do testing
please!
![Page 44: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/44.jpg)
Testing doesn’t just apply to existing software, at the
endWe test systems, not
software.We test early, we test often but it might not be called
testing
![Page 45: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/45.jpg)
Testers Need to Learn How to Code
Some, perhaps many will have to
![Page 46: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/46.jpg)
Testing is about measuring
achievement, not quality
Align systems delivery with project goals and risks and you’ve got a seat on the
board
![Page 47: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/47.jpg)
Testers don’t own testing anymore
Be a TestMaster"Assurance through the
lifecycle"
![Page 48: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/48.jpg)
New Model Testing changes what
testers & developers need to
learnRoles of developers/testers become
blurredPerhaps they are not different disciplines
after all?Why have a tester certification?
![Page 49: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/49.jpg)
The phase after development is REWORK, not
TestingTesting must align with
development, not compete with it or rescue it
![Page 50: A New Model For Testing](https://reader035.fdocuments.us/reader035/viewer/2022081715/53fd88148d7f72a81c8b4941/html5/thumbnails/50.jpg)
A New Model for Testinghttp://dev.sp.qa/download/newModel
@paul_gerrard
Paul [email protected]
gerrardconsulting.com
Programme Chair 2014