QA is Broken, Fix it!
-
Upload
ffrees-family-finance -
Category
Career
-
view
18 -
download
0
description
Transcript of QA is Broken, Fix it!
Stephen Blower
@badbud65
QA is Broken.Fix It!
"In every chain of reasoning, the evidence of the last conclusion can be no greater than that of the weakest link of the chain, whatever may be the strength of the rest.“
Thomas Reid's ”Essays on the Intellectual Powers of Man” 1786
How many different development/test
methodologies are in existence?
Question:
Spiral Model of Software Development (1986)
Rational Unified Process (1994)V-Model (1982)
Waterfall (1984)
Scrum (1995)
Crystal Clear (1996)
Extreme Programming (1996)
Adaptive Software Development (1995)
Feature Driven Development (1995)
Dynamic Systems Development Method (1995)
Agile Manifesto Published (2001)
Test Driven Development (2003)Rapid Application Development (2004)
Goal-Driven Software Development (2006)Behaviour Driven Development (2009)
IntroductionNo one right way
We all desire perfection but, the road to success is often misleading with self confessed Oracles.
Fantasy Reality
No one Development Team is the SameNumerous Tools
Numerous Methodologies
Diverse Levels of Experience and Skill Sets
Wide ranging levels of Enthusiasm and Energy
Optimistic and Pessimistic Attitudes Towards Change
Strict Process Driven or Flexible and Informal Practices
Personal Ownership or Strict Hierarchical Structures
Development Processes Should be about the People
Not How to Control the People
The ideas I would like you to take away from this talk
• Software development hasn’t advanced as far as people like to believe
• Too many ideas, little consensus and no proven methodologies
• No one software development methodology is the holy grail
• Change is difficult, move forward slowly and throw away broken processes
What is QA and what do I mean by Broken?Quality Assurance?
Dictionary Definition:
Quality, defines a “Certain Level of Excellence”.
“Excellence” meaning “Extremely Good”
“Assurance” means “Keeping Promises”
So Quality Assurance means:
Keeping Promises to Supply Products or Services to a Certain Level of Excellence.
~Johanna Rothman put this really well in a talk, she said that when testers tell her that they do “Quality Assurance", she asks questions like these:
1. Do testers have the authority and cash to provide training for programmers who need it?
2. Do testers have the authority to settle customer complaints? Or to drive the handling of customer complaints?
3. Do testers have the ability and authority to fix bugs?
4. Do testers have the ability and authority to either write or rewrite the user manuals?
5. Do testers have the ability to study customer needs and design the product accordingly?
If not, the quality of the product and of the complaining customer's experience in dealing with it, are not under the testers' control.
This exert is from a Paper Called: The On-going Revolution in Software Testing: Cem Kaner
We are all doing it wrong, because no one knows how to do it right, including me.
A Brief History of Software
DevelopmentHow did we get into this mess?
A Brief History of Software Development Practices
The Agile™ Manifesto states:
We are uncovering better ways of developing software by doing it and helping others do it.
Through this work we have come to value: Individuals and interactions over processes and toolsWorking software over comprehensive documentationCustomer collaboration over contract negotiationResponding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more.
When was agile first used?
Quote: From: Iterative and Incremental Development: A Brief History
Craig Larman of Valtech & Victor R. Basili of University of Maryland
Gerald M. Weinberg:
“We were doing incremental development as early as 1957, in Los Angeles, under the direction of Bernie Dimsdale [at IBM’s Service Bureau Corporation]. He was a colleague of John von Neumann, so perhaps he learned it there, or assumed it as totally natural. I do remember Herb Jacobs (primarily, though we all participated) developing a large simulation for Motorola, where the technique used was, as far as I can tell, indistinguishable from XP.”
XP wasn’t published until 1996, nearly a full 40 years later.
1957! Yes 1957!
Don’t Blame Royce, Blame the US Military.
Winston W Royce: “Risky and Invites Failure
1984 : American Department of Defense for System Software Development created DoD STD 2167
It promoted a one-pass document-driven waterfall process.
Anecdotally: US Military just took this picture to develop the Waterfall model
Why did Waterfall gain popularity?
HierarchicalTop down structured approach
Strict controlsCreates illusions of control
Now we have 101 ways of doing the same thing:
Developing Software
Software development is still in its teens and as such there is not any empirical evidenced based approach to software testing.
So which methodology is the right one?
SDLC 100 Success Secrets - Jeremy Lewis Typical Snake Oil - Not one of the 100 (Supposedly) Asked Questions are anything you couldn’t answer yourself if you have a modicum of experience within software development.
Common Failings•Not knowing what a PROBLEM is
•Not knowing how to change
•Not knowing what change looks like
•Not understanding the model/s chosen fully
•Not spending the time to implement the change
•No real desire for the change, just a general feeling
•Choosing the wrong model or choosing one model
•Not having a detailed delivery plan
•Not having clearly defined goals
“A goal without a plan is just a wish.”
Antoine de Saint Exupéry
Assessing the Current
SituationAssess for Success
What is a Problem?Something is not working? Yes?
Not strictly true
Who says it’s not working?
In what way is it not working?
How do you make a problem go away?
a) Change the differenceb) Change the perception
c) Change the desire
Jerry Weinberg calls people who rush to solve problems as a "solution problemmer". Not a problem solver, but a solution problemmer.
Basically: "If our solution doesn't work, *you should change your problem*.“
“A difference between what is perceived and what is desired.”
More specifically:
A difference /according to some person/ between what is perceived /by some person/ and what is desired /by some person/.
“An undesirable situation that is significant to and maybe solvable by some agent, though probably with some difficulty.”
Definitions and Stories, from and inspired on Skype conversation with Michael Bolton
Test - Serve the Business
Testing is a service role. Feel good about that. The service you provide is vital. Service implies clients—the people you serve. Your success is measured primarily by how well you serve your clients'
desires and best interests.
Project Managers
PMs are entitled to know your process and influence it. You serve the project manager by reporting your status on demand, reporting important problems fast, and not being a bottleneck to the project.
Document Writers/BAs
Like you, the people who write the documentation get incomplete information. You can help them understand how the product really works. Writers can help you, too. As they do their research on the product, they will learn things that you don't know.
Technical Support
Whatever problems are left in the product they become a burden for the people who provide technical support. You serve support by alerting them to aspects of the product that may trouble the user. If you work with them, they can help make the case that a bug should be fixed.
Senior Management and Share Holders
You serve the business. That's why you must be careful not to sound or act like a quality fanatic instead of a reasonable person. Express test status reports in crisp, operational terms, so that executives feel they have a basis on which to make decisions.
Marketing/Sales
Marketing & Sales need to know whether anything in the product is inconsistent with the key benefits the product is supposed to provide to customers. A bug that seems minor to others might be critical to marketing & sales.
End User
In your heart, you serve the people who will make use of the product. Their satisfaction is in the best interests of your project, of course. But there is also a special satisfaction that goes with being the primary user advocate on the project team.
Developers
You make the developer's job easier by providing good bug reports, as soon as possible. Strive to know your craft and know the product so you don't waste the developer’s time with mistaken or frivolous reports.
Lessons Learned in Software Testing: Section 1.3. Lesson 3
Talk to Everyone
“Talk to someone about themselves and they'll listen for hours.”
Brush up your social skills
Understand what is wrong
Access morale
Gather information uncritically
Understand desire to change
Gain constructive feedback
“Any fool can criticize, condemn and complain--and most do.”
Management opinions
Stay Neutral
Don’t take sides
Be open to discussion
Constantly invite feedback
Discuss opinions rigorously
All opinions are weighed equally
At this stage, they are
Everyone has an opinion
Be able to defend your arguments in a rational way. Otherwise, all you have is an opinion. - Marilyn vos Savant
To gain a fuller understanding
Get you hands dirty and test
Get familiar with the tools
Get familiar with the processes
This is essential
To gain respect
To gain direct experiences
To understand the products
Also get to understand your competitors
33% No – 30% Probably and 37% Definitely – That’s Scary
Delving Deeper• How do teams work together? Are they working closely or are they
siloed? How does information flow between teams? Is it managed in any way? Are people expected to source the information themselves or does the project manager ensure the right people are kept in the loop at the right time?
• Are people in teams listened to? Does one person in a team feel they are not listened to yet another has a totally different opinion? Why is that?
• How have changes been implemented previously? Did they work, if so how did they work? If not, why not?
Delving Even Deeper• What is the current process? Is it documented? Does everyone know what the
process is?
• Is there a roadmap? Do people know what is happening in two weeks’ time?
• Does test know when the next product is coming in to test? Are test consulted about the test schedule? Is there a test schedule?
• Are test involved at the design stage? Are test involved in any stage of the development process? Is test only involved when the product is code complete?
Creating you plan for change
Ideas are now growing roots
Ideas for change based on:
Experience with previous change
Practical hands on experience
Everyone's opinions
Personal experience
Change is now inevitable
Not just an idea
There’s a time line
This shits now getting real, baby.
Argue and discuss changes:
Convincingly
Collaboratively
Respectfully
Understand objections:
Is it practical
Is it appropriate
Be prepared to:
Throw away plans
Change plans drastically
“No man is an Island” - John Donne
James Bach
Summary: Assessing the Current Situation
• Talk to everyone and Listen to everyone• Do not weigh any option above any other• Stay neutral, be uncritical and stay positive• Continuously gain feedback• Gauge desire for change• Gain support and buy-in• Keep your enthusiasm levels high• Understand current processes, what works and what doesn’t• Understand how to communicate effectively• Learn to argue in a Socratic style• Interact directly, do the job of the testers• Start planning early• Create, modify, scrap plans and constantly evaluate plans
NOTE: I talk about measuring the success of new changes in the last section
Planning how to Deliver
ImprovementsPlan for Success
Think outside the box, while retaining what’s useful in the box
ALL! Changes for improvement identified
Don’t be fooled
Some ideas will be scraped
Some ideas won’t get buy-in
Continue to talk, discuss & collaborate
It’s not about you
It’s about improvement
Keep management in the loop
Create a realistic time line
They don’t like surprises
Don’t: “Over Promise and Under Deliver” or “Under Promise and Over Deliver”
Get buy-in
To agree with; to accept an idea as worthwhile.
To communicate a vision and create buy-in.
From the supporters
From opposes, this is essential
From management
From the ideas people
From the people who desire change
From the people who hate change
“Deep and sustainable change... requires changes in behavior among those who do not welcome the change.” ― Douglas B. Reeves
Process, Process, Process!
Avoid like the plague
Process changes are needed : Yes
Process for sake of process : No
Processes needed : Yes
But streamlined, not reams of docs
The won’t be read anyway
No Best Practices PLEASE!
“A “best practice” is an idea that a consultant thinks he can sell to a lot of people. There is no assurance that this idea has ever succeeded in practice, and certainly no implication that it has been empirically tested and found superior (best) to competing ideas under general conditions.”
Cem Kaner
Lets not call them Best Practices, lets call them Good Practices
Best: The highest quality, excellence, absolute qualifier, context independent
Practice: Habitual or customary performance
Therefore: Best Practice is: The highest quality of habitual performance with no context
Lloyd Roden Definition
Summary: Planning how to Deliver Improvements
• Keep talking• Get Buy-in from everyone and anyone you can• Be prepared to constantly evaluate plans for change• Collaborate• Keep your mind on the end game, Improving Test• Introduce small changes early• Always have you eye on the time line• Do not “Over Promise and Under Deliver” • Do not “Under Promise and Over Deliver” • Be realistic with your goals and time line• Don’t write numerous reams of documentation, they won’t be read• Don’t create numerous new processes, they will hate you for it• Don’t create ANY Best Practices• Get agreement
Inspire and Measure
Deliver for Success
Leadership comes in small acts as well as bold strokes - Carly Fiorina
Enable, Empower and Enthuse
Lessons Learned in Software Testing:Read it, don’t expect others to though
Lean Coffee sessions:Explore testing ideas
Pair Testing:Learn together, challenge
Challenge testing ideas:Challenge the status quo
Challenge, push and drive:Question, question, question
Encourage, nay demand feedback:Focus on those who don’t feedback
Praise with sincerity:Don’t praise for sake of praise
“Some men (sic) have thousands of reasons why they cannot do what they want to, when all they need is one reason why they can” ― Martha Graham
Measurement, ARGH!
Measure for improvement
Measure sparingly
Be aware of error margins
Don’t set targets to be achieved
Metrics give you information
It’s primarily about PEOPLE not metrics
Gain feedback on metrics
Ensure test are involved with metrics
Recommended Reading: Measuring and Managing Performance Organisations : Robert D Austin.
"When a measure becomes a target, it ceases to be a good measure“ - Dame Ann Marilyn Strathern
Did you take away these ideas from this talk?
• Software development hasn’t advanced as far as people like to believe
• Too many ideas, little consensus and no proven methodologies
• No one software development methodology is the holy grail
• Change is difficult, move forward slowly and throw away broken processes
Conclusion
• There’s no such thing as best
• There’s no such thing as perfection
• Don’t create process after process
• Don’t create any Best Practices
• Don’t create reams of documentation
• Change, challenge, and learn, together
• It’s about people, not how to control people
“If you meet the Buddha, kill him.” - Zen Master Linji – Blog: Adam Goucher on Heuristics
Questions?
He must be very ignorant for he answers every question he is asked - Voltaire