The Survey Says: Testers Spend Their Time Doing...
-
Upload
techwellpresentations -
Category
Software
-
view
26 -
download
0
Transcript of The Survey Says: Testers Spend Their Time Doing...
9/11/2014
1
© 2013 IBM Corporation
@Al_Wagner
IBM Rational
Testers spendtheir time doing…
The
1 © 2013 IBM Corporation
Session Abstract
How can testers contribute more to the success of their project and their company? How can they focus on asking the right questions, improving test planning and design, and finding defects so the business releases a quality product―even though there’s always one more fire to extinguish or one more request to fulfill? There aren’t enough hours in the day to do it all.
Join Al Wagner as he reveals recent survey results showing where testers actually spend their time and where testers think their time would be better spent. Compare your own experience with what 250 test professionals from around the world reported. You may be surprised how prevalent testing challenges really are. Learn what techniques and technologies are available to help today’s test professionals execute what they were actually hired to do—test software.
Return to your organization with an increased understanding of how other testers are dealing with their testing bottlenecks and what activities your peers view as the best use of their valuable time.
9/11/2014
2
2 © 2013 IBM Corporation
Why run a survey?
Aside from talking to directly to customers, we were…
watching forums and blogs
reading papers
following social media channels
listening to the Analysts
What we found was a small sampling of data harvested from anecdotes and discussions between small groups.
We wanted to hear directly from the Testers!
We wanted to hear from a lot of Testers!
3 © 2013 IBM Corporation
Agenda – Survey Says
Setting the stage
Challenges testers face
Help is available
Putting it in context
9/11/2014
3
4 © 2013 IBM Corporation
Who responded?250 Testing Professionals
from around the world!
Testing Experience
Testing Roles
Quality Managers, Test
Managers, and Test Leads
with more than 6 years
experience!
5 © 2013 IBM Corporation
What and how are they testing?
250 Testing Professionals
dedicated to improving
software quality
Types of applications being tested
Development methodology followed
43% cite Scrum as their main
development methodology.
56% of organizations are
using some variation of
iterative development.
9/11/2014
4
6 © 2013 IBM Corporation
What tools and techniques are being used?
61% of Testers surveyed use
documents & spreadsheets to manage
software quality and test execution.
Is this effective? You decide!
7 © 2013 IBM Corporation
Agenda
Setting the stage
Challenges testers face
Help is available
Putting it in context
9/11/2014
5
8 © 2013 IBM Corporation
What’s The Rush?Better – Cheaper - Faster
In The Past
• Organizations looked to find a
balance between speed, cost &
risk.
• Development teams were not as
agile as they are today; testing
never seemed to have enough
time.
Today
• Organizations are asking teams to
release higher quality software
sooner.
• Development teams are becoming
more and more agile; testing still
never seems to have enough
time.
9 © 2013 IBM Corporation
Change in Release Frequency
How have Release Cycles changed?
…over last 3 years
32% of Test Professionals surveyed have
seen an increase in release frequency
over the past 3 years.
34% of those who
expressed no change were
already releasing quarterly
or better.
9/11/2014
6
10 © 2013 IBM Corporation
What is the ratio of Testers to Programmers on a delivery team in your organization?
11 © 2013 IBM Corporation
Testing takes as long as it takes! Really?
Testing will take
longer when you are
not “actually” testing.
How Much of Your 40-Hour Workweek Is
Spent on Unplanned, Non-Testing Activities?
Most Disruptive
non-Testing Activities!
35% of Testers are
spending at least ½ of their
week not testing!
9/11/2014
7
12 © 2013 IBM Corporation
How Confident Are You in Your Product's Quality?
44% of Test Professionals
surveyed have little to no confidence
in the quality of the software
being released.
13 © 2013 IBM Corporation
Where and How Often Do You Find Defects?
40% of Testers shared that defects are
frequently found in production!
Shouldn’t it be a priority to find defects sooner?
9/11/2014
8
14 © 2013 IBM Corporation
What change do you feel will make the biggest difference in your organization?
15 © 2013 IBM Corporation
Agenda
Setting the stage
Challenges testers face
Help is available
Putting it in context
9/11/2014
9
16 © 2013 IBM Corporation
Testers want to spend
more time on…
1
2
3
4
5
6
7
8
We asked 250 Testers where they would like to spend less time and where they felt their time would be better spent?
17 © 2013 IBM Corporation
1
2
3
4
5
6
7
8
Survey Says!
I want to Spend MORE time
9/11/2014
10
18 © 2013 IBM Corporation
Things to remember when automating tests…
Not everything should be automated!
Not everything can be automated!
Test automation is not a silver bullet!
19 © 2013 IBM Corporation
Are you testing what’s under the surface?
Build and Deploy
Software components
built by developers.
Software being
checked by Testers
9/11/2014
11
20 © 2013 IBM Corporation
Test Automation
Are we automating the right tests?
Are we automating the tests rights?
Traditional Test Automation*
* http://www.agilecoachjournal.com/index.php/2014-01-28/testing-2/the-agile-testing-pyramid/
User Interface
tests
Service
tests
Unit
tests
21 © 2013 IBM Corporation
Prioritizing your test automation effort
Focus on…
Manual /
Exploratory
Testing
pushing
tests
lower
Unit Tests
Tests written before code (TDD);
instant feedback to the developer on
code being written; defects isolated to
specific line(s) of code; run as part of
build automation process
Integration Tests
Tests message exchange, behavior, and
performance between integrated and dependent
components/systems; defects isolated to
specific application component, continuously
executed as part of deployment automation;
lower effort to maintain as application changes
User Interface tests
Checks functionality via the user interface without
understanding the application architecture (black box);
most fragile of automated test types; lacks ability to
isolate defect; most time consuming to create and
maintain; dependent on user interface being available
and stable
9/11/2014
12
22 © 2013 IBM Corporation
Unattended test executionTeams need to automate tests at every layer
23 © 2013 IBM Corporation
Separate what the test accomplishes from how it is executed
–Keywords abstract the details of how to execute a logical unit of function
–Business scenarios compose keywords into a flow
Better leverage the skills in your organization
–Subject matter experts can focus on the business scenarios independent of implementation
–Test automation engineers can focus on automating
Reduce test (scenario and automation) maintenance
–Keyword behavior isn’t reproduced in multiple tests; update fewer places when application changes
–Reduce test cases because overlap becomes apparent when abstraction is raised to keyword level
Durable, maintainable, resilient to changeTesters want maintainable test automation
9/11/2014
13
24 © 2013 IBM Corporation
A business process may be available through multiple channels– The goal is the same across channels
– The flow is similar across channels
– The UI gestures differ depending on channel
A business process may span multiple channels– Different steps in the flow occur on different
channels
– Consider applications taking advantage of mobile capabilities such as location and near field communication
Multichannel testing is about effectively creating, managing, and executing tests of these kinds of processes
Validating business flows across multiple channelsEffectively creating, managing and executing tests
Keywords make supporting channels easy!
25 © 2013 IBM Corporation
Testers want to spend
LESS time on…
1
2
3
4
5
6
7
We asked 250 Testers where they would like to spend less time and where they felt their time would be better spent?
9/11/2014
14
26 © 2013 IBM Corporation
1
2
3
4
5
6
7
8
Survey Says!
I want to Spend LESS time
27 © 2013 IBM Corporation
Standing up Test Environments is a key inhibitorApplications are getting more complex
• Increasing expense of standing up
complex environments
• Delays caused by lack of system
availability
• Third-party services can be
especially costly
• Testers spend more time setting up
tests than testing
30-50%The average amount of time testing teams
spend on setting up test environments,
instead of testing
$5-30 million
The typical investment to build a single
test lab for a Fortune 500 company. Most
have dozens …
9/11/2014
15
28 © 2013 IBM Corporation
How do you mitigate delays?
Beware… these approaches are risky and
potentially damaging to the business.
29 © 2013 IBM Corporation
Solving the Test Environment dilemmaMaking production-like test environments a reality
Today’s test environments must be: Realistic and complete (production-like)
Inexpensive to stand up
Available on demand
Agile test environments require four enabling capabilities Test Data Management
Service Virtualization
Deployment Automation
Orchestration
9/11/2014
16
30 © 2013 IBM Corporation
Time Spent on Test Data Management(more than one response allowed)
64% creating test data manually!
34% cloning production and manually cleansing!
31 © 2013 IBM Corporation
Refresh test data
speeding testing and
application delivery
Reduce Risk
Mask sensitive
information for
compliance and
protection
Reduce Cost
Automate creation of
realistic “right sized” test
data to reduce the size
of test environments
• Understand what test data is needed for test cases
• Create “right-sized” test data by subsetting
• Ensure masked data is contextually appropriate to the data it replaced so as not to impede testing
• Easily refresh & maintain test data by developers and testers
• Automate test result comparisons to identify hidden errors
• Support custom & packaged ERP applications in heterogeneous environments
Improve application quality and delivery Agile testing needs continuous access to test data
Speed Delivery
9/11/2014
17
32 © 2013 IBM Corporation
Dealing With Unavailable Software & Services(more than one response allowed)
Survey says… “Top 2 tactics”:
• Manual creation and maintenance of mocks
• Expensive; lack of scalability; takes time away from testing
• Defer testing until dependent software is available
• Adds risk; decreases test coverage; potential project slip
33 © 2013 IBM Corporation
What is Service Virtualization?Agile teams need to make the unavailable available for testing
Virtual components simulate the behavior of a service or application during testing
– System dependencies are a key challenge in setting up test environments
– Unavailable/inaccessible services slow down the testing process
– Costly 3rd party access fees can significantly impact testing availability
Virtual components run on commodity hardware, private cloud, public cloud
Each developer and tester can easily have their own test environment
Developers and testers can continue to use current testing procedures and tools
Streamline creation of production-like environments and negative testing
Heterogeneous Environments
Public CloudPrivate Cloud
Data Warehouse MainframeEnterprise
Service Bus
Directory
Identity
File
systems
Collaboration
App Under TestRouting
Service
Third-party
Services Portals
Content
ProvidersEJB
Shared
ServicesArchives
Business
Partners
Messaging
Services
Databases Mainframe
applications
App Under Test
Third-party
Services
Packaged apps, messaging services, etc.
Virtual Components
9/11/2014
18
34 © 2013 IBM Corporation
Deployment AutomationAgile teams need to accelerate application delivery across the lifecycle
Automated, consistent deployments and rollbacks of applications
Orchestration of changes across servers, tiers and components
Configuration and security differences across environments
Clear visibility: what is deployed where and who changed what
Integrated with middleware, provisioning and service virtualization
Key Features:
• Multi-Tier Application Models
• Easy Process Designer
• Track what is where:
Inventory
• Scalable Distributed
Automation
• Quality Gates and Approvals
• Integrations Replace Custom
Scripting
Deployment Automation
35 © 2013 IBM Corporation
Agenda
Setting the stage
Challenges testers face
Help is available
Putting it in context
9/11/2014
19
36 © 2013 IBM Corporation
Dev-Test-Ops (aka. DevOps)Enterprise capability for continuous software delivery that enables organizations to seize market opportunities and reduce time-to-customer feedback
Accelerate
software delivery –
for faster time to value
Balance speed, cost,
quality and risk –
for increased capacity
to innovate
Reduce time to
customer feedback –
for improved customer
experience
Continuous
Customer
Feedback &
Optimization
Collaborative
Development
Continuous Release
and Deployment
Continuous
Monitoring
Continuous
Business Planning
Continuous
Testing
Operate Develop/ Test
Deploy
Steer
DevOpsContinuous Feedback
37 © 2013 IBM Corporation
All the world’s a “test” stageand all Testers merely players; They have their exits and their entrances;
The Continuous Delivery Pipeline
Development Testing Staging ProductionBusiness
Owners Customers
From ideation to implementation…• everyone contributes to quality!
• everyone contributes to a positive end user experience!
• everyone contributes to the success of the business!
9/11/2014
20
38 © 2013 IBM Corporation
Continuous Testing:
• helps extend agile and lean practices across
the delivery lifecycle
• removes traditional testing bottlenecks such as
unavailable test environments to increase
efficiency
• enables teams to test earlier and with greater
coverage at lower cost
• provides project teams with continuous
feedback on software quality to reduce
business risk
A comprehensive Continuous Testing solution
covers quality management, functional UI and
integration test automation, performance testing,
and service virtualization.
Organizations need Continuous TestingAccelerating the delivery of higher quality software
Resolve today’s testing
challenges with confidence
39 © 2013 IBM Corporation
Orchestration and automation for the deployment of applications,
middleware configuration, and database changes into development,
test and production environments.
Databases InternalMessages
Third-partyServices
virtual components
Simultaneously
test across
multiple test
stages
Dev QA
Service
Virtualization
Combining continuous testing withContinuous Testing enables Continuous DeliveryRepeatable, & traceable test and deployment automation
Test
Automation
Test Environments
Dynamic Infrastructure
Virtualize unavailable
yet dependent software
& systems
Continuously test in
production-like env.
Quality
Management
Manage quality across
the lifecycle
Deployment
Automation
Deploy private, secure
right sized test data
Test Data
Management
Deploy just built
applications
Continuous
Delivery
Orchestrate the
delivery pipeline
integrated with
9/11/2014
21
40 © 2013 IBM Corporation
Top 6 take aways
1
2
3
4
5
Not every test can be automated.
Not every test should be automated.
At least 1/3 of the delivery lifecycle
is testing. Testers do matter!
Test what matters earlier.
Test it continuously!
Deploy what you have.
Virtualize the rest!
6
Everyone contributes to quality!
You can’t test quality in.
Continuous Testing enables
Continuous Delivery
41 © 2013 IBM Corporation
The
Download your copy today!
ibm.biz/TestingSurveyResults
How do you stack up to what
the others are saying?
9/11/2014
22
42
ibm.biz/continuoustesting
43