Agile Testing Analytics
-
Upload
qasymphony -
Category
Software
-
view
156 -
download
0
Transcript of Agile Testing Analytics
![Page 1: Agile Testing Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062503/587615071a28ab306c8b660f/html5/thumbnails/1.jpg)
Agile Testing Analytics
STARWEST 2016
![Page 2: Agile Testing Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062503/587615071a28ab306c8b660f/html5/thumbnails/2.jpg)
Presenter
Jonathan Alexander CTO of QASymphony
![Page 3: Agile Testing Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062503/587615071a28ab306c8b660f/html5/thumbnails/3.jpg)
SDLC
QASymphony - A Robust Testing Platform
Monitor PPM
Require-ments
Build
Test
Defects
Deploy
Automation
![Page 4: Agile Testing Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062503/587615071a28ab306c8b660f/html5/thumbnails/4.jpg)
Our Solutions
For Test Case ManagementqTest is a powerful,
elegant, scalable test management solution that works with small
teams and allows large enterprises to coordinate
and track hundreds of projects across many
locations.
For Unscripted Testing
qTest eXplorer is a ground-breaking test
documentation tool that supports exploratory
and unscripted testing AND saves time when performing traditional
manual testing.
For Enterprise Reporting
Insights 2.0 gives the testing team a self-
service business intelligence tool to
consolidate, manage and analyze all your
testing data.
![Page 5: Agile Testing Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062503/587615071a28ab306c8b660f/html5/thumbnails/5.jpg)
Our Customers
![Page 6: Agile Testing Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062503/587615071a28ab306c8b660f/html5/thumbnails/6.jpg)
In This Session
Analytics - What and Why
Quality Analytics
Coverage and Risk Analytics
Velocity Analytics
Test Case Optimization
How to Get Started
![Page 7: Agile Testing Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062503/587615071a28ab306c8b660f/html5/thumbnails/7.jpg)
Analytics - Data Sources
Many potential data sources
Need to create links: dev tickets -> code check-ins, test cases, defects, support cases
![Page 8: Agile Testing Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062503/587615071a28ab306c8b660f/html5/thumbnails/8.jpg)
The Goal: Leverage Data for Improvement
● Use analytics to improve:
○ Test coverage
○ Forecasting completion dates
○ Efficiency and most effective use of resources
○ Test case quality
○ Productivity
● Think of analytics as an objective input to the planning process
![Page 9: Agile Testing Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062503/587615071a28ab306c8b660f/html5/thumbnails/9.jpg)
Quality Analytics
● Core:○ Test result %s by project/release
■ separate out latest runs○ Defect priority and status %s
● Extra:○ Test results by day or week○ Defect status/priority crosstab○ Defects per test run○ Defect leakage (found by users)
● Tips:○ Use color-coding to identify potential issues○ Put manual and automated results side-by-side
![Page 10: Agile Testing Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062503/587615071a28ab306c8b660f/html5/thumbnails/10.jpg)
Coverage Analytics● Core:
○ Test cases by requirement■ Latest run results■ Breakdown by type
○ Defects by requirement
● Extra:○ Test case complexity○ Test time per requirement○ Last date of test run(s)
● Tip:○ Use data visualization to spot risks
![Page 11: Agile Testing Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062503/587615071a28ab306c8b660f/html5/thumbnails/11.jpg)
Velocity Analytics● Core:
○ Requirements inflow rate○ Test case creation rate○ Test run rate (cases & steps)○ % tests complete and blocked○ Defects opened and closed
● Extra:○ Avg. and total testing time spent○ Mean time to test(s) created, run, passed○ Forecast time and defects remaining
● Tip:○ Breakdown analytics by tester
![Page 12: Agile Testing Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062503/587615071a28ab306c8b660f/html5/thumbnails/12.jpg)
Test Case Optimization● Start to think of test cases like source code
● Track manual and automated test cases, exploratory scripts
● Track analytics that will help optimize your test case library○ Days since last run
■ Archive test cases that are not used anymore○ Flapping (# of times consecutive runs have different results)
■ Examine these tests and code/functional areas, might indicate need to refactor one or the other
○ Percentile complexity (steps and time spent per test)■ Refactor highly complex tests for greater efficiency and
more pinpoint understanding of results○ Cumulative execution time
■ Automate the manual tests that are taking the most time■ Refactor automated tests that are running the longest
![Page 13: Agile Testing Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062503/587615071a28ab306c8b660f/html5/thumbnails/13.jpg)
How To Do This Yourself● Setup a Test Analytics Reporting Server
○ Use an open source or 3rd party BI tool (such as qTest Insights)○ Keep it simple
● Identify team members that will Work on Test Analytics○ Depending on tool(s) may need technical and “analyst”○ Commit to a certain # of hours per week or per month
● Start with Requirements, Test Results, and Defects data○ For most companies data size will be very manageable○ Don’t tackle big data problems (partition data if necessary)
● Start with Quality analytics, then add Coverage, then Velocity○ Focus on weekly project reports
● Add More Detail and More Data Sources Over Time
![Page 14: Agile Testing Analytics](https://reader035.fdocuments.us/reader035/viewer/2022062503/587615071a28ab306c8b660f/html5/thumbnails/14.jpg)
Thank You for Listening!Questions?