Test Status Reporting: Focus Your Message for Executives
-
Upload
techwellpresentations -
Category
Technology
-
view
186 -
download
1
description
Transcript of Test Status Reporting: Focus Your Message for Executives
T7 Concurrent Class
10/3/2013 11:15:00 AM
"Test Status Reporting: Focus
Your Message for Executives"
Presented by:
Stephan Obbeck
KROLL Consulting AG
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Stephan Obbeck
KROLL Consulting AG
Senior consultant and managing director of the test center at KROLL Consulting AG,
Wiesbaden, Germany, Stephan Obbeck has more than fifteen years of IT experience with
hands-on software testing and is a certified project and test manager. After starting his career
as a tester, Stephan now invents test organizations for leading telecommunication and media
companies in Europe and manages test projects. As an architect for test automation, Stephan
supports organizations in moving from manual testing to automation.
1
03. October 2013
Stephan Obbeck
KROLL Consulting AG
Wiesbaden, Germany
Test Status Reporting: Focus Your Message for Executives
Agenda
Firmenvorstellung 2KROLL Consulting AG
View this presentation as animated Prezi at
http://prezi.com/j0ufgev9cukz
2
Management - and Test Reports
• Too Many Information� Management members pretend to have not much time -
keep them thinking like this!
� Your status report is not the only one a manager has to read
• Too Technical Details
� Most management members do not understand the
technical details of a test project
� If they do not understand, they rarely ask for more
information
� Decisions are made on data which is not understood
• Contradictory Information
� Data is tricky - it might fight back if you have contradictory
information
� Irregular reporting mechanism and content during a project
(this week reporting about progress, next week about
defects)
3KROLL Consulting AG
Status Reports - Layers of Communication
4KROLL Consulting AG
• Information Transfer FACTS� Obvious Status of the project
� Transfer of Information
� Actual Situation and next steps
• Non-Information Transfer EMOTIONS� Project advertisement
� Underline Requests you might have by data
� Influence Management decisions regarding your test project
• Reduced Complexity COMPACTYour recipient does not have much time
Focus your message - bring it to the point
� key facts on one page
� statistics understandable without additional information
� Make sure the right decision is taken from the report
3
Management Summary Report
5KROLL Consulting AG
• Status of your project� Colored status
• Progress of your work� Tasks during reporting period
� Tasks for the next reporting period
• Maximum two statistics that
describe:
� Progress of the test project
� Situation of your main project risks
Detailed Report
6KROLL Consulting AG
• Status of your project� Additional statistics and features
� Overview of bugs
� Statistics for specific stakeholders
� =
4
Status Reports - Structure of Test Reports
7KROLL Consulting AG
Keep report as
short as possible
Use graphics and
charts whenever possible
Avoid technical details
needing explanation
Focus on what you really
want to say
Prepare Your Test Report: Recipients
During Test Planning –
Plan Your Reporting
• To whom do you report?� What is his / her position?
� Which information does he / she need?
� How deep is the technical understanding?
� To whom do you HAVE to report?
� To whom do you WANT to report?
8KROLL Consulting AG
5
Prepare Your Test Report: Content
During Test Planning –
Plan Your Reporting
• What do you want to report?� There are tons of data information you can report.
� Which numbers really give value to management?
� How can you influence management?
• How do you want to report?� Plan your project and keep your reporting in mind.
� Reporting the progress?
� Reporting the risks or better the tasks?
9KROLL Consulting AG
Stakeholder Analysis (WHO?) - Questions
10KROLL Consulting AG
Stakeholder are all persons /
organizations who might be affected by your project.
• This analysis helps you to find out, who might have interest /
or requests reports from your test project:� What is the position of the stakeholder?
� How is the stakeholder affected by your project?
� What is the stakeholder's opinion to your project?
� How does the stakeholder see the project?
6
Stakeholder Analysis (WHO?) - Rating
11KROLL Consulting AG
• Rate each stakeholder by� Interest
� Power
• Use a matrix to evaluate your results.
• Visualize your stakeholders and
their relation to the project.
Stakeholder Analysis (WHO?) - Matrix
12KROLL Consulting AG
7
Risk Analysis (WHAT?) – for Benefit
A risk analysis identifies areas of danger to
your project and how they can be defeated.
• Report your success. (What have you already achieved?)
• Report your areas of risk. (What can make your project fail?)
• How can you measure your risk?
• How can numbers and diagrams
illustrate your risk?
13KROLL Consulting AG
Risk Analysis (WHAT?) - Matrix
• Analyse your risks and evaluate� Likelyhood (How likely is it that this risk occurs?)
� Consequences (How big are the consequences of this risk?)
• Use a matrix to group your risks
• Visualize the most important risks for your project
14KROLL Consulting AG
8
Risk Analysis (WHAT?) - Visualization
15KROLL Consulting AG
Example
Risk: High number of defects might be found� Your team can handle 20 defects per week. If more occure, you need
additional testers.
� Report number of defects from the beginning on with a bar chart.
Plan Your Work Packages Right (HOW?)
16KROLL Consulting AG
Try to build work packages
that have the same size by either
• Testcases with equal number of steps.
• Group Testcases to bundles of same
estimated working time.
• Use parameters for level of completeness
for each working package (see later slide).
• If your work packages have a comparable size� You ease your test planning.
� You have better control on your project status.
� Your status report can use linear progress information.
9
• How to present the overall
project status?
• Some of the most common used
status methods:� “Signal Status” with red, yellow, green
� Project tendencies
� Subjective project progress
General Status Reporting
17KROLL Consulting AG
Status Reports often use a “signal status“
• Green:
Project on track
•• Yellow: Yellow:
Slight delays - project re-planning necessary
• Red:
Main project targets can not be reached within
project parameters
• Problem:� Yellow (middle) is a safe harbor.
� I have seen projects being on yellow for weeks and months.
The Yellow Comfort Zone
18KROLL Consulting AG
10
Signal Status with Even Number
BETTER:
Use a reporting system with an
even number of status fields (e.g. 4 or 6)
• Green: Project on track
•• Yellow:Yellow: Slight delays - no management attention
necessary (e.g. setup daily defect meetings)
• Orange: Significant problems - management
attention necessary (e.g. additional resources)
• Red: Main project targets can not be reached
within project parameters
19KROLL Consulting AG
Avoid Project Tendency
Often a tendency is added to the status:
Which direction the project heads to.
"Status will go up next week, when we work over the weekend“
"Status will go further down, if we do not get another tester“
• Do not report trends - manage your risks!� Every manager should know that a risk coming to reality
will have a negative impact on the project.
� Report the risk with the actions need to be taken.
� If a risk is resolved by actions, your project will be
in scope again!
� Add a risk management matrix to your report.
� Ensure management can see the risks and their actions
on a first look at the report / risk matrix.
20KROLL Consulting AG
11
Risk Management
Example of a Risk Management Matrix:
A simple way how to manage a risk (Risk #01):
• Risk: Time schedule tight. If many bugs will be found during
testing and need to be documented, time for testing might
be too short.
• Action taken: We will work next weekend.
• Result: At the weekend we can test 25 test cases and the
project will be back on track on Monday.
21KROLL Consulting AG
90% Syndrom
Joe has to write a test concept.
At the end of every day you ask him about the status.
What answer will you get as the days went by? 90%
Longer tasks often stick in an incomplete status of 80 - 95%.
There are always some remaining things to be done...
22KROLL Consulting AG
12
Level of Completeness
• Split longer tasks ( > 1 day) � Sub tasks (<= 1 day)
� Assign level of completeness
to each sub task.
• Add a constant
"time elapsed" fraction
for each reporting period
during the project:� If your project is 4 days long (=100%) and
you report every day, your time fraction is 25%
• Report elapsed time and level of completeness
• At least one of these two should have a constant progress to
manage your test report by progress status reports.
23KROLL Consulting AG
Manage your Test with Status Reports
24KROLL Consulting AG
13
Improve your Status Reports
25KROLL Consulting AG
By using different Charts
• Excel - and other tools - offer dozens of different chart types.
• Which one to choose?� Support the message you want to transfer to management.
� One Eye Catcher for your message.
� Use colors to emphasize your main message.
• Green = OK
• Red = Attention
� I prefer basic chart layouts,
do not be too creative.
Pie Chart and Bar Chart
26KROLL Consulting AG
• Pie Chart� Basic chart to show different status information
� Low expressiveness
� Just a snapshot status
• Bar Chart� Same information available as in pie chart for each bar
� History information available (PAST)
� History information to follow a trend (FUTURE)
14
Test Progress Evaluation
27KROLL Consulting AG
If your goal is to finish 2000 test cases in calendar week 32 the history in
the bar chart can help you to justice if you are on track.
Green arrow: Your project is on track.
Red arrow: Your project is not on track and you need to
take corrections (new target date)
Yellow / orange arrow: You look fine but you should be careful...
Test Progress
28KROLL Consulting AG
In general your test progress
• does not follow a straight line
• follows an S-Curve:
� Starting flat
� Increases in the middle
� Flats down at the end
15
Test Progress Analysis Methods
29KROLL Consulting AG
• This analysis method works if you have a
constant work progress for each of your
reporting periods.
• If this is not the case (different number of
testers, etc.):� Use a time elapsed progress report.
� Add an expected target line to your diagram to identify your
planned progress during the project.
Test Automation
30KROLL Consulting AG
Three reports should be created out
of your test automation
16
Management Summary
• Purpose:� Total test results of the test run
� Basis for decisions
• Adressee:� Managers
� Decision Makers
• Structure:� Not more than one or two pages
� Use graphics to evaluate the result quickly
� Use colors to illustrate important messages
� Just a passed / not passed information
� No (or very short) explaining texts for errors
31KROLL Consulting AG
Error Report
• Purpose:� List of errors found during the test run.
� Checklist for analysts to work through all
anomalies and review it.
• Addressee:� Developers, Testers and Software Analysts
� Project Manager
• Structure:� Show all results that are not OK.
� Show the test case and the object under test.
� State a text explaining the anomaly: The IS
state and the expected SHOULD result.
32KROLL Consulting AG
17
Inspection Report
• Purpose:� Complete list of all test results for each test
cases and objects under test.
� Complete overview of the test run - all test
cases have been executed.
• Addressee:� Project Manager
� QA Inspector
� Auditing Department
• Structure:� Show ALL results of the test run
(OK and not OK).
� Show all test cases and all objects under test.
� Explaining text might be used as well.
� Might be created as a data file for a test
database.
33KROLL Consulting AG
Summary - You‘re almost there…
34KROLL Consulting AG
Don’t forget
• Keep your report compact.
• Plan your reporting before kicking off
the project:� WHO? (Stakeholder Analysis)
� WHAT? (Risk Analysis)
� HOW? (Project Planning)
• Avoid reporting mechanism needing explanation.
• Use your project control mechanism in your report.
• Organize your reporting from test automation.
18
I appreciate your feedback
35KROLL Consulting AG
THANK YOU
• The presented methods are taken from various management
certification workshops and are my own experiences in
different projects.
• If you pick up one of these ideas and try it in your
organization, feel free to drop me an email and share your
experiences with me:
Thank You
STEPHAN OBBECK
KROLL Consulting AG
Schenkendorfstraße 5
D-65187 Wiesbaden
www.kroll-consulting.de [email protected]