Risk Management Process Metrics - Attwater Consulting Conf Pres/Risk Management... · Risk...
Transcript of Risk Management Process Metrics - Attwater Consulting Conf Pres/Risk Management... · Risk...
Risk Management Metrics
International Council on Systems Engineering
Risk Management Working Group Mark Powell, Chair
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 2
Proper Attribution• A project in work by the INCOSE RMWG• Paper Authored by Barney B. Roberts and
Richard Kitterman• Originally Presented at INCOSE
International Symposium July 2005 in Rochester, NY
• Presenter: Mark Powell, RMWG Chair
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 3
The Problem with Risk Management
• A good risk management process results in nothing happening
• How does one measure process performance?• Multiple choice:
• If a project meets its performance goals, then …• A. It’s risk management process was successful• B. The project had a run of good luck• C. The project was under-constrained• D. All of the above
• If a project overruns its cost commitments, then …• A. Its risk management process failed• B. The project got a bad roll of the dice• C. The project was over constrained• D. All of the above
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 4
Risk Management Measurements?
• What do you measure?• How do you measure it?• How do we know what is a “good”
measurement, or a “bad” measurement?
• INCOSE Chartered RMWG to Investigate RM Metrics
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 5
Potential RM Metrics• Comparison Metrics
• Other standards• Using CMMs• Problem: Assumes that the others are “good”
• Return on investment Metrics• Cost of investment of risk management process
execution ratio-ed to the reduction of risk• Estimate the risk without a mitigation plan, then ratio to
estimated risk after planned mitigation, compare to • Problem: both are estimates, inadequate historical basis
• Efficiency and Effectiveness Metrics• Measure attributes of the process that indicate
efficiency and effectiveness• Problem: Has promise, but very limited experience
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 6
Potential Metrics Continued• Staleness Metrics
• Measure the lag in the flow of products through the risk management process – too long in one step of the process is “bad”
• Problem: Not specific to the quality of the process -- will also measure the lag in a bad process
• Trending Metrics• Measure the change in the number of risks in various
categories over time• Problem: Assumes that all risks are equal – one “very
bad” risk may overwhelm many other “bad” risks and give a false sense of security
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 7
Potential Metrics Concluded• Results of RMWG Investigation
• The best measure – actually compare estimates to outcomes
• Problem: Statistical in nature and requires sampling and analyses of many cases to develop models and relationships
So, what to do? A proposal follows.So, what to do? A proposal follows.
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 8
RMWG RM Metrics Proposal• RM Metrics Classified by Usage Frequency
• Infrequent Metrics• Usually before or after a project• When significant performance issues are noted• During the development of a Risk Management process
• Continuous Metrics• Measure the process during execution• Measure the quality of the products during execution• Attempt to make interim corrections if needed
• On-demand Metrics• When a measurable result is available, compare to
expectations• Ad Hoc or Periodic
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 9
Infrequent Metrics and Measurements
• Compliance to organization’s standards• Build a compliance matrix – extract “Shall” statements• Compare project’s process against the organization's
standardsProject
Compliant? “Shall” Statement Project RM Plan Paragraph
5 The identification of risk shall be actively encouraged at all levels in the Project Team Pg. 2, Section 1.6
4 Risk identification shall be accomplished by entering the information into the Kepler Risk Database through the Kepler Risk Tracking Tool.
Pg. 2, Section 1.6
All identified risks shall be reviewed by Project Management on a cyclical basis and accepted, assigned to a Risk Owner for action, rejected or retired.
Pg. 2, Section 1.6
Corrective Action?
Corrective Action?
A scale of 1 to 5 indicating level of compliance
A scale of 1 to 5 indicating level of compliance
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 10
Infrequent Metrics and Measurements
• Compliance to “Best Practices”• No consensus on what are “Best Practices”• Use CMMs as a substitute
FA 1.1 FA 1.2 FA 2.5Manage
RiskFA 3.3 FA 3.4
LEVEL 5
LEVEL 4
LEVEL 3
LEVEL 2
LEVEL 1
LEVEL 0
Quantitative effectiveness goals are establishedContinuous process improvement
Evidence that work is accomplished
Performance is planned and tracked
Standard process is definedPrograms tailor the standard process
Measurable goals are establishedPerformance is predicted
GENERICPRACTICES &ATTRIBUTES
FA- SPECIFICPRACTICES
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 11
Sample CMM Assessment Management Category
Leve
l 2 G
ener
ic P
ract
ices
are
per
form
ed
Res
ults
are
at l
east
of a
dequ
ate
utili
ty
Leve
l 3 G
ener
ic P
ract
ices
per
form
ed
Res
ults
are
of a
t lea
st s
igni
fican
t util
ity
Leve
l 4 G
ener
ic P
ract
ices
per
form
ed
Res
ults
are
at l
east
of m
easu
rabl
y si
gnifi
cant
util
ity
Leve
l 5 G
ener
ic P
ract
ices
per
form
ed
Res
ults
are
of o
ptim
um u
tility
Cap
abili
ty M
atur
ity L
evel
BY
Ass
essm
ent T
ool
Project 01A ### # ### 0Project 01B 2Project 02 2Project 03 2Project 04 2Project 05 1Project 06 2Project 07 2Project 08 2
Level 1 Level 2 Level 3 Level 4 Level 5
Leve
l 4 S
peci
fic P
ract
ices
are
per
form
ed
Leve
l 5 S
peci
fic P
ract
ices
are
per
form
ed
Res
ults
are
at l
east
of m
argi
nal u
tility
Leve
l 1 S
peci
fic P
ract
ices
are
per
form
ed
Leve
l 2 S
peci
fic P
ract
ices
are
per
form
ed
Leve
l 3 S
peci
fic P
ract
ices
are
per
form
ed
####
The CMM Questionnaire Assessment Tool showed projects interviewed as Level 2 or less
The CMM Questionnaire Assessment Tool showed projects interviewed as Level 2 or less
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 12
Continuous Metrics
• Compliance to Plan• Performance
• Effectiveness• Efficiency• Staleness
• Trending
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 13
Continuous Metrics: Compliance to Plan
• Compliance to RM Plan• Is the project actually doing what it said it
would do?• A simple compliance matrix• Shall statements from the RM Plan for the
project versus evidence that the activities are actually performed
• Determine corrective action
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 14
Continuous Metrics: Performance
• Performance – measure the performance of the process• Effectiveness
• Effective: No or very few unforeseen “problems” occur• Approach: How many “problems” occurred that were
never identified as risks• Efficiency
• Efficient: Catching risks early when it is more cost effective to mitigate them
• Approach: Measure the time between when a risk was identified and when it became a problem
• Staleness• How many risk products are “stuck” in a process step and
how long have they been there
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 15
The Effectiveness Metric• Performance – Effectiveness
• Premise: An effective risk management system will prevent unexpected problems
• PE, Process Effectiveness is the ratio of problems encountered, Np, that were not identified as risks, to the risks identified, Nr
PE = 1 – Np/(Np + Nr)• Measure of goodness, 90% good, 80% watch,
70% Action• Action, causal analysis and process
improvement
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 16
The Efficiency Metric• Performance – Efficiency
• Premise: An efficient risk management system is one in which the planning and mitigation of risks occurs well before they become problems
• For n realized risks, Pe, Process efficiency, is the average time lapse between all risks’ identification date, TID, and the time that it is realized, TR,
• Pe = Σ(TR,i – TID,i)/n, • Measure of goodness, 90% good, 80% watch, 70%
Action• Action, causal analysis and process improvement
i=1,n
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 17
The Staleness Metric• Performance – Staleness
• Residence time for risks in major steps• Short residence times: < ~30 days, are “Excellent”
long residence times: ~180 are “Very Poor”• Measure of goodness: 90% good, 80% watch, 70%
Action• Action: directed project management attention to
insure actions• Example measures:
• First Latency: Time identified to time first action by project management
• Second Latency: Time from assignment to a Risk Owner to time the project “Accepts” the risk mitigation plan
• Subsequent Latencies: Lateness tracked against dates on the steps in the risk mitigation plan
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 18
Sample Latency Measurement
IDRisk
Coordinator Evaluates
Project Mgr Decision
Risk Owner Develops Mitigation
Risk Coordinator Evaluates
Project Mgr Decision
Track Progress on mitigation
steps
Retire
Retire
Reject
Accept W/O Mitigation
Accept, Mitigation Required
Time, tt = 0
First Latency Calculation, t1, Residency in “Active” Status, Time from ID to first decision by project
t = t1
Second Latency Calculation, t2, Residency in Mitigation Planning
t = t2
Third through n Latency Calculations, tn, “Lateness” on mitigation steps
30
20
10
030 60 90 120 180
Latency, Days
Num
ber
t1 t2Mitigation Steps
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 19
Trending MetricsDevelopment Risks
12
5 4 3 2 1 0 0
14
16 16 14 1413 13
10 8
16 8 10 10
10 1014 16
40
5
10
15
20
25
30
PDR
09/2
4/98
10/2
0/98
11/2
5/98
01/1
4/99
03/0
9/99
05/2
1/99
06/0
3/99
08/1
7/99
Time
Num
ber o
f Ris
ks
# Low
# Medium
# High
Risk Mix by Level
44
19 14 14 11 80
52
5957
50 5252
54
42
4
2229
36 37 40
58
4
42
0
10
20
30
40
50
60
70
80
90
100
PDR
09/2
4/98
10/2
0/98
11/2
5/98
01/1
4/99
03/0
9/99
05/2
1/99
06/0
3/99
Time
Perc
ent
% Low
% Medium
% High
• Body Count versus time• Measure change• Goodness is more
vague on this one• No change is “bad”• Increasing risk numbers
may be “bad”• A decreasing trend in the
red and yellow is good• Action, directed project
management attention to insure actions
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 20
More Trending Metrics• Waterfall charts• Measure
• Latency (Calculated elsewhere)• Errors in prediction of impact or effectiveness of mitigation
• Goodness: Miss residual risk by “half a color”• Action: directed project management attention to correct
Ris
k Le
vel
Time
Mitigation Plan Mitigation Effectiveness
AMEs Planned
AMEs Actual
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 21
Example: Consumption of Reserve
• Comparison of Expected Value of risks to available reserve
Risk Item (Or WBS)
Risk Lien in $K
1 35$ 2 155$ 3 76$ 4 265$ 5 46$ 6 52$ 7 354$ 8 66$ 9 78$ 10 465$ 11 321$
SUM 1,913$ RSS 749$
Current Reporting Period
Consumption Of Reserve
$-
$500
$1,000
$1,500
$2,000
$2,500
1 2 3 4 5 6 7 8 9 10 11Reporting Period
Res
erve
s
Total Risk Soft LiensAvailable Reserve
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 22
On-Demand Metrics: Results
• Based on risks that have been accepted by the project either with or without mitigation
• Should those risks be realized, the impact is measured
• Compare the measured impact with the predicted impact
• Action, causal analysis and process improvement
Distribution for Program/Cost
Values in Billions
0.000
0.200
0.400
0.600
0.800
1.000
4.28 4.325 4.37 4.415 4.46
4.34.3
4.28 4.325 4.37 4.415 4.46
70% 30% 4.3651
Actual
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 23
Summary of Metrics• Infrequent
• Measure the quality of the process• Compliance to the organization’s standards• Comparison with Best Practices
• Continuous• Compliance to plan• Performance
• Effectiveness• Efficiency• Staleness
• Trending• On-demand
• Results - for “accepted” risks that are realized, compare actual risk outcomes with predictions.
Risk Management Metrics, Project Risk Symposium 2006; © 2006 Mark A. Powell Slide # 24
INCOSE RMWG Recommendation
• A Proper Combination of Metrics should be Selected• Individual Metrics are Not Sufficient• Should Include Each Frequency Type
• RM Process Performance is Measurable and Hence Manageable
• RM Process Performance Can be Justified