Presentation from January 8, 2008 Dinner Meeting

8/9/2019 Presentation from January 8, 2008 Dinner Meeting

1/20

1

Courtney LaneINCOSE WMA Dinner Meeting

January 8, 2008

Its More Than Just Numbers:How to Implement a Qualitative

Risk Analysis Approach

This document is confidential and is intended solely for the use and information of the client to whom it is addressed.


2/20

2

P urpose

Describe the difference between qualitative andquantitative risk analysis

Discuss common problems associated with qualitativeanalysis

Examine best practices for implementing a qualitative riskanalysis approach


3/20

3

S tandard Risk Management P rocess

DevelopStrategy

This presentation will focus on the analysis sub-process. This sub-processshould be defined in an organizations risk management strategy

Analyze

PlanMonitor

andControl

Identify


4/20

4

C omponents of Risk Analysis

Probability of occurrence

Impact to a projects cost, schedule, and technicalobjectives

Timeframe

Handling method

B y identifying these components, project managers will be able to determinethe magnitude of the risk and effectively prioritize handling of the risks


5/20

5

D ifference Between Qualitative and QuantitativeAnalysis

Qualitative vs. Quantitative

U se ranges to selectprobability and impactrankings

Absolute numericalestimates are used for impact and probability

Focuses on overall level of individual risks (e.g. high,medium, low)

Allows projects toaggregate individual risksto determine the overalllevel of project risk

Allows programs to quicklyprioritize risks based on anumber of criteria

U ses complicated analysissuch as Monte Carlosimulation to prioritizerisks


6/20

6

W hy use Qualitative Analysis over Quantitative?

Lack of historical data

Lack of the necessary time and resources to conductcomplicated analysis

Desire to use criteria for prioritization that is difficult toquantify

Timeframe

Critical dependencies Political considerations

The type of analysis appropriate for an organization is dependent upon therisk management maturity level of the organization


7/20

7

Risk Management Maturity ModelLevel C haracteristics1: Chaotic isks are ignored or managed in an ad hoc fashion

2: Reactive Processes definedManagement uncomfortable discussing riskMost efforts spent on resolving issues

3 : Proactive Processes standardized across the organizationManagement comfortable with risk managementRisk management data used in decision making processes

4 : Predictive U se of quantitative analysis to predict future resultsCost/benefit analysis implemented

5 : Optimization Opportunities to exceed project objectives are identified andmanagedS ource: Modified from SEI s Capability Maturity Model

M ost organizations are not at an appropriate maturity level toimplement quantitative risk analysis


8/20

8

Top P roblems Related to Qualitative Analysis

#1U sing inappropriate criteria to determineprobability and consequence levels

#2 Trying to quantify qualitative data

#3Relying on risk scores generated by riskmanagement tools to determine a list of toprisks


9/20

9

P roblem #1: Inappropriate C riteria

Criteria is often too detailed

Subject matter experts lack the necessary expertise toappropriately select the criteria

Easy to game the analysis since justification of criteriaselection is extremely subjective

P oor criteria definitions make it difficult for risk analysts toaccurately reflect impacts to cost, schedule, and performance


10/20

10

S olution: Understand Your Audience

Determine what questions decision makers want the riskmanagement process to answer

Assess the level of expertise of those performing the riskassessments

Select criteria that balances the needs and abilities of both parties


11/20

11

Ex amples of Qualitative Analysis C riteriaNegligible

(A)Marginal

(B)

S ignificant(C )

C ritical(D )

C atastrophic(E )

C ost Minimal or noimpact

$2 5K- $5 0K $5 0K- 100 K $100 K- 25 0K > $25 0K

S chedule Schedule slip < 7 days

Schedule slipbetween 7 and 1 4 days

Schedule slipbetween 1 4 and21 days

Schedule slipbetween 21 and3 0 days

Schedule slipgreater than 1month

P erformance Minimal or noimpact

No impact to keyperformanceparameter;

degradedcapabilityperformance

Degradedenterprisecapability

performance

Degradedcapabilityperformance with

impacts to onekey performanceparameter

Degradedcapabilityperformance with

impacts to morethan one keyperformanceparameter

C riteria is too detailed and thus difficult to assess


12/20

12


(A)Marginal

(B)S ignificant

(C )C ritical

(D )C atastrophic

(E )


10% of overallbudget

S chedule Minimal or noimpact

Additionalresourcesrequired; able tomeet need dates

Minor slip in keymilestones; notable to meet needdates

Major slip in keymilestone or critical pathimpacted

Cant achieve keyteam or major project milestone


Acceptable withsome reduction inmargin

Acceptable withsignificantreduction inmargin

Acceptable; noremaining margin

U nacceptable

S ource: Risk Management Guide for DoD Acquisition

C riteria is extremely subjective and could be easily gamed


13/20

13


(A)Marginal

(B)

S ignificant(C )

C ritical(D )

C atastrophic(E )


Cost impactscontained withinthe project

Cost impactscontained withinthe program

Cost impactscontained withinthe organization

Cost impactsaffect externalpartners andcould jeopardizeprogram

S chedule Minimal or noimpact

Impacts to criticalpath containedwithin the project

Impacts to criticalpath containedwithin the program

Impacts to criticalpath of more thanone program inthe organization

Impacts thecritical path of external partnersand could

jeopardizeprogram


Impacts to criticalfunctions of the

project


program


organization

Impacts to criticalfunctions of

external partnersand could jeopardize theprogram

C riteria definitions are easy to assess and impact levels areconsistent across criteria


14/20

14

P roblem #2: Quantifying Qualitative D ata

Engineers often feel more comfortable using numberswhen performing analysis

Assign numbers to levels of impact U se specific probabilities instead of ranges

Risk scores often determined by multiplying impact by probability

However, most risk analysis uses uncalibrated ordinalscales

Cannot perform math on uncalibrated scales Gives a false sense of rigor Decisions made on inaccurate data

P erforming math on qualitative data may lead to inaccurateconclusions and a false sense of rigor


15/20

15

S olution: D o Not Include Numbers in Analysis

Avoid assigning numbers to levels of impact andprobability

U se probability ranges U se letters to represent levels of impact and probability

U se the highest level of individual impacts to determineoverall impact (dont average impacts)

U se a risk matrix to determine overall level of risk


16/20

16

Ex ample Qualitative Risk Analysis

O verall level of riskis Red


17/20

17

P roblem #3: O veremphasizing Risk S cores

If problem #2 is not overcome, this becomes a muchlarger problem

Even if numerical scores are not used, rankings could beincorrect

Risk tools can generate a list of top risks based on locations on therisk matrix

Cannot rely solely on the placement on the risk matrix

Dependencies, political issues and timeliness factors also play apart

O veremphasis of risk scores to determine top risks resultsmay result in poor decisions made by senior leaders


18/20

18

S olution:

Tailor list of top risks based on the audience

Consider all factors, not just likelihood and consequence,when creating the list

Do not use arbitrary thresholds to determine handlingstrategies

Aim to manage all yellow and red risks


19/20

19

S ummary and C onclusion

Select the appropriate type of analysis based on the riskmanagement maturity level of the organization

Typical challenges associated with qualitative analysiscan be overcome:

K now your audiences Avoid using numbers to perform analyses Consider other factors when selecting list of top risks

Q ualitative analysis can provide leaders with enoughinformation to make sound decisions if implemented properly


20/20

20

C ontact Information

Courtney LaneSenior Consultant

Booz Allen [email protected]

7 03- 96 1-800 9

Presentation from January 8, 2008 Dinner Meeting

Documents

Transcript of Presentation from January 8, 2008 Dinner Meeting