20101116 Risk Matrix Hkieb Print

7/27/2019 20101116 Risk Matrix Hkieb Print

1/52

The Risk of Using Risk Matrix in

Assessing Safety Risk

Joint Seminar of HKARMS, HKIE-MMNC,CILTHK, and IMechE

16 November 2010

Vincent Ho

HAKRMSSafety Engineering Lecture Series 2.2


2/52


3/52

HKARMS

Skepticism

In defense of using some popular methods for safety and decisionanalysis, you may have heard (or said) the following:

ur me o s s rucure an orma

It helps us build consensus

It can be done quickly and within budget. Its easily understood by senior management

Its a proven method (proven meaning somebody else did it

This is the best (or only practical) tool we have used

If someone can be an expert of a risk analysis method after a one-

33

33

,


4/52

HKARMS

Takeaways

Risk management methods vary widely among industries but the mostpopular risk assessment methods are/may be the least effective

There is a strong placebo effect in analysis - even a completelyineffective method would feel like it worked, particular when it is easyto maste

Even in organizations with extensive performance metrics, one of themost important measures is almost always ignored the effectiveness

We will not complete a risk assessment of using risk matrix

44

ong u ope s a w s mua e your n ng n e

effectiveness of using risk matrix and the associated risks


5/52

HKARMS

Topics to be Covered

Understanding Risk

Background of the Risk Matrix Application

Issues in Using Risk Matrices

55Power Point will be available at www.hkarms.org


6/52


7/52

HKARMS Understanding Risk

Background of the Risk MatrixApplication

Types of Risk Matrices


Definitions of Risk

SafeguardsHazardRisk=

Risk is never zero by increasing safeguards, as long as hazard is present

Conceptually good but difficult to use in assessing risk

eConsequencxikelihoodLRisk=

Classical, most popular but most misleading More useful in hazard analyses

77


8/52


9/52

HKARMSHKARMS Understanding Risk




Definitions of Risk

eConsequencyUncertaintRisk =

potential situation or event

Without uncertainty or damage, there is no risk n or er o ave a rs s ua on, o eemens mus e presen

Anybody can guess extent of damage/Consequence but withdifferent levels of uncertainties

99 This definition has been my favorite


10/52





Definitions of Risk

From ISO 31000:2009Risk Management Principles and Guidelineson Implementation; ISO 73: Risk Management - Vocabulary

Published as a standard on the 13th of November 2009 for the

implementation of risk managements s e ne as e e ec o uncertaintyon o ec ves

to be applicable and adaptable for "any public, private orcommunity enterprise, association, group or individual."

1010

How often does your risk management systemmention the word uncertainty?


11/52





Sources of Uncertainties

Stochastic uncertainties, parameter uncertainties, modelinguncertainties

No access to the whole truth (e.g., failure rates, consequence)

Impossible to explicitly specify all conditions

Inade uate or incorrect information on conditions

Inconsistent interpretation and classification of events

Lack of success data (for number of demands and

Limited data sample size; realised risk and unrealised risk

Imperfect mathematical and computer modelling of reality

1111

In probabilistic (or quantitative) risk assessments,uncertainty is measured by level of belief; i.e., probability


12/52





Quantitative Definition of Risk

In general, risk is used to answer: What can go wrong?

a are e amage e ecs How likely is it that this will happen?

What are the uncertainties? Thus, risk can be thought to be consisting

of four elements: Scenarios or accident sequences

Safety System 3

Consequence Likelihood Uncertainties

Bad Thing

1212

Safety System 2


13/52





Use of Probabilities in Risk Assessments

In a QRA, we know what theConsequences (damage effects)IE Safety Sys1 Sys 2 Sys 3

BAD THING

.01/YR

SAFE

SAFE.1

we want to know the Likelihood of

these contributing factors

UNSAFE STATE

Sys 1

SAFE

FAIL

..5

.1

,

accident sequences using event

tress and fault trees, then apply

Failure

Valve OperatorFailure

Pump

0.5

SELS2.3

reaching end states by each

accident sequence Test &Maintenance

Failure toStart or STBYFailure Rate

Failureto Run

0.1 0.10.3

1313

0.10.10.1

Total Risk is the sum of all paths leading to Unsafe State


14/52





Quantitative Definition of Risk

Scenario Likelihood Consequence

s1

s2

L1

L2

C1

C2

s3

L3

C3

s

L

C

Risk = {}

For each Si, Riski = Li x Ci

1414

Total risk of the system is R =i Li x Ci

L is expressed by probability of frequency in handling uncertainties


15/52

Background of the Risk Matrix

Application


16/52





The Beginning of The Risk Matrix Era

The arm race in the 1950s and 1960sgenerated a large number of systems that

operate

A System Safety Program grew out of the

improve safety and system survivability

This proactive system-level approach

replaced the reactive, fly-fix-fly approach

1962: System Safety Engineering for the Development of Air Force

1616

Ballistic Missiles

1969: MIL-STD-882A, System Safety Program Requirements (882Dis now being revised)


17/52

HKARMSHKARMS

Understanding Risk




- -System Safety Program Requirements

To achieve acceptable risk through a systematic approach ofhazard analyses, risk assessments, and risk management

The Mid-Std-882 series have introduced

RAMS criteria for system design hazard analysis tools

Hazard logging system

Documentation to satisfy approval authority

1717


18/52





Hazard Evaluation

The complexity of ahazardanalysis depends on the scope,application and industry

MIL-STD-882 suggests the use of worksheets with look up table or riskmatrices to characterise the risk impact of hazards in terms of the

likelihoodand consequence mainly as preliminary screening analysis The application of the worksheet/ risk matrix approach to evaluate

hazards has since become very popular in almost everywhereincludin safet anal sis terrorism risk anal sis roect risk

management, traffic safety, climate impact, ERM, etc.

1818


19/52





Worksheet Method

For qualitative screening purposes or rank-ordering of hazard scenarios

Each row is one hazard scenario (almost an accident sequence!) that givesone unique set of likelihood/ consequence /risk, which are then expressed as

ns or casses

Information contained must be adequate and concise different analysts

should be able to arrive the same set of likelihood/consequence/risk classes,,

1919 Strictly speaking, a worksheet type analysis isa Hazard Analysis, not a Risk Analysis


20/52

HKARMS

Understanding Risk




, ,Consequence

Contract No:

System:

Subsystem:Hazard Analysis Work Sheet

Prepared by: Date:

Reviewed by: Date:

Authorised by: Date:

Ref

No.

Hazard Scenario

Description/

Consequence

Op.

Mode

Existing

Safeguard/

Control Measure

Risk Impact Proposed Mitigation

Measures/Control

Residual

Impact Comment/

ResolutionStatus Responsibility

Days

Remained

OpenC L R U C L R U

LikelihoodConsequence

RISK ClassRISK Class

2020 Need look up tables to quickly look up the relationship


21/52


22/52





Risk Matrix Defines Your Risk Appetite

H

MH

ih

ood

MLLikel

L

2222ImpactConsequence


23/52

HKARMS

Large Appetite for Risk Plan for All Extreme Risks

Understanding Risk




quence

quence

asingC

ons

asingC

ons

High Risk

Increasing Likelihood Increasing LikelihoodIncr

Incr e um

Lows verse

ence

Negligibleence

ing

Conse

q

ing

Conse

q

2323 Increasing Likelihood Increasing LikelihoodIncrea

Increa


24/52





Application in Risk Control

nce

se

que

gCon

Hazard Scenario

reasin

2424 Increasing LikelihoodIn

c


25/52





Demonstrate Reduction in Risk Ranking

Original RiskPossible Residual Risk

nce

Risk Controlse

que

r n c p e sRisk EliminationRisk Avoidanceg

Con

Risk TransferRisk Reduction r

easin

2525 25

Increasing LikelihoodIn

c


26/52





Pattern of Your Risk Profile

Does your Risk Map look more like oneof these charts?

Clustering or equally-spread risk

mapping means that the risk matrixe

Risk profile changes as safety and riskmanagement program mature, why noton

sequen

your risk matrix?

2626 Likelihood

updated its risk matrix?


27/52


28/52


29/52





MIL-STD-882 Mishap Severity Categories

2929

U d t di Ri k


30/52





MIL-STD-882 Mishap Mishap Probability Levels

3030

UnderstandingRisk


31/52

HKARMS

Understanding Risk




- -Values

-Based

3131

UnderstandingRisk


32/52





Typical Risk Matrix

ConsequenceLikelihood

Insignificant1

Minor2

Moderate3

Major4

Catastrophic5

Almost Certain A S

Likely BM S S H H

Moderate CL M S H H

Unlikely DL L M S H

= -

Rare E L L M S S

3232

S = Significant risk - senior management attention needed

M = Moderate risk - management responsibility must be specifiedL = Low risk - manage by routine procedures

Action-Based

UnderstandingRisk


33/52





Typical Risk Matrix

Priority-Based

3333

UnderstandingRisk


34/52

HKARMSHKARMS

Consequences

Understanding Risk




Likelihood Severe

(1)

Major

(2)

Medium

(3)

Minor

(4)

Negligibl

e

(5)

certain

(A)

Likely

(B)H H M M L

Possible

(C)H M M L L

Unlikely M M L L T

Rare

(E)M L L T T

E Extreme risk Immediate action required; this level of risk needs detailed research and

planning by senior management.H High risk Action plan is required as soon as practicable by senior management.

M Moderate risk Action plan is required by Area/Department Manager within reasonable timeHybrid

3434

L Low risk Managed by routine procedures and employees under supervision.

T Trivial risk Unlikely to need specific application of resources.

THERE IS NO STANDARD RISK MATRIX

Understanding Risk


35/52

HKARMSHKARMSg




Risk Matrix using Scores

3535

This type of scoring matrix allows adding up

of hazard risks


36/52


(What Can Go Wrong?)

Understanding Risk


37/52

HKARMS




Quantitative Input

Conse uenceEqv Fatality 0 0.005 0.07 1 15

C1 C2 C3 C4 C5

Freq 0.00 0.02 0.26 3.87 38.73F1 31.62 0.05 0.59 8.37 122.38 1224.75

F2 3.16 5.00E-03 0.06 0.84 12.24 122.48

F3 0.32 5.00E-04 5.92E-03 0.08 1.22 12.25lihood

. - . - . - . - . .

F5 3.16E-03 5.00E-06 5.92E-05 8.37E-04 0.012 0.12F6 3.16E-04 5.00E-07 5.92E-06 8.37E-05 1.22E-03 0.012

F7 0.00 5.00E-08 5.92E-07 8.37E-06 1.22E-04 1.22E-03

Like

3737

Is the design of your risk matrix based on

Risk Acceptability Limits?



38/52

HKARMSHKARMS Background of the Risk MatrixApplication



Issues in Designing Risk Matrix

Infer Risk and

Value Equivalence

3838



39/52

HKARMS Background of the Risk MatrixApplication



MIL-STD-882 Mishap Severity Categories

Introducing value of l ife,

or cost of preventing

, .

3939


B k d f th Ri kM t i


40/52





Level Descriptor Description

1 Insignificant Superficial injury/illness, no treatment or first aidonly, low financial loss (less than $5k), requires no

.

2 Minor Medically treated injury/illness, medium financial loss($5k to $50k). Short term environmental damage &

minor remediation.3 Moderate LTI, no permanent impairment, 20 shi fts lost, major financial loss ($100k -$500k),Long term environmental impact & majorremediation.

5 Catastro hic Fatalit toxic release off -site with detrimental effectAre these values scalable?

4040

huge financial loss (More than $500k), long termenvironmental damage & major remediation

For OSH use, not suitable for systems involving mass public


Backgroundof theRiskMatrix


41/52

HKARMS Background of the Risk MatrixApplication




4141 Some typical mistakes




42/52





Are these values scalable?

4242 BE CAREFUL WHEN DESIGNING A RISK MATRIX




43/52




(Bad) Example of Using Risk Matrix

Hazard Consequence Prob Severity Risk

Class

Pump Room fire Both pumps fail Med High A

Severity

Probability Low Med Hi h

Low D C B

Medium C B A

High B A A

4343

Pump fire is medium probability in this facility

Losing both redundant pumps will lead to plant damage




44/52




(Bad) Example of Using Risk Matrix

Med Low C

Hazard Consequence Prob Severity Risk

Class

Pump A on fire Pump A damaged

Severity

Probability Low Med HighLow D C B

Medium C B A

-

High B A A

4444

(scenarios or rows in worksheet) with a lower risk for each sub-item

MIS-USE CAN CREATE FALSE SENSE OF SAFETY

HKARMSHKARMS


45/52

HKARMSHKARMS

Advantages of Worksheet/Risk Matrix

Hmmm, this is a RiskClass B hazard. RiskAnalysis is so easy!!!

Everybody has done at least one

, Detailed analyses not required Good for compliance check and ensure consideration of

Useful in evaluating a large number of alternatives withobvious differential risks

4545

HKARMS


46/52

Disadvantages of Worksheet/Risk Matrix

Results can be inconsistent between users Difficult to verify assumptions and results

Difficult to identify common mode failures,system interactions, cascaded failures,

complex situation, etc. Cannot compare alternatives in same risk

class

Cannot yield the total risk of a hazard, let

alone for a system Can easily become paper safety and give a false sense of safety/security

4646

DO NOT TREAT WORKSHEET/RISK MATRIX ANALYSIS AS

THE END GAME

HKARMSHKARMS


47/52

Management Process

What can go wrong? (Issues, how, ) What are the damage effects?

What are the uncertainties?

4747


Background of the Risk MatrixA li i


48/52

Application



What's Wrong with Risk Matrices?

Source: What's Wrong with Risk Matrices? T. Cox, Risk Analysis Vol 28, suggests the

following problems making risk matrix unsuitable to correctly assess risks: Poor Resolution. Typical risk matrices can correctly and unambiguously compare only a

small fraction (e.g., less than 10%) of randomly selected pairs of hazards. They can assignidentical ratings to quantitatively very different risks (range compression)

Errors. Risk matrices can mistakenly assign higher/lower qualitative ratings toquan a vey sma er arger rs s. or r s s w nega vey correae requences anseverities, they can be worse than useless, leading to worse-than-random decisions

Suboptimal Resource Allocation. Effective allocation of resources to risk-reducing

Ambiguous Inputs and Outputs. Categorizations of severity cannot be madeobjectively for uncertain consequences. Inputs to risk matrices (e.g., frequency and severitycategorizations) and resulting outputs (i.e., risk ratings) require subjective interpretation, and

4848

different users may obtain opposite ratings of the same quantitative risks. These limitationssuggest that risk matrices should be used with caution, and only with careful explanations of

embedded judgments. Lock-on effect


Background of the Risk MatrixA li ti


49/52

Application



Evidence of Effective Risk Management

Source: The Failure of Risk Management: Why It's Broken and How to Fix It

Douglas Hubbard suggests :

an empirical science it arises from experience

Employing quantitative modelling techniques to model risks

quantifying risks

Models should be built iteratively, testing each assumption against observation

o yng or rs managemen o e gven approprae vs y n organsa ons

Creating an organisation-wide approach to managing risks. This ensures thatorganisations will tackle the most important risks first, and that its risk

4949

managemen u ge w e spen n e mos e ec ve way




50/52

Application



Evidence of Effective Risk Management

a s cs ase on arge sampes e use o s epen s on e ava a y o sorca orother data that is similar to the situation at hand

Direct evidence this is where the risk management technique actually finds some problem

Component testing even if one isnt able to test the method end-to-end, it may be possible totest specific components that make up the method. For example, it may be possible to validate

the risk matrix with known accidents or situations Check of completeness organisations need to ensure that their risk management methods

cover the entire spectrum of risks, else theres a danger that mitigating one risk may increasethe probability of another

nerna compeeness covern a par s o e or ansa on

External completeness covering all external entities that the organisation interacts with

Historical completeness this involves covering worst case scenarios and historical data

5050

occur together; those that may lead to common-mode failure discussed earlier.


51/52

Whether risk matrix is friend or a foe depends on.

it well or you might be better off without it

51

End




52/52

Application



Scoring Behavior & Error

Source: Problems with scoring methods and ordinal scales in risk assessment

Evan Hubbard suggests : Popular weighted scores add error to unaided human judgment - even if scales

are we e ne - y n ro ucng an ex reme roun ng error . ox

The use of scales simply obscures (doesnt alleviate) the lack of information andpotential disagreements - it creates an illusion of communication (D. Budescu)

scale. Two scales that each define a 1 in the same way (e.g. 1=impact lessthan $1M), will elicit different responses for a 1 depending on how many otherchoices there are (C. Fox)

The anchoring effect means even the random order of assessments has an effecton judgments

5252

Scoring methods are usually simple, but

our behavior in using them is not

20101116 Risk Matrix Hkieb Print

Documents

Transcript of 20101116 Risk Matrix Hkieb Print