Measuring RiskWhat Doesn’t Work and What

Does

aliadocorp.com

222

Our Company Aliado – Name’s Root is “Ally;” How We Value Our

Relationships with our Customers

Leading the Way Since 2008

Professional Service Company /Management Consultancy

Core Aliado Leaders have over 20 Years Professional Service Experience from Big 5

Our Company

Consultants servicing our technical expertise: Security Infrastructure Risk Management

Our most important Assets - People Integrity, Principles, Values Reputation, Reputation, Reputation Brand

Trusted Advisor

Professional Services – Not Products

The Problem If your risk analysis and risk management don’t work, then that

is your biggest risk.

Risk management (or, for that matter, any management methodology) itself rarely has performance metrics applied to it. The most popular methods have no published controlled experiments where the improvement on forecasts and decisions was actually measured (although anecdotal case studies are common).

Mis - Communication – IT Security and Business Managers have no common language to discuss risk. IT Security talks in terms of high, medium, low or ordinal scales and Business Managers talks in terms of quantitative numbers.

5

Issues with a “Risk Map” Does your “Risk Map” look more like the top

or bottom chart? If more like the top, how do the errors mentioned earlier compare to the variance among the clustered responses?

Clustering means that all the previous errors mentioned before make up a large part of the difference between scores of individual risks.

How does this address correlations, common mode failures, and cascade failures? These factors can make a few “low risk” items add up to one very big risk.

The “math” in these methods don’t even remotely approximate the relationships one might build in a quantitative model.

Risk maps like this may be ok for initial brainstorming, but don’t make critical decisions based on it.

Like

lihoo

d

Impact

Extreme

◦ Extreme likelihood of security controls being compromised with the possibility of catastrophic financial losses occurring as a result.

An asset with a vulnerability that was demonstrated to be exploitable and subsequently led to the compromise of sensitive

information would be designated with this rating.

High

◦ High likelihood of security controls being compromised with the potential for significant financial losses occurring as a result. This

rating would be given if a vulnerability was found to be exploitable and potentially affect the confidentiality, availability, and/or

integrity of a given asset.

Elevated

◦ Elevated likelihood of security controls being compromised with the potential for material financial losses occurring as a result.

Assets with a finding that led to information disclosure, for example, but not necessarily a full compromise, would be assigned this

rating.

Moderate

◦ Moderate likelihood of security controls being compromised with the possibility of limited financial losses occurring as a result. A

system with a vulnerability whose impact was reduced by factors such as configuration settings or difficulty of exploitation would

be assigned this rating.

Low

◦ Low likelihood of security controls being compromised with negligible impact as a result. This rating signifies either the non-

existence of vulnerabilities or those that have minimal impact.

Information Security Threat Rating Scale – Qualitative Information

Errors in Expert Judgment▶ Human expertise is an important input in and it is

hard to completely automate. But there are certain types of errors in human judgment we know how to measure and control for:

• Overconfidence – Their chance of being right is much less than they believe• Influence by irrelevant factors – Factors like the order in which you consider

projects, whether it is a 5-point scale or a 10-point scale, or how much other people in the room smile all affect your judgments

• Inconsistency – When given the same sets of problems to evaluate, experts have a hard time giving the same answers; also, their memory is reconstructed so that they believe they always had one preference when in fact they didn’t

• Misinterpretation – We tend to interpret cues about risks, measurements and decision problems in a way that is logically and mathematically irrational

7

Sure, You Feel Better, But…Studies have shown that it is very easy for a decision-making process to increase confidence in forecasts and decisions even if measured outcomes (return on decisions, forecasts, etc.) are not improved – or even made worse

Gathering more information makes you feel better but, at some point, begins to reduce decision quality while confidence continues to increase. (Tsai C. 2008)

Interaction with others also increases decision confidence but, again, at some point decisions are not improved while confidence continues to increase (Heath C., Gonzalez R. 1995)

Formal training in detecting lies makes individuals slightly worse at detecting lies in controlled experiments – but there confidence in their judgments increases dramatically. (Kassin, S.M., Fong, C.T. 1999)

An experiment with AHP shows confidence increased whether decisions are improved or degraded. (Williams M. et. al. 2007)

Almost all popular business methodologies show no correlation to financial performance of the firm (N. Nohria et. al. 2003)

Scales are simple. But our response behaviors when we use them are not. Typical scales combine several complex, subtle errors

The use of scales simply obscures (doesn’t alleviate) the lack of information and potential disagreements - he calls this an “illusion of communication” (Budescu)

Popular weighted scores add error to unaided human judgment. Scale error is added even if scales are “well defined” by introducing an extreme rounding error. It is possible to have one risk 10 or 50 times greater than another risk end up in the same final group. (Cox)

“Partition dependence” creates an unanticipated relationship among choices on a scale. Two scales that each define a “1” in the same way (e.g. 1=“impact less than $1M), will elicit different responses for a 1 depending on how many other choices there are.

Treating ordinal scales like linear scales that can be added and multiplied introduces an error of “assumed ratios”. They assume relative values of the scales roughly approximates real world relationships when an analysis of historical data shows they do not.

Scale Errors

First, Do No Harm

Method Gut Feel

Weighted Score Preference Theory Models

Quant. Models

Measured Improvement to Judgment?

Baseline No: Remove no errors and add new errors

No: AHP has known math problems; might improve consistency

Yes: Proven w/controlled tests

Does it quantify risk?

Only intuitively

No, it attempts to describe risk;

No, but it can quantify risk aversion

Yes

Determines High-Payoff Measures?

No No: Turns some good measures into scores

No Yes (w/AIE)

Net Reduction in Error?

Baseline No: Probably Worse Maybe Slightly Better – Maybe not

Best

“Gut Feel” is the baseline. Anything that “works” has to show an improvement on this. Measured sources of error : inconsistency, overconfidence, various biases, inaccurate estimates

The worst case is not “gut feel” – some methods add more error The best case isn’t perfection – just measurably reduced error compared to gut feel

Aliado provides a Methodology--Applied Information Economics—that IT Security and IT Business Leaders can understand◦ A statistical and probability application that allows

an organization to measure their risk accurately on an ongoing basis that provides tangible results in quantifying the risk on any risk landscape component.

The Solution

What Does Work?

“Calibrate” experts to realistically assess probabilities. “Do the math”– don’t rely intuition entirely.

◦ Use the “calibrated” judgments of experts in Monte Carlo simulations.◦ Simple historical models usually outperform human judges.◦ Compute the “Expected Value of Information” to identify important

measures. Improve unaided human judgment with statistical

“smoothing”. Try rational incentives to encourage better expert

judgment. Document basic decision criteria - especially risk vs. return.

Model The Current State of Uncertainty – Initially use calibrated

estimates and then actual measurements

Applied Information Economics

Optimize Decision – Use the quantified Risk/Return boundary of the Decision makers

to determine which decision is preferred.

Define the Decision and Identify Relevant Variables. Set up the

“Business Case” for the decision, using these variables –

Calib

rati

on T

rain

ing

Compute the value of additional Information – Determine what to

measure and how much effort to spend on measuring it.

Measure where the information value is high – Reduce uncertainty

using any of the methods

No

Yes

Is there significant value to more information?

Prob

abili

ty o

f a

nega

tive

RO

I

Return

0%

10%

20%

30%

40%

50%

0% 50% 100% 150% 200%

Event A

Event B

Monte Carlo: Yes, it Works

Demand

%Orders LostOR

Lost Revenue

• Performance metrics for decision analysis tools is very sparse, but favors Monte Carlos.

• One researcher in the oil industry found a correlation between the use of quantitative risk analysis methods and financial performance – and the improvement in performance started when they started using the quantitative methods. (F. Macmillan, 2000)

• Data at NASA from over 100 space missions showed that Monte Carlo simulations beat other methods for estimating cost, schedule and risks (Published this in The Failure of Risk Management and OR/MS Today)

The Value of Information

*),|(),...|(),|(max)(1 1 1

,,21

,1 EVrpVrpVrpVrpEVIk

i

z

j

z

jijjlijj

z

jijji

The formula for the value of information has been around for almost 60 years. It is widely used in many parts of industry and government as part of the “decision analysis” methods – but still mostly unheard of in the parts of business where it might do the most good.

What it means:

1. Information reduces uncertainty2. Reduced uncertainty improves decisions3. Improved decisions have observable consequences with measurable value

The Impact of Computing Information Value

Traditional M

easurement P

riorities

Val

ue o

f In

form

atio

n

The Priority of Measurements is Reversed: This calculation reveals that most organizations will consistently focus on low-value measurements and ignore high-value measurements - this is the “measurement inversion”

Only a Few Measurements Are Really Needed: We also found that, if anything, fewer measurements were required after the information values were known.

Some Additional Empirical Measurements are almost always needed: I found that 97% of the models I built justified further measurement according to the information values.

Prob

abil

ity

of a

ne

gati

ve R

OI

Return

0%

10%

20%

30%

40%

50%

0% 50% 100% 150% 200%

Documenting Risk Aversion

Acceptable Risk/Return Boundary

Investment Region

• Our risk tolerance changes much more frequently than we are aware and for arbitrary reasons. One study showed that being around people who smile make us more likely to take risky bets. Others show that simply remembering past events that made us angry make us more risk tolerant while recalling events where we were afraid made us more risk averse.

• The simplest element of Harry Markowitz’s method “Modern Portfolio Theory” is documenting how much risk an investor accepts for a given return. Documenting our appetite for risk makes it less vulnerable to capricious change.

• The “Investment Boundary” states how much risk an investor is willing to accept for a given return. For our purposes, we modified Markowitz’s approach a bit.

Investment

Objections I Have Heard• Some managers have told me they wish they could quantify the risks of

their decisions more rigorously• But they cite various reasons why they believe they can’t:

– Concepts like “risk” (as well as “quality”, “flexibility”, etc.) are fundamentally immeasurable

– They can better evaluate an investment “by experience” i.e. in relation to other investments they’ve seen

– Some are skeptical about statistics (“you can prove anything with statistics”)

– Any approach that involves statistics will seem too “theoretical” to top management

– “We don’t have enough data”– “We can’t compute a precise probability”

• Each of these are refuted by the evidence

Copyright HDR 2008 dwhubbard@hubbardresearch.com

Business Case:◦ Retail Firm fined 2.1Billion by FTC for failing Audit◦ FTC required Retail Firm to have a independent Consulting Firm to provide Risk

Assessment◦ CSO had a Big 4 Firm as their Auditor and was not confident with them from the

fine from the FTC◦ CSO heard of our Quantitative Risk Assessments and understood the value of a

tangible assessment that would provide him a real value of the risks they had and could be accepted by the FTC.

Scope:◦ Provide Risk Assessment across 4 Major lines of Business◦ The elements in scope across all their lines of business were: Customer Applications Prescription Applications Data Warehouses Mobile

Data Mainframe Windows Servers Financial Applications iSeries SystemsNetwork Devices Internet Applications UNIX Backup Tapes Proprietary Data Applications Stratus Databases Personal SystemsHR Applications Wireless Remote Access Back Office Applications Test SystemsPrint Operations LDAP Systems Credential

◦ Threat communities – Cyber Criminals, Privileged Insiders, Non – Privileged Insiders, Malware

Case Study – Retail Firm

◦ Provide the Information risk for each line of business and the overall aggregate risk for the Retail Firm. Our client was able to see that their overall aggregate risk was in line with their expectations

that they were willing to accept for their business.

◦ Identify Risk by Threat Community Cyber Criminals by far was the biggest risk to our client. Privileged Insider was second most significant with Non – Privileged Insider than Malware.

◦ Loss Event Type Confidentiality consisted most of loss at 290M due to sensitive customer info Availability was distant second at $530K Integrity was last at $2100

◦ Risk by Asset Group Personal Systems had the biggest exposure accounting for 48% of risk exposure Internet Applications at 16% Windows Servers (Internet facing) at 11%

◦ Best opportunities for risk reduction Access Privileges – 33M Personal System Security – 34M Mobile Media Restrictions – 20M

Case Study – Retail Firm

Business Case ◦ Federal Executive Agencies face significant management and technical

challenges when measuring the contribution of IT investments to mission results as required by the Clinger-Cohen Act.

◦ VA Information Security Program had an approved new infrastructure initiative that will mitigate IT Security – related risk across the department. The risks include reducing the cost and frequency of virus, unauthorized access, fraud and other type of losses.

◦ VA wanted to have a methodology that could meet the Clinger Scope

◦ Provide the best rollout strategy for VA Public Key Infrastructure investment that will optimize the value of PKI

◦ Combination of its optional investments will reduce the greatest losses at a reasonable cost.

◦ Determine the Effectiveness of the Information Security Program over time

Case Study – Infrastructure IT Security Investment – Veterans Affair

Rollout of VAPKI should occur in a particular order and should be implemented when a certain criterion is met. That criterion is 1.1% is the expected reduction in fraud cost per person where VAPKI is implemented. In other words, if the annual fraud cost per person were $500 then the VAPKI cost per person must be less than $5.50 per person to justify rolling it out to that facility.

VA accelerated the anti-virus roll out by six months The Information Security Program should reduce by 75% to 95%

the expected losses for all security incidents through 2006 estimated somewhere between $1.1 billion and $2.4 billion.

One major optional investment (certain parts of Intrusion Detection) did not reduce losses and therefore should not be

made. This is about a $30 million cost avoidance.

Case Study – Infrastructure IT Security Investment Results– Veterans Affair

AIE determined that VA would make the best investment decisions by taken seven key measurements. Those measurements will allow VA to determine the effectiveness of the Information Security Program over time.

◦ Annual Fraud losses due to internal unauthorized access - $80M to $180M

◦ Number of Pandemic Virus attacks per year – 2 to 4◦ Average number of people affected by a virus – 25K to 60K◦ Percentage productivity loss due to virus outbreak – 15% to 60%◦ Percentage of veterans affected – 2% to 15%◦ VAPKI initial cost of VA wide roll out - $1.3M to $2M◦ VAPKI annual cost of VA wide roll out - $1.1M to $1.3M

Case Study – Infrastructure IT Security Investment Results– Veterans Affair

Final Tips Regardless of how quantitative your environment

might be, be a skeptic about how your organization assesses decisions and risks – ask how they know it works (and consider the consequences if it doesn’t)

Your judgment also has a performance that can – and must – be measured

Considered against the size and risk of decisions, better risk analysis will be one of the best investments in your portfolio

Questions?Contact:

Karen AldridgeRegional Saleskaldridge@aliadocorp.comwww.aliadocorp.com

Information Security Threat Rating Scale