Phil HugginsPrivate Conference Spring 2008

RiskUncertaintyCardinals versus OrdinalsProbability Estimation AccuracySuppressing UncertaintyFoxes and HedgehogsUncertainty Calibration

A state of uncertaintySet of possible ?negative?

outcomes

A set of possibilities with quantified probabilities and consequences

Lack of complete certaintyMore than one possibilityTrue outcome not known

A set of probabilities assigned to a set of possibilities

Cardinal -> Size Cardinal -> A number or %

Ordinal -> Position in Set Ordinal -> High/Medium/Low or 1/2/3/4/5

Ordinals often have different meanings depending on the person

An estimate of realityThe usefulness of an estimate has

two measures

Calibration ▪ How close to reality is the estimate

Discrimination▪ The range of the estimate

We’re the ‘experts’

Client doesn’t want doubt

Client pushes the risk of the decision onto you

Hitting targets for risk reduction focuses on goals not outcomes

‘The fox knows many things but the hedgehog knows one big thing’ – Isiah Berlin

Hedgehogs• Probability is a deductive exercise• Highly confident in their estimates• Intolerant of uncertainty

Foxes • Probability is an ad-hoc estimation• Low confidence in their estimates• Tolerant of uncertainty

Foxes are generally better calibrated and discriminated

Less ‘foxy’ foxes are slightly worse at discrimination

Best overall at making short term predictions in field of expertise

Hedgehogs worst overall when making long term predictions in field of expertise

Hedgehogs win less but win big

90% Confidence Interval (CI) ‘You are 90% certain the probability of

occurrence is between 60% and 50%’ 95% sure below the upper bound 95% sure above the lower bound

Ask yourself 20 general knowledge questions asking for numerical answers Assign lower and upper bounds to your 90% CI of being

correct Ask yourself 20 true/false questions

Set a value to how certain you are 50%, 60%, 70%, 80% 90%, 100%

General Knowledge You expected to get 9 out of 10 right (90% CI) What did you get? You probably got less than that right

True / False Convert % to decimal (50% = 0.5) Add them up Compare to number of questions you got right If number of correct answers was 2.5 or more

less then you are overconfident

Take multiple tests in successionEquivalence Bets (next slide)Pros and Cons

Consider two pros why you should be confident and two cons why you could be wrong

Avoid Anchoring 95% certain below upper bound 95% certain above lower bound

Two choices:1. Win £1000 if answer falls within your upper and

lower bounds.2. Spin a dial split into 90% and 10% slices. You win

£1000 if it finishes on the 90% slice

Most people (80%) prefer to spin the dial Ask yourself which you would prefer

Preferring 1 is over confident Preferring 2 is under confident

Just pretending to bet money makes you better

After 3 tests 70% of people are perfectly calibrated

After 3 tests 20% of people are significantly improved

10% of people never really get it

Testing has shown that most people relied on to make estimates are in the first two groups already

Thanks for listening

http://blog.blackswansecurity.com