Evaluation and expression of variability and uncertainty

Andy HartThe Food and Environment Research Agency

andy.hart@fera.gsi.gov.uk

Outline

• What are variability and uncertainty and why

do they matter?

• Evaluating variability and uncertainty

• Expressing variability and uncertainty

• (Dealing with uncertainty in decision-making)

What are variability & uncertainty?

• Variability = heterogeneity in the state of the world

– variability of adverse effects

• e.g. magnitude/severity, incidence, duration, recovery, mortality

– variability in the factors influencing the effects

• e.g. exposure, sensitivity, etc.

• Uncertainty = limitations in knowledge

– uncertainty about factors influencing effects

– uncertainty about how those factors combine

• Uncertainty may be reduced by additional information

Some common types of uncertainty

• Measurement uncertainties– Variation and bias

• Sampling uncertainty– Due to limited numbers of measurements

• Extrapolation uncertainty– When estimating model inputs from partially-relevant data

• Model structure uncertainties– Which factors to include, how to combine them

• Disagreement between experts

• Ambiguity in definition of terms and concepts

• Ignorance (the “unknown unknowns”…)

Why do variability and uncertainty matter?

• Because they are important for decision-making:

• Ignoring variability and uncertainty may lead to sub-optimal decisions

Benefits

Risks

Net risk/benefit

Benefits

Risks

Net risk/benefit

Why do variability and uncertainty matter?

• Public trust in decision-making

Trust requires openness…

…openness requires recognition of uncertainty

BSE – Phillips Report

What do risk managers need to know?

• What are the possible outcomes?

• How likely are they?

• In a risk assessment, this might be:

– How often do people exceed a safe dose?

– How much do they exceed it by?

– How many people will show adverse effects?

– How certain are our estimates of this?

Best estimates are not enough!

1 2 3 4 5 6 7 8 9

• What other outcomes

are possible ?

• How likely are they ?

Understanding uncertainty enables risk-managers to:

� choose policies with a high probability of success

� identify when more information is needed

Approaches for evaluation of variability and uncertainty

• Qualitative

• Quantitative

– Deterministic

– Probabilistic

• Tiered refinement of evaluation

Qualitative evaluation

• Expert judgement

• Conceptual models

• Logical argument

• Weight of evidence

• Scoring methods – semi-quantitative

Implicit or explicit

EFSA tabular approach to evaluating uncertainty

• Systematic and transparent

• Practical for everyday use

• Also taken up in guidance for REACH

Source of uncertainty Direction & magnitude

One row for each source of uncertainty - -/+++

- - -/+

etc.

Overall uncertainty of outcome - - -/++

Deterministic methods

• Use calculations with fixed values to assess risk

• Variability and uncertainty addressed by:

– Standardised uncertainty factors

– Worst-case assumptions

– What-if calculations

– Simple sensitivity analysis

• Quantifies alternative outcomes but not their frequency or

probability – characterise qualitatively

• Difficult to assess combined effect of multiple assumptions

on conservatism of result

Probabilistic methods

• Use probability distributions to represent

variability and uncertainty

• ‘Propagate’ variability & uncertainty through the

model calculations

• Quantify variability & uncertainty of the outcome

(a) Probability density function

0.0

0.1

0.2

0.3

0.4

0 1 2 3 4 5 6

Exposure of 97.5th percentile consumer

�g/kg bw/day

Pro

babili

ty d

ensity

0

20

40

60

80

100

0 1 2 3 4 5 6

Exposure, �g/kg bw/day

Perc

entile

consum

er

95% confidence

interval for exposure

of 80th percentile

consumer

Probabilistic methods

• Distributions may be based on data or expert

judgement

• Possible to include quantification of model

uncertainty, e.g. alternative dose-response models

• Requires additional assumptions (e.g. distribution

type) that are themselves uncertain

Probabilistic & deterministic

e.g. Pesticide risks to birds

food intake per day (wet wt)

concentration on

food type

fraction of food

type in diet

fraction of diet

obtained in

treated areasexposure

(mg/kg/day)

initial

concentration

LD50

(mg/kg)

Toxicity-Exposure

Ratio

daily energy

expenditure

body weight

gross energy

content of food

moisture content

of food

assimilation efficiency

application

rate

residue per

unit dose

0 0.2 0.4 0.6 0.8 10

1

2

3

4

5

6

7

Fre

que

ncy

0 0.2 0.4 0.6 0.8 10

5

10

15

20

25

30

PT

Pro

bability

Density

0 0.2 0.4 0.6 0.8 10

0.5

1

1.5

2

2.5

3

Fre

que

ncy

0 0.2 0.4 0.6 0.8 10

0.5

1

1.5

2

2.5

3

3.5

PD Insect - May

Pro

bab

ility D

ensity

0.0001 0.01 1 100 100000 0

20%

40%

60%

80%

100%

Dose [mg / kg BW / d]

% o

f sp

ecie

s

108 106

Mallard Duck

Bobwhite Quail

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Fre

qu

ency

0 0.2 0.4 0.6 0.8 10

0.5

1

1.5

2

2.5

3

3.5

4

AE Plants

Pro

bability D

ensity

25 30 35 40 45 50 55 600

0.02

0.04

0.06

0.08

0.1

Pro

ba

bili

ty D

en

sit

y Male BW

1.2 1.4 1.6 1.8 21.6

1.8

2

2.2

2.4

2.6

log10

BW [g]

log

10 D

EE

[kJ / g

]

Best Fit95% Prediction IntervalData

18 20 22 24 26 28 300

0.05

0.1

0.15

0.2

0.25

0.3

0.35

18 20 22 24 26 28 30

Gross Energy Insects [kJ / g]

0 0.2 0.4 0.6 0.8 10

1

2

3

4

5

6

Fre

qu

en

cy

0 0.2 0.4 0.6 0.8 10

0.5

1

1.5

2

2.5

3

3.5

Moisture Content Insects

Pro

bab

ility D

ens

ity

10 20 100 2000

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

Pro

babili

ty D

ensity

RUD Insects

0 1 2 3 4 50%

20%

40%

60%

80%

100%

% o

f skyl

ark

s

FIR / BW

0.01 1 100 10000 0%

20%

40%

60%

80%

100%

% o

f s

ky

lark

s

Toxicity Exposure Ratio (TER)

% o

f s

ky

lark

s

Sin

gle

Fo

od

Ite

m

Hart, Roelofs, Crocker, Mineau

+ unquantified uncertainties

�

Different levels of evaluation

Deterministic: range of alternative outcomes

Probabilistic: shows both the range of outcomes and their probabilities

Scoring: relative index, no units

‘high’‘medium’‘low’ Qualitative: descriptive categories

�������������� �����

Tiered refinement of evaluation

• Quantification may not always be necessary

• Start simple

• Do no more than is needed to reach a decision

• Focus refinement on key sources of variability & uncertainty

• Always evaluate the unquantifieduncertainties

Refine assessment

Evaluate options

for risk

mitigation?*

Collect data

to reduce key

uncertainties?

Yes

Assess some elements

probabilistically?*

Select options

for refinement of

assessment

Adequate

certainty?

No

Qualitative assessment

End assessment.

Take decision.

Assess key quantifiable

elements deterministically

No

Adequate

certainty?

End assessment.

Take decision.

Yes

Assess more elements

deterministically?*

Sensitivity analysis

• Sensitivity analysis identifies which inputs contribute

most to variability and uncertainty of outcome

– Major sources of uncertainty � potential targets

for research to reduce uncertainty

– Major sources of variability � potential targets for

risk management (if controllable)

– Unimportant variables � consider omitting to

simplify model

Evidence and uncertainty

• Evidence is inversely related to uncertainty

� can we use probability as a metric for evidence?

• Assess the impact of each piece of evidence on the probability distribution for the quantity of interest

– quantify impacts using expert elicitation

– combine lines of evidence quantitatively

� probability distribution for overall outcome

e.g. Turner et al., JRSS A, 2009

Evidence and uncertainty

Experimental

study

Quantitative result

Various sources of bias & uncertainty

Elicit expert judgements of impact on outcome

Combine distributions

� adjusted result

Evidence from other studies…

Expressing variability & uncertainty

• Narrative descriptions

• Defined terminologies

• Quantitative expressions

Qualitative expression

• Narrative descriptions of variability & uncertainty

• Open to variable interpretation

• Communicates only that outcomes are variable & uncertain – not how different or likely they are

�������� ������� ������ �������� ��������������������� ����� ��������� ��������� ��������� ����������� ��������� ���������� ��������� ����������������� ���������� �������������� ���������� ����������

������������ ������������ �������������

Defined terminologies

• E.g. International Panel on Climate Change

Less than 1 out

of 10 chance

Very low

confidence

About 2 out of 10

chance

Low confidence

About 5 out of 10

chance

Medium

confidence

About 8 out of 10

chance

High confidence

At least 9 out of

10 chance

Very High

confidence

< 33% probabilityUnlikely

< 10% probabilityVery unlikely

< 1% probabilityExceptionally

unlikely

33 to 66%

probability

About as likely as

not

> 66% probabilityLikely

> 90% probabilityVery likely

> 99% probabilityVirtually certain

Chance of a statement being correct Probability of an outcome occurring

Defined terminologies

• Advantages

– Consistent use and interpretation of terms

– Increased information

– Promotes quantitative thinking

• Limitations

– Loss of information from quantitative assessments

– Boundary effects

– May pre-empt or provoke risk management judgements

– Different terms and definitions needed for different types and dimensions of risk

Quantitative expressions

• Scores (relative measure of risk)

• Deterministic estimates, e,g. “worst case”

• Upper and lower bounds

• Confidence/credibility intervals

• Probability distributions

– Graphical (various

formats)

– Tabular summary

Relate to selected outcomes – need

to express their

probability at least qualitatively

Can be challenging to

communicate

Addressing the unquantifieduncertainties

• “The report of the risk assessment should

indicate any constraints, uncertainties,

assumptions and their impact on the risk

assessment” (CODEX Working Principles on

Risk Analysis)

• Key requirements:

– Comprehensive and systematic

– Characterise impact on outcome

• EFSA tabular approach or similar

Dealing with uncertainty in decision-making

• Resolving uncertainty is part of risk management

• Expressing uncertainty is not “unhelpful”

• Uncertainty of outcome vs. confidence in

decisions

Resolving uncertainty is part of risk management

• “responsibility for resolving the impact of

uncertainty on the risk management decision lies

with the risk manager, not the risk assessors”

(CODEX Working Principles on Risk Analysis)

• This is because the degree of certainty and

precaution required is a societal issue and

depends on many considerations outside the

scope of the scientific assessment (e.g.

economic, legal, cultural etc.)

Expressing uncertainty is not “unhelpful”

• Risk managers are often unreceptive:

– “Confidence intervals are too wide to be useful”• Manager needs to know the uncertainty is high

– Shows how much reliance can be placed on the science

– Enables manager to make an informed decision

– “Just give me your best estimate”• Best estimates are not enough

– “Showing uncertainty will scare the public”• Concealing uncertainty is worse in the long term

• Need to develop better ways to communicate

Uncertainty of outcome vs. confidence in decisions

• Never claim certainty of outcome

– Outcome is always uncertain

– Credibility and trust require openness (Phillips)

• It is possible to have a high level of confidence

that a decision is optimal even though the

outcome is highly uncertain

– it is certainly better to play Russian roulette

with one bullet than two, even though the

outcome of this decision is uncertain

Possible topics for discussion:

• Can we be more systematic in evaluating

uncertainties?

• Would standardised definitions for different levels

of risk and uncertainty be useful?

• Can we express expert judgements about

evidence and uncertainty in terms of probabilities

and distributions?

• When are quantitative methods helpful?

• How can we interact better with risk managers

about uncertainty?