Understanding and Using Uncertainty Information in Weather Forecasting Susan Joslyn University of...

59
Understanding and Using Uncertainty Information in Weather Forecasting Susan Joslyn University of Washington
  • date post

    20-Dec-2015
  • Category

    Documents

  • view

    215
  • download

    1

Transcript of Understanding and Using Uncertainty Information in Weather Forecasting Susan Joslyn University of...

Understanding and Using Uncertainty Information in Weather

Forecasting

Susan Joslyn

University of Washington

Acknowledgements

Earl Hunt

David Jones

Limor Nadav-Greenberg

John Pyles

Adrian Raftery

Karla Schweitzer

McLean Slaughter

Meng Taing

Jeff ThomassonThis research was supported by the DOD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-10745

Forecast Uncertainty

• Available for some time

• Rarely communicated in public forecasts

• Underused by weather forecasters

Forecast Uncertainty

• Difficult to understand- Forecasters claim

• People make mistakes when reasoning with probability

• Format: Frequency (1 time in 10) is better than

Probability (10% chance)

Forecast Uncertainty

• Useful for deterministic forecasts decision?

Theoretically

Practically useful?

• It doesn’t matter how good the information if people can’t or won’t make use of it.

Goals for Psychology Team

 • Establish uncertainty information is useful Threshold forecast (forecasters & general public)- high wind advisory for boater safety

• What is best presentation format to enhanceUnderstanding?Decisions?

Three Major Lines of Inquiry

1. Does probability information improve threshold forecast? Study 1

2. Does display format (visualization) matter?

Study 2

3. Does the wording matter? Studies 3-4

(probability/ frequency)

Study1 Does Probability Information Improve

Threshold Forecast?

Participants: Advanced atmospheric science students

Task: • Forecast wind speed and direction

• Decide whether to issue high wind advisory

(winds > 20 knots)

Within Subject Design

Historical data Radar Imagery

Satellite Imagery

TAFs and current METARs

Model output

(AVN, MM5 & NGM)

Historical data Radar Imagery

Satellite Imagery

TAFs and current METARs

Model output

(AVN, MM5 & NGM)

+

Chart showing probability

of winds > 20 k

Condition 1 Condition 2

Probability of Winds ≥ 20k

Within Subjects Design

Historical data Radar Imagery

Satellite Imagery

TAFs and current

METARsModel output (AVN, MM5 & NGM)

Historical data Radar Imagery

Satellite Imagery

TAFs and current

METARsModel output (AVN, MM5 & NGM)

+Chart showing probability

of winds > 20 k

Condition 1 Condition 2

• Same participants, same weather• Only difference is probability product

Results

Threshold Forecast: • People posted fewer wind advisories with

probability product.

• Similar ability to discriminate between high wind and low wind event (sensitivity).

Results: Percent Advisories

Y= % times

forecasters posted advisory

X= probability

of winds

> 20K

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0-10 10-30 30-50 50-70 70-90 90-100

Probability (winds > 20 k) range given by Model

Percent Advisories

With Probability ProductWithout Probability Product

Conclusion: Uncertainty Information IS Beneficial for Threshold

• Increased advisories when high winds were very likely

• Decreased advisories when high winds were unlikely-fewer false alarms

• Increase trust in warnings!

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0-10 10-30 30-50 50-70 70-90 90-100

Probability (winds > 20 k) range given by Model

Percent Advisories

With Probability ProductWithout Probability Product

Study 2 Does Display Format Matter?

• 3 different visualizations of 90% predictive interval

• Range of likely wind speeds

• All conditions included median wind speed chart

deterministic forecast

3 Visualizations: Between subjects

1. 90% Upper bound: warmer colors = higher wind speed

• “ observed wind speeds will be higher only 1 time in 10”• worse case scenario: highest likely winds

3 Visualizations

1. 90% Upper bound:• wind speeds will be higher only 1 time in 10warmer colors = higher wind speed

2. Margin of error:

• range of wind speeds between UB & median • display of uncertainty in the forecast

warmer colors = more uncertainty

3 Visualizations

1. 90% Upper bound:wind speeds will be higher only 1 time in 10warmer colors = higher wind speed

2. Margin of error:

range of wind speeds between upper bound and median warmer colors = more uncertainty

3. Box plot:

median 90% Upper bound90% lower boundWind Speed in knots

Wind speed in knots

Method

• Participants: Atmospheric Science students (replicated on NOAA Forecasters)

• Practice: Learned how to read charts

• Test: - Forecast wind speeds - Threshold: high wind advisory (winds >20 knots)

- Rate uncertainty in forecast

Results: Wind Speed Forecast

• UB forecast significantly higher wind speeds• Display provided a high anchor (Tversky & Kahneman, 1982)

0 0.5 1 1.5 2 2.5

Box Plot

Upper bound

Margin of Error

Knots above the Median

1.55

2.02

1.17

Results: High Wind Advisories

Likelihoodof high winds

Box PlotUpperBound

Margin of Error

HIGH Median > 20K 98.44% 94.45% 91.67%

MEDIUM Median 15-20K 32.40% 31.24% 27.95%

LOW Median <15 K

3.57% 3.97% 2.38%

People in the box plot condition:• posted significantly more advisories• most in high likelihood situations

Results: Uncertainty Rating

Box plot.81

Upper Bound .89

Margin of Error .97

• MoE best for detecting relative uncertainty• They learned: “The wider the range the greater the uncertainty”

• Ratings in the MoE significantly more highly correlated to range

correlation

Conclusion: Format Matters

Box Plot better threshold forecast wind speed: no bias (salient high and low anchors)

MoE detect relative uncertainty in forecast

Upper higher winds speeds: bias (anchor)

Bound no benefit to threshold forecast

Study 3 & 4 Does Wording Matter?

• Participants:

Psychology undergraduates

• Frequency is easier to understand than probability (Gigerenzer, 1995, 1999, 2000)

– Research on complex problems – Is that true of simple expressions of uncertainty?

Does Wording Matter?

There is a 10% chance that the wind speeds will be greater than 20 knots.

Method

Procedure: Fill out questionnaire rating expressions of uncertaintyDecide whether or not to post a high wind advisory

Suppose that there is a 10% chance that the wind speeds will be greater than 20 knots.

“How likely are the wind speeds to be greater than 20 knots? (please fill in a bubble)” Very Unlikely Very Likely O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O Would you issue a small craft advisory (winds equal or greater than 20 knots)? ___Yes ___No

Method

Procedure: Fill out questionnaire rating expressions of uncertaintyDecide weather to post a wind advisory

Suppose that there is a 10% chance that the wind speeds will be greater than 20 k.

“How likely are the wind speeds to be greater than 20 knots? (please fill in a bubble)” Very Unlikely Very Likely O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O Would you issue a small craft advisory (winds equal or greater than 20 knots)? ___Yes ___No

Method

Procedure: Filled out questionnaire rating expressions of uncertaintyDecide weather to post a wind advisory

Suppose that 1 time in 10 the wind speeds will be greater than 20 knots.

“How likely are the wind speeds to be greater than 20 knots? (please fill in a bubble)”Very Unlikely Very Likely O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O Would you issue a small craft advisory (winds equal or greater than 20 knots)? ___Yes ___No

Study 3

2 Variables: Wording & Likelihood

Probability Frequency

10% chance = 1 time in 10

90% chance = 9 times in 10

Study 3: Likelihood of High Wind Held Constant

1 time in 10 wind speeds = 9 times in 10 wind speeds

will be greater than 20 knots will be less than 20 knots

Results: Reversal Error

• Rate from wrong side of scale

Suppose that there is a 90% chance that the wind speeds will be less than 20 knots.

“How likely are the wind speeds to be less than 20 knots? (please fill in a bubble)”

O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O <---very unlikely very likely ------>

• They completely misunderstand the phrase

• Most in “90% (9 in 10) less than” wording

• Which is it? High likelihood? Less than?

Reversal error

Study 4

Manipulated Less / Greater

Less Greater10% chance less 10 % chance greater

Added 2 levels of likelihood

Less Greater10% chance less 10 % chance greater1 in 10 less 1 in 10 greater30% chance less 30% chance greater 3 in 10 less 3 in 10 greater 70% chance less 70% chance greater 7 in 10 less 7 in 10 greater 90% chance less 90% chance greater 9 in 10 less 9 in 10 greater

Equivalent Expressions

Less Wording Greater Wording10% chance less 10 % chance greater1 in 10 less 1 in 10 greater30% chance less 30% chance greater 3 in 10 less 3 in 10 greater 70% chance less 70% chance greater 7 in 10 less 7 in 10 greater 90% chance less 90% chance greater 9 in 10 less 9 in 10 greater

Equivalent Expressions

Less Wording Greater Wording10% chance less 10 % chance greater1 in 10 less 1 in 10 greater30% chance less 30% chance greater 3 in 10 less 3 in 10 greater 70% chance less 70% chance greater 7 in 10 less 7 in 10 greater 90% chance less 90% chance greater 9 in 10 less 9 in 10 greater

Results: Reversal Error

More often in “less than” wording (4x as likely)

Mean reversal error

per person

Less than .41

Greater than .10

High vs. low likelihood does not matter

Frequency wording does not help

Results: Wind Advisories

Percent of Advisories Posted

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4

Likelihood of winds exceeding 20 k

Percent of Advisories Posted

freq greater

10% 30% 70% 90%

Percent of Advisories Posted

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4

Likelihood of winds exceeding 20 k

Percent of advisories.

freq greater

freq less

Results: Wind Advisories

10% 30% 70% 90%

Percent of Advisories Posted

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4

Likelihood of winds exceeding 20 k

Percent of advisories.

freq greater

freq less

prob greater

Results: Wind Advisories

10% 30% 70% 90%

Results: Probability “less” is worst

10% 30% 70% 90%

Reversal error subjects eliminated from this analysis

Percent of Advisories Posted

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4

Likelihood of winds exceeding 20 k

Percent of advisories.

freq greater

freq less

prob greater

prob less

10% 30% 70% 90%

Conclusion: Wording Matters

• “Less than” wording is difficult (reversal errors)

• Wind speed advisories in “probability less” - too many advisories in low ranges

- too few in high ranges

• Frequency protects against posting errors generated by “less than” wording

Conclusions

• Probability information improves threshold forecasts– Many end-user weather decisions are yes/no threshold decisions

• The right display format – Improves understanding

• MoE communicates relative uncertainty

– Improves weather decisions• Box Plot increases warnings in high likelihood• Box Plot unbiased wind speed forecast

• Wording matters– “Less than” is confusing– Frequency helps sometimes

• NOT in reversal errors• HELPS in posting advisories

The End

Results: Percent Advisories

Y= % times

forecasters posted advisory

X= probability

of winds

> 20K

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0-10 10-30 30-50 50-70 70-90 90-100

Probability (winds > 20 k) range given by Model

Percent Advisories

With Probability ProductWithout Probability Product

Results: Percent Advisories

• Y= % times

forecasters posted advisory

• X= probability

of winds

> 20K

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0-10 10-30 30-50 50-70 70-90 90-100

Probability (winds > 20 k) range given by Model

Percent Advisories

With Probability ProductWithout Probability Product

Results: Percent Advisories

• Y= % times

forecasters posted advisory

• X= probability

of winds

> 20K

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0-10 10-30 30-50 50-70 70-90 90-100

Probability (winds > 20 k) range given by Model

Percent Advisories

With Probability ProductWithout Probability ProductExpected Response

Results: Percent Advisories

• Y= % times

forecasters posted advisory

• X= probability

of winds

> 20K

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0-10 10-30 30-50 50-70 70-90 90-100

Probability (winds > 20 k) range given by Model

Percent Advisories

% Observed Winds

> 20 k

With Probability ProductWithout Probability Product% times observed winds> 20ktsExpected Response

Study 1: Rating

• 10% was rated significantly higher

Probability condition:

10% chance (M=1.32) 90% chance (M=.99)

O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O

Frequency condition:

1 in ten (M=1.06) 9 out of 10 (M=.98)

O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O

Study 2: Rating

10 was rated higher--did not reach significance

10% (1 in 10) greater (M=1.25) 90% (9 in 10)less (M=.97)

O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O

10% (1 in 10) less (M=.98) 90% (9 in 10)greater (M=.88)

O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O

Study 1: Reversal Error

Mean reversal error per person

90% (9 times)

less than.83

10% (10 times) greater than

.33

User Needs & Understanding

• Naval Forecasters • Terminal Aerodrome Forecast (TAF) posted at regular intervals while fulfilling other duties

Microphone

& recorder

Method

• Talk-aloud while creating TAF

Numerical Model: MM5

Satellite

Synoptic Pattern Comparison

1. Compare position of low in the model & satellite

2. Assess differences in movement and position

3. Adjust forecast accordingly

Compare Predicted to Observed Values

1. Access NOGAPS predicted pressure for current time 29.69

2. Access current local pressure and 29.69

subtract from NOGAPS - 29.64

.05

3. Access NOGAPS predicted pressure for 29.59

forecast period and subtract error amount - .05

4. Forecast 29.54

Results

• Naval forecasters rely heavily on models (1/3-1/2 source statements referred to models)

• Statements implying understanding of model uncertainty

Model biases and strengths

Initialization of model run Strategies for determining uncertainty

Evaluation of degree of uncertaintyAdjusting model predictions

Conclusions

• Uncertainty?• Error in deterministic forecast?• Subsequent questionnaire study: confidence

is related to their assessment of model performance

Probability ProblemThe probability that a woman getting

a mammogram has breast cancer is 1%. If the woman has breast cancer the probability is 80% that she will have a positive mammogram.

If the woman does not have breast cancer the probability that she will still have a positive mammogram is 10%.

You have a patient that has a positive mammogram (no symptoms)--what is the probability she has breast cancer.

Frequency ProblemTen out of every 1,000 women

have breast cancerOf those 10 women with breast

cancer 8 will have a positive mammogram

Of the remaining 990 women without breast cancer, 95 will still have a positive monogram

You have a sample of women who have positive mammograms in your screening (no symptoms)

How many of these women will actually have breast cancer?

Results: Probability “less” is worst

Reversal error subjects eliminated from this analysis

Percent of Advisories Posted

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4

Likelihood of winds exceeding 20 k

Percent of advisories.

freq greater

freq less

prob greater

prob less

10% 30% 70% 90%