Verification of ECMWF deterministic forecasts at Météo-France
Probability Forecasts and Verification
-
Upload
murphy-eaton -
Category
Documents
-
view
15 -
download
0
description
Transcript of Probability Forecasts and Verification
Probability Forecasts and Verification
Uncertainty
• Forecasts have uncertainty that should be taken into account:– Observational Uncertainty (initial conditions)
• Instrument errors, bias in frequency of measurements, representativeness, reporting errors, random errors, precision errors, conversion errors …
– Model Uncertainty• Insufficient spatial resolution, truncation errors,
parameterization errors, coding errors, initialization errors …
Deterministic and Probabilistic Forecasts
• Deterministic Forecast: Specific Outcome is Predicted
• Probabilistic Forecast: Range of Possible Outcomes Accompanied by Confidence Estimate
Ex: Tercile Probabilities
Probability Forecast
Probabilistic Verification
• Requires Several Forecast Cases – Cannot Verify a Single Probabilistic Forecast
• Compare Forecast Distribution to a Distribution of Observations
• Rank Probability Score (RPS)– Rank Probability Skill Score (RPSS)
• Relative Operating Characteristics (ROC)• Reliability (Not Really Verification)
Rank Probability Score
Rank Probability Skill Score
July 1 startDJF prec forecastRPSS
NMME
CFSv2
Relative Operating Characteristics
Relative Operating Characteristics
Hit Rate = hits/(hits+misses)
False Alarm Rate = false alarms/(hits+false alarms)
Relative Operating Characteristics
For Each Threshold Build aContingency Table SummingOver All Forecasts
Relative Operating Characteristics
Reliability
• Prediction System is Reliable if Predicted Probabilities Agree with Observed Frequencies– Ex: Climatological Probability has Perfect
Reliability (33%, 33%, 33%)
• Reliability Diagram Measure whether the Prediction System is Reliable, Overconfident or Underconfident
• Reliability Diagram also Shows Resolution and Sharpness
Reliability Diagram
Reliability Diagram
T2m October Starts JFM Verification
T2m October Starts JFM Verification
Probabilistic Verification
• Requires Several Forecast Cases – Cannot Verify a Single Probabilistic Forecast
• Compare Forecast Distribution to a Distribution of Observations
• Rank Probability Score (RPS)– Rank Probability Skill Score (RPSS)
• Relative Operating Characteristics (ROC)• Reliability (Not Really Verification)