Post on 17-Dec-2015
Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and
Temperature Forecasts
Tom HopsonLuca Delle Monache, Yubao Liu, Gregory Roux,
Wanli Wu, Will Cheng, Jason Knievel, Sue Haupt
Army Test and Evaluation Command:Dugway Proving Ground
Dugway Proving Grounds, Utah e.g. T Thresholds
• Includes random and systematic differences between members.
• Not an actual chance of exceedance unless calibrated.
Xcel Energy Service Areas
Wind Farms (50+)~3200 MW
Northern States Power (NSP)Public Service of Colorado (PSCO)Southwestern Public Service (SPS)
3.4 million customers (electric)Annual revenue $11B
Copyright 2010 University Corporation for Atmospheric Research
WRF RTFDDA Model DomainsEnsemble System (30 members)
D1 = 30 kmD2 = 10 km
0-48 hrs0-48 hrs
Real Time Four Dimensional Data Assimilation (RTFDDA)
41 vertical levels
Vary:•Multi-models•Lateral B.Cs.•Model Physics•External forcing
Yubao Liu -- yliu@ucar.edu for further questions
Goals of an EPS
• Predict the observed distribution of events and atmospheric states
• Predict uncertainty in the day’s prediction• Predict the extreme events that are possible on a
particular day• Provide a range of possible scenarios for a
particular forecast
OutlineI. Brief overview of: 1) quantile regression (QR),
2) logistic regression (LR), 3) umbrella post-processing procedure, 4) “analog Kalman filter” (ANKF)
II. 2nd moment calibration via rank histogramsIII. Skill score comparisons and improvements
with increased hindcast dataIII. Example of blending approachesIV. Conclusions
Example of Quantile Regression (QR)
Our application
Fitting T quantiles using QR conditioned on:
1) Ranked forecast ens
2) ensemble mean
3) ensemble median
4) ensemble stdev
5) Persistence
Hopson and Hacker 2012
Logistic Regression for probability of exceedance (climatological thresholds)
f (z) =1
1+ e−z
z=β0 + β1x1 + ...+ βkxk
Prob
abili
ty/°
K
Temperature [K]
Prob
abili
ty/°
K
Temperature [K]
ForecastPDF
climatologicalPDF
Step I: determineclimatological quantiles
Prob
abili
ty
Temperature [K]
Step 3: use conditioned CDF tointerpolate desired quantiles
1.0
0.5
.75
.25
prior
posterior
Step 2: calculate conditionalprobs for each climat quan
Prob
abili
ty
Temperature [K]
0.5
.75
.25
1.0
Final result: “sharper” posterior PDFrepresented by interpolated quans
Hopson and Hacker 2012
T [K
]
Timeforecastsobserved
Regressor set: 1. reforecast ens2. ens mean3. ens stdev 4. persistence 5. LR quantile (not shown)
Prob
abili
ty/°
K
Temperature [K]
climatologicalPDF
Step I: Determineclimatological quantiles
Step 2: For each quan, use forward step-wisecross-validation to select best regress setSelection requires: a) min QR cost function, b) binomial distrib at 95% confidenceIf requirements not met, retain climatological “prior”
1.
3.2.
4.
Step 3: segregate forecasts based on ens dispersion; refit models (Step 2) for each range
Time
forecasts
T [K
]
I. II. III. II. I.Pr
obab
ility
/°K
Temperature [K]
ForecastPDF
prior
posterior
Final result: “sharper” posterior PDFrepresented by interpolated quans
Hopson and Hacker 2012
National Security Applications Program Research Applications Laboratory
Significant calibration regressors
3hr Lead-time 42hr Lead-time
Station DPG S01
National Security Applications Program Research Applications Laboratory
RMSE of ensemble members
3hr Lead-time 42hr Lead-time
Station DPG S01
Timet = 0day-1day-2day-6 day-5 day-4 day-3day-7
OBS
PRED
KF-weight
KF
Delle Monache et al. 2010
Analog Kalman Filter (ANKF)• Deterministic method applied to each individual ensemble
• KF weighting run in analog space
Timet = 0day-1day-2day-6 day-5 day-4 day-3day-7
OBS
PRED
KF-weight
KF
ANKF
AN
“Analog” Spaceday-4day-7day-5 day-3 day-2 day-1day-6
PRED
OBS
farthest analog
closest analog
NOTEThis procedure is applied independently at each
observation location and for a given forecast time
Delle Monache et al. 2010
OutlineI. Brief overview of: 1) quantile regression (QR),
2) logistic regression (LR), 3) umbrella post-processing procedure, 4) “analog Kalman filter” (ANKF)
II. 2nd moment calibration via rank histogramsIII. Skill score comparisons and improvements
with increased hindcast dataIII. Example of blending approachesIV. Conclusions
42-hr dewpoint time seriesBefore Calibration After Calibration (QR)
Station DPG S01
Original ensemble QR
LR ANKF
Rank Histograms15hr lead wind forecasts
Skill measures used:1) Rank histogram (converted to scalar measure)2) Root Mean square error (RMSE)3) Rank Probability Score (RPS)4) Relative Operating Characteristic (ROC) curve
Skill Scores
SS =Aforc −ArefAperf −Aref
Comparing to original ensemble forecast, but with bias removed => “reference forecast”
Blue - QR
Red - ANKF
Green - LR
Skill Score ComparisonFor wind farm CEDC, 3hr lead forecasts
Reference forecast: original wind speed ensemble w/ bias removedData size: 900pts
Rank Histogram scalar
QR QR
LR LR
RMSE
Skill Scores Dependence on Training Data Size
Upper dashed – 900ptsSolid – 600ptsLower dashed – 300pts
Reference Forecast:Original wind speed ensemble w/ bias removed
ROC
QR QR
LR LR
RPSS
Skill Scores Dependence on Training Data Size (cont)
Reference Forecast:Original wind speed ensemble w/ bias removed
Upper dashed – 900ptsSolid – 600ptsLower dashed – 300pts
23
RPS ROC
RMSE Brier Score
Wind farm TWBT
1
1 1
112
12 12
1224
24 24
2436
36 36 48
4848
48
36ANKFQRQR + ANKF
24
original ANKF
QR QR + ANKF
TWBT6-h
Summary “step-wise cross-validation”-based post-processing framework provides a method to ensure forecast skill no worse than climatological and persistence
Also provides an umbrella to blend together multiple post-processing approaches as well as multiple regressors, and to diagnose their utility for a variety of cost functions
Quantile regression and logistic regression useful tools for improving 2nd moment of ensemble distributions
See significant skill gains with increasing “hindcast data” amount for a variety of skill measures
Blending of post-processing approaches can also further enhance final forecast skill (e.g. ANKF and QR) by capturing “best of both worlds”
Further questions: hopson@ucar.edu or yliu@ucar.edu