Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature...
-
Upload
rosamond-hamilton -
Category
Documents
-
view
214 -
download
1
Transcript of Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature...
![Page 1: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/1.jpg)
Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and
Temperature Forecasts
Tom HopsonLuca Delle Monache, Yubao Liu, Gregory Roux,
Wanli Wu, Will Cheng, Jason Knievel, Sue Haupt
![Page 2: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/2.jpg)
Army Test and Evaluation Command:Dugway Proving Ground
![Page 3: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/3.jpg)
Dugway Proving Grounds, Utah e.g. T Thresholds
• Includes random and systematic differences between members.
• Not an actual chance of exceedance unless calibrated.
![Page 4: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/4.jpg)
Xcel Energy Service Areas
Wind Farms (50+)~3200 MW
Northern States Power (NSP)Public Service of Colorado (PSCO)Southwestern Public Service (SPS)
3.4 million customers (electric)Annual revenue $11B
Copyright 2010 University Corporation for Atmospheric Research
![Page 5: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/5.jpg)
WRF RTFDDA Model DomainsEnsemble System (30 members)
D1 = 30 kmD2 = 10 km
0-48 hrs0-48 hrs
Real Time Four Dimensional Data Assimilation (RTFDDA)
41 vertical levels
Vary:•Multi-models•Lateral B.Cs.•Model Physics•External forcing
Yubao Liu -- [email protected] for further questions
![Page 6: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/6.jpg)
Goals of an EPS
• Predict the observed distribution of events and atmospheric states
• Predict uncertainty in the day’s prediction• Predict the extreme events that are possible on a
particular day• Provide a range of possible scenarios for a
particular forecast
![Page 7: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/7.jpg)
OutlineI. Brief overview of: 1) quantile regression (QR),
2) logistic regression (LR), 3) umbrella post-processing procedure, 4) “analog Kalman filter” (ANKF)
II. 2nd moment calibration via rank histogramsIII. Skill score comparisons and improvements
with increased hindcast dataIII. Example of blending approachesIV. Conclusions
![Page 8: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/8.jpg)
Example of Quantile Regression (QR)
Our application
Fitting T quantiles using QR conditioned on:
1) Ranked forecast ens
2) ensemble mean
3) ensemble median
4) ensemble stdev
5) Persistence
Hopson and Hacker 2012
![Page 9: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/9.jpg)
Logistic Regression for probability of exceedance (climatological thresholds)
f (z) =1
1+ e−z
z=β0 + β1x1 + ...+ βkxk
![Page 10: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/10.jpg)
Prob
abili
ty/°
K
Temperature [K]
Prob
abili
ty/°
K
Temperature [K]
ForecastPDF
climatologicalPDF
Step I: determineclimatological quantiles
Prob
abili
ty
Temperature [K]
Step 3: use conditioned CDF tointerpolate desired quantiles
1.0
0.5
.75
.25
prior
posterior
Step 2: calculate conditionalprobs for each climat quan
Prob
abili
ty
Temperature [K]
0.5
.75
.25
1.0
Final result: “sharper” posterior PDFrepresented by interpolated quans
Hopson and Hacker 2012
![Page 11: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/11.jpg)
T [K
]
Timeforecastsobserved
Regressor set: 1. reforecast ens2. ens mean3. ens stdev 4. persistence 5. LR quantile (not shown)
Prob
abili
ty/°
K
Temperature [K]
climatologicalPDF
Step I: Determineclimatological quantiles
Step 2: For each quan, use forward step-wisecross-validation to select best regress setSelection requires: a) min QR cost function, b) binomial distrib at 95% confidenceIf requirements not met, retain climatological “prior”
1.
3.2.
4.
Step 3: segregate forecasts based on ens dispersion; refit models (Step 2) for each range
Time
forecasts
T [K
]
I. II. III. II. I.Pr
obab
ility
/°K
Temperature [K]
ForecastPDF
prior
posterior
Final result: “sharper” posterior PDFrepresented by interpolated quans
Hopson and Hacker 2012
![Page 12: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/12.jpg)
National Security Applications Program Research Applications Laboratory
Significant calibration regressors
3hr Lead-time 42hr Lead-time
Station DPG S01
![Page 13: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/13.jpg)
National Security Applications Program Research Applications Laboratory
RMSE of ensemble members
3hr Lead-time 42hr Lead-time
Station DPG S01
![Page 14: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/14.jpg)
Timet = 0day-1day-2day-6 day-5 day-4 day-3day-7
OBS
PRED
KF-weight
KF
Delle Monache et al. 2010
Analog Kalman Filter (ANKF)• Deterministic method applied to each individual ensemble
• KF weighting run in analog space
![Page 15: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/15.jpg)
Timet = 0day-1day-2day-6 day-5 day-4 day-3day-7
OBS
PRED
KF-weight
KF
ANKF
AN
“Analog” Spaceday-4day-7day-5 day-3 day-2 day-1day-6
PRED
OBS
farthest analog
closest analog
NOTEThis procedure is applied independently at each
observation location and for a given forecast time
Delle Monache et al. 2010
![Page 16: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/16.jpg)
OutlineI. Brief overview of: 1) quantile regression (QR),
2) logistic regression (LR), 3) umbrella post-processing procedure, 4) “analog Kalman filter” (ANKF)
II. 2nd moment calibration via rank histogramsIII. Skill score comparisons and improvements
with increased hindcast dataIII. Example of blending approachesIV. Conclusions
![Page 17: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/17.jpg)
42-hr dewpoint time seriesBefore Calibration After Calibration (QR)
Station DPG S01
![Page 18: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/18.jpg)
Original ensemble QR
LR ANKF
Rank Histograms15hr lead wind forecasts
![Page 19: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/19.jpg)
Skill measures used:1) Rank histogram (converted to scalar measure)2) Root Mean square error (RMSE)3) Rank Probability Score (RPS)4) Relative Operating Characteristic (ROC) curve
Skill Scores
SS =Aforc −ArefAperf −Aref
Comparing to original ensemble forecast, but with bias removed => “reference forecast”
![Page 20: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/20.jpg)
Blue - QR
Red - ANKF
Green - LR
Skill Score ComparisonFor wind farm CEDC, 3hr lead forecasts
Reference forecast: original wind speed ensemble w/ bias removedData size: 900pts
![Page 21: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/21.jpg)
Rank Histogram scalar
QR QR
LR LR
RMSE
Skill Scores Dependence on Training Data Size
Upper dashed – 900ptsSolid – 600ptsLower dashed – 300pts
Reference Forecast:Original wind speed ensemble w/ bias removed
![Page 22: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/22.jpg)
ROC
QR QR
LR LR
RPSS
Skill Scores Dependence on Training Data Size (cont)
Reference Forecast:Original wind speed ensemble w/ bias removed
Upper dashed – 900ptsSolid – 600ptsLower dashed – 300pts
![Page 23: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/23.jpg)
23
RPS ROC
RMSE Brier Score
Wind farm TWBT
1
1 1
112
12 12
1224
24 24
2436
36 36 48
4848
48
36ANKFQRQR + ANKF
![Page 24: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/24.jpg)
24
original ANKF
QR QR + ANKF
TWBT6-h
![Page 25: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/25.jpg)
Summary “step-wise cross-validation”-based post-processing framework provides a method to ensure forecast skill no worse than climatological and persistence
Also provides an umbrella to blend together multiple post-processing approaches as well as multiple regressors, and to diagnose their utility for a variety of cost functions
Quantile regression and logistic regression useful tools for improving 2nd moment of ensemble distributions
See significant skill gains with increasing “hindcast data” amount for a variety of skill measures
Blending of post-processing approaches can also further enhance final forecast skill (e.g. ANKF and QR) by capturing “best of both worlds”
Further questions: [email protected] or [email protected]
![Page 26: Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ceb5503460f949b75be/html5/thumbnails/26.jpg)