Probabilistic QPFs for the Indian Monsoon using Reforecasts
description
Transcript of Probabilistic QPFs for the Indian Monsoon using Reforecasts
![Page 1: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/1.jpg)
Probabilistic QPFs for the Indian Monsoon
using Reforecasts
Tom Hamill
NOAA / ESRL
NOAA Earth SystemResearch Laboratory
![Page 2: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/2.jpg)
Outline• Background
– Why reforecasting?– NOAA’s reforecast data set.– How skill can be overestimated using the conventional
method of applying metrics– Monsoon climatology– Logistic regression review
• Results– Stepwise elimination : logistic regression results– Brier skill scores and forecast reliability, before
and after calibration using logistic regression– Some examples of actual calibrated forecasts
![Page 3: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/3.jpg)
Problem with current ensemble forecast systemsForecasts may be biased and/or deficient in spread,
so that probabilities are mis-estimated. “Calibration” (statistical correction) needed.
Heavy rain in anarea where none ofthe ensemble memberspredicted it.
http://www.spc.noaa.gov/exper/sref/
![Page 4: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/4.jpg)
This article onreforecasting in the Bulletinof the AmericanMeteorological Society is a goodplace to start foran overview.
![Page 5: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/5.jpg)
NOAA’s reforecast data set
• Model: T62L28 NCEP GFS, circa 1998
• Initial States: NCEP-NCAR Reanalysis II plus 7 +/- bred modes.
• Duration: 15-day integrations every day at 00Z from 19781101 to now. (http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/refcst/week2).
• Data: Selected fields (winds, hgt, temp on 5 press levels, precip, t2m, u10m, v10m, pwat, prmsl, rh700, heating). NCEP/NCAR reanalysis verifying fields included (Web form to download at http://www.cdc.noaa.gov/reforecast). Data saved on 2.5-degree grid.
• Experimental precipitation forecast products: http://www.cdc.noaa.gov/reforecast/narr .
![Page 6: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/6.jpg)
2.5°
1.0°
Reforecast data was archived on global domain. For this experimentwe saved forecast total precipitation,column precipitable water, and sea-levelpressure tendency)on coarse and fine grids, as shown,for May 15 - Oct 15, 1979-2007.
![Page 7: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/7.jpg)
Overestimating skill: a review of the Brier Skill Score
Brier Score: Mean-squared error of probabilistic forecasts.
BSf=
1n
pkf −ok( )
2
k=1
n
∑ , ok =1.0 if kthobservation≥threshold0.0 if kthobservation< threshold
⎧⎨⎩
Brier Skill Score: Skill relative to some reference, like climatology.1.0 = perfect forecast, 0.0 = skill of reference.
BSS =BS
f−BS
ref
BSperfect
−BSref =
BSf−BS
ref
0.0 −BSref =1.0 −
BSf
BSref
![Page 8: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/8.jpg)
Overestimating skill: another example
5-mm threshold
Location A: Pf = 0.05, Pclim = 0.05, Obs = 0
BSS =1.0 −BS
f
BSclim =1.0 −
.05 −0( )2
.05 −0( )2=0.0
Location B: Pf = 0.05, Pclim = 0.25, Obs = 0
BSS =1.0 −BS
f
BSclim =1.0 −
.05 −0( )2
.25 −0( )2=0.96
Locations A and B:
BSS =1.0 −BS
f
BSclim =1.0 −
.05 −0( )2 + .05 −0( )2
.25 −0( )2 + .05 −0( )2=0.923
why not 0.48?
![Page 9: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/9.jpg)
An alternative BSSSay m overall samples, and k categories
where climatological event probabilities are similar in this category. ns(k) samples assigned to this category. Then form BSS from weighted average of skills in the categories.
BSS =ns k( )
mk= 1
nc
 1 -BS
fk( )
BSc l i m
k( )
Ê
Ë
ÁÁÁÁÁ
ˆ
¯
˜̃˜̃˜̃
(for more details on all of this, see Hamill and Juras, QJRMS, October C, 2006)
![Page 10: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/10.jpg)
Monsoon precipitation climatology
![Page 11: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/11.jpg)
Monsoon precipitation climatology
![Page 12: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/12.jpg)
Monsoon precipitation climatology
![Page 13: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/13.jpg)
Monsoon precipitation climatology
![Page 14: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/14.jpg)
Logistic regression
• Predictors tested: √(ensemble-mean precip), precipitable water, SLP tendency
• Observed data: Indian precipitation analyses on 1-degree grid, 1979-2004
• Stepwise elimination to determine which predictors are useful.• Train with data +/- 10 days around date of interest. Cross
validated, so, for example, regression coefficients for 1979 were trained on 1980-2004 data.
• Test June 1 - October 1, 1979 - 2004.
P(Obs >Threshold) =1.0 −
1.01.0 + exp β0 + β1x1 +L + βnxn{ }
where x1, xn are model predictors, betas are fittedregression coefficients. Used NAG library routine.
![Page 15: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/15.jpg)
Which predictors in logistic regression with stepwise elimination? Day 1
For every day of the monsoonseason, a stepwise linear regression was run to determinewhich predictors provided a reduction in error. As shown,a power-transformed ensemble-mean forecast precipitation was uniformly selected as an important predictor. Precipitablewater was occasionally selected,and sea-level pressure changewas virtually never selected. Based on these results, all subsequent logistic regressionanalyses will be based on usingonly one predictor, the power-transformed ensemble-meanprecipitation amount.
![Page 16: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/16.jpg)
Which predictors in logistic regression with stepwise elimination? Day 3
The same conclusionis reached whenconsidering otherforecast leads.
![Page 17: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/17.jpg)
Brier Skill Scores
Confidenceintervals areso small theydon’t show upon the plot.
![Page 18: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/18.jpg)
Reliability, Ens. Relative Frequency, 1 and 5 mm
5, 95 percent confidenceintervals via block bootstrap.
solid lines: frequencydistribution of climatology
![Page 19: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/19.jpg)
Reliability, Ens. Relative Frequency, 10 and 25 mm
![Page 20: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/20.jpg)
Reliability, Logistic Regression, 1 and 5 mm
![Page 21: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/21.jpg)
Reliability, Logistic Regression, 10 and 25 mm
![Page 22: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/22.jpg)
Map of Logistic Regression BSS, Day 1
Note: This methodof calculating BSSlumps all samplesat a particular gridpoint together fordates between 1 Mayand 1 October. To theextent that theclimatological eventprobability varies overthis range of dates, theBSS may be somewhatinflated. Please readHamill and Juras,QJRMS, Oct (c) 2006.
![Page 23: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/23.jpg)
Map of Logistic Regression BSS, Day 2
![Page 24: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/24.jpg)
Map of Logistic Regression BSS, Day 3
![Page 25: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/25.jpg)
Map of Logistic Regression BSS, Day 4
![Page 26: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/26.jpg)
Map of Logistic Regression BSS, Day 5
![Page 27: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/27.jpg)
Monthly variations of skill, 1 mm
![Page 28: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/28.jpg)
Monthly variations of skill, 10 mm
More skill laterin the monsoonseason.
![Page 29: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/29.jpg)
Logistic regression forecast example #1, 1-day lead
![Page 30: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/30.jpg)
Logistic regression forecast example #1, 3-day lead
![Page 31: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/31.jpg)
Logistic regression forecast example #2, 1-day lead
![Page 32: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/32.jpg)
Logistic regression forecast example #2, 3-day lead
![Page 33: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/33.jpg)
Logistic regression forecast example #3, 1-day lead
![Page 34: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/34.jpg)
Logistic regression forecast example #3, 3-day lead
![Page 35: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/35.jpg)
Logistic regression forecast example #4, 1-day lead
![Page 36: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/36.jpg)
Logistic regression forecast example #4, 3-day lead
![Page 37: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/37.jpg)
Conclusions
• Precipitation probabilities estimated directly from the ensemble are very unreliable and unskillful because of substantial model deficiencies (coarse resolution, sub-optimal model physics, methods of generating ensemble, limited ensemble size).
• With a large data set of past forecasts (“reforecasts”) using the same model that is run operationally (here, a 1998 version of NCEP’s GFS), the model forecasts can be post-processed to yield reliable and somewhat skillful probabilistic forecasts.
• The 1st-generation NOAA reforecast model is out of date, but there is growing interest worldwide in producing reforecast data sets with current models (e.g., ECMWF will produce limited reforecasts starting in 2008).
![Page 38: Probabilistic QPFs for the Indian Monsoon using Reforecasts](https://reader034.fdocuments.us/reader034/viewer/2022050909/56815a72550346895dc7d33f/html5/thumbnails/38.jpg)
References(downloadable from www.cdc.noaa.gov/people/tom.hamill/cv.html) • Hamill, T. M., J. S. Whitaker, and X. Wei, 2004: Ensemble re-forecasting: improving medium-range forecast skill
using retrospective forecasts. Mon. Wea. Rev., 132, 1434-1447.
• Hamill, T. M., J. S. Whitaker, and S. L. Mullen, 2006: Reforecasts, an important dataset for improving weather predictions. Bull. Amer. Meteor. Soc., 87, 33-46.
• Hamill, T. M., and J. S. Whitaker, 2006: Probabilistic quantitative precipitation forecasts based on reforecast analogs: theory and application. Mon. Wea. Rev.,134, 3209-3229.
• Hamill, T. M., and J. Juras, 2006: Measuring forecast skill: is it real skill or is it the varying climatology? Quart. J. Royal Meteor. Soc., 132, 2905-2923.
• Wilks, D. S., and T. M. Hamill, 2007: Comparison of ensemble-MOS methods using GFS reforecasts. Mon. Wea. Rev., 135, 2379-2390.
• Hamill, T. M., and J. S. Whitaker, 2006: Ensemble calibration of 500 hPa geopotential height and 850 hPa and 2-meter temperatures using reforecasts. Mon. Wea. Rev., 135, 3273-3280
• Hagedorn, R, T. M. Hamill, and J. S. Whitaker, 2007: Probabilistic forecast calibration using ECMWF and GFS ensemble forecasts. Part I: 2-meter temperature. Mon. Wea. Rev., accepted.
• Hamill, T. M., R. Hagedorn, and J. S. Whitaker, 2007: Probabilistic forecast calibration using ECMWF and GFS ensemble forecasts. Part II: precipitation. Mon. Wea. Rev., accepted.