CTB Science Plan For Multi Model Ensembles (MME) Suru Saha Environmental Modeling Centre...
-
Upload
collin-hodges -
Category
Documents
-
view
216 -
download
2
Transcript of CTB Science Plan For Multi Model Ensembles (MME) Suru Saha Environmental Modeling Centre...
CTB Science Plan
For
Multi Model Ensembles (MME)
Suru Saha
Environmental Modeling Centre
NCEP/NWS/NOAA
Vision A multi model ensemble forecast system that
leverages the best national and international models for improved predictions on intraseasonal, interannual
and decadal time scales
Background• Studies have shown that the forecast skill of a multi-
model system is higher than that of most of the individual models
• To maximize the amount of model information for NOAA’s seasonal forecasts, a multi-model ensemble strategy will be pursued by the CTB
Current Status of IMME• There has been progress on a possible International
MME (IMME) with the CFS and the EUROSIP models (ECMWF, UKMO and MeteoFrance)
• Preliminary discussions have taken place between NCEP and ECMWF on protocols for the transfer of the hindcast data, as well as real time forecasts.
• Each center will make available new hindcasts, if and when, they upgrade their seasonal forecast system.
Current Status of NMME• There has been little progress on a possible National
MME (NMME) with the CFS and the NCAR, NASA and GFDL models
• Results from a previous version of the GFDL model and data assimilation system did not show significant additional skill to the CFS, for the basic metrics of US surface temperature and precipitation.
Current Status of NMME (contd)• GFDL has made some new hindcasts.
Unfortunately, the CTB has no resources to evaluate these hindcasts.
• Regarding the NCAR model, Ben Kirtman has been funded to make the hindcasts using the same strategy as the CFSRR
• Regarding the NASA model, there has been no update on the completion of their hindcasts.
Gaps• There are three major gaps in executing a NMME
strategy• Firstly, skill levels for all models are barely above
climatological probabilities for US surface temps and precipitation, at monthly and seasonal time scales.
• Unless there is a significant improvement in these metrics, the value of a NMME may not be sufficient to justify the major cost of making hindcasts to support an operational NMME.
Gaps (contd)• Secondly, the computational and human costs of
preparing hindcasts for evaluation of skill and calibration is significant, and not presently funded by NOAA for models other than the CFS.
• Thirdly, the cost and logistics of running a NMME for all models, in operations, are cosiderable and has not yet been supported by NOAA.
Future Strategy of the NMME (Option 1)
• With less than a 100% investment, including modest computer resources and a staff of 1 full time and 2-3 part time personnel, the CTB could support the current strategy of:
• All participating centers make hindcasts of 2 initial months (for the mid-May and mid-November release of CPC’s official monthly and seasonal forecast) over the period 1979-present (or more years, if possible).
Future Strategy of the NMME (Option 1)• A dataset of prescribed output variables from these hindcasts
(full fields) will be made available to the CTB consolidation team. They will make an objective evaluation of these monthly and seasonal hindcasts, to determine if skill is added to the CFS for the basic metrics of US T & P.
• Systematic error correction will be done for each tool separately, and the regression coefficients will be determined under a three-years-out Cross-Validation approach.
• If the weight assigned to a model is near zero, then the consolidation will be redone without that model and the originators of the model will be notified.
Future Strategy of the NMME (Option 1) contd• If there is additional skill to the CFS, the participating center
will work with EMC in porting the frozen model and data assimilation systems to NCEP. Random hindcasts will be made using this system to ensure that there is reproducibility of the results on the NCEP operational/development computers.
• This is important for the application of calibration and weights for consolidation (which are derived from the retrospective forecasts) to real time forecasts which will be made on the NCEP computer.
• This process will involve computer and human resources, which will need to be funded.
Future Strategy of the NMME (Option 1) contd• If the results are reproducible, then the participating
center will complete the rest of the 10 calendar months with the exact same frozen system.
• They will make the same datasets available to the CTB consolidation team, for further evaluation to determine if skill is being added to the CFS for these months.
• If there is additional skill to the CFS, then EMC will work with NCO (computer operations) to implement the NMME into operations at NCEP.
Future Strategy of the NMME (Option 2)• With 100% investment, including a dedicated NOAA
mainframe computer, with a huge number of nodes, disk space, tapes, and a support staff on the order of 20 to 25 full time personnel, CTB could support porting all participating systems (including models and assimilation systems) to the CTB computer.
• CTB would then proceed with making the hindcasts as described above. This could be done with individual models or with combinations of model physics and numerics within the Earth System Modeling Framework (ESMF) to form a true NMME.
Future Strategy of the NMME (Option 2) contd
The seven year cycle• Start CFS Reanalysis at same time as porting all the other
models (GFDL, NCAR, NASA) to the NOAA NMME computer.
• Make sure Reanalysis (CFS) initial conditions can be used in all these models (GFDL, NCAR, NASA), or make Reanalysis for each model system.
• This implies that improvements that have been made to these models and data assimilation systems every 7 years will be utilized in the NMME, in a continuous fashion, hopefully leading to an increase in skill of the NMME every time.
Future Strategy of the NMME (Option 2) contdThe seven year cycle
• Make hindcasts (2 months, mid-May and mid-November release) for each model system separately or in an optimal combination of different physics parameterizations and/or different dynamic cores
• Evaluate to see if NMME has additional skill to the CFS (CTB Consolidation team)
• If additonal skill exists, make hindcasts for remaining 10 calendar months
• Redo evaluation for additional skill to the CFS for these months
• If additional skill exists, then transition to NCEP operations.
Why the Need for Long Hindcast Datafor S/I prediction ?
Huug van den Dool (CPC)
and Suranjana Saha (EMC)National Centers for Environmental Prediction,
NWS/NOAA/DOC
“Statistical correction of today’s numerical forecasts using a long set
of reforecasts (hindcasts) dramatically
improves its forecast skill”
(Hamill, Whitaker and Mullen 2006, BAMS)
(Single model, multi-membered ensemble, NWP)
Data Used for this study• DEMETER : 7 European Coupled GCMs• CFS• 1981-2001 (21 years of hindcasts)• 4 initial months• Monthly Data
Focus:
Skill in Tmp2m over US lead 3
February forecasts from November initial conditions
Explained Variance (%) for US monthly T2m
Feb 1981-2001; lead 3 (Nov starts)Verification : Climate Division data)
SE
corr
CFS ECM PLA MET FRA
UKM INGV LOD CER
0 years
2.1 1.2 0.0 0.0 0.0 0.4 0.2 0.0
8years
4.3 7.1 1.4 1.4 7.5 1.4 0.4 2.2
21years
11.2(0.33 corr)
8.0 0.4 0.4 8.6 0.6 0.1 0.5
The 3 best models are operational models
Skill Estimates for 8-years are not stable
Best skill obtained for all 21 years
SE
corrCFS ECM PLA MET
FRAUKMO
INGV LODYC
CERF
MME (EW)
0 years
2.1 1.2 0.0 0.0 0.0 0.4 0.2 0.0 0.2
8years
4.3 7.1 1.4 1.4 7.5 1.4 0.4 2.2 3.8
21years
(all)
11.2 8.0 0.4 0.4 8.6 0.6 0.1 0.5 2.0
Explained Variance (%) for US monthly T2m
Feb 1981-2001; lead 3 (Nov starts)Verification : Climate Division data)
Skill of MME with equal weights for ALL models is far less than the skill of the 3 best models
Explained Variance (%) for US monthly T2m
Feb 1981-2001; lead 3 (Nov starts)Verification : Climate Division data)
SE
corr
CFS ECM PLA MET FRA
UKM ING LOD CER MME (EW)
MME3
0 years
2.1%
1.2 0.0 0.0 0.0 0.4 0.2 0.0 0.2 0.9
8
years
4.3 7.1 1.4 1.4 7.5 1.4 0.4 2.2 3.8 8.6
21
years
11.2 8.0 0.4 0.4 8.6 0.6 0.1 0.5 2.0 17.0
(corr=
51%)
Skill of MME with equal weights for 3 best models improves with 21 years
Need more years to determine the SE where/when
interannual stand dev is large
8-year 21-year
8yr minus 21yr
CFS US TMP2M LEAD 3 FORECAST VERIFYING IN FEB
8-year 21-year
MME, ALL MODELS, EQUAL WEIGHT
US TMP2M LEAD 3 FORECAST VERIFYING IN FEB
8yr minus 21yr
Skill of MME is very much less
than CFS when all models are
included
MME, 3 BEST MODELS, EQUAL WEIGHT
US TMP2M LEAD 3 FORECAST VERIFYING IN FEB
8-year 21-year
8yr minus 21yr8yr minus 21-year
Skill of MME for 3 best models is the most for 21 years
Explained Variance (%) for US monthly PrecipFeb 1981-2001; lead 3 (Nov starts)Verification : Climate Division data)
SE
corrCFS ECM PLA MET
FRAUKM INGV LOD CER
0 years
0.7 0.4 1.1 1.1 NA 1.1 0.1 0.9
8years
5.1 0.6 1.9 1.6 NA 1.8 0.1 1.5
21years
7.5 1.0 3.0 2.2 NA 3.3 0.1 1.9
Not much Skill in any of the models for US precipitation
Skill in SST Anomaly Prediction for Nino-3.4[DJF 97/98 to AMJ 04]
5-member CFS reforecasts
50
60
70
80
90
100
1 2 3 4 5 6
Forecast Lead [Month]
An
om
aly
Co
rrel
atio
n [
%]
CFS
CMP
CCA
CA
MAR
CON
Skill in SST Anomaly Prediction for Nino-3.4 [DJF 97/98 to AMJ 04]
50
60
70
80
90
100
1 2 3 4 5 6
Forecast Lead [Month]
An
om
aly
Co
rre
lati
on
[%
]
CFSCMPCCACAMRKCON
15-member CFS reforecasts
Skill in SST Anomaly Prediction for Nino-3.4[DJF 81/82 to AMJ 04]
5-member CFS reforecasts
50
60
70
80
90
100
1 2 3 4 5 6
Forecast Lead [Month]
An
om
aly
Co
rrel
atio
n [
%]
CFS
CMP
CCA
CA
MAR
Skill in SST Anomaly Prediction for Nino-3.4 [DJF 81/82 to AMJ 04]
50
60
70
80
90
100
1 2 3 4 5 6
Forecast Lead [Month]
An
om
aly
Co
rre
lati
on
[%
]CFSCMPCCACAMRK
15-member CFS reforecasts
“ONLY 1 BIG ENSO EVENT”
ALL ENSO EVENTS
CONCLUSIONS• Without SEC (systematic error correction) there is no skill
by any method (for presumably the best month: Feb)
• With SEC (1st moment only), there is skill by only a few models (5 out of 8 are still useless)
• MME not good when quality of models varies too much
• MME3 works well, when using just three good models
CONCLUSIONS (contd)
• CFS improves the most from extensive hindcasts (21 years noticeably better than 8) and has the most skill. Other models have far less skill with all years included.
• Cross validation (CV) is problematic (leave 3 years out when doing 8 year based SEC?)
• Need more years to determine the SEC where/when the inter annual standard deviation is large