Predictability & Prediction of Seasonal Climate over North America
description
Transcript of Predictability & Prediction of Seasonal Climate over North America
![Page 1: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/1.jpg)
23-27 Oct. 2006 NOAA 31st Annual Climate Diagnostics and Prediction Workshop
Predictability & Prediction of Predictability & Prediction of Seasonal Climate Seasonal Climate over North Americaover North America
Lisa GoddardLisa Goddard, Simon Mason, Ben Kirtman, , Simon Mason, Ben Kirtman, Kelly Redmond, Randy Koster, Wayne Higgins, Kelly Redmond, Randy Koster, Wayne Higgins,
Marty Hoerling, Alex Hall, Jerry Meehl, Marty Hoerling, Alex Hall, Jerry Meehl, Tom Delworth, Nate Mantua, Gavin SchmidtTom Delworth, Nate Mantua, Gavin Schmidt
(US CLIVAR PPAI Panel)(US CLIVAR PPAI Panel)
![Page 2: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/2.jpg)
Time Series of Prediction Skill
operational
Potential predictability
Research forecasts
(1) Understand the limit of predictability
(2) Identify conditional predictability (e.g. state of ENSO or Indian Ocean)
(3) Document the expected skill to judge potential utility of the information for decision support
(4) Set a baseline for testing improvements to prediction tools and methodologies
(5) Set a target for real-time predictions.
(Courtesy of Arun Kumar & Ants Leetmaa)
![Page 3: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/3.jpg)
Real-time prediction skill…North America, 1-month lead, seasonal terrestrial climate
• Provide a template for verification- What are the best metrics? Best for who?
- Pros & cons of current metrics- Can we capture important aspects of variability (e.g. trends, drought periods)?
• Estimate skill of real-time forecasts- How predictable is N. America climate?
- Benefit of multi-model ensembling?
• Provide baseline against which we can judge future advances- How best to archive/document for future comparison?
- Are we missing something? (i.e. statistical models)
![Page 4: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/4.jpg)
Forecast DataDynamical models (single):• CCCma – Canadian Centre for Climate Modeling and Analysis• KMA – Korean Meteorological Agency• MGO- Main Geophysical Observatory, Russia• NASA/GMAO-National Aeronautics and Space Administration, USA• RPN – Canadian Meteorological Centre• ECHAM4.5 – MPI (run at IRI)• CCM3.6 – NCAR (run at IRI)• ECMWF –European Center for Medium Range Weather Forecasts• Meteo-France – Meteorological Service, France• LODYC- Laboratoire d'Océanographie Dynamique et de Climatologie,
France• Met Office – UK Meteorological Office• MPI – Max Planc Institute for Meteorology, Germany• CERFACS – European Centre for Research and Advanced Training in
Scientific Computing, France• INGV-Instituto Nazionale di Geofisica e Vulcanolgia, Italy• NOAA-CFS – National Oceanic Atmospheric Administration, USAMulti-Model of dynamical models (simple average)
Statistical models (from CPC): CCA, OCN (others?)Multi-Model of dynamical + statistical models
![Page 5: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/5.jpg)
Model NX NY NM L S
CCCma-GCM2 96 48 10 0.5-3.5 Mar1969-Dec2003 by 3
CCCma-GCM3 128 64 10 0.5-3.5 Mar1969-Dec2003 by 3
KMA 144 73 6 2.5-8.5 Jan1979-Dec2002
MGO 144 73 6 0.5-3.5 Nov1978-Nov2000 by 3
NASA-GMAO 144 90 6 1.5-3.5 Feb1993-Nov2002 by 3
RPM 192 96 10 0.5-3.5 Mar1969-Dec2000 by 3
ECHAM4.5 128 64 24 0.5-6.5 Jan1958-Dec2002
CCM3.6 128 64 24 0.5-6.5 Jan1958-Dec2002
ECMWF 144 71 9 0.5-5.5 Feb1958-Nov2001 by 3
Meteo-France 144 71 9 0.5-5.5 Feb1958-Nov2001 by 3
LODYC 144 71 9 0.5-5.5 Feb1974-Nov2001 by 3
MetOffice 144 71 9 0.5-5.5 Feb1959-Nov2001 by 3
MPI 144 71 9 0.5-5.5 Feb1969-Nov2001 by 3
CERFACS 144 71 9 0.5-5.5 Feb1980-Nov2001 by 3
INGV 144 71 9 0.5-5.5 Feb1973-Nov2001 by 3
CFS 192 94 15 0.5-8.5 Jan1981-Dec2003
Forecast Data
![Page 6: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/6.jpg)
Model NX NY NM L S
CCCma-GCM2 96 48 10 0.5-3.5 Mar1969-Dec2003 by 3
CCCma-GCM3 128 64 10 0.5-3.5 Mar1969-Dec2003 by 3
KMA 144 73 6 2.5-8.5 Jan1979-Dec2002
MGO 144 73 6 0.5-3.5 Nov1978-Nov2000 by 3
NASA-GMAO 144 90 6 1.5-3.5 Feb1993-Nov2002 by 3
RPM 192 96 10 0.5-3.5 Mar1969-Dec2000 by 3
ECHAM4.5 128 64 24 0.5-6.5 Jan1958-Dec2002
CCM3.6 128 64 24 0.5-6.5 Jan1958-Dec2002
ECMWF 144 71 9 0.5-5.5 Feb1958-Nov2001 by 3
Meteo-France 144 71 9 0.5-5.5 Feb1958-Nov2001 by 3
LODYC 144 71 9 0.5-5.5 Feb1974-Nov2001 by 3
MetOffice 144 71 9 0.5-5.5 Feb1959-Nov2001 by 3
MPI 144 71 9 0.5-5.5 Feb1969-Nov2001 by 3
CERFACS 144 71 9 0.5-5.5 Feb1980-Nov2001 by 3
INGV 144 71 9 0.5-5.5 Feb1973-Nov2001 by 3
CFS 192 94 15 0.5-8.5 Jan1981-Dec2003
Forecast Data: JJA & DJF (1981-2001)
![Page 7: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/7.jpg)
Verification Data & Metrics
OBSERVATIONAL DATA: 2.5x2.5 deg• 2m T: CRU-TSv2.0 (1901-2002)• Precipitation: CMAP (1979-2004)
VERIFICATION MEASURESMetrics consistent with WMO - SVS for LRF (Standardised Verification
System for Long Range Forecasts) • Deterministic information :
- MSE & its decomposition - correlation, mean bias, & variance ratio
• Probabilistic information: - Reliability diagrams, regionally accumulated- ROC areas for individual grid boxes
![Page 8: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/8.jpg)
Mean Squared Error
![Page 9: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/9.jpg)
Mean Squared Error
Pro:* Gives some estimate of uncertainty in forecast
(i.e. RMSE).
Con:* Can not infer frequency of large errors unless
precise distributional assumptions are met.
Recommendation:* Perhaps simple graph or table showing
frequency of errors of different magnitudes would be appropriate.
![Page 10: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/10.jpg)
Correlation : TemperatureDJF 1981-2001
![Page 11: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/11.jpg)
Correlation : TemperatureJJA 1981-2001
![Page 12: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/12.jpg)
Correlation : PrecipitationDJF 1981-2001
![Page 13: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/13.jpg)
Correlation : PrecipitationJJA 1981-2001
![Page 14: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/14.jpg)
Correlation
Pros:* Commonly used; familiar* Gives simple overview of where models are
likely to have skill or not
Con:* Merely measure of association, not of forecast
accuracy
Recommendation:* Avoid deterministic metrics
![Page 15: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/15.jpg)
Event occurs? Warning given?
Hit Yes Yes
Miss Yes No
False Alarm No Yes
Correct Rejection No No
FORECASTS
OBSERVATIONS Yes No
Yes Hit Miss
No False alarm Correct Rejection
![Page 16: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/16.jpg)
Example
Ensemble forecasts of above-median March – May rainfall over north-eastern Brazil
FORECASTS (80%)
OBSERVATIONS Wet Not wet Total
Wet 5 2 7
Not wet 1 7 8
Total 6 9 15
1
false-alarm rate 13%8
5
hit rate 71%7
![Page 17: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/17.jpg)
![Page 18: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/18.jpg)
ROC Areas : DJF TemperatureBELOW-NORMAL
0.50.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
![Page 19: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/19.jpg)
ROC Areas : DJF TemperatureABOVE-NORMAL
0.50.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
![Page 20: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/20.jpg)
ROC Areas : JJA TemperatureABOVE-NORMAL
0.50.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
![Page 21: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/21.jpg)
ROC Areas : JJA TemperatureBELOW-NORMAL
0.50.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
![Page 22: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/22.jpg)
ROC Areas : DJF PrecipitationABOVE-NORMAL
0.50.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
![Page 23: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/23.jpg)
ROC Areas : DJF PrecipitationBELOW-NORMAL
0.50.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
![Page 24: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/24.jpg)
ROC Areas : JJA PrecipitationABOVE-NORMAL
0.50.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
![Page 25: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/25.jpg)
ROC Areas : JJA PrecipitationBELOW-NORMAL
0.50.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
![Page 26: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/26.jpg)
ROC Areas
Pros:
* Can treat probabilistic forecasts
* Can be provided point-wise
* Can distinguish ‘asymmetric’ skill
Cons:
* Fails to address reliability
![Page 27: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/27.jpg)
RELIABILITY
![Page 28: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/28.jpg)
RELIABILITY
![Page 29: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/29.jpg)
Reliability
Pros:
* Treats probabilistic forecasts
* Relatively easy to interpret
* Provides most relevant information on usability of forecast information over time
Cons:
* Difficult to provide for individual grid points, especially for short time samples
![Page 30: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/30.jpg)
Temperature Trends over North America%
-Are
a C
over
ed b
y “A
bove
-Nor
mal
”
![Page 31: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/31.jpg)
Temperature Trends over North America%
-Are
a C
over
ed b
y “A
bove
-Nor
mal
”
![Page 32: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/32.jpg)
Observed Precipitation over North America1998-2001
Anomalies relative to1981-1997
Percent difference relative to 1981-1997
1 2 3 4 1 2 3 4
JJA DJF
Frequency (# years out of 4)for precipitation in BN category
![Page 33: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/33.jpg)
Frequency of Below-Normal PrecipitationJJA 1998-2001
1 in 4
2 in 4
3 in 4
4 in 4
OBSERVATIONS
![Page 34: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/34.jpg)
Frequency of Below-Normal PrecipitationDJF 1998-2001
1 in 4
2 in 4
3 in 4
4 in 4
OBSERVATIONS
![Page 35: Predictability & Prediction of Seasonal Climate over North America](https://reader036.fdocuments.us/reader036/viewer/2022081519/568144d4550346895db19f5f/html5/thumbnails/35.jpg)
Summary
• What’s an appropriate template?- Skill metrics should be flexible (i.e. user defined “events”, categories, thresholds)
- Probabilistic forecasts must be treated probabilistically!!!
• How are we doing?- Could be better. Encouraging performance estimates by some measures, but inadequate performance on important aspects of climate variability. - Missing elements necessary for seasonal prediction?
• Baseline??