4th WGNE Workshop on Systematic Errors in Weather and Climate Models

Met Office, Exeter, UK, April 15-19, 2013

Fanglin Yang and Glenn White

NCEP-EMC Global Modeling Branch

Travel Report

Travel support for Fanglin Yang by NCEP EMC and IMSG is gratefully appreciated.

Exeter Cathedral

Met Office

Years built 1112-1400

3

Outline

• Workshop Overview ( will skip since Glenn has given a very comprehensive review).

• Fanglin’s workshop presentation

• Points of Interest

4

Workshop Overview The workshop was aimed to understand the nature and cause of errors in models used for weather and

climate prediction. Most of the presentations were focused on the comparison of CIMP5 models, on the progress of transpose-AIMP project, on the diagnosis of NWP model errors, and on metrics of NWP model verification.

Interactions, and results that can potentially benefit NCEP GFS1. Beth Ebert from CAWCR (Australia), representing JWGFVR, presented the Progress and Challenges in

NWP Forecast Verification. She mentioned there is a second-phase intercomparison of deterministic NWP forecast verification led by WMO. I discussed with her how NCEP can get involved in this project (http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html). She indicated that Yuejian Zhu from EMC ensemble group is representing NCEP in JWGFVR. I also discussed with her how to better use observations instead of analyses to verify NWP forecasts.

2. Discussed with Roger Sanders in Met Office on the use of AMSU-A data in Met Office data assimilation. He mentioned that Met Office is only using AMSU-A channel-4 data over ocean. They found that including AMSU-A channels (1-3 and 15) in DA gave worse forecasts. The usefulness of these channels is also seasonal dependent. The GFS DA uses all AMSU-A channels. My poster presentation showed that by doubling the observational errors of these channels over 40S-80S latitudes GFS forecast skills in the SH was improved. We may need to consider further limiting the usage of AMSU-A data in our DA system.

3. Jane Mulcahy at Met Office presented a poster showing the impact of increasing aerosol complexity in NWP forecasts. I communicated with her how aerosol effect is treated in current operational GFS, and how an offline aerosol model (GOCART) is set up at NCEP to forecast dusts. One interesting idea coming out of the conversation is to use climatological aerosol to parameterize the indirect effect of aerosol on cloud and radiation in NWP models.

5

4. Hursinski from Colorado evaluated water vapor analyses in a few NWP centers against GPSRO and AIRS retrieval. I noted GFS is an outlier among the models he compared. GFS tends to be too moist in the middle to upper atmosphere (see Fig. 1 at the end of this report). I asked if he had made the same comparison using forecasts as well to understand the root cause. He indicated that he will update the comparison using forecast data and will share with us his result.

5. Bushell from Met Office showed in a poster the impact of Kim scheme of convective gravity-wave parameterization on QBO in Met Office unified model (test version). I discussed with him the performance and ease for NWP implementation of a few parameterizations of non-orographic gravity waves available in the community written, for instance, by Alexander and Baldwin (used in UIUC GCM), by Kim et al. (used in KMA), by Hines (used in GFDL), and by Warner and McIntyr (used in ECMWF). He indicated the Kim and Warner and McIntyr are probably the easier ones to use.

6. Discussed with Shaocheng Xie from PCMDI on how to use DOE ARM ground observations for verifying GFS cloud and radiation, and for guiding model physics development. Dr. Xie produced a set of Best Estimate of Radiation and Clouds from a long record of ARM observations at multiple stations. This dataset has long been widely used in climate modeling community. I will contact Dr. Xie to obtain this dataset for our use. Dr. Xie also showed his interest in using our global model verification tool for their Cloud-Associated Parameterizations Testbed project.

A few interesting observations1. Met Office improved their seasonal forecasts dramatically, mostly because of an increase of NAO forecast skill.2. Bill Large from NCAR stated that in coupled A-O GCMs surface wind stress in the mid-latitudes from AGCM is either

too weak or does not have the right wind direction. It has to be enlarged to enhance ocean mixing to deepen ocean mix-layer depth.

3. Disuke Hotta from JMA showed how to use diurnal tide to validate NWP models.4. Yi Huang from Monash University showed that almost all NWP models have too less super cooled cloud water in

the southern high latitudes.5. Mark Rodwell from ECMWF showed how to use ISCCP, CERES and CLIPSO data to verify NWP model clouds and

radiation. We should also think how to use these data to verify GFS.

6

Outline

• Workshop Overview

• Fanglin’s workshop presentation

• Points of Interest

8

JJA-2012 Temp Differences between GFS and ECMWF, Analyses

GFS analysis is much warmer than ECMWF analysis in the lower troposphere at the Southern mid to high latitudes. The largest difference is found in the lower troposphere above the boundary layer.

Zonal Mean Temperature

850 hPa Temp

Austral Winter

9

Differences of 850hPa temperature between GFS 00Z-cycle analyses and guess. The latter are 18Z GDAS 6-hour forecasts.In general, the analyses are 0.1~0.5 degrees warmer than the guess in the Southern mid to high latitudes.

JJA-2012 DJF 2012/13

GFS A - G: 850hPa Temperature

10

• Compared to ECMWF, GFS cloud cover is about 7% less in the NH and 13% in the SH.

• GFS has a fast spin-down of cloud cover in the first 24 to 48 hours in all regions.

Mean Cloud CoverCases of February 2013

NH SH

Tropics

11

Findings:

• GFS analysis is one to two degrees warmer than ECMWF analysis in the Southern lower troposphere at middle to high latitudes in all seasons of the year.

• GFS analysis is warmer than its guess (GDAS 6-hour forecast) in the same region.

• GFS global cloud cover is about 10% less than ECMWF cloud cover. The deficit is the largest in the Southern Hemisphere (~13%).

• GFS cloud cover has a quick spin-down in the first 24 to 48 hours of forecast. The spin-down is primarily caused by the reduction of high clouds.

12

Sensitivity Experiment I: expgsi

Double the observation errors in GSI for AMSU-A Channels 1-4 and 15 (and ATMS equivalents if nchanl=22) in the Southern Hemisphere between the latitudes of 40oS and 80oS.

The model used for the experiment is GFS T382L64 with 3D-VAR GSI . A new control run at this configuration was made for clean comparison. Both runs were carried out on ZEUS.

1 June 2012 ~ 31 August 2012

13

Analysis Difference in Temperature: expgsi - control

Zonal Mean Temperature

850 hPa Temp

By inflating the observation errors of AMSU-A channels over the Southern mid to high latitudes we reduced the analysis temperature increment by one to two degrees. This change reduced the analysis temprature difference between GFS and ECMWF.

14

GFS Forecast Skills : SH 500-hPa Height Anomaly Correlation

http://www.emc.ncep.noaa.gov/gmb/wx24fy/wgne/prexpgsi/

Day-5 AC increased by ~0.01 Increases of AC are significant in the first 5 days, except for day 4.

No significant changes were found in NH scores, see

15

SH Wind RMSE

http://www.emc.ncep.noaa.gov/gmb/wx24fy/wgne/prexpgsi/

Significantly reduced SH wind and temperature RMSE at all levels and at almost all forecast length.

No significant impact was found in NH and the tropics.

SH Temperature RMSE

16

Roger Sanders at Met Office commented on the use of AMSU-A data in Met Office data assimilation. He mentioned that Met Office is only using AMSU-A channel-4 data over ocean. They found that including AMSU-A channels (1-3 and 15) in DA gave worse forecasts. The usefulness of these channels is also seasonal dependent.

The GFS DA uses all AMSU-A channels. My poster presentation showed that by doubling the observational errors of these channels over 40S-80S latitudes GFS forecast skills in the SH were improved. The DA group is experimenting how to better make use of AMSU-A channels in our DA system.

17

Outline

• Workshop Overview

• Fanglin’s workshop presentation

• Points of Interest

18

Impacts of increasing aerosol complexity in short and medium-short-range NWP forecasts

Jane Mulcahy, David Walters and Sean MiltonMet Office

• Investigate the potential benefits of including more complex aerosol representations and their direct and indirect effects in short and medium-range global forecasts. The different aerosol representations evaluated include three dimensional monthly mean speciated aerosol climatologies (current operational global NWP configuration), fully prognostic aerosols modelled using the CLASSIC aerosol scheme and finally, initialised aerosols using assimilated aerosol fields from the GEMS (Global and regional Earth system Monitoring using satellite and in-situ data) project.

CNTRL: “Historical” aerosol climatology (Cusack et al. 1998)CLIM: Monthly mean speciated climatologies derived from HadGEM2

simulations, Direct Effect Only.AER_dir: Full prognostic CLASSIC aerosol, Direct Effect OnlyAER_dir_indir: Full prognostic CLASSIC aerosol, Direct & Indirect EffectsINIT_dir: Initialised CLASSIC aerosol, Direct Effect OnlyINIT_dir_indir: Initialised CLASSIC aerosol, Direct & Indirect Effects

Experiments Using Met Office Unified Model

19

Impacts of increasing aerosol complexity in short and medium-short-range NWP forecasts

Key initial findings:• AER_DIR_INDIR simulation reduces the RMSE of a number of key forecast metrics

including T850, Z500 in the tropics and NH as well as NH PMSL after day 10. Including the direct effect only leads to an increase in RMSE in most cases.

• Small degradation in the mean error in temperature and height in mid-troposphere but improves bias at surface and upper levels.

• Aerosol climatology with indirect effects has a neutral or positive impact on most of the standard NWP verification metrics.

T 850hPa NH Mean Errors Z 500hPa NH Mean Errors

20

Comparisons of Observed & Modeled Water Vapor Variability to Assess Hydrological Cycle in Models

E. R. Kursinski and A. L. Kursinski Moog Broad Reach, Golden, CO, USA

GPS RO & AIRS HUMIDITY COMPARISON with ANALYSES• ECMWF & AIRS are quite similar • NCEP differs the most • GPSRO reveals highest % of extremely wet & dry air • Moisture analyses stay close to NWP model’s moisture climatology • Improving the assimilation impact requires that the NWP model moisture climatology be close to

the observed climatology

346 hPa 547hPa

Standard NWP verification

21

1st WGNE meeting

1960 now1970 19901980 201020001950

Mean error, RMSE,

S1 score

500 mb and 1000 mb

Anomaly correlation

Multiple vertical levels

WMO Lead Centre for Ensemble

Verification

Surface

Standard verification WMO CBS-

EXT(85)

WWRP/WGNE Joint Verification Working Group

WMO Lead Centres for Long Range Forecast

VerificationWMO Lead Centre for

Deterministic Verification

Progress and challenges in NWP forecast verificationBeth Ebert: The Center for Australian Weather and Climate Research

http://www.wmo.int/pages/prog/arep/wwrp/new/Forecast_Verification.html

It seems NCEP-EMC’s participation in WMO verification activities is limited.

Spatial Verification Methods Intercomparison

• Conclusions from 1st phase– Different methods have different strengths– All address bias

• 2nd phase (ongoing)– Complex terrain – MAP D-PHASE / COPS dataset– Wind and precipitation, timing errors 22

Category Scales with skill

Location errors

Intensity errors

Structure errors

Occurrence (hits, misses, false alarms)

Traditional (gridpoint) × × × Neighbourhood × × Scale separation × × Features based × Deformation × × ×

Uncertainty in observations

• As models improve, can no longer ignore observation error!

• Remove observation bias errors where possible• Effects of random obs error on verification

– “Noise” leads to poorer scores for deterministic forecasts– Ensemble forecasts have poorer reliability & ROC

• What can we do?– Error bars in scatter plots– Quantitative reference to “gold standard”

• Correct for systematic error in observations• RMSE – Ciach & Krajewski (Adv. Water Res.,1999)• Categorical scores – Briggs et al. (MWR, 2005), Bowler (MWR, 2006)

– Multiple observation sources

23

24

Examples of

How to Use Observations (Clouds and Radiation) to Aid Model Development

25

The ‘too few, too bright’ tropical low-level cloud problem in CMIP5 models.C. Nam*, S. Bony, J.L. Dufresne & H. ChepferLaboratoire de Météorologie Dynamique LMD/IPSL, Paris*Now at the University of Leipzig

CMIP5 Models & ObservationsCMIP5: AMIP experiments from 06/2006 – 12/2008.– Observations: Combine active and passive satellite instruments tounderstand the vertical structure of multi-layered clouds.– CALIPSO (GOCCP): Total/High/Mid/Low & 3D cloud fraction.– Parasol Reflectances– CERES (EBAF): Cloud Radiative Effect– ERA-Interim Re-analysis: Large scale environmental properties.

26

Cloud Radiative Effects Tropics (30oN to 30oS):

Longwave CRE: modelsare positively biased (27Wm-2 CERES).

Shortwave CRE: modelsare positively biased(-46 Wm-2 CERES).

LW CRE shows better spatial variability & correlation withobservations than SW CRE

27

Cloud Cover

Standardized Deviations (Normalized)

Tropics (30oN to 30oS):

Low-level: All but onemodel is negativelybiased (30%CALIPSO).

Mid-level: models bothpositively andnegatively biased(13%CALIPSO).

High-level: modelspositively biased(34%CALIPSO).

28

SW Cloud Radiative Effect

Low Cloud Cover

• Models generally have much stronger CRE, for a given cloud cover, than found in CERES observations.

• Over-estimate optical thickness of low-level clouds, particularly in shallow cumulus regimes.

29

Comments: similar type of analyses can, and should, be carried out for the GFS to better understand model errors in clouds and radiation, and to guide the development of GFS physical parameterizations.

New Satellite Datasets for Diagnosing Model Errors

UK Met Office ECMWF MPI-M MeteoFrance

Roger SaundersMet Office

and ESA CCI

“Climate Modelling User Group“

© Crown copyright Met Office

Model development: is the model improving? Look at SW radiation.

Model 1-CERES

• Development of new Hadley Centre climate model• Shortwave radiation at TOA• ISCCP model improves• CERES model gets worse

Model 2-CERES

Model 1-ISCCP Model 2-ISCCP

• Development of new Hadley Centre climate model• Shortwave radiation at TOA• ISCCP model improves• CERES model gets worse

Climate model assessment

• Attempt to use multiple obs data sets to assess improvement/deterioration in model• Uses range of available observations as “uncertainty”• Process-by-process: here consider clouds, radiation and precipitation•This is the process gone through to assess a new version of a climate model

Better Worse Neutral

Cloud, Radiation and Precip Parameters assessed

Welcome to the Met Office

© Crown copyright Met Office

Andy Brown

Director of Science and WGNE co-chair

© Crown copyright Met Office

Commonality of errors cross time and space scales:example of missing mid-level cloud

Bodas-Salcedo et al Illingworth et al.

Climate model versus ISCCP Mesoscale NWP versus CloudNet

Systematic Biases in Microphysics: observations and parametrization

Ian Boutle, Steven Abel, Peter Hill, Cyril Morcrette

QJ Roy Met Soc, 2013, doi:10.1002/qj.2140

Currently assume that cloud is homogeneous for microphysics calculations

Climate grid-box (~125km)NWP grid-box (~25km)

GOES SW Satellite Image, SE Pacific during VOCALS

Parametrizing f (cloud)

• Continued increase at all scales

• Strongest at smallest scales (1/3 power law)

PDF Shape

• Gamma or log-normal distributions both provide good descriptions of the cloud and rain variability

• Gamma is slightly better, but the maths is more tractable with log-normal!

Cloud Rain

This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC

LLNL-PRES-605075

On the correspondence between short- and long-timescale systematic errors in the Transpose-AMIP II and CMIP5/AMIP

4th WGNE workshop on systematic errors in weather and climate models, April 15-19, 2013, Met Office, Exeter, UK

Hsi-Yen MaProgram for Climate Model Diagnosis and Intercomparison

Lawrence Livermore National Laboratory

withS. Xie (LLNL), K. Williams (Met Office), J. S. Boyle (LLNL), S. Bony (IPSL), H. Douville

(Metro France), S. Fermepin (IPSL), S. A. Klein (LLNL), B. Medeiros (NCAR), S. Tyteca (Metro France), M. Watanabe (AORI, U of Tokyo), and D. Williamson (NCAR)

Total cloud fraction biases (ISCCP simulator)

Strong similarity between hindcast and climate bias patterns

Biases in net shortwave at TOA (SWAbs)

Strong correlation to cloud biases Negative biases may imply poor simulations of cloud condensate