Update on Dropout Related COPC Action Items Presented by Dr. Bradley Ballish Co-Chair JAG/ODAA

Update on DropoutRelated COPC Action Items

Presented byDr. Bradley Ballish

Co-Chair JAG/ODAA

14 May 2009COPC Meeting

NAVOStennis Space Center

May 14, 2009 COPC Dropout Update 2

Outline• Participants and resources

• Recent dropout examples

• COPC Action Item (AI) 2008-1.5

• COPC AI 2008-2.14

• Dropout team findings

• Summary

• Next steps

• Background slides


Participants and Resources• NCEP’s Jordan Alpert• NCEP’s DaNa Carlis• NCEP’s Krishna Kumar• NCEP’s Bradley Ballish• NCEP is investing in an additional

FTE/contractor• NRL’s Rolf Langland• FNMOC’s Chuck Skupniewicz• The NCEP dropout team meets weekly with

Steve Lord and reports quarterly to the NCEP/OD


Recent Dropout Examples

FNMOCDropout GFS

Dropout

To view how the model errors change with time at different levels, use thethe website of NCEP’s DaNa Carlis:http://www.emc.ncep.noaa.gov/gmb/dcarlis/directory, and click onDropout_cases/south_hem/10apr200900z_dropoutSH for the above GFS dropout, see the next slide for differences in 48 hour GFS and ECMWF forecasts, where the forecast differences suggests multiple initial condition errors

http://www.emc.ncep.noaa.gov/gmb/dcarlis/directory


48 Hour GFS forecast errorversus verifyingGSI analysis

48 Hour ECMWFforecast errorversus verifyingECWMF analysis

48 hour GFS and ECMWF forecastdifferences, whichoften show severallarge difference areas in the SH


AI 2008-1.5COPC Action Item 2008-1.5: Develop a

monitoring system to analyze differences between the NCEP and FNMOC global models and the ECMWF global model in real-time and make this real-time system available to OPCs as a daily tool.

Status

• Experts from NCEP, FNMOC and AFWA have had brainstorming meetings and email exchanges

• Initial requirements have been defined


AI 2008-1.5 Draft Requirements

• NCEP and FNMOC will generate internal warnings when their global models have extreme localized differences from ECMWF forecasts on a 1x1 degree grid

• NCEP and FNMOC will develop a real time warning system to show when integrated forecast differences of their global model with the ECMWF (such as height correlations) exceed normal limits

• FNMOC and NCEP will develop graphical tools to allow 24x7 staff to study the divergence in forecasts to show likely analysis problem areas


AI 2008-1.5 Next Steps

• Requirements sign off (6-2009)

• Identification of project manager (7-2009)

• Work schedule completion (8-2009)

- Design TBD

- Construct TBD

- Test TBD

- Deploy TBD


AI 2008-2.14• COPC Action Item 2008-2.14: CSAB will

oversee the work of the “Dropout Team” to continue examining the performance dropout issue and how lessons learned can be applied to numerical model performance improvements. This activity shall include establishing periodic interface with ECMWF to exchange QC methodologies/best practices. Recommendations for QC of data, acquisition of new data sources and any other relevant issues shall be presented to the Spring 2009 COPC

Status• Lessons learned have been documented• The ECMWF has been somewhat responsive

to attempts at coordination but follow through has been sporadic

• Recommend closing this AI


Dropout Team Findings• General aspects related to the ECMWF comparison

– The NCEP GFS can avoid most dropouts when initialized by the ECMWF analysis globally or in select sensitive areas (see background slide 20)

– The ECMWF analysis is 4DVAR, runs at later times with a larger data time window and has a higher resolution model

– The ECMWF observational data QC is more comprehensive than NCEP’s

• For almost all dropouts studied, simple, obvious data QC problems have not been found to explain the dropout– Most dropouts tend to involve subtle analysis differences in sensitive

areas that grow with time in the forecasts– Observational data denial tests, where various data types are

removed (such as satellite winds or select radiances), improve some cases and help identify problems

• For more details, see the two papers presented at the AMS 2009 annual meeting, see: http://ams.confex.com/ams/89annual/techprogram/paper_142644.htm and

http://ams.confex.com/ams/pdfpapers/142649.pdf

http://ams.confex.com/ams/89annual/techprogram/paper_142644.htm

http://ams.confex.com/ams/pdfpapers/142649.pdf


Dropout Team Findings• Specific deficiencies identified

– Analysis: The GSI analysis agrees more with observational outliers (observations with large differences to the model background) than the ECMWF

– Analysis: The GSI and FNMOC analyses have higher heights than the ECMWF analysis possibly due to:

• Satellite radiance bias correction differences• ECMWF thinning of aircraft data which have an

overall warm bias


Dropout Team Findings• Specific deficiencies identified

– Data: The GSI and FNMOC analyses are affected more by the slower speeds of satellite winds due to QC differences

– Data: Tests show that the use of non-radiance data in the GSI analysis sometimes hurts forecast skill in the Southern Hemisphere and is being researched to define the error sources

– Data: OPC weather station dictionaries (locations and elevations) are inconsistent with each other and published WMO tables, impacting analyses and forecasts


Summary• Dropouts continue to be a problem for

forecast skill

• Proposed requirements for AI 2008-1.5 (real time analysis of NCEP and FNMOC differences to the ECMWF) were given, need approval and follow up actions

• The dropout team has documented various causes of dropouts and recommends closing of AI 2008-2.14


Next Steps• Complete monitoring tool work

• Review capabilities of ECMWF observational data base (ODB)

• Continue diagnostic work

• Focus on– Satellite winds– Aircraft temperature data (bias correction a

higher priority for ECMWF as well)– Satellite radiance data


Background Slides

Proposed Plans for NCEP & FNMOC


Proposed Plans• NRL could use adjoints of the

analysis/model to direct the OPCs towards better QC of satellite winds

• NCEP will test bias correction of surface pressure data

• JCSDA could test using satellite radiance data as QC checks on altitudes of satellite winds


Proposed Plans (continued)

• NCEP will consider development of track-checking of marine surface and radiosonde data

• NCEP will work with the NWS TOC to develop a website showing information on bufr data types at the TOC

• JCSDA could develop graphical tools to show integrated satellite radiance data (height and total precipitable water)


Proposed Plans (continued)• NCEP will use baroclinic instability rates

and ensemble tools to help analyze model sensitivity to analysis differences

• AFWA could make reports every 6 months comparing station dictionary metadata entries for the OPCs with recommendations for corrections


Collaborative Dropout Work• FNMOC’s Dr. Rolf Langland and Chuck Skupniewicz

have met with the NCEP staff and shared email exchanges resulting in useful information. Dialog continues

• Dr. Langland’s website http://www.nrlmry.navy.mil/adap-bin/tcs_adap.cgi, with adjoint estimates of how analysis differences impact NOGAPS 24 hour forecasts, is useful in showing sensitive areas in the analysis

• Dr. Langland’s studies of height biases versus the ECMWF and his finding that aircraft data in the Pacific have negative impact based on adjoint estimates are important results

• OPCs have been comparing station metadata dictionaries for weather observing sites and making corrections

http://www.nrlmry.navy.mil/adap-bin/tcs_adap.cgi


Trough in central Pacific shows differences between ECMWF(no dropout) and GFS (had dropout)

Ovrly “patch” boxECM in this areabut GSI elsewhereresulted in 16 point improvement in 5 day 500 hPa AC score


Divergence of Forecast Measures

• Develop a system to warn OPCs when the current divergence (difference) in forecasts exceeds expected limits

• Divergence in forecasts measures should include anomaly correlation scores and RMS differences

• Metrics may be for limited areas such as the Northern Hemisphere, North American, or other theaters of interest

• Metrics should include scores on heights and winds at a few pressure levels and precipitation

• Normal limits could be derived from past periods that excludes poor forecast cases


Actions for Large Forecast Differences

When we detect that comparable forecasts such as the 00Z ECMWF/GFS have abnormally large differences:

• Alert forecasters• Check divergence in other forecasts e.g. 18Z

GFS vs 00Z GFS or FNMOC vs GFS• Analyze forecast maps to find problem areas

in the analysis• Check for possible data problems


Satellite Radiance Observation Plots• Codes have been developed at NCEP to plot

radiance derived temperatures and other observational data differences to the model background or analysis from a given channel for a particular satellite

• These are useful for analyzing dropout cases, but are too time consuming as this involves many satellite types and channels (each with different vertical profiles)

• These tools depend on the accuracy of our radiance bias corrections

• More work is needed to get simpler summaries e.g. height differences at standard levels from all channels integrated together from each satellite


A Detection and Warning System for Problems in Classes of Observational Data

• An operational system will be developed to warn us if any class of data develops a large change in bias or RMS differences versus the model background

• History has shown that classes of data can develop problems, such as US wind profilers from 9 to 11 April 2008, see the next slide. The GFS has a .67 500 hPa AC 5 day score from 18Z 9 April 2008

• Satellite radiances must be included, see the following slide

• Findings of bad data must be shared with other OPCs


RMS Wind Differences to Background for Profiler Data300 to 200 hPa 00Z GDAS April 2008

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12

Day in April 2008

RM

S D

iffer

ence

in m

/sec

An error caused USwind profilers to haveeast longitudes or wronglocations in China


GOES 11 Channels 1-4Contribution to GSI Penalty Function

Channels 1-3 show jumparound 12Z 9 Apr 2009


• ECM runs (blue) are a good representation for ECMWF analysis• OVRLY runs (green) with ECM psuedo-obs over the Central Pacific drastically improve two October 2007 dropout cases (102200 & 102212).

5 Day Anomaly Correlation Scores at 500 hPa for Dropout CasesECM Performs Better than GFS (NH)

2007-2008

0.4

0.5

0.6

0.7

0.8

0.9

1

2007102112 2007102200 2007102212 2007102312 2008021712 2008030112 2008030400 2008030412

Initialization Date

50

0 m

b A

no

ma

ly C

orr

ela

tio

n

GFS

ECMWF

ECM

OVRLY


0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

2008011100 2008011212 2008020300 2008030312 2008031800 2008031812 2008042512 2008042600 2008052200 2008062512

Initial Date

50

0 m

b A

no

ma

ly C

orr

ela

tio

nGFS

ECMWF

ECM

CNTRL

InterpECMGES

ECM runs (blue) in the SH do almost as well as ECMWFCNTRL runs (green) improve upon GFS scores 9 of 10 times but only alleviates about half of the dropoutsInterpECMGES runs (purple) improve 4 of 10 cases over the CNTRL run

SH 5-day 500 hPa Anomaly Correlation Scores


Comparison of how the GSI and ECM Analyses fit Observations

• Statistics are made on how the GSI and ECM analyses fit the observations for different observation types as a function of pressure and analysis differences, for different regions

• This study shows that each analysis fits certain observation types differently, but does not conclude which performs better

• These analysis fits to data need to be rerun with higher resolution ECMWF analyses

30

GFS vs. ECM Fits to Radiosondes

Percent Better Draw for GSI and ECM Analyses for Radiosonde Temperatures with 2-4 Degree Analysis Difs 12Z April 2008

0

10

20

30

40

50

60

70

80

90

SFC-900

900-800 800-700 700-600 600-500 500-400 400-300 300-200 200-100 100-000

Pressure Category in hPa

Be

tte

r D

raw

in %

GSI

ECM

Percent Better Draw for GSI and ECM Analyses for all Radiosonde Temperatures 12Z April 2008

0

10

20

30

40

50

60

70

SFC-900

900-800 800-700 700-600 600-500 500-400 400-300 300-200 200-100 100-000


Be

tte

r D

raw

in %

GSI

ECM

Percent Better Draw for GDAS and ECM Analyses for all Radiosonde Winds 12Z April 2008

0

10

20

30

40

50

60

70

80

SFC-900

900-800 800-700 700-600 600-500 500-400 400-300 300-200 200-100 100-000


Be

tte

r D

raw

in %

GSI

ECM

Percent Better Draw for GDAS and ECM Analyses for Radiosonde Winds with 5-10 m/sec Analysis Difs 12Z April 2008

0

10

20

30

40

50

60

70

80

90

100

SFC-900

900-800

800-700

700-600

600-500

500-400

400-300

300-200

200-100

100-000


Be

tte

r D

raw

in %

GSI

ECM

31

Satellite Wind Speed Biases OBS-ANL Apr 2008 12Z ECM (Red), GSI (Blue) m/sec

JMA Type 245

-1.2 -0.7 -0.2 0.3

SFC-900

900-800

800-700

700-600

600-500

500-400

400-300

300-200

200-100

JMA IR Type 252

-1.2 -0.7 -0.2 0.3 0.8

700-600

600-500

500-400

400-300

300-200

200-100

MODIS IR Type 257

-1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4

SFC-900

900-800

800-700

700-600

600-500

500-400

400-300

300-200

MODIS WV Type 259

-1.2 -0.7 -0.2 0.3

600-500

500-400

400-300

300-200

Note ECM biases are mostly negative compared to GSI, meaning ECM winds are stronger than GSI winds at satellite wind locations


• The GSI analysis systematically has higher heights than ECMWF at 200 hPa – note much red and little green

• This possibly could be due large numbers of aircraft observations with warm biases, which warm the analysis and could be affecting the satellite radiance bias corrections

Systematic Height Differences in the GSI vs. ECMWF


Rolf Langland (NRL Monterey)shows systematic height differences between all models and ECMWF (shown is ECMWF-NCEP at 500 hPa), apparently from the satellite window coverage of ECMWF (12-h) vs others (6-h).

A Height difference plot arranged with time (October to December 2007) vs longitude, averaged over 35-65N latitudes.

The range of the systematic bias is ±12 m

34

Aircraft vs Sonde GSI Draws to Temps between 200-300 mb

# Aircraft >> # Sondes, thus warm aircraft data overwhelms the GSI/GFS system

Aircraft Tdiff (obs-ges)

Aircraft Tdiff (obs-anl)SOND Tdiff (obs-anl)

SOND Tdiff (obs-ges)


Suru Saha’s website displays model fits to RAOBS in North America showing the GFS analysis and guess maintain a warm bias throughout most of the troposphere that may be related to large numbers of aircraft with warm biases


Proposed Aircraft Temperature Bias Corrections

• In the November 2008 issue of BAMS, Ballish and Kumar analyzed systematic temperature differences between radiosondes and aircraft

• Biases vary with aircraft types, pressure and the aircraft phase of flight

• The next slide shows their proposed temperature bias corrections to ACARS temperatures for January 2007


Proposed ACARS Temperature Bias Corrections January 2007

-1.4-1.2

-1-0.8-0.6-0.4-0.2

00.20.40.6

757-

222

757-

223

757-

24A

PF

737-

3H4

757-

232

767-

34A

F

A30

0F4-

60

737-

522

MD

-11F

737-

832

MD

-88

757-

251

A31

0-20

3

767-

332

A31

0-32

4

Aircraft Types

Bia

s C

orre

ctio

ns in

Deg

rees

C

SFC-700 700-500 500-300 300-150


Model Climate Impact from Aircraft Warm Temperatures

• The next slide courtesy of Dick Dee of ECMWF shows the increase in the number of aircraft reports versus time at ECMWF

• The temperature bias of the ECMWF analysis and background seem to be affected by the large increase in the number of aircraft temperatures along with other factors

• The NCEP GSI may have more bias impact as it does not thin aircraft data and its satellite radiance bias corrections are anchored to the analysis as truth as opposed to radiosondes as truth


Model Climate Bias Impact From Warm Aircraft Temperatures

Global-mean departures of analysis (blue) and background (red) from radiosonde temperatures (K) at 200hPa, and number of obs/day (x10-4, green)

Global-mean departures of analysis (blue) and background (red) from aircraft temperatures (K) at 200hPa, and number of obs/day (x10-4, green)


Eady Baroclinicity Index (EBI)

The EBI has higher amplitude and noise in the ECM analyses (b) versus operational GSI (a), but shows less amplitude at hour 24 (d) than the operational GFS (c). The EBI may be useful for studying and predicting forecast error growth rates and may help warn of a coming dropout.

a

b

c

d

5-day AC=0.67

5-day AC=0.86

Update on Dropout Related COPC Action Items Presented by Dr. Bradley Ballish Co-Chair JAG/ODAA

Documents

Transcript of Update on Dropout Related COPC Action Items Presented by Dr. Bradley Ballish Co-Chair JAG/ODAA