1

Thomas Karl Director National Oceanic and Atmospheric Administration

National Climatic Data Center

AMS 88th Annual MeetingTown Hall Meeting January 21,2008

NOAA Reanalysis

User Needs

AMS Town Hall Meeting

2

Outline

Purpose of MeetingOverview of NOAA Reanalysis EffortsClimate Forecast System Reanalysis and

Reforecast (CFSRR):– Data Availability Plans– Data Access Optimization

User Input and Open Discussion

3

Has the proposed file organization and groupings reflect current user needs and expectations?

Which data need to be placed on-line given e.g., 150TB of disk? Are we on the right track? Do we need to re-group data sets to

create needed products – such as initialization files, analysis, or by forecast projections?

Can some other mechanism (i.e. LDM) provide some CFSRR output to users in near real-time, shifting the load from the archive?

NCDC and NCEP are committed to provide the highest resolution if possible. Tonight’s discussion is a focus on user needs tempered by available resources.

Tonight’s discussion is a focus on user needs

Open Forum and User Input

4

New NOAA Reanalysis Projects

1) Historical SFC Reanalysis (Compo et al.,)– 1850 to present. ~60TB

2) Post WW-II Reanalysis (NCEP)– 1944 to present. ~235TB

3) Climate Forecast System Reanalysis and Reforecast (CFSRR) Project (Saha et al.)– 1978-2008 Reanalysis and Seasonal Reforecast:

915TB (Grib1)

Development of a Reanalysis Clearinghouse– Capability for an “on-going analysis of the climate

system”

5

CFSRR Overview (Saha, EMC/NCEP) Coupled Climate Forecast System (CFS) Reanalysis and Reforecast

(CFSRR) (Atm - Ocn - Land – SeaIce) has two parts:

– Reanalysis: 31-year (1979-2009) T384L64 (~32 km and 64 hybrid

layers for the atmosphere, 0.50 and 40 levels for the ocean, 4 soil

levels)

– Reforecast: 28-year (1982-2009) T126L64 (~32 km and 64 hybrid

layers for the atmosphere, 0.50 and 40 levels for the ocean, 4 soil

levels)

– 6 hourly Reforecast for 1 year

NCDC and NCEP are seeking community input for archive and access

priorities to this massive dataset

Approximately 915 Terabytes in GRIB1.

Note: this presentation assumes conversion is needed to Grib1 as many user

applications cannot yet deal with GRIB2 directly (including NCDC).

6

Proposed CFS Reanalysis Data

CFS High Resolution Initial Conditions: 72 TB CFS Low Resolution Initial Conditions: 35 TBFull Data Ingest : 46TB (satellite radiances,

RAOB, in-situ, etc.)Pgbh Pressure Grib Files 0.5 x 0.5 Hourly: 91TB Flxf T382 Gaussian Hourly: 25TBOcnh 0.5 x 0.5 Hourly: 18TBDiabf 1.0 x 1.0 Hourly: 27TB Ipvh 0.5 x 0.5 Hourly: 17TBMonthly: 1TB

TOTAL: 332 TB

7

1. Siganl : 3-D Hybrid Analysis

2. Sfcanl : Surface Analysis

3. Ocnanl : 3D Ocean Analysis

4. Pgbh : 3-D Pressure level data

5. Flxf : SFC fluxes, radiative fluxes, precip.

6. Ocnh : 3-D Ocean data

7. Ipvh : 3-D Isentropic level data

8. Diabf : Diabatic Heating, Moistening rates, etc.

9. Egyh : Energetics, u’ v’, TKE, etc.

Proposed CFS Reanalysis Hi-Resolution File Level Organization

T382L64 + Ocean (1/4 x ½) 6 hourly = 72TB T126L64 + Ocean (1/2 x 1) 6 hourly = 35 TB

0.5 x 0.5 degree Hourly= 91TB

T382 Gaussian Hrly= 25TB

0.5 x 0.5 degree Hourly= 18TB

1.0 x 1.0 6-Hourly: 27TB

0.5 x 0.5 degree Hourly: 17TB

0.5 x 0.5 Monthly: 1 TB

8

CFS 28-Year Reforecast Data1982-2009

6-hourly Pressure Grib: 412TB6-hourly Ocean : 102TB6-hourly Ipv: 78TB Monthly and Time-series: 3TB

1.0 x 1.0 degree for first 6 months of forecast

2.5 x 2.5 for next 6 months of forecast

TOTAL: 595 TB

9

National Academies Recommendation

National Research Council, Board of Atmospheric Sciences and Climate (BASC):

“Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather

and Climate Forecasts”

Recommendation 3.4:

The NOAA National Operational Model Archive and

Distribution System (NOMADS) should be maintained and

extended to include (a) long-term archives of global and regional

ensemble forecasting systems and their native resolution, and

(b) re-forecast datasets to facilitate post-processing”

10

Open Discussion

Proposed Data and System Access Priorities

11

1) Basic ftp and http scripting services to the highest resolution data though the NCDC tape archive services.

2) NOMADS on-line sub-setting services for most requested data (tonight’s discussion).

3) Advanced GIS and other Web Based Portal Services as resources and time permit

Issue: Data access may be throttled based on number of concurrent users, I/O restraints, and communications bandwidth at NCDC.

CFSRR System Access Prioritization 1

12

Reanalysis Data Access Prioritization 1

Highest Resolution CFSRR data provided– Low Res users can subset or use Post WW-II– Access assisted by user feedback as to file

level organization (this meeting!)Historical “20th Century” SFC Reanalysis

– 60TB (Grib and HDF) FY10 Post WW-II U/A Reanalysis

– Low resolution 6-hourly (235TB Grib-1)– When funded

13

FEEDBACK and QUESTIONS

Suranjana.Saha@noaa.gov

CFSRR PI

Glenn.Rutledge@noaa.gov

nomads.ncdc.noaa.gov

David.Easterling@noaa.gov

OPEN DISCUSSION

14

Backup Slides

15

• To overcome a deficiency in model data access, some of the Nations top scientists are actively engaged in a grass-roots framework to share data and research findings over the Internet.

• NCDC, NCEP and GFDL initiated the NOAA Operational Model Archive and Distribution System.

• NOMADS is a distributed data services pilot for format independent access to climate and weather models and data.

Overview

16

• foster research within the geo-science communities (ocean, weather, and climate) to study multiple earth systems using collections of distributed data;

• promote model evaluation and community feedback;

• develop institutional partnerships and access via distributed open standard technologies.

• Establish a unified climate and weather model archive providing format independent access to retrospective models;

NOMADS GoalsNOMADS Goals

The NOAA Operational Model Archive and Distribution System

17

Direct Client Access

NCEP Dual Ingest & QC

Portals

GDS and TDS

Live Access Server

GrADS, Ferret, MatLab, IDL, IDV, Web browsers or any OPeNDAP enabled client

• Project ACCESS: NASA,

GMU, GMU & OPeNDAP• Project GALEON Unidata• NOMADS Web-Plotter,

http, ftp & binary subsetting• GIS access (WCS, W*S)

NCDC Archive

CEOS-Grid

NOAA-wide LAS“sister-servers”

Exploratory Grid Projects w/ Globus

Collaboration Focus

The NOMADS System Design

Multiple paths to format independent data access

18

NCDC HDSS Access System(HAS = NCDC Archive)

NOMADSModel data

CLASSSatellite

NEXRADRadar CDO, GIS

ServicesIn-situ

Data requestfrom accessservice—eg,NOMADS

Data delivery to access service—eg,NEXRAD

A very simplified diagram of data flow from NCDC‘s HDSS tape robotics system & incoming sources

Other incomingdata

MajorAccessServices

NOAA Customers