Scientific Investigations; Support from Research Data

Archivesfor

Joint Office for Science Support

26 February, 2002Steven Worley

SCD/DSS

Key Steps of Scientific Investigations

(archivist point of view)• Formulate the questions• Review the state of understanding• Search and discover data• Access data• Analyzes data• Share new findings • Archive results• Document new understandings

Search and Discover Data

• How? Web based Information Server• Features

– 5K+ html pages (metadata)– All datasets are described (530, 20 TB)– Location of all data files in MSS– Access options are identified– Higher level information

• Catalogs• Project specific descriptions

Always current dataset descriptions

Features

• Organization Navigation

• Archive Navigation

• Pull down menus

• Search

Dataset Page

• Title and Brief description

• Systematic Navigation

• Metadata highlights

• Period of Record

• Usage

• Variables

• Related Sites (NOAA)

• Contact Person

• Related Datasets

Archive Content• Weather Center Operational Analyses

– NCEP beginning in 1976• Many products – global, regional, different

resolutions, full atmosphere stack, all surface grids

– ECMWF beginning in 1980• Ditto

• Atmospheric Reanalyses– NCEP/NCAR 1948 – 2001– NCEP Version II, 1979 – 2000– ECMWF ERA15, 1979 – 1993– ECMWF ERA40, 1958 – 2002– NCEP N.America Reanalysis

NCEP/NCAR Global Atmospheric Reanalysis Data Product Period of

Record Temporal Res.

Spatial Res. (dg)

Update Cycle

# Levs.

# Vars.

Major Variables

Analysis on Pressure Sf c.

1948- 6/ 2001

6 hr 2.5 1-2 mn 17 7 u,v,z,t,rh

Analysis on Sigma Sf c.

1948- 6/ 2001

6 hr 192x94 Gaussian

1-2 mn 28 6 u,v,t,sph,rel.vort,

Analysis on Theta Sf c.

1948- 6/ 2001

6 hr 2.5 1-2 mn 11 10 N**2, ab.vort,u,v, t,rh,pot.vort

Surf ace Flux Fields

1948- 6/ 2001

6 hr 2.5 1-2 mn 12 Clouds, rad.flx, soil.moist,heat.flx precip

Monthly Mean Anal. P. Sf c.

1948- 2000

1 mn 2.5 1-2 mn 17+ 36 u,v,z,t,rh

CD-ROMS 1953- 1999

12 hr, 1 day, 1mn

2.5 3-6 12 u,v,z,t,rh,heat.flx, rad,flx,precip

model qc’ed observations are returned f orecasts, once every 5 days a f orecast fi elds, 6 hr, available out to 8 days

Outstanding Features• Three different coordinate surfaces• Very long analysis, 2+ Terabytes size• Unrestricted distribution• CD-ROMS are very popular

Countries Receiving Reanalysis CDROMs

Highlights• Over 8900 CDROMs 1997-09/2001

• Recipients; U.S. 46%, Japan 11%, (Canada, UK) 4%, (Germany, India) 3%, (Australia, S.Korea, Spain, Mexico, Norway, Russia, France) 2%

Archive Content

• Observational dataset – Comprehensive Ocean-Atmosphere Data

Set (COADS), beginning in 1794– Global land surface observations, beginning

in 1930’s, some earlier– Global upper air measurements, beginning

in 1946

• Many more supporting datasets and analyses

1920 1930 1940 1950 1960 1970 1980 1990 2000 2010

Surface Observation Archives

U.S. Navy SPOT

U.S. Air Force GTS

UW Antarctic, Greenland (1987-98)

Australian Synoptic

Russian Synoptic

AWS Global

U.S. Air Force Global, Davis

U.S. and Canada Hourly

U.S. Hourly

NCEP ADP Global Synoptic

1971-1996

1973-1980

1980-2001

1939-1982

1936-1986

1930-1973

1967-1980

1976-2001

1938-2001

1975-2001

JOSS & SCD Archives, Mutual Support for Science

EXAMPLE

GCIP: GEWEX Continental-Scale International Project

• JOSS•in situ data•model derived soundings•.gif images of 2D model data

• SCD• model output data (Eta, MAPS, GEM)• model derived soundings

The JOSS and SCD metadata for GCIP are well linked.

Illustrates dedicated staff efforts.

Future: Can/should we develop better ways to insure scientist make the connection?

Access Data

• MSS – all the data– No restrictions - open to all with NCAR

computing accounts– Great service to the SCD computers– Service to UCAR via LANs/WANs

Data Access

• Directly from Web Information Server– Open to the public – All data are NOT available here

• Popular collections• Smaller datasets• 100 Gigabytes total

– Now monitoring data downloads

Data Access

• Customized data packages– Persons without MSS access– Persons needing large datasets– Persons needing subsets of large

datasets– One-off jobs, not done in real time

• Package delivery– FTP download– Media (tape(s), CD-ROM)

0

200

400

600

800

1000

1200

1400

1600

Use

rs

1983 1985 1987 1989 1991 1993 1995 1997 1999 2001

Unique Users Served Annually

Cus. Orders CDROMS MSS Data Server

TBytes of Data Provided TBytes of Data Provided

0

5

10

15

20

25

Tera

by

tes

1995 1996 1997 1998 1999 2000 2001

Years

SCD Resarch Data Delivered

Estimate

TB

5.6 TB in 1995 to about 20.5 TB in 2001

Analyze Data• Provide software read the data

– Simple programs – users must be programmers

• SCD/VETS provides much more– NCL (graphics, file manipulations, and

computations)

• Community Data Portal– Real time access to subsets of large datasets (SCD and other UCAR archives)

– Real time comparison of datasets– Real time basic analysis computations

• Scientists “still” design much of their own

Share new findings / Archive results

• Input Elements– Existing archived datasets– Other archives or new field data

• Scientific Research– High level data processing– Dataset integration and comparison

• Output Elements– New data products

• Shared with colleagues• Openly distributed to the public

NCEP Operational Analyses blended with QSCAT Satellite data

Wind Stress Curl, 01/24/2000 1800 UTC

a) NCEP Operational ONLY

b) NCEP + QSCAT swaths

c) OI blend of NCEP + QSCAT

Blending by Colorado Research Associates

We archive all three products.

a b

c

Key Steps of Scientific Investigations

(archivist point of view)• Formulate the questions• Review the state of understanding• Search and discover data• Access data• Analyzes data• Share new findings • Archive results• Document new understandings