Hnilo RDAP11 Data Archives in Federal Agencies

The National Climate Model Portal

Overview & NCDC Archive and Access Challenges

Dr. Jay Hnilo NCMP Senior Scientist

NOAA’s Cooperative Institute for Climate and Satellites (CICS-NC)National Climatic Data Center (NCDC) Asheville, NC 28801

ASIS&T Research Data Access and Preservation Summit ASIS&T Research Data Access and Preservation Summit Denver CO. March 31, 2011Denver CO. March 31, 2011

M. Sutton 1995

The National Oceanic and Atmospheric Administration

2

NOAA National Climate Model Portal

NCMP

Background: NOMADS - A Data Access System

NCMP and NOMADS - Goals and Motivation

NCDC Archive Processes - Archive Processes

- Distributed Access Philosophy

OutlineOutline

3


NCMP

Until 2002 there existed no long-term archive for Climate and Weather models in NOAA.

Retrospective analysis and model inter-comparison are necessary to verify and improve short term NWP models, seasonal forecasts, climate simulations, assessments and detection.

University and Institutional research goes largely untapped by NOAA scientists. Effort is wasted on data receipt and format issues with no infrastructure to collaborate.

BackgroundNOMADS Data Access System


4


NCMP

• In 2002 to overcome a deficiency in model data access, some of the Nations top scientists actively engaged in a grass-roots framework to share data and research findings over the Internet.

• NCDC, NCEP and GFDL initiated the NOAA Operational Model Archive and Distribution System.

• NOMADS is a distributed data service providing format independent access to climate and weather models and associated data.



5


NCMP

• foster research within the geo-sciencecommunities (ocean, weather, and climate)to study multiple earth systems using collections of distributed data,

• promote model evaluation and product development

• develop institutional partnerships via distributed open technologies.

• provide distributed access to models and associated data. Begin to scale to petabyte.

BackgroundProject GoalsProject Goals

BackgroundProject GoalsProject Goals

6


NCMP

Pare down large file sizes of high resolution data and products- and provide flexible inter-operable access.

(re-) Group different data sets to create needed products – such as initialization files for model development, analysis, or by forecast projection.

Subset and aggregate the data: - in parameter space

- in physical space - in temporal space

BackgroundMotivation: Tools for Users

BackgroundMotivation: Tools for Users

7


NCMP

NCDC Archive Motivation for Archive Stewardship

NCDC Archive Motivation for Archive Stewardship

8


NCMP

NOAA-wide procedure to identify, appraise, and decide what scientific records are preserved in a NOAA Facility. Then a Submission Agreement (SA) outlines details of dataset.

Reviewed by a cross-NOAA working group- the Environmental Data Management Committee (EDMC). Long-term stewardship the goal.

Criteria developed using guidelines from National Archives and Records Administration (NARA), and National Research Council (NRC) reports on NOAA data management, and from other related reviews.

Technology to provide access and value-added products to deep archive an on-going NCDC activity (NOMADS-NCMP etc.).

NCDC Archive Archive Procedures

NCDC Archive Archive Procedures

9


NCMP

The Submission Agreement and “What to Archive Process” allows data center to make informed planning decisions

Provides a formal way to be selective about how data are supported

Documents the justification for allocating archive support for the data

Data Reduction policies and recommendations now underway with NOMADS (e.g., remove fcsts > 5 years).

NCDC Archive Benefits of Archive Procedures

NCDC Archive Benefits of Archive Procedures


NCMP

Private orpublished

results

Searchtools

Metadata

Ontologies

Public and private

catalogue

Workflowgeneration

tools

Private virtual

workspaces

Sharedvirtual

workspaces

Monitoring & controlservices

Workfloworchestration

engine

Observing System Simulation Experiments

Other (e.g.Unique

Instrumentation)

ModelingSystems

AnalysesDatasets

Eventdetection

D D D

M

Earth Systems Modeling Framework

National Climate Model Portal

NCMP

D data

Q Q/A

M model

A analysis

Observing Systems

Real-time data streams

Mid

dle

wa

re, a

ccess p

roto

cols, se

cure

da

ta tra

nsp

ort

Use

r au

the

ntica

tion

, acce

ss con

trol lo

gic

Me

tad

ata

voca

bu

larie

s, on

tolo

gy sta

nd

ard

s

Users

NOAA

Climate

Services

Portal

Ed

uca

tion

an

d tra

inin

g, u

ser su

pp

ort

Compute serversNCDC ArchiveDigital

libraries

Task

Reanalysis and

Climate Clearing-

houseCommunity

vetted observational

database

GEOUS-GEO

Adaptive

DOE Earth SystemGrid &iRODS

Q A

Pre / Post Processing

Rutledge/Meacham/Fontaine 2006

U.S. GEO Modeling Infrastructure VisionU.S. GEO Modeling Infrastructure Vision

Access

Data Ingest

NCDC Archive

National Climate Model Portal

Web Based Data Services

Community

• Priority Technologies & Partners - GO-ESSP Community - NCSP National Climate Service Portal

- NCPP National Climate Prediction and

Projections Center (ESRL prototype)- OPeNDAP, OGC, CF, NetCDF, TDS…- iRODS Renaissance Computing Institute- ESGF Earth System Grid Federation- IPCC & LLNL/PCMDI Archive

The NOMADS-NCMP System

12


NCMP

British Atmospheric Data Centre◦ Bryan Lawrence – Director, British Atmospheric Data Centre

Geophysical Fluid Dynamics Laboratory◦ V. Balaji, Head, Modeling Group, Princeton/GFDL

The German Climate Computing Centre◦ Michael Lautenschlager (NeRC Grid)

Lawrence Livermore National Laboratory◦ Dean Williams, PCMDI, Chief Archive Services/CMIP5 , ESGF

National Center for Atmospheric Research◦ Don Middleton, Senior Manager, Enabling Technologies, ESGF

Pacific Marine Environmental Laboratory◦ Steve Hankin (Unified Access Framework, DMIT)

NOAA/Earth Systems Research Laboratory◦ Cecelia Deluca (National Climate Projection and Prediction NCPP prototype)

NOAA/National Climatic Data Center◦ Glenn Rutledge, (Program Manager NOMADS/NCMP)

Global Organization for Earth Systems Science Portals

Related Workshop: 2011 GO-ESSP Workshop

Related Workshop: 2011 GO-ESSP Workshop

NCDC hosts the2011 GO-ESSP WorkshopMay 9-10, Asheville NC

http://go-essp.gfdl.noaa.gov/


NCMP

Questions?

[email protected]@noaa.gov

NCDC Asheville, NC

http://nomads.ncdc.noaa.gov

M. Sutton 1995

NCMP


Thank you

Hnilo RDAP11 Data Archives in Federal Agencies

Documents

Transcript of Hnilo RDAP11 Data Archives in Federal Agencies