DMAC The Data Management and Communications Subsystem of the U.S. Integrated Ocean Observing System...

DMACThe Data Management and Communications Subsystem

of theU.S. Integrated Ocean Observing System (IOOS)

Steve HankinNOAA/Pacific Marine Environmental Laboratory

DMAC Steering TeamDMAC Modelers’ Caucus

Nov ‘06 TECO-WIS 2

GOOS - Global Ocean ObservationsGTSPP, ARGO, Satellites, VOS, others

Why not do something comparableWhy not do something comparablefor the coastal ocean?for the coastal ocean?


GCOOS

CeNCOOS

NANOOS

AOOS

PacIOOS

SECOORA

MACOORASCCOOS

NERA

CaRA

Coastal Componentof IOOS

GLOS

National Backbone• Federal Agencies Responsible• EEZ & Great Lakes• Core variables required by RAs& Fed Agencies• Network of sentinel & reference stations• Standards/Protocols

Regional Coastal Ocean Observing Systems• Regional Associations responsible• Involve private & public sectors • Inform Federal Agencies of user needs• Enhance the backbone based on user needs• Incorporate sub–regional systems

US IOOS Coastal System:“National

Backbone”

Regional Associations


Satellites

Aircraft

Fixed Platforms

Ships

Drifters & Floats

AUVs

ObservingSubsystem

Integrated Ocean Observing System

Metadata standards

Data discovery

Data transport

Online browse

Data archival

DMACSubsystem

Modeling & AnalysisSubsystem

Safe & EfficientMaritime Operations

Homeland Security

Natural Hazards

Climate Change

Public Health

Ecosystem Health

Living MarineResources

Societal Goals


Many Variables – the IOOS Core Set

• Physical– Salinity, temperature, air-sea heat flux– Sea and water level, water depth (Shore line)– Surface waves (height, direction, freq), currents & sea ice

• Chemical– Contaminants & dissolved nutrients– Dissolved oxygen

• Biological– Fish species & abundance– Chlorophyll, ocean color & phytoplankton abundance/species– Zooplankton abundance/species– Pathogens

• Interdisciplinary– Optical properties (Atmospheric & in-water)– Bottom character and benthic habitats


The Governance Challenge

• Many independent observing system activities

– federal, state, county, municipal, academic & private players

• Independent funding paths

• Widely varying program missions

• A planning office -- Ocean.US -- established to coordinate the efforts

Effective governance depends upon a workable data management and communications (DMAC) plan


1.Heterogeneous

data

2. Looselyfederated

organizations

3. Large,distributedholdings

……competingcompetingcharacteristics characteristics ruled out the ruled out the most most straightforward straightforward solutions for IOOSsolutions for IOOS


1.Heterogeneous

data

2. Looselyfederated

organizations


No simple data standard can be No simple data standard can be designed that all ocean data designed that all ocean data

providers will utilizeproviders will utilize


No management No management structure exists structure exists to enforce to enforce adoption of a adoption of a complex, data complex, data management management solutionsolution

1.Heterogeneous

data

2. Looselyfederated

organizations



Data cannot be Data cannot be centrally managed at centrally managed at

a single locationa single location

1.Heterogeneous

data

2. Looselyfederated

organizations



The DMAC Plan – a community effort

• Brought together …• 6 Federal agencies• 6 Universities• 4 Regional/State agencies• 3 Private sector orgs.

• Steering Committee guided2 Outreach Teams, 4 Expert Teams

• Data Facilities Outreach• User Outreach

• Data Discovery & Metadata• Data Transport• Data Archive & Access• Applications & Products

Steering Comm. convened June 2002

Public drafts circulated Dec. 2003

The DMAC Plan – a community effort• Four levels of review

1. Solicited expert review2. Public workshop3. Public email-based review4. Federal Registry comments

• Drafts broadly circulated• ExCom, NORLC, IWG, USGST,

Ocean.US Workshops, Regional Summits, Professional Society newsletters, email lists and websites

• 150 reviewers• 6 Federal Agencies• 22 Universities & Institutes• 13 Private Sector• 13 Regional/State agencies• 5 International

• An implementation plan (a roadmap to detailed specification)

Published March 2005


The DMAC Plan is a framework for interoperability among independent,

heterogeneous systems, large and small

• Independence: Organizations will manage their own data

• Standards: enabling innovation!

• “Web services”: the common language to be spoken by pre-existing systems


How DMAC selects standards

“Adapt, adoptand only as a last resort

develop”

Classified by IOOS maturity levels1. R&D

2. Pilot

3. Pre-operational

4. operational

Strong preference for open standards


DMAC planning process continues …

Steering TeamExpert Teams

Transport and AccessMetadata and Data DiscoveryArchive

CaucusesPrivate sectorInternationalModelingRegional

Working GroupSystems Engineering

Interagency Oversight Working Group


Data Transport

Harmonize developments from 2 communities

1. GIS – utilizing OGC W*F standards

2. FES* – utilizing OPeNDAP


develop”

* FES = fluid earth systems

Expert Team -- 10 membersacademia (4), federal (4), private sector (2)


Digression on OPeNDAP …”DAP” == Data Access Protocol

Since 1992 - a “web service” before the term existed

• “Files” vanish “datasets”, instead

• On the server– netCDF, GRIB, HDF, IEEE, … or relational databases– aggregations (multi-terabyte datasets)

• On the client– “open” and read subsets– any netCDF application is a web client (may need to relink)– Matlab, IDL, IDV, GrADS, Ferret, CDAT, …

• Widely used in research modeling communities


OGC OPeNDAP Gateway

Grids 2 - WCS 3 - CF GALEON

Time series 1 - WFS/GML 2 - DAPPER

Profiles 1 - WFS/GML 2 - DAPPER

Tracks 2 - WFS/GML 1 - DAPPER

Points 3 - WFS/GML 1 - (CDM)

... … …

IOOS maturity levels1. R&D2. Pilot3. Pre-operational4. operational

Data Transport


Metadata andData Discovery

Team approach– Coordination with organizations/programs such as

MMI*, ISO, FGDC*, GCMD*, ORION* and QARTOD*

– Engage data providers in the process

Determine marine metadata content requirements– data lifecycle: identify, assess, access, utilize and

archive

– adaptive to data type and method of data collection

Compare metadata elements across multiple standards – FGDC, ISO, DIF, EML, ESML, SensorML, MarineXML, etc.

• MMI = Marine Metadata Interoperability project

• FGDC = US Federal Geographic Data Committee

• GCMD = NASA Global Change Master Directory

• ORION = US NSF-funded ocean research initiative

• QARTOD = Quality Assurance Real Time Ocean Data


develop”


Starting point: “Guide for IOOS Data Providers, v1.0”

Initial guidance contains:– Required vocabularies

• IOOS keywords, core variables, national backbone programs– Recommended vocabularies

• ISO 19115 topic categories, GCMD keywords, IHO codes, OBIS species, CF standard names

– Metadata standards• ISO 19115, FGDC CSDGM, GCMD DIF, OBIS

– Minimal list of elements• Discovery, access, data transport, archive, consumer use,

data management– Data discovery systems

• NSDI Clearinghouse, Geospatial OneStop, GCMD, OBIS

Metadata andData Discovery

DMAC Modelers’ Caucus – Challenges

1. Advancement and use of CF (netCDF & OPeNDAP)

Further work:

a. unstructured gridsb. Geospatial datumsc. Nested grids

2. Downscaling and splicing of neighboring domains

3. Rich model metadata

4. OSSEs – couple modeling and observing system design

5. Model-data validation and model-model comparison Modeling and Analysis Steering Team (MAST)

>> Convened Oct. 25, 2006 <<Community Modeling Networks (CMNs) to follow


“Community Standards for Unstructured Grids”

UCAR/Unidata, Boulder CO -- >> October 16-18 <<

NOAA-IOOS funded • Standards for unstructured coastal model

outputs– A community process– Standardize file contents, formats Standardize network access protocols

• A demonstration testbed will be created

• Technical discussions will be open– on a moderated web site

http://cf-pcmdi.llnl.gov/


DMAC and WIS – Closely related challenges

Both seek to

• Link organizationally-independent data suppliers

• Address a broad range of user needs

• Integrate heterogeneous data types

• Merge real-time (push) and delayed mode (pull)

• Build upon standards

WORLD METEOROLOGICAL ORGANIZATIONWeather Climate Water


DMAC areas in need of particular attention(to what degree can WIS solutions be adopted?)

• Real time (push) ( IDD/LDM? )

• Access control• Service registries• Data assembly (regional and national)

WORLD METEOROLOGICAL ORGANIZATIONWeather Climate Water

DMAC and WIS – Closely coupled solutions


kahm-sa-hahm-ni-da

(Thank you)

DMAC The Data Management and Communications Subsystem of the U.S. Integrated Ocean Observing System...

Documents

Transcript of DMAC The Data Management and Communications Subsystem of the U.S. Integrated Ocean Observing System...