Post on 18-Dec-2015
ESIP Federation Meeting 15-18 July 2008
Data and Metadata Standards for GHRSST: present to future
ESIP Federation Meeting 15-18 July 2008
Univ. of New Hampshire, Durham, NH
Ed Armstrong
Jet Propulsion Lab/CalTech
Physical Oceanography DAAC
Pasadena, California
ESIP Federation Meeting 15-18 July 2008
Data throughput…G
b D
ay-1
ESIP Federation Meeting 15-18 July 2008
User Base
Power Users:• NAVO
• UK Met Office
• NASA JPL
• Canada Meteorological Agency
• Danish Meteorological Institute
2021371 621184
62141
147
12327112327
1391184
5
894818
2741
234
2
4
3
2
25
1
159
4
Australia Belgium Business Organizations
Brazil Canada Chile
Commercial Czech Republic Germany
Denmark Educational Estonia
Spain Finland France
US Government Greece Croatia/Hrvatska
Indonesia India International Organizations
Italy Japan Jet Propulsion Laboratory
Korea, Republic of Morocco US Department of Defense
Mexico National Aeronautics and Space Administration (US) Networks
Netherlands Norway New Zealand
(Non-Profit) Organizations Peru Poland
Portugal Romania Russian Federation
Sweden Singapore Turkey
Taiwan United Kingdom United States
Unknown (numeric host id) South Africa
ESIP Federation Meeting 15-18 July 2008
Product Specifications - 2003
• netCDF 3.6 • Self describing format
– Popular in the ocean modeling and NWP community
• Direct access– Improves usability: L2P products can have over 20 arrays
• Portable– GHRSST is an international project with data produced on a variety of platforms
• “bundled” with CF/COARDS compliant metadata• Limitations:
– No internal compression. Externally compressed (bzip2 or gzip)– No chunking
ESIP Federation Meeting 15-18 July 2008
Metadata Specifications - 2003
• NASA Directory Interchange (DIF) format • Compact, well defined user history, simple to implement• A granule level XML metadata (File Record [FR]) for each file made by producer
– typical entries are dynamic such as time and position – e.g., <Southernmost_Latitude></Southermost_Latitude>
• A product level XML metadata (Dataset Description [DSD])– typical entries are more static contacts, document links, and summary statements
– e.g., <Summary></Summary>
• Some “custom” tags added. For example, <file_compression>• These are stored as separate databases at the JPL GDAC Master Metadata Repository (MMR)
– mySQL relational database for search, query, and assembly
ESIP Federation Meeting 15-18 July 2008
Metadata Specifications - 2003
• FGDC conversion• All GHRSST products are transferred to NOAA Longterm Stewardship and Reanalysis Facility (LTSRF) after a period of 30 day• NOAA has requirement for FGDC compliant metadata for its products. No data ingested without metadata.• A conversion of the FR and DSD records to FGDC is performed on file-by-file basis
– Mappings from DIF to FGDC done using XSLT style sheets– For L2P data multiple files of same product type in same FGDC record. Up to 288 files per FGDC record.
ESIP Federation Meeting 15-18 July 2008
Mapping Anomaly
• Some mappings are not clean. FR record:<Personnel>
<Role>Technical Contact</Role>
<First_Name>Ed</First_Name>
<Last_Name>Armstrong</Last_Name>
<Email>ghrsst@podaac.jpl.nasa.gov</Email>
<Address>
Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA 91109 USA
</Address>
</Personnel>
• mapped FGDC record:<address>
Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA 91109 USA
</address>
<city>
Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA 91109 USA
</city>
<state>
Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA 91109 USA
</state>
<postal>
Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA 91109 USA
</postal>
<country>
Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA 91109 USA
</country>
ESIP Federation Meeting 15-18 July 2008
Measurements of SST (netCDF CF1.3)
Dep
th
10m
1 mm
1 m
10 m
SSTskin – SST10m (K)
1. Night-time (or strong winds) profile in red2. Day time situation, strong solar radiation and light winds
sea_surface_skin_temperature
sea_surface_subskin_temperature
sea_water_temperature(Attribute:depth)
Sea_surface_foundation_temperature
-0.17 0 0.5 1.0 1.5 2.0 2.5 3.0
Nig
ht-
tim
e (
or
str
on
g w
ind
s)
pro
file
in
red
surface_temperature
ESIP Federation Meeting 15-18 July 2008
CF 1.3 Specification for SST standard names
• surface_temperature (SSTint)
• sea_water_temperature (SSTdepth)
• sea_surface_skin_temperature; K
• sea_surface_subskin_temperature; K
• sea_surface_foundation_temperature: K
ESIP Federation Meeting 15-18 July 2008
Future product development
• NetCDF version 4• Benefits
– Internal compression and chunking (from HDF storage model)– Support for Parallel I/O– Backward compatibility with v3.6– The lure of “new stuff”
• Advantages must be weighted against a format that has proven metrics and use
– Operationally it is difficult to implement a new file format– The user may (will) complain. New software and libraries to install etc.
• Evaluated this summer, with a decision by next GHRSST Science Team Meeting June 2009 for adoption
ESIP Federation Meeting 15-18 July 2008
Future metadata requirements
• European Space Agency will require metadata for next generation Sentinel satellite derived products to be ISO 19115-2 compliant. This directly impacts GHRSST.
• Also required for using GHRSST products in WMO Integrated Global Observing Systems (WIGOS) systems (a GEOSS component)
• What to change in our metadata production system ?• We have an investment in something that works. We can’t break everything and start
from scratch.• The way forward is likely to generate mappings to ISO 19115 similar to FGDC
conversion.– Currently reviewing our metadata architecture (FR and DSD XML records) to see what needs to
be added/modified– Review must be completed in the next months. A written protocol by the end of the year.– What part of ISO 19115 to conform to?? North American profile…Remote Sensing extensions.– ISO metadata embedded in netCDF v4 GHRSST file ??
ESIP Federation Meeting 15-18 July 2008
Summary
• GHRSST has demonstrated significant achievements with regard to data and metadata standards on a relatively modest budget
• Although focused on SST discipline, is serves as a model for a working GEOSS• Supporting Science and Applications through well thought out and implemented best practices
• One of the challenges will be to evolve with new technologies wrt to format and metadata standards
• ISO 19115– Relatively straightforward once the mappings are determined
• netCDF v4 – More challenging. More investigation needed.
• Others: Search, Discovery, Imaging– THREEDS: data cataloguing– JPL Datacasting: RSS-like data subscription– Swath granule subset– ECHO metadata inventory– OPeNDAP– WMS POET
ESIP Federation Meeting 15-18 July 2008
Backups
•
ESIP Federation Meeting 15-18 July 2008
(A)ATSR reduces uncertainty in global average SST (John Kennedy, Nick Rayner)
ESIP Federation Meeting 15-18 July 2008
GDAC overview
ESIP Federation Meeting 15-18 July 2008
• Polar Orbiting infrared has high accuracy & spatial resolution • Geostationary infrared has high temporal resolution• Microwave Polar orbiting has all-weather capability
GHRSST-PP L4 products
(A. Harris)
ESIP Federation Meeting 15-18 July 2008
SST system quality control and uncertainty estimation
AnalysisProduct
s net
CD
F f
orm
at
Input SST
data of Varied format& with
no errors
Datamerging
andanalysis
SST Obs.
Products net
CD
F f
orm
at
Add ancillary
QC SSES& dynamic
flags
The GHRSST-PP Strategy
SSTObservations
ApplicationsL3
L1P Radiance data
ESIP Federation Meeting 15-18 July 2008
GDS-v2.0Rev 0.1June 2008Generate L2P
data products (L2Pc -> L2P)
Generate L4 analysed data products (e.g.,
SSTfnd) based on L2P data input.
Satellite SST data
Ingest & QC input data streams
Auxiliary data
Reference data fields
Generate L3Pdata products
Reference data fields
Analysed L4 SST data products
Rolling and long-term
archive system
Derive Single Sensor error
statistics (SSES)
Satellite and in situ MDB
Single Sensor Error
Statistics
Generate MDB records
MDB/SSES system
MDB recor
ds
Diagnostics, metrics and GDS data product
validation
HR-DDS archive GDAC &RDAC
L4 SST HR-DDS data granules
L2P HR-DDS data
granules
Extract HR-DDS L2P granules
User data access through LAS/DODS HR-
DDS server
Extract HR-DDS L4 SST granules
HR-DDS system
In situ data
CDR SST data products
Re-Analysis f L2P/L3P/L4
L3P SST data products
Reference data fields
L2P/L2Pc SST data products
Metricsdefinition
s
GMPE SST ensembleproducts
Generate GMPE
products
Reference data fields
Main data flow
HR-DDS data flow
MDB/SSES data flow
RAN data flow
GMPE data flow
Reference data fields
Auxiliary data
GMPE systemRAN system