The Unified Access Framework for Gridded Data … the 1 st year focus of NOAA’s Global Earth...
-
Upload
imogen-greene -
Category
Documents
-
view
214 -
download
0
Transcript of The Unified Access Framework for Gridded Data … the 1 st year focus of NOAA’s Global Earth...
The Unified Access FrameworkThe Unified Access Frameworkfor Gridded Datafor Gridded Data
… … the 1the 1stst year focus of NOAA’s Global year focus of NOAA’s Global Earth Observation Integrated Data Earth Observation Integrated Data
EnvironmentEnvironment (GEO-IDE)(GEO-IDE)
Steve Hankin, NOAA/PMELSteve Hankin, NOAA/PMELand the NOAA UAF teamand the NOAA UAF team
NOAA UAF-gridNOAA UAF-grid 22
Enterprise-wide integration of data Enterprise-wide integration of data is a very hard problem! is a very hard problem!
NOAA UAF-gridNOAA UAF-grid 33
Heterogeneous requirementsHeterogeneous requirements
Data Providers with different outlooksData Providers with different outlooks(different "corporate cultures”)(different "corporate cultures”)
Data Users with diverse needs and skillsData Users with diverse needs and skills
Corporate cultures and people’s habits are Corporate cultures and people’s habits are notoriously difficult to changenotoriously difficult to change
… … and by the way, getting funds for integrationand by the way, getting funds for integrationis often a hard sellis often a hard sell
NOAA UAF-gridNOAA UAF-grid 44
Integrating NOAA’s dataIntegrating NOAA’s datais such a challengeis such a challenge
weather forecast weather forecast (time critical)(time critical)
fisheries management fisheries management (regulatory aspects)(regulatory aspects)
nautical chartingnautical chartingclimate, ocean, atmosphere monitoringclimate, ocean, atmosphere monitoring… …
modeling & observationsmodeling & observations
research & operationsresearch & operations
NOAA UAF-gridNOAA UAF-grid 55
The accepted approach:The accepted approach:build a “system of systems”build a “system of systems”
Connect existing data systems using aConnect existing data systems using aService Oriented ArchitectureService Oriented Architecture(“wrap” existing systems with loosely coupled, (“wrap” existing systems with loosely coupled, standardized services)standardized services)
NOAA UAF-gridNOAA UAF-grid 66
Global Earth ObservationGlobal Earth Observation
Integrated Data EnvironmentIntegrated Data Environment
GEO-IDE Con-ops GEO-IDE Con-ops outlines such a planoutlines such a plan
2005-06, 66 pages2005-06, 66 pages
NOAA UAF-gridNOAA UAF-grid 77
Seed fundingSeed funding ٨٨ available this yearavailable this year
How do you approach a very big problem with a How do you approach a very big problem with a very small amount of money?very small amount of money?
finallyfinally
NOAA UAF-gridNOAA UAF-grid 88
1.1. Define requirementsDefine requirements
2.2. Create enterprise architecture documentsCreate enterprise architecture documents
3.3. Assemble volunteer teams to implement Assemble volunteer teams to implement
NOT !NOT !
NOAA UAF-gridNOAA UAF-grid 99
An alternative approachAn alternative approach
Don't Solve ProblemsDon't Solve Problems-- Copy Success -- Copy Success
""Switch: How to Change Things When Change Is HardSwitch: How to Change Things When Change Is Hard", ", Chip and Dan Heath (psychologists)Chip and Dan Heath (psychologists)
http://www.fastcompany.com/magazine/142/switch-how-to-change-things-when-change-is-hard.html
Feb. '10 - Ocean SciencesFeb. '10 - Ocean Sciences NOAA UAF-gridNOAA UAF-grid 1010
Projects: (too many to name)Projects: (too many to name)
DataDataformats:formats:
netCDF GRIB HDF
ServiceServicestack:stack:
netCDF-CF-DAP-THREDDS-WMS
Applications:Applications: Matlab ArcGIS Ferret
GrADS Google Earth IDV LAS ERDDAP …
Users: (too many to name)Users: (too many to name)
…
NOAA UAF-gridNOAA UAF-grid 1111
UAFUAF(*)(*)-grid-grid(initial focus: gridded data)(initial focus: gridded data)
– netCDF netCDF – CF CF – OPeNDAP OPeNDAP – THREDDS THREDDS – WMS (& WCS) WMS (& WCS)
(*) Unified Access (*) Unified Access FrameworkFramework
---------- an abstract data an abstract data modelmodel
---------- metadata conventionsmetadata conventions
---- network transportnetwork transport
---- network topologynetwork topology
---- renderingrendering
NOAA UAF-gridNOAA UAF-grid 1212
Who is using this approach?Who is using this approach?
Climate, ocean, atmosphere modelersClimate, ocean, atmosphere modelers– IPCC, GFDL, NCAR, …IPCC, GFDL, NCAR, …
Satellite programsSatellite programs– GHRSST, PathFinder, CoastWatch, …GHRSST, PathFinder, CoastWatch, …
NCEP weather and ocean forecastsNCEP weather and ocean forecasts(GRIB via NOMADS)(GRIB via NOMADS)
Coastal (“HF”) radarCoastal (“HF”) radar
A growing list of observations programsA growing list of observations programs– Argo, OceanSites, SAMOS&GOSUD (soon), …Argo, OceanSites, SAMOS&GOSUD (soon), …
NOAA UAF-gridNOAA UAF-grid 1313
Some rough spotsSome rough spots
need better compliance with CF conventionsneed better compliance with CF conventions
"files" need to be aggregated into logical "files" need to be aggregated into logical datasetsdatasets
need richer discovery-level metadataneed richer discovery-level metadata
NOAA UAF-gridNOAA UAF-grid 1414
NcMLNcMLa tool for fixing compatibility problemsa tool for fixing compatibility problems
e.g. adding a CF “standard_name” attribute to GRIB datae.g. adding a CF “standard_name” attribute to GRIB data
<variable name="vorticity"><variable name="vorticity"> <attribute <attribute name="standard_name" name="standard_name" value="atmosphere_absolute_vorticity“ value="atmosphere_absolute_vorticity“ /> /></variable></variable>
The file, itself, is untouched. But access to the file through the The file, itself, is untouched. But access to the file through the services is made to conform to standards.services is made to conform to standards.
NOAA UAF-gridNOAA UAF-grid 1515
more NcMLmore NcML
e.g. aggregating 3 years of data into 1e.g. aggregating 3 years of data into 1
<aggregation type="joinExisting“<aggregation type="joinExisting“ dimName="TimeAxis"> dimName="TimeAxis"> <netcdf location=“year1.nc" ncoords=“365"/> <netcdf location=“year1.nc" ncoords=“365"/> <netcdf location=“year2.nc" ncoords=“365"/> <netcdf location=“year2.nc" ncoords=“365"/> <netcdf location=“year3.nc" ncoords=“365"/> <netcdf location=“year3.nc" ncoords=“365"/>
</aggregation></aggregation>
The files are untouched. But the services see a single logical The files are untouched. But the services see a single logical file.file.
NOAA UAF-gridNOAA UAF-grid 1616
How users will find UAF datasetsHow users will find UAF datasets
Less mature …Less mature …
UAF web crawler extracts THREDDS metadataUAF web crawler extracts THREDDS metadata– Ingests into a dbaseIngests into a dbase– other metadata also ingested as available other metadata also ingested as available
ISO 19115 records generated ISO 19115 records generated Datasets and services registered Datasets and services registered – GCMD, Geospatial One-Stop, GEOSS, data.gov GCMD, Geospatial One-Stop, GEOSS, data.gov
Much metadata work will remain beyond year 1.Much metadata work will remain beyond year 1.
NOAA UAF-gridNOAA UAF-grid 1717
The emerging UAFThe emerging UAF
The UAF THREDDS catalog:The UAF THREDDS catalog:
~400 “clean” datasets~400 “clean” datasets.. growing as the team aggregates .. growing as the team aggregates and configures using ncMLand configures using ncML– team members become coachesteam members become coaches
lasting solutions must be implemented lasting solutions must be implemented at the data provider sitesat the data provider sites
The UAF TeamThe UAF Team– 8-10 active participants8-10 active participants– self-selectedself-selected
5 months into the project …5 months into the project …
NOAA UAF-gridNOAA UAF-grid 1818
The emerging UAFThe emerging UAF
Users access data through Users access data through their client applicationstheir client applications– Without downloading filesWithout downloading files
Configuration of clients is Configuration of clients is underwayunderway
Some examples …Some examples …Take differences
Scripts
Live Access Server Live Access Server (LAS)(LAS)
NOAA UAF-gridNOAA UAF-grid 1919
Model 1: UMASS-ECOMModel 1: UMASS-ECOM Model 2: UMAINE-POMModel 2: UMAINE-POM Data: SST 2008-Sep-08 07:32Data: SST 2008-Sep-08 07:32
Desktop access in MatlabDesktop access in Matlab
NOAA UAF-gridNOAA UAF-grid 2020
Desktop access in FerretDesktop access in Ferret
NOAA UAF-gridNOAA UAF-grid 2121
Desktop access in GrADSDesktop access in GrADS
NOAA UAF-gridNOAA UAF-grid 2222
Desktop access in IDVDesktop access in IDV
NOAA UAF-gridNOAA UAF-grid 2323
Access in ArcGISAccess in ArcGISusing the Environmental Data using the Environmental Data
Connector (EDC)Connector (EDC)
NOAA UAF-gridNOAA UAF-grid 2424
Access in Google EarthAccess in Google Earth
NOAA UAF-gridNOAA UAF-grid 2525
WMS provides WMS provides access to many access to many GIS applicationsGIS applications
NOAA UAF-gridNOAA UAF-grid 2626
Browser access using ERDDAP. Browser access using ERDDAP. And from there …And from there …
NOAA UAF-gridNOAA UAF-grid 2727
.mat.mat
REST URL access to data subsetsREST URL access to data subsetsin several simple formatsin several simple formats
(accessible through home-grown scripting of many types)(accessible through home-grown scripting of many types)
.kml
.mat
.nc
NOAA UAF-gridNOAA UAF-grid 2828
Next year’s target: observational data
Collections of time series & profiles arealready on-line
UAF home page … coming soon
NOAA UAF-gridNOAA UAF-grid 2929
““No standard should be approved without having No standard should be approved without having been used to implement a few projects of realistic been used to implement a few projects of realistic complexity.”complexity.”
The Rise and Fall of CORBA (Henning, 2006)
The UAF strategy buildson historical lessons
NOAA UAF-gridNOAA UAF-grid 3030
Thank youThank you