2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software...
-
Upload
dwight-townsend -
Category
Documents
-
view
226 -
download
1
Transcript of 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software...
![Page 1: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/1.jpg)
![Page 2: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/2.jpg)
22
![Page 3: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/3.jpg)
33
![Page 4: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/4.jpg)
ROMS/COAWST NcML fileROMS/COAWST NcML file
44
![Page 5: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/5.jpg)
55
![Page 6: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/6.jpg)
Exploiting IOOS: A Distributed, Standards-Based Framework and
Software Stack for Searching, Accessing, Analyzing and Visualizing Met-Ocean Data
Exploiting IOOS: A Distributed, Standards-Based Framework and
Software Stack for Searching, Accessing, Analyzing and Visualizing Met-Ocean Data
Rich Signell (USGS-CMG)Filipe Fernandes (SECOORA)
Kyle Wilcox (Axiom Data Science)Andrew Yan (USGS-CIDA)
Rich Signell (USGS-CMG)Filipe Fernandes (SECOORA)
Kyle Wilcox (Axiom Data Science)Andrew Yan (USGS-CIDA)
Regional IOOS DMAC Meeting: Silver Spring, 5/28/2015
![Page 7: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/7.jpg)
ObjectivesObjectives
• Set up a standards-based framework for easy and efficient access to insitu and ocean model data
• Provide a high-level search and browse web interface for program datasets, for scientists, end users and program managers
• Contribute to a growing standardized data search, access and use infrastructure that supports all geoscience
• Set up a standards-based framework for easy and efficient access to insitu and ocean model data
• Provide a high-level search and browse web interface for program datasets, for scientists, end users and program managers
• Contribute to a growing standardized data search, access and use infrastructure that supports all geoscience
![Page 8: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/8.jpg)
Why not just use ERDDAP?Why not just use ERDDAP?
• Two reasons:• 1. Unstructured grid models• 2. Curvilinear grid models
• Two reasons:• 1. Unstructured grid models• 2. Curvilinear grid models
![Page 9: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/9.jpg)
UGRID Conventions on GitHubUGRID Conventions on GitHub
![Page 10: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/10.jpg)
SGRID Conventions: github/sgridSGRID Conventions: github/sgrid
![Page 11: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/11.jpg)
IOOS Model Data Interoperability DesignIOOS Model Data Interoperability Design
ROMS
ADCIRC
HYCOM
SELFE
NCOM NcML
NcML
NcML
NcML
NcML
Common Data Model
OPeNDAP+CF
WCS
NetCDF Subset
THREDDS Data Server
Standardized (CF-1.6, UGRID-0.9) Virtual Datasets
NonstandardModel Output Data Files
Web Services Matlab
Panoply
IDV
Clients
NetCDF-Java
Libraryor Broker
WMS
ncISO
ArcGIS
NetCDF4-Python
FVCOM
PythonERDDAP
NetCDF-Java
SOS
Geoportal Server
GeoNetwork
GI-CAT
Observed data (buoy, gauge, ADCP, glider)
Godiva2
pycsw-CKANNcML
GridUgridTimeSeriesProfileTrajectoryTimeSeriesProfile
NonstandardData Files
CatalogServices
![Page 12: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/12.jpg)
Interoperable Model Comparison in Matlab(using nctoolbox)
Interoperable Model Comparison in Matlab(using nctoolbox)
![Page 13: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/13.jpg)
compare_secoora_model_sections.mcompare_secoora_model_sections.m
![Page 14: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/14.jpg)
3D visualization of data with IDV3D visualization of data with IDV
![Page 15: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/15.jpg)
NECOFS Access in ArcGIS(using the dap2arc python toolbox)
NECOFS Access in ArcGIS(using the dap2arc python toolbox)
![Page 16: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/16.jpg)
USGS CMG PortalUSGS CMG Portal
![Page 17: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/17.jpg)
NetCDF Point Subset ServiceNetCDF Point Subset Service
![Page 18: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/18.jpg)
Iris Python tools from the UK Met OfficeIris Python tools from the UK Met Office
![Page 19: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/19.jpg)
Automated model comparisonAutomated model comparison
![Page 20: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/20.jpg)
![Page 21: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/21.jpg)
Getting your model results connectedGetting your model results connected
• Find someone with a THREDDS Data Server or install your own
• Drop your files in a directory, and add an NcML file that starts with “00_dir” (e.g. “00_dir_roms.ncml”) to aggregate, standardize and describe the dataset: Sample ROMS NcML file
• If you want your data to end up in the portal, add “CMG_Portal” to the “project” attribute: <attribute name=“project”value=“CMG_Portal”/>
• If you want your datasets to be discoverable, submit a PR on list of thredds catalogs being scanned on github
• Full instructions on the USGS-CMG Portal Github Wiki
• Find someone with a THREDDS Data Server or install your own
• Drop your files in a directory, and add an NcML file that starts with “00_dir” (e.g. “00_dir_roms.ncml”) to aggregate, standardize and describe the dataset: Sample ROMS NcML file
• If you want your data to end up in the portal, add “CMG_Portal” to the “project” attribute: <attribute name=“project”value=“CMG_Portal”/>
• If you want your datasets to be discoverable, submit a PR on list of thredds catalogs being scanned on github
• Full instructions on the USGS-CMG Portal Github Wiki
![Page 22: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/22.jpg)
A few problems… PackagingA few problems… Packaging
• Ipython notebooks are a great way to document model skill assessment workflows (Filipe will talk about this)
• But python environment uses a lot of tricky packages. How to make this easy for folks?
• Conda and binstar to the rescue! (Filipe will talk about this)
• Ipython notebooks are a great way to document model skill assessment workflows (Filipe will talk about this)
• But python environment uses a lot of tricky packages. How to make this easy for folks?
• Conda and binstar to the rescue! (Filipe will talk about this)
![Page 23: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/23.jpg)
A few problems… WMSA few problems… WMS
• ncWMS works great for CF compliant data• Unstructured grids are not CF compliant.• Staggered grids are not CF compliant. • ncWMS doesn’t work for unstructured grid data
(FVCOM, ADCIRC, SELFE), and doesn’t work for staggered grid velocities in models like ROMS, WRF and Delft3D
• sci-wms to the rescue, using UGRID conventions for unstructured grid (pyugrid), and SGRID conventions for staggered grid (pysgrid). (Kyle will talk about this)
• ncWMS works great for CF compliant data• Unstructured grids are not CF compliant.• Staggered grids are not CF compliant. • ncWMS doesn’t work for unstructured grid data
(FVCOM, ADCIRC, SELFE), and doesn’t work for staggered grid velocities in models like ROMS, WRF and Delft3D
• sci-wms to the rescue, using UGRID conventions for unstructured grid (pyugrid), and SGRID conventions for staggered grid (pysgrid). (Kyle will talk about this)
![Page 24: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/24.jpg)
Key Infrastructure ComponentsKey Infrastructure Components
• Common data models for “feature types” (structured, staggered and unstructured grids, time series, profiles, swaths) (Unidata CDM, UGRID, SGRID)
• Standard web data services for delivering these common data model “feature types” (OPeNDAP/CF/UGRID/SGRID, WMS, SOS, WFS, ERDDAP/tabledap, ERDDAP/griddap)
• Standard catalog services for the metadata (OGC CSW, OpenSearch)
• Tools for easy delivery of data in standard services
• Tools for easy search, access and use of data in standard services (in all major environments: Python, ArcGIS, R, Matlab, JavaScript)
• Common data models for “feature types” (structured, staggered and unstructured grids, time series, profiles, swaths) (Unidata CDM, UGRID, SGRID)
• Standard web data services for delivering these common data model “feature types” (OPeNDAP/CF/UGRID/SGRID, WMS, SOS, WFS, ERDDAP/tabledap, ERDDAP/griddap)
• Standard catalog services for the metadata (OGC CSW, OpenSearch)
• Tools for easy delivery of data in standard services
• Tools for easy search, access and use of data in standard services (in all major environments: Python, ArcGIS, R, Matlab, JavaScript)
![Page 25: 2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.](https://reader035.fdocuments.us/reader035/viewer/2022062217/56649e865503460f94b8a14e/html5/thumbnails/25.jpg)
Infrastructure BenefitsInfrastructure Benefits
• What are the benefits?– Less time wasted messing with data,
more time spent on science– More skill assessment of models– More usage and more appropriate
useage of model results – Faster feedback to modelers =>
improved models– Better science, better models =>better
world
• What are the benefits?– Less time wasted messing with data,
more time spent on science– More skill assessment of models– More usage and more appropriate
useage of model results – Faster feedback to modelers =>
improved models– Better science, better models =>better
world