And now for something completely different - HYCOM.org · And now for something completely...
Transcript of And now for something completely different - HYCOM.org · And now for something completely...
![Page 1: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/1.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
And now for something completely different …
(data management?)
![Page 2: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/2.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Ashwanth Srinivasan (RSMAS)Steve Hankin (PMEL)
HYCOM National MeetingOctober, 2004
HYCOM Data Management& Services
HYCOM Data Management& Services
A community of contributors, including …Peter Cornillon, OPeNDAP PI (URI)
Mike Clancy, US GODAE Server (FNMOC)
IPRC, Unidata, MBARI, COLA, DOE/PCMDI, NOAA/GFDL&CDC&PFEL
Jon Callahan, Joe Mclean, Kevin O’Brien (U.Washington/JISAO)
Roland Schweitzer, Yonghua Wei (contractors) Ansley Manke (NOAA/PMEL)
James Gallagher, Dan Holloway, et. al. (URI)Phil Sharfstein, Dave Dimitriou (FNMOC)
A community of contributors, including …Peter Cornillon, OPeNDAP PI (URI)
Mike Clancy, US GODAE Server (FNMOC)
IPRC, Unidata, MBARI, COLA, DOE/PCMDI, NOAA/GFDL&CDC&PFEL
Jon Callahan, Joe Mclean, Kevin O’Brien (U.Washington/JISAO)
Roland Schweitzer, Yonghua Wei (contractors) Ansley Manke (NOAA/PMEL)
James Gallagher, Dan Holloway, et. al. (URI)Phil Sharfstein, Dave Dimitriou (FNMOC)
![Page 3: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/3.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Part I. The Requirements, the Vision, the ToolsSteve Hankin, NOAA/PMEL
Part II. Current System and CapabilitiesAshwanth Srinivasan, RSMAS
Part III. Plenary discussion: goals and priorities
HYCOM Data ManagementHYCOM Data ManagementHYCOM Data Management& Services
HYCOM Data Management& Services
Part IIb. - Product serving activities at the IPRCPeter Hacker
![Page 4: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/4.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Outline
Part I.• Quick Review• Data Management Requirements• Around the corner (FDS)
Part II.Current System and Capabilities
Part III.Plenary data management priorities
![Page 5: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/5.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Web access to model outputs and observations from distributed sites
Live Access Server (LAS)
Quick Review
Live Access Server (LAS)
Quick Review
![Page 6: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/6.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
![Page 7: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/7.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
netCDFnetCDF
![Page 8: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/8.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Compare variables from different models and data products
![Page 9: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/9.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
![Page 10: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/10.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Compute simple analyses
![Page 11: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/11.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
![Page 12: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/12.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Ocean obs via LAS(incl. custom “constraints”)
![Page 13: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/13.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
GTS Ocean obs onUS GODAE Server
![Page 14: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/14.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
• XML Metadata contains the “intelligence”• Back end applications do the real work• OPeNDAP provides remote data access
LAS -- an “Information Product Server”
Internet
2
3
LocalData
LAS productserver(perl)
NetScape
Temp
Dep
th 4
Internet(OPenDAP)
RemoteData
Meta-data
LASUI
1LAS UIserver(java)
workapplication
Ferret,Ferret,CDATCDAT
or otheror other
![Page 15: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/15.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
OPeNDAP: network access to data and “semantic metadata”
![Page 16: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/16.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Query available data sets:
Query variables in data set “model_1”:
Query space-time domain:
Request a subset of data as a file: (“asc” for ASCII format)
“Scripted” access to LAS
>lasls >lasls http://cpu/LAShttp://cpu/LAS
>lasls >lasls http://cpu/LAShttp://cpu/LAS model_1model_1
>lasls >lasls http://cpu/LAShttp://cpu/LAS model_1 model_1 sstsst
>lasget >lasget --x 20:60 x 20:60 --y 20:60 y 20:60 --t 11t 11--DecDec--2000 2000 --f f ascaschttp://cpu/LAShttp://cpu/LAS model_1 model_1 sstsst
![Page 17: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/17.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
HYCOM information & data portal
InformationInformationaccessaccess
Uniform data Uniform data accessaccess
Live Access ServerLive Access Server
homehome
Area 1Area 1 Area 2Area 2 Area 3Area 3
Project 1Project 1 Project 2Project 2 Project 3Project 3 Project 4Project 4
Exp 1Exp 1
Exp 2Exp 2
Exp 1Exp 1
Exp 2Exp 2
![Page 18: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/18.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Outline
Part I.• Quick Overview (LAS & OPeNDAP)• Data Management Requirements• Around the corner (FDS)
Part II.Current System and Capabilities
Part III.Plenary data management priorities
![Page 19: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/19.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Our goal:
Create a framework for sharing data that permits HYCOM modeling projects (and others) to subset, browse, analyze and inter-compare model outputs and compare to observations
![Page 20: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/20.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Why is this important?
• Facilitate collaboration between partners(especially basin scale coastal models)
• Facilitate model validation
• Reach other users; increase the visibility of the project
![Page 21: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/21.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Outline
Part I.• Quick Overview (LAS & OPeNDAP)• Data Management Requirements• Around the cornerPart II.
Current System and Capabilities
Part III.Plenary data management priorities
![Page 22: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/22.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
FDS“Ferret Data Server”
• OPeNDAP output for …– any variable available from LAS– with uniform metadata standards
• Built on a Java framework from COLA(called ‘Anagram’)
![Page 23: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/23.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
FDS
• We can configure an LAS to present HYCOM output variables, regridded to the Levitus grid (2001) …
![Page 24: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/24.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
On-the-fly regriddingfrom curvilinearto rectilinearcoordinates
![Page 25: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/25.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Similarlyfrom hybrid-Zto rectilinear
![Page 26: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/26.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
FDS
• The rectilinear-grid “view” of that data is now available via OPeNDAP for a number of purposes …
![Page 27: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/27.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
HYCOM data at the desktop(for Matlab, IDL, Ferret, GrADS, …)
The remote dataset is just a “filename”http://server/FDS/my_HYCOM_dataset
The dataset will be exactly as LAS presents it.
![Page 28: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/28.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Comparison between datasets becomes straightforward:
Servers request regridded data from one another via FDS.
LASUI
NetScape
DataLAS
site 1
Meta
MetaVIRTUAL server
DataLAS
Meta
DataLAS
site 2
Meta
site 4
DataLAS
Meta
site 3
LASuser
interface
Meta Meta
Meta
Collaborating “sister” servers
![Page 29: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/29.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
HYCOM data at the desktop(for Matlab, IDL, Ferret, GrADS, …)
You can ask FDS to perform simple analyses:
Vertical average of variable “TEMP” http://server/FDS/_expr_{my_output}{Tave=TEMP[Z=@AVE]}
(reduces data volumes transferred over the net)
![Page 30: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/30.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
FDS
• FDS employs “delayed analysis”– It appears that the whole dataset has been
transformed– In reality the selected subset is transformed
on-demand• Arbitrary regridding on-demand, too (with
cautions)
![Page 31: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/31.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Demanding (large) products may be created in “batch”
mode(scheduled for V6.5)
![Page 32: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/32.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
OutlinePart I.• Quick Overview (LAS & OPeNDAP)• Data Management Requirements• Around the corner (FDS)
Part II.Current System and Capabilities
Ashwanth Srinivasan
Part III.Plenary data management priorities
![Page 33: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/33.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
OutlinePart I.• Quick Overview (LAS & OPeNDAP)• Data Management Requirements• Around the corner (FDS)
Part II.Current System and Capabilities
Ashwanth Srinivasan
Part III.Plenary: data management priorities
![Page 34: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/34.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Plenary Discussion:Data Management Priorities
• Immediate priorities– To make HYCOM basin-scale outputs readily
usable by coastal modeling groups• Curvilinear? hybrid-Z? rectilinear?• Formats (CF ….)? Format for finite element
modeling sites?– How much data kept on-line ?– Are we succeeding ?
How to gather feedback ?
![Page 35: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/35.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Plenary Discussion:Data Management Priorities
• 2nd tier priorities– A “data portal” for all project participants ?
• Basin-scale and regional data, model outputs, and descriptions
• How can we make it easy/tempting/compelling to get ALL models on-line ?
– Other suggestions? How else can we facilitate coastal modeling activities?
• Who and what specific needs ?
![Page 36: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/36.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Plenary Discussion:Data Management Priorities
• ‘Next’ Priorities– What reference data sources for comparisons?
• Moored time series? GTS surface obs? Climatologies? Met models? Other ocean models?
– Model-data comparison techniques ?• Sparse data techniques? Dense data techniques? Error fields?
Correlations?
– Comparison between neighboring domains?– Ensemble comparison techniques ?– Metrics useful for comparison ?
![Page 37: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/37.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Plenary Discussion:Data Management Priorities
• ‘Next’ Priorities– “3 time axis” data management– Special user interfaces? Education? GIS-
style?– Are there other applications to make
compatible with the HYCOM framework ?
– Other suggestions?(Break out of the box!)
![Page 38: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/38.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Plenary Discussion:Data Management Priorities
• “Technical” priorities– Visualizations of time series, profiles,
sections, … from curvilinear data ? (done)– Graphics on ‘native’ curvilinear coordinates ?– Automate synchronization of Web portal with
model outputs– Higher efficiency storage techniques
(e.g. HDF with “chunking” and compression)?
![Page 39: And now for something completely different - HYCOM.org · And now for something completely different ... different models and data products. Oct. 2004 HYCOM Nat’l Meeting Steve](https://reader034.fdocuments.us/reader034/viewer/2022050410/5f8776614787415ec449f7ea/html5/thumbnails/39.jpg)
Oct. 2004 HYCOM Nat’l Meeting Steve Hankin
Plenary Discussion:Data Management Priorities
• Suggestions:– .push capability?– .LAS production of native HYCOM format– outputs available daily?– Mercator projection of maps?– How to encourage regional modeling sites to make
data available (via OPeNDAP, LAS, FTP,or other)?– .– .– .