HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual...
-
Upload
chester-lawrence -
Category
Documents
-
view
213 -
download
0
Transcript of HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual...
![Page 1: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/1.jpg)
HDF OPeNDAP Project Update
MuQun Yang and Hyo-Kyung Lee
The HDF Group
March 31, 2009
1Annual briefing to ESDIS04/20/23
![Page 2: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/2.jpg)
Outline
• HDF5 – OPeNDAP work review• HDF4 – OPeNDAP handler progress report• Future work plan
04/20/23 Annual briefing to ESDIS 2
![Page 3: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/3.jpg)
Users
OPeNDAP Clients
IDV
OPeNDAP Servers
HDF4 Files
Hyrax AIRS
DAP
libnc-dap
Visualization Tools
Access data via OPeNDAP
HDF HANDLER
04/20/23 3Annual briefing to ESDIS
![Page 4: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/4.jpg)
HDF5 – OPeNDAP Work
4Annual briefing to ESDIS04/20/23
![Page 5: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/5.jpg)
HDF5-OPeNDAP handler
• Update with new Hyrax• Help GES DISC serve TES and OMI data• Finish HDF5 – DAP2 mapping document and
submit it as a technical note to NASA SPG
04/20/23 Annual briefing to ESDIS 5
![Page 6: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/6.jpg)
04/20/23 Annual briefing to ESDIS 6
HDF5-Friendly OPeNDAP Client Library
• Retrieve the key HDF5 information• Easy to access other HDF5 data (e.g. EOS5
swath)
![Page 7: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/7.jpg)
Example: Groups in HDF5Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it.
HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5
7Annual briefing to ESDIS04/20/23
![Page 8: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/8.jpg)
04/20/23 Annual briefing to ESDIS 8
MLS swath via GrADS
CO at these elevations is a measure of pollution
lower stratosphere
upper stratosphere
Before
After
CO level near Beijing, China before, during and after Olympics
![Page 9: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/9.jpg)
HDF4 – OPeNDAP Work
9Annual briefing to ESDIS04/20/23
![Page 10: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/10.jpg)
Current HDF4-OPeNDAP handler
• Contributed by JPL in 1998• Part of the OPeNDAP distribution• Visualization client tools cannot display most
HDF-EOS2 and HDF4 data
04/20/23 Annual briefing to ESDIS 10
![Page 11: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/11.jpg)
• Enhance the HDF4-OPeNDAP handler
• More NASA HDF-EOS2 and HDF4 data can be accessed via visualization clients
11Annual briefing to ESDIS04/20/23
Our work
![Page 12: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/12.jpg)
Challenges – more products
12Annual briefing to ESDIS04/20/23
Hundreds of HDF-EOS2 and HDF4 Data products
![Page 13: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/13.jpg)
Challenges – Requirement from Clients
• CF conventions
- Clients expect a certain data structure
- Clients expect some key attributes
13Annual briefing to ESDIS04/20/23
![Page 14: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/14.jpg)
• Various HDF-EOS2 Grid projections• Geographic• Sinusoidal• Polar stereographic• …
04/20/23 Annual briefing to ESDIS 14
Challenges – Grid
Data [4][12]Lon[12]Data [4][8]Lon[4][8]
Geographic
Sinusoidal
![Page 15: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/15.jpg)
• HDF-EOS2 Swath- Dimension map
04/20/23 Annual briefing to ESDIS 15
Challenges - SwathData field has three columnsLongitude field has two columns Longitude should have three columns
![Page 16: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/16.jpg)
Challenges – various products • “Special” HDF-EOS2 files
• Hybrid HDF-EOS2 files- Some HDF4 objects that cannot be read by EOS
library
• HDF4 data products
04/20/23 Annual briefing to ESDIS 16
![Page 17: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/17.jpg)
Enhanced HDF4 Handler
04/20/23 Annual briefing to ESDIS 17
• Follow CF conventions Generate the correct DAP data type. Add shared geo-location information. Insert additional key attributes.
How: Parse Metadata or Use EOS2 library
![Page 18: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/18.jpg)
Implementation choice
• Use Metadata parser or HDF-EOS2 library• Metadata parser
• Pro: simple
• Con: limited products
• HDF-EOS2 library• Pro: maximum products
• Con: install HDF-EOS2 library first
04/20/23 Annual briefing to ESDIS 18
![Page 19: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/19.jpg)
Solution Comparison
19Annual briefing to ESDIS04/20/23
![Page 20: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/20.jpg)
Quick Implementation
• Parse Metadata – hdf5_handler
• Use HDF-EOS2 library – h4toh5
20Annual briefing to ESDIS04/20/23
![Page 21: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/21.jpg)
Current Status
• MODIS / AIRS Grid and Swath
• Other Projections (Polar, Sinusoidal)
• Testing
21Annual briefing to ESDIS04/20/23
![Page 22: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/22.jpg)
HDF-EOS2 in CF via DAP
2204/20/23
AIRS Grid AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G08303124144.hdfhttp://daac.gsfc.nasa.gov/daac-bin/whom/mk_page_cgi.pl?PATH=datapool/AIRS/Level3_V005/01_AIRX3STD/2008/10MODIS GridMOD08_D3.A2008284.005.2008289105354.hdfhttp://modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiring.html
AIRS Swath ftp://airspar1u.ecs.nasa.gov/data/s4pa//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/
AIRS.2008.11.25.143.L2.RetStd.v5.2.2.0.G08331121849.hdfMODIS Swath
MOD06_L2.A2001222.0025.004.2003127111841.hdfhttp://modis-atmos.gsfc.nasa.gov/MOD06_L2/acquiring.htmlAnnual briefing to ESDIS
![Page 23: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/23.jpg)
More challenges: Missing Attributes
• MODIS Grid – Rain fall
23Annual briefing to ESDIS04/20/23
No Unit!
• AURA OMI Grid - Ozone
![Page 24: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/24.jpg)
More challenges Match dimension names
• Horizontal geolocation information:
- Non-standard names ( e.g. nlon, nlat)
How to match?• Vertical geolocation information:
- Users provide the geolocation name and vaule but no conventions are enforced.
How to know?
04/20/23 Annual briefing to ESDIS 24
![Page 25: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/25.jpg)
More challenges – Missing Values• In a swath file, a data field has three dimensions. But no dimension value for the third dimension.
25Annual briefing to ESDIS04/20/23
MISR_AM1_TC_CLASSIFIERS_P020_O000995_F06_0011.hdf
![Page 26: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/26.jpg)
A Solution to challenges
• Provide an option for the handler to generate fully CF-compliant output
• Create a separate XML input file for the handler and feed it into the handler
• Insert attributes, renaming rules, and missing values by putting them in the XML input
• This requires a knowledge about the characteristics of data in advance
26Annual briefing to ESDIS04/20/23
![Page 27: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/27.jpg)
One Solution An input file from data provider
• Example: Attributes, Renaming, Values
27Annual briefing to ESDIS04/20/23
<xml><variable name=‘RrLandRain’> <attr name=‘unit’ type=‘String’> inches</attr></variable><rename from=‘pressStd’ to=‘level’></rename><insert name=‘NAltitudeDim’ size=10 type=‘Int8’>0,1,2,3,4,5,6,7,8,9</insert></xml>
![Page 28: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/28.jpg)
Future Work (Short Term)
• Expand test-suite with data pool
• Release through OPeNDAP website
• Support for Swath via Parsing
• Support for Point data type via HDF-EOS2 library
28Annual briefing to ESDIS04/20/23
![Page 29: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/29.jpg)
Future Work (Long Term)
• General HDF4 Files Support
• Using an input XML file to provide full
CF support
• HDF-EOS2 to DAP2 Mapping Document
29Annual briefing to ESDIS04/20/23
![Page 30: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/30.jpg)
Thank You!
October 15-18, 2008 Workshop Name (edit footer to change) 30
![Page 31: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/31.jpg)
Acknowledgements
This work was supported by the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A, NNX08A077A and NNX06AG75A.
Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.
October 15-18, 2008 31Workshop Name (edit footer to change)
![Page 32: HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, 2009 1 Annual briefing to ESDIS10/31/2015.](https://reader031.fdocuments.us/reader031/viewer/2022020417/56649f0e5503460f94c23092/html5/thumbnails/32.jpg)
Questions/comments?
October 15-18, 2008 Workshop Name (edit footer to change) 32