1. The HDF Group HDF-OPeNDAP Project Update Joe Lee and Kent Yang The HDF Group James Gallagher OPeNDAP, Inc. The 13th HDF and HDF-EOS Workshop November 3-5, 2009 November 3-5, 2009 HDF/HDF-EOS Workshop XIII 1 www.hdfgroup.org

2. Outline Introduction to OPeNDAP Whats New? o HDF5-OPeNDAP Handler (minor) o HDF4-OPeNDAP Handler (major) November 3-5, 2009 HDF/HDF-EOS Workshop XIII 2 www.hdfgroup.org

3. Access HDF via FTP Aqua/Aura Users IDL AIRS/OMI FTP Local HDF Files November 3-5, 2009 FTP Clients FTP Servers HDF/HDF-EOS Workshop XIII HDF Files 3 www.hdfgroup.org

4. OPeNDAP Concept Typical Remote Access View Data (User) FTP/HTTP Remote Data (HDF4/5) Visualization Tools tools (e.g., grads) (gradsdap) Handler (hdf4/5_handler) Client Library (libnc_dap) Server (Hyrax) Data Access Protocol (DAP) via Internet November 3-5, 2009 HDF/HDF-EOS Workshop XIII 4 www.hdfgroup.org

5. Access HDF via OPeNDAP Aqua/Aura Users IDV HyraxHANDLER HDF AIRS/OMI libnc-dap DAP Visualization Tools November 3-5, 2009 OPeNDAP Clients OPeNDAP Servers HDF/HDF-EOS Workshop XIII HDF Files 5 www.hdfgroup.org

6. Whats New? HDF5-OPeNDAP Handler Work November 3-5, 2009 HDF/HDF-EOS Workshop XIII 6 www.hdfgroup.org

7. HDF5 Handler Update Aura Users IDV HDF5 HANDLER Update Hyrax OMI Swath Grid libnc-dap DAP Visualization Tools November 3-5, 2009 OPeNDAP Clients OPeNDAP Servers HDF/HDF-EOS Workshop XIII HDF5 Files 7 www.hdfgroup.org

8. HDF5-OPeNDAP Handler Update Support Aura HDF-EOS5 OMI Swath Access some files that follow HDF5 Dimension Scale Specification Change in Short Name Option RadiativeCloudFraction A3RadiativeClou November 3-5, 2009 HDF/HDF-EOS Workshop XIII 8 www.hdfgroup.org

9. Summary: Updated HDF5 Handler Official Release: October 6, 2009 http://opendap.org/hdf5_handler November 3-5, 2009 HDF/HDF-EOS Workshop XIII 9 www.hdfgroup.org

10. Whats New? HDF4-OPeNDAP Handler Work November 3-5, 2009 HDF/HDF-EOS Workshop XIII 10 www.hdfgroup.org

11. HDF4-OPeNDAP Handler Work Problems with the old HDF4 Handler Our Work Limitations Future Work November 3-5, 2009 HDF/HDF-EOS Workshop XIII 11 www.hdfgroup.org

12. Old HDF4 Handler Problem OPeNDAP Visualization Tools CAN NOT display most NASA HDF-EOS2 and HDF4 data November 3-5, 2009 HDF/HDF-EOS Workshop XIII 12 www.hdfgroup.org

13. Old HDF4 Handler Problem Aqua Users IDV Hyrax Old HDF4 HANDLER AIRS Grid libnc-dap DAP Visualization Tools November 3-5, 2009 OPeNDAP Clients OPeNDAP Servers HDF/HDF-EOS Workshop XIII HDF4 Files 13 www.hdfgroup.org

14. Our Work Let Visualization Clients Access HDFEOS2 and HDF4 files November 3-5, 2009 HDF/HDF-EOS Workshop XIII 14 www.hdfgroup.org

15. Whats Missing? My house costs 1. Data Computer Science int my_house = 1; // Use boolean to save? My house in Hawaii costs $1M. Information Earth Science Geo-Location, Geo-Location, Geo-Location Unit, Scale factor November 3-5, 2009 HDF/HDF-EOS Workshop XIII 15 www.hdfgroup.org

16. Whats Missing? Clients expect certain data structures o DAP Grids and Shared dimensions Clients expect some key attributes o CF conventions November 3-5, 2009 HDF/HDF-EOS Workshop XIII 16 www.hdfgroup.org

17. Solution: Metadata Parser Goal: Meet the Clients Requirements Parse the structMetadata to o generate the DAP Grids and Shared Dims o generate the CF-compliant attributes November 3-5, 2009 HDF/HDF-EOS Workshop XIII 17 www.hdfgroup.org

18. New HDF4 Handler (parser) Aqua Users IDV libnc-dap New HDF4 HANDLER (parser) Hyrax Old HDF4 HANDLER AIRS Grid DAP Visualization Tools November 3-5, 2009 OPeNDAP Clients OPeNDAP Servers HDF/HDF-EOS Workshop XIII HDF4 Files 18 www.hdfgroup.org

19. Demo: Parser Solution Problem: Only a few HDF-EOS2 files work! AIRS Grid AIRS Swath MODIS Swath MODIS Grid AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G08303124144.hdf ftp://airspar1u.ecs.nasa.gov/data/s4pa//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/AIRS.2008.11.25.143.L2.RetStd.v5.2.2.0.G0 http://daac.gsfc.nasa.gov/daac-bin/whom/mk_page_cgi.pl?PATH=datapool/AIRS/Level3_V005/01_AIRX3STD/2008/10 MOD08_D3.A2008284.005.2008289105354.hdf MOD06_L2.A2001222.0025.004.2003127111841.hdf 8331121849.hdf http://modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiring.html XIII November 3-5, 2009 HDF/HDF-EOS Workshop 19 www.hdfgroup.org http://modis-atmos.gsfc.nasa.gov/MOD06_L2/acquiring.html

20. New Challenges for HDF4 Handler Hundreds of HDF-EOS2 and HDF4 products o Grid Projections o Swath Dimension Map November 3-5, 2009 HDF/HDF-EOS Workshop XIII 20 www.hdfgroup.org

21. New Challenges HDF-EOS2 Grid projections Geographic Sinusoidal Polar Stereographic etc. HDF-EOS2 Swath Dimension Map November 3-5, 2009 HDF/HDF-EOS Workshop XIII 21 www.hdfgroup.org

22. Solution: Use HDF-EOS2 Library Goal: Support more HDF-EOS2 products Use HDF-EOS2 Library to o handle different Grid projections o handle Swath dimension map November 3-5, 2009 HDF/HDF-EOS Workshop XIII 22 www.hdfgroup.org

23. New HDF4 Handler (Parser) New HDF4 Handler (EOS2 library) New HDF4 HANDLER (parser) (EOS2 library) Aura Users IDV MODIS Swath w/ Dim. Map MODIS Swath Hyrax libnc-dap DAP Visualization Tools November 3-5, 2009 OPeNDAP Clients OPeNDAP Servers HDF/HDF-EOS Workshop XIII HDF4 Files 23 www.hdfgroup.org

24. Demo: HDF-EOS2 Library Solution Swath with Dimension Map November 3-5, 2009 HDF/HDF-EOS Workshop XIII 24 www.hdfgroup.org

25. Two Solutions Metadata Parser HDF-EOS2 Library Pro: Simple installation Pro: More products Con: Limited products Con: HE2 library installation November 3-5, 2009 HDF/HDF-EOS Workshop XIII 25 www.hdfgroup.org

26. Two Solutions - Implementation Adapted the implementation of previous software packages Metadata Parser HDF-EOS2 Library hdf5_handler h4toh5 tool November 3-5, 2009 HDF/HDF-EOS Workshop XIII 26 www.hdfgroup.org

27. Solution Comparison November 3-5, 2009 HDF/HDF-EOS Workshop XIII 27 www.hdfgroup.org

28. Live Demo Tutorial Example o http://hdfeos.org/software/idv.php Demo Servers o http://hdfgroup.org/projects/opendap o http://hdfgroup.org/projects/opendap4 November 3-5, 2009 HDF/HDF-EOS Workshop XIII 28 www.hdfgroup.org

29. Limitations 7 Reasons for Visualization Failures Sample Files for Visualization Failures o http://hdfgroup.org/projects/opendap4/limits.html November 3-5, 2009 HDF/HDF-EOS Workshop XIII 29 www.hdfgroup.org

30. The Tip of the Iceberg The current HDF4 handler CAN NOT visualize most of NASA HDF-EOS2 / HDF4 files November 3-5, 2009 HDF/HDF-EOS Workshop XIII 30 www.hdfgroup.org

31. More Challenges for HDF4 Handler Tackle limitations and other challenges. Handle NASA HDF4 (e.g.,CERES) files. In 2010 November 3-5, 2009 HDF/HDF-EOS Workshop XIII 31 www.hdfgroup.org

32. Summary: Enhanced HDF4 Handler Official Release on Oct. 6, 2009. http://opendap.org/hdf4_handler Support some HDF-EOS2 Grid / Swath files Use HDF-EOS2 library Future work: support more NASA HDF-EOS2 / HDF4 files November 3-5, 2009 HDF/HDF-EOS Workshop XIII 32 www.hdfgroup.org

33. The HDF Group Thank You! November 3-5, 2009 HDF/HDF-EOS Workshop XIII 33 www.hdfgroup.org

34. Acknowledgements This work was supported by cooperative agreement number NNX06AC83A, NNX08AO77A and NNX06AG75A from the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space Administration. November 3-5, 2009 HDF/HDF-EOS Workshop XIII 34 www.hdfgroup.org

35. The HDF Group Questions/comments? November 3-5, 2009 HDF/HDF-EOS Workshop XIII 35 www.hdfgroup.org