Towards Long-Term Archiving of NASA HDF-EOS and HDF Data
Data Maps and the Use of Mark-Up Language
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Ruth Duerr, Mike Folk, Muqun Yang, Chris Lynnes, Peter Cao
Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Outline
• Background
• Data Mapping Project Description
• Plans and Early Results
Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Outline
• Background
• Data Mapping Project Description
• Plans and Early Results
Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
A Concern
• The majority of the data from NASA’s Earth Observing System (EOS) have been archived in HDF Version 4 (HDF4) or HDF-EOS 2 format.
• HDF files have a complex internal byte layout, requiring one to use the API to access HDF data
• Long-term readability of HDF data depends on long-term allocation of resources to support the API
Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
A Proposal from the Workshop Last Year
• Chris Lynnes noted that What was needed was a map to the
contents of an HDF file The output of the HDF4 tools (e.g., hdfls,
hdp, etc.) already provide much of the information needed
Extending these tools to create a map to the contents of the file might be feasible
Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Outline
• Background
• Data Mapping Project Description
• Plans and Early Results
Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Data Mapping Project Description
• Assess and categorize NASA holdings of HDF4 data
• Investigate methods of mapping HDF4 files• Develop requirements for tools to create
maps of HDF4 files• Create a prototype tool to create maps• Test the utility of these maps by developing 2
independent tools that use the maps to read real data
Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Data Mapping Project Description (continued)
• Assess the utility of this approach
• Document our findings
• Present results and options for proceeding to the user community
• Evaluate the effort required for a full solution that meets community needs
• Submit a proposal for that effort
Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Outline
• Background
• Data Mapping Project Description
• Plans and Early Results
Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Assess and Categorize NASA Holdings
While the volume of NASA data stored in HDF4/HDF-EOS2 format is measured in PB; the fraction of the total number of NASA data sets archived in HDF4/ HDF-EOS2 is “small”
• NASA provided a starter list of data sets held
• NASA data centers were requested to provide a list at a project briefing
• Results from each DAAC being compared to ECHO assessment of data sets using a .hdf extension
Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Assess and Categorize NASA Holdings (continued)
• Examples of each of the hdf4 data sets have been obtained and examined*
• Information kept summarized below:
• Product id/name• Data Center• Product Version• Multi-file product?• HDF/EOS info (if any)
HDF/EOS version Point info Swath info Grid info
• HDF info Version Raster image info Palette SDS info V data info Annotation
* For the most part
Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Assess and Categorize NASA Holdings (continued)
• Very preliminary findings Roughly 50/50 split between HDF-EOS
and plain HDF Point data is relatively rare and when found
is not accompanied by swath or grid data No indexes yet While a few products use the image types,
there are no palettes yet
Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Investigate Methods of Mapping HDF4 Files
• NSIDC and GES-DISC have provided THG sample data files• Preliminary priorities for capabilities to tackle:
Contiguous SDS Contiguous SDS with unlimited dimension Chunked SDS Compressed SDS Chunked and compressed SDS SDS and attributes Vdata and attributes Annotation Vgroup Raster image and attributes
Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Investigate Methods of Mapping HDF4 Files
• NSIDC and GES-DISC have provided THG sample data files• Preliminary priorities for capabilities to tackle:
Contiguous SDS Contiguous SDS with unlimited dimension Chunked SDS Compressed SDS Chunked and compressed SDS SDS and attributes Vdata and attributes Annotation Vgroup Raster image and attributes
Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Develop Requirements for Tools to Create Maps
• Maps will be XML-based
• A draft of a map format specification has been started
Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Create a Prototype Tool to Create Maps
• An iterative process is being used to create the prototype
• Each iteration adds the next capability from the prioritized list shown earlier
• At this point, the tool just creates a text description
Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Communications Plan
• Bi-weekly telecons with our sponsors (may move to monthly)
• Briefing to NASA Data Center managers held, expect to provide periodic updates
• Brief community at the HDF-Workshop and other relevant meetings (e.g., AGU)
• Submit a paper to the special issue of IEEE Transactions of Geoscience and Remote Sensing devoted to Data Archiving and Distribution
• Public wiki established but not yet populated
Presented at the HDF and HDF-EOS Workshop XI - Nov. 6-8, 2007Landover, Maryland
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Summary
• We’ve started a project to assess and prototype the ability to create maps to the contents of HDF4 files that allow programmers to develop code to read data without using the HDF APIs
• We welcome community involvement
Top Related