HDF
-
Upload
the-hdf-eos-tools-and-information-center -
Category
Technology
-
view
146 -
download
2
description
Transcript of HDF
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 1
Mike Folk,HDF Grouphttp://hdf.ncsa.uiuc.edu/
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
HDFHDF
HDF-EOS Workshop IIHDF-EOS Workshop IISept. 22, 1998Sept. 22, 1998
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 2
I. OverviewI. Overview
• What is HDF?
• HDF software
• HDF objects
• Who uses HDF?
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 3
What is HDF?What is HDF?
• A data file format for managing scientific data
in heterogeneous environments
• I/O software, utilities, search and access tools
• A standard used widely among scientists
• Enabling data interoperability since 1988
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 4
Requirements for scientific dataRequirements for scientific data
• “Scientific” data
– A variety of data types and structures
– Large structures
– Metadata in a variety of forms
• Portability
– I/O library works on many machines
– Data easily moved from machine to machine
• Fast I/O
• Efficient storage
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 5
University of Illinois at Urbana-Champaign
Why use HDF?Why use HDF?
• Share scientific data
– in heterogeneous computing environments
– with others of like interests
• To use software that understands HDF
• To improve I/O performance
• To improve storage efficiency
• To use an open standard
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 6
HDF Object TypesHDF Object Types
Palette
8-bit raster
24-bit raster
Scientific Data Sets (SDS)(multidimensional arrays)
Vdata (tables)
lat | lon | temp
----|-----|-----
12 | 23 | 3.1
15 | 24 | 4.2
17 | 21 | 3.6March 15, 1990. Simulation with
k=10.0, beta=1.22e3. Calculate
the magnitude ...
Vgroup
AnnotationThis HDF file containsone example of eachobject type
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 7
An HDF File: A Collection ofAn HDF File: A Collection of
Scientific Data ObjectsScientific Data Objects
HDF file containing four 3-D arraysHDF file containing four 3-D arrays
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 8
Mixing HDF Objects in One FileMixing HDF Objects in One File
March 15, 1990. Simulation with
k=10.0, beta=1.22e3. Calculate
the magnitude ...
3-D array3-D arrayRaster imageRaster image
2-D array2-D array
groupgroup
RasterRaster
imageimage
palettepalette
annotationannotation
HDF fileHDF file
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 9
The HDF File StructureThe HDF File Structure
Primitive Data Object
tag ref offset length
data element
tag ref offset length
data element
Data descriptor (DD):Data descriptor (DD):
Data:Data:
(Everything else in an HDF file is build out of these.)(Everything else in an HDF file is build out of these.)
(12 bytes)(12 bytes)
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 10
Example of hierarchical HDF structureExample of hierarchical HDF structureRaster Image Group (original format)Raster Image Group (original format)
dim's 2 offst lenimage 3 offst lenpalette 3 offst len 4 offst lenRIG
tag ref
Blksize
Nextblk
2
400 x 600
Data elements:
tag ref
palette
image
dim's
3
3
Index:
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 11
Other features of interestOther features of interest
• Data compression
• The data part of an object can be stored in
“special” ways
– As a series of linked-blocks
– As a portion of an external file
– In chunks, or tiles
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 12
Utilities and applications forUtilities and applications formanipulating, viewing, andmanipulating, viewing, andanalyzing data in HDF files.analyzing data in HDF files.
A software library:A software library:–– High-level, object-specific APIs. High-level, object-specific APIs.–– A low-level interface. A low-level interface.
A physical file format for storingA physical file format for storingscientific data.scientific data.
General Applications
ApplicationProgramming
Interfaces
Low-levelInterface
HDFfile
HDF SoftwareHDF Software
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 13
HDF Applications SoftwareHDF Applications Software
• Free software
– NCSA HDF library and utilities
– Other software
• Commercial/other software that “understands”
– all of HDF (Noesys, IDL)
– certain HDF objects (MATLAB, AVS)
– certain HDF applications (SHARP, WIM)
• http://hdf.ncsa.uiuc.edu/tools.html
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 14
University of Illinois at Urbana-Champaign
What platforms does HDF run on?What platforms does HDF run on?
• Sun: Solaris, SunOS
• SGI: Indy, Power Challenge, Origin, Cray C90,
YMP, T3E
• HP9000, HP-Convex Exemplar
• IBM: RS6000, SP2
• DEC: Alpha/Digital UNIX, OpenVMS
VAX: OpenVMS
• Intel Pentium: Solarisx86, Linux, FreeBSD 2.2,
Windows: NT/95
• PowerPC: Mac-OS
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 15
HDF ObjectsHDF Objects
Palette
8-bit raster
24-bit raster
Scientific Data Sets (SDS(multidimensional arrays
Vdata (tables)
lat | lon | temp
----|-----|-----
12 | 23 | 3.1
15 | 24 | 4.2
17 | 21 | 3.6March 15, 1990. Simulation with
k=10.0, beta=1.22e3. Calculate
the magnitude ...
Vgroup
AnnotationThis HDF file containsone example of eachobject type
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 16
Array Number Type
Dimension X,Y,Z
SDS attributes
DimensionAttributes
DimensionNumber Type
Dimension Scale
DimensionAttributes
DimensionNumber Type
Dimension Scale
DimensionAttributes
DimensionNumber Type
Dimension Scale
Dimension X Dimension Y Dimension Z
Scientific Data Set
Array
SDS: Scientific Data SetSDS: Scientific Data Set
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 17
Writing an SDSWriting an SDS
• Open a file and start SD interface SdstartSdstart
• Create or open an SDS SDcreate/Sdselect
• Write out the SDS Sdwritedata
• Terminate access to the SDS SDendaccess
• Terminate access to SD interface SDend
and close the file
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 18
Class
Vdata Name
Class_1
General vdata
Records
Field_3Field_2 Field Names
2.6 0.00
Fields
3.5 1.22
1.5 23.50
8
5
2
5.3 6.93
Field_1
6
7 12.303.8
Vdata: HDF table structureVdata: HDF table structure
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 19
Vgroup System OrganizationVgroup System Organization
Vgroup
Data
Vgroup Vgroup
Data
Data
Data
Data
Vgroup
UNIX File System OrganizationUNIX File System Organization
Directory
File
Directory Directory
File
File
File
File
Directory
Vgroup: HDF Grouping StructureVgroup: HDF Grouping Structure
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 20
VgroupVgroup
March 15, 1990. Simulation with
k=10.0, beta=1.22e3. Calculate
the magnitude ... 2-D array2-D array
Vgroup
RasterRaster
imageimageannotationannotation
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 21
Image Array
Required
Array Name
Pixel Type
Width
Height
General Raster
name = IMAGE_1
Image Attribute
Optional
Palette
GR: HDF general raster imageGR: HDF general raster image
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 22
8-bit Raster Image Pixel
Palette
0 00000000 00000000 00000000
1 00000001 00000001 00000001
2 00000010 00000010 00000010
... . . .
192 11000000 11000000 11000000
. . .
253 11111101 11111101 11111101
254 11111110 11111110 11111110
255 11111111 11111111 11111111
Color Look-up Table (Color Components)
Entry red green blue
HDF PaletteHDF Palette
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 23
“This is a file label.”
“This is a file description.”
“This is another file label.”
“This is another file description.”
File Annotations
"This is an SDS label." "This is RIS24 description." "This is a RIS8 description."
Object Annotations
HDF File
RIS8RIS24SDS
HDF AnnotationsHDF Annotations
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 24
Who uses HDF?Who uses HDF?
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 25
A Sampling of HDF UsersA Sampling of HDF Users
NCSA-affiliated Science teams NCSA-affiliated Science teams Visualization, exchange, archivingVisualization, exchange, archiving
Vendors: Mathworks, Fortner Vendors: Mathworks, Fortner Visualization, data analysis, exchange,Visualization, data analysis, exchange,
Software, RSI, etc. Software, RSI, etc. data managementdata management
TRAPPIST TRAPPIST Exchange, analysis, visualization ofExchange, analysis, visualization of
(Euro consortium) (Euro consortium) non-destructive testing data (STEP)non-destructive testing data (STEP)
NIST Reactor & Cold Neutron NIST Reactor & Cold Neutron Data exchange, archivingData exchange, archiving
Res FacilityRes Facility
Stanford Univ EE DeptStanford Univ EE Dept Persistent storage of simulation dataPersistent storage of simulation data
Johns Hopkins Appl Physics LabJohns Hopkins Appl Physics Lab Planetary data exchange, archivePlanetary data exchange, archive
Inst Adv Tech in the HumanitiesInst Adv Tech in the Humanities Image annotationImage annotation
Comp Graphics Unit ManchesterComp Graphics Unit Manchester Image processing, visualizationImage processing, visualization
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 26
Major User #1: EOSDISMajor User #1: EOSDIS
•• User support for scientists, data producers, etc.User support for scientists, data producers, etc.
•• Library and file structure improvementsLibrary and file structure improvements
•• HDF tools, utilities, access softwareHDF tools, utilities, access software
•• Software maintenance and QASoftware maintenance and QA
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 27
Major User #2: ASCIMajor User #2: ASCI
• ASCI Data Models and Formats (DMF) Group
– open standard exchange format and I/O library for ASCI
– DOE tri-lab ASCI applications
• HDF requirements
– large datasets (> a terabyte)
– ASCI data types, especially meshes
– good performance in massive parallel environments
– emphasis on HDF 5
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 28
II. Major HDF ProjectsII. Major HDF Projects
• Support, maintenance, QA
• Java applications
• Remote access
• HDF5: next generation
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 29
Support, Maintenance, QASupport, Maintenance, QA
•• User supportUser support
•• Library and file structure improvementsLibrary and file structure improvements
•• Software maintenance and QASoftware maintenance and QA
•• Documentation, tutorials, etc.Documentation, tutorials, etc.
•• HDF tools and utilitiesHDF tools and utilities
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 30
Java applicationsJava applications
• A Java HDF API
– Basis for tools that access HDF
• A Java HDF Viewer
– HDF browser/visualizer
• Java Scientific Data Server Prototype
– Lessons learned about scientific data servers
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 31
Java HDF API (JHI)Java HDF API (JHI)
• Java HDF Interface
– Complete Java interface to the HDF library
– Analogous to the HDF F77 interface
– To be used by Java apps to access HDF files
– http://hdf.ncsa.uiuc.edu/java-hdf-html/
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 32
Java-based HDF Viewer (JHV)Java-based HDF Viewer (JHV)
• client-based HDF viewer
– browsing
– data viewing (“spreadsheet”)
– image viewing and plotting
– image processing
– animation
– data import/export
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 33
JHVJHV
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 34
Remote Data AccessRemote Data Access
• Accessing Scientific Info Using Networks
– Scientific data is large and complex,
– Need to locate objects & subsets within large files
– The “download and browse” approach is infeasible
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 35
The SDB: Web-based Server-side DataThe SDB: Web-based Server-side Data
BrowserBrowser
• Remote interactive browser for data and
metadata
– “A conversation with the data”
– Views provided at increasing levels of detail
– Returns subsets of data
– Performs simple data translations
– Adaptable to different domains (EOS, Astronomy)
– Reads HDF, netCDF, CDF, FITs formats
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 36
Sample SDB applicationsSample SDB applications
• DIAL (Data & Info Access Link)
– HDF-EOS data access
• Distrib. Ocean Data System (DODS) WP-ESIP
– High-speed remote subsetting earth science datasets
• Boeing
– Extract space/time info from large image collections
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 37
Other workOther work
• Java applets for scientific data browsing
• A VRML server for Radio Astronomy data
• Interoperable search with Z39.50
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 38
HDF5: Next generation HDFHDF5: Next generation HDF
• Features
– Large arrays and files (>2GB)
– Simple, comprehensive data model
– Emphasis on Parallel I/O
– Alternate storage structures
• Collaborations
– Mesh data standard for ASCI physics
– Integrate with commercial object store (DLI)
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 39
Basic HDF5 data objectBasic HDF5 data object
Record
int8 int4 int16 float32
Dimensionality: 5 x 3 x 4
Number type:
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 40
III. HDF and HDF-EOSIII. HDF and HDF-EOS
• HDF profiles and HDF-EOS
• HDF configuration record project
• Future directions?
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 41
HDF Profiles and HDF-EOSHDF Profiles and HDF-EOS
• HDF does not support specific application areas.
• To share files, users must agree on how to
organize them.
• HDF user groups create “profiles” describing
how to organize data in their HDF files.
• Profiles standardize domain-specific HDF files
• Example: HDF-EOS
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 42
HDF-EOS profilesHDF-EOS profiles
• Profiles for Earth remote sensing data and in-
situ measurements
• Includes
–– Standard metadata for EOS dataStandard metadata for EOS data
–– API and library that reads/writes HDF-EOS filesAPI and library that reads/writes HDF-EOS files
–– Utilities to simplify the work of analyzing andUtilities to simplify the work of analyzing and
visualizing HDF-EOS files.visualizing HDF-EOS files.
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 43
General ApplicationsHDF-EOS APIHDF-EOS API
ApplicationProgramming
Interfaces
Low-levelInterface
HDFfile
HDF-EOS software layersHDF-EOS software layers
HDF-EOS ApplicationsHDF-EOS Applications
HDF-EOS
profiles
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 44
“HDF Configuration Record” (HCR)“HDF Configuration Record” (HCR)
To simplify the tasks of defining,To simplify the tasks of defining,
comparing, and producingcomparing, and producing
HDF-EOS filesHDF-EOS files
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 45
HCRHCR
• Formal descriptions of HDF objects
• Based on ODL (Object Description Language)
• Supports HDF-EOS swath, grid, point
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 46
HCR of SwathHCR of Swath
/* Project XYZ */
/* First version defined on June 10th, 1998 */
OBJECT = SWATH
NAME = SCAN1
OBJECT = Dimension
NAME = GeoTrack
Size = 1200
END_OBJECT = Dimension
OBJECT = Dimension
NAME = GeoCrossTrack
Size = 205
END_OBJECT = Dimension
OBJECT = Dimension
NAME = DataX
Size = 2410
END_OBJECT = Dimension
END_OBJECT = SWATH
END
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 47
HCR Utilities:HCR Utilities:
• Convert HCR ↔↔↔↔ HDF-EOS
• Edit HCR and HDF-EOS
• Compare HCR with HDF-EOS file
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 48
HCR editHCR edit
swath pageswath page
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 49
HCR editHCR edit
ODL pageODL page
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 50
Future possibilitiesFuture possibilities
• Deploy HCR?
• HDF as an archive format?
• Java tools for HDF-EOS?
• HDF5?
– HDF5 ↔↔↔↔ HDF4 conversion
– HDF4 API on HDF5
– HDF-EOS on HDF5
NCSA/Univ of Illinois at Urbana-Champaign
HDFHDF 51
HDF InformationHDF Information
• HDF Information Center
– http://hdf.ncsa.uiuc.edu/
• HDF Help email address
• HDF users mailing list