Www.hdfgroup.org The HDF Group HDF Project Update Mike Folk, Elena Pourmal, Kent Yang The HDF Group...

www.hdfgroup.org

The HDF Group

1

HDF Project Update

Mike Folk, Elena Pourmal, Kent Yang The HDF Group

Briefing to ESDISOctober 16-17, 2012

10/16/2012 HDF Briefing to NASA

www.hdfgroup.orgHDF Briefing to NASA 2

Project Goals

• Support for NASA HDF data providers and data users, and the EOSDIS Core System (ECS)

• Maintenance and quality assurance for HDF4 and HDF5 libraries and utilities

• Evolving the HDF4 and HDF5 libraries and utilities to meet new NASA EOS requirement

• Integration with complementary technologies and application domains

• Prototype study of web technologies to access HDF data

10/16/2012

www.hdfgroup.org

The HDF Group

3

Support activities for NASA HDF data providers and

users and ECS


www.hdfgroup.org4

Activities

• Help Desk• Outreach

• Mailing lists, Web site• Conferences, Workshops, Other

• EOS support• netCDF support and collaboration• Later today:

• HDF OPeNDAP support• Support for NASA data products

• SMAP, ICESat-2, MEaSURES)


www.hdfgroup.org5

HELP DESK


www.hdfgroup.org6

The HDF Group Help Desk

• High-priority support for ESDIS users• Fast response• Problem investigation and resolution• Performance tuning• Access to HDF developers

• Basic support for entire HDF user community, as resources permit

• [email protected]


mailto:[email protected]

www.hdfgroup.org7

Help Desk issues resolution time

< 4 hours

< 1 day

< 2 days

< 1 week

< 2 weeks

< 1 month

< 6 month

not closed

0

10

20

30

40

50

60

HDF Issues Resolution Time

201020112012*

Num

ber

of is

sues

* Number of messages up do August 31, 2012


www.hdfgroup.org8

Help Desk issues by years

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012

*0

100

200

300

400

500

600

700

800

900

1000

HDF4 and HDF5 issues by years

HDF5HDF4

Nu

mb

er

of

issu

es

* Number of messages up do August 31, 2012


www.hdfgroup.org9

OUTREACH


www.hdfgroup.org10

Outreach

• Web pages improvements• FAQ support for HDF4 and HDF5

http://www.hdfgroup.org/HDF5-FAQ.html

http://www.hdfgroup.org/HDF-FAQ.html • Later today: Detailed talk on support for EOS

(Kent)• Organizing material based on the ESDIS

survey results and feedback from the users• Example: Tools by category (work in progress)

• How do I edit, copy, import/export data with HDF?




http://www.hdfgroup.org/HDF-FAQ.html

http://www.hdfgroup.org/HDF-FAQ.html

www.hdfgroup.org11

Tools by category


www.hdfgroup.org12

Outreach

• Mailing list and [email protected]

http://www.hdfgroup.org/news/

[email protected]

http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/










www.hdfgroup.org13

[email protected]

• Started in Summer 2007• Monitored by the members of the HDF group• Number of messages:

• Most discussed topics:• Parallel HDF5• Advanced HDF5 features (drivers, complex

types, performance)• .NET and C++ wrappers, HL libraries

2012 to Sep 30 2011 2010

703 1,334 1,721



www.hdfgroup.org14

[email protected]

• Members of the FORUM help with• Releases testing on a regular basis• Maintaining CMake build systems on platforms

beyond Windows• Answering questions

• The Help Desk focuses on ESDIS and other paying customers while referring others to the FORUM for topics that require domain knowledge or specific HDF5 usage

• Reporting defects• Securing funding for The HDF Group



www.hdfgroup.org15

Conferences, Workshops, Other

• 2011 IDV/THREDDS Workshop, July 2011• Learned how to use IDV and THREDDS

• 10th NASA ESDSWG Conference, Nov 2011• Poster on HDF and netCDF interoperability

• 2012 ESIP Summer Meeting, July 2012• Presentations

• HDF/HDF-EOS Data for Dummies• What does Open Source mean for HDF?

• Poster• What’s new at hdfgroup.org?


www.hdfgroup.org16


• 15th HDF and HDF-EOS Workshop, April 2012• Five tutorials, nine presentations and lightning

talks, and one special session• Materials: http://www.hdfeos.org/workshops/ws15/

agenda.php


http://www.hdfeos.org/workshops/ws15/agenda.php



www.hdfgroup.org17


• Advised ICESat-2, MOPITT and several MEaSURES projects on interoperability with netCDF-4

• Participated in the SMAP ISO-19115 work• Participated in ESDSWG SPG telecons• Joined several ESDSWG mailing lists related

to HDF


www.hdfgroup.org18

NETCDF SUPPORT AND COLLABORATION


www.hdfgroup.org19

NetCDF-4 support

• Help Unidata netCDF-4 developers• Frequent communications with netCDF-4

developers• Provided a fix for ncdump critical to the ESDIS

users• Test netCDF-4 daily with the HDF5

development code • Plan to participate in Unidata Advanced

netCDF Workshop on October 26, 2012


www.hdfgroup.org20

EOS SUPPORT


www.hdfgroup.org21

EOS Support

• Submitted several bugs and the corresponding fixes to the HDF-EOS team

• EOS2 and EOS5 are tested daily with HDF4 and HDF5 development code

• HDF-EOS website• Actively maintain the website• Improve the quality of the example codes and

plots to access NASA HDF and HDF-EOS data via MATLAB, IDL and NCL

• More information in a separate talk


www.hdfgroup.org

The HDF Group

22

Maintenance and quality assurance for HDF4 and

HDF5 libraries and utilities


www.hdfgroup.org23

Activities

• Manage and resolve issues• Perform maintenance releases of HDF software

• Support OSs and compilers identified by ESDIS• Address issues and software evolution requests

from ESDIS• Perform SQE and SQA activities

• Put in place missing processes• Improve existing processes• Build in software quality by performing testing

and using the best software development practices


www.hdfgroup.org24

HDF ISSUES


www.hdfgroup.org25

Issues and their priorities

• Must Fix• Data corruption• Portability• Backward and Forward

Compatibility• Funded Requests

• Fix after “Must Fix”• Power User Requests• Tools• Library issues• Build Infrastructure

• When resources permit• Wrappers• HL Libraries• Other

Need your input on priorities!


www.hdfgroup.org26

HDF4 opened and closed issues

10/16/2012

2003 2004 2005 2006 2007 2008 2009 2010 2011 20120

10

20

30

40

50

60

70

HDF4 open and closed issues

OpenedClosed

Num

ber

of I

ssue

s

HDF Briefing to NASA

www.hdfgroup.org27


10/16/2012

20032004

20052006

20072008

20092010

20112012

0

100

200

300

400

500

600


OpenedClosedClosed High Priority

Num

ber

of is

sues

HDF Briefing to NASA

www.hdfgroup.org28

SQE AND SQA AT THE HDF GROUP


www.hdfgroup.org29

Activities

• CMMI appraisal and process improvement action plan (Dec. 2011)

• Created SQE position• Focus on

• Implementing CMMI best practices and procedures

• Enforcing current practices and procedures

• Focused on improving software testing


www.hdfgroup.org30

Activities

• Worked on establishing a culture of code reviews for the HDF libraries and tools

• Improved releases planning and monitoring for HDF5

• Used static analysis tools to identify defects• Coverity (used at THG)• Klocwork (used at LLNL)


www.hdfgroup.org31

HDF MAINTENANCE RELEASES


www.hdfgroup.org32

General maintenance for 2011-2012

• Implemented CMake for all HDF software• Worked on closing reported issues• Focused on performance enhancements• Improved HDF documentation

• http://www.hdfgroup.org/HDF5/doc/Advanced.html • Explored the needs for internal documentation• Preventive measures:

• Coverity static analysis; Rational tools; valgrind• Have been exploring new testing tools to sustain

testing for HDF products (CDash, Bamboo, Cloud)


http://www.hdfgroup.org/HDF5/doc/Advanced.html

http://www.hdfgroup.org/HDF5/doc/Advanced.html

www.hdfgroup.org3310/16/2012 HDF Briefing to NASA

2011 May Jun Jul Aug Sep Oct Nov DecHDF4

HDF5 1.8.7 1.8.8

HDF-Java 2.8

h4h5 tools

CF – code freeze 6 weeks for HDF5 4 weeks for other products

2012 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

HDF4 4.2.7 patch1 4.2.8

HDF5 1.8.9 CF 1.8.10

HDF-Java

CF 2.9

h4h5 tools

2.2.1

Maintenance Releases 2011 – 2012

www.hdfgroup.org34

HDF4 maintenance releases

• HDF 4.2.7 released in February 2012• Extra functions to support H4 mapping project• Support for Linux PPC64 with IBM XL Fortran• Minor bug fixes and documentation improvements

• HDF 4.2.8 release in August 2012• Support for H4mapping project

• Information about releases available from http://www.hdfgroup.org/ftp/HDF/prev-releases/ReleaseFiles/


http://www.hdfgroup.org/ftp/HDF5/prev-releases/ReleaseFiles/



www.hdfgroup.org35

HDF5 maintenance releases

• HDF5 1.8.7-1.8.9• Many improvements in tools

• ESDIS request to handle “/” in attributes names• Fortran 2003 support• Fortran wrappers for Dimension Scales APIs• Information about releases available from


• HDF5 1.8.10 November 15, 2012• Many improvements in h5diff, minor bug fixe



www.hdfgroup.org36

H4h5tools maintenance releases

• 2.2.1 released February, 15th, 2012• Based on HDF 4.2.7 and HDF5 1.8.8• Fixed bugs reported by ESDIS users• Upgraded to HDF5 1.8 APIs• Updated documents


www.hdfgroup.org37

Java maintenance releases

• 2.8 release December 2011• Addressed all issues reported by ESDIS users

• 2.9 release December 2012• Based on HDF 4.2.8 and HDF5 1.8.10• ESDIS requests:

• Show groups/attributes in creation order• Export data to a binary/ascii file without having

to open the object in the TableView


www.hdfgroup.org38

FUTURE DIRECTIONS


www.hdfgroup.org39

General maintenance for 2013-2014

• Document and improve maintenance processes and procedures

• Focus on documentation for HDF4 and HDF5• Improve quality

• Establish code review procedures for all HDF products

• Expand static analysis to all HDF software packages

• Enhance testing framework


www.hdfgroup.org40

Future Maintenance Releases


2013 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov DecHDF4 CF 4.2.9

HDF5 CF 1.8.11 CF 1.8.12

HDF-Java

CF 2.10

h4h5 tools

CF 2.2.2

CF – code freeze 6 weeks for HDF5 4 weeks for other products

2014 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

HDF4 CF 4.2.10

HDF5 CF 1.8.13 CF 1.8.14

HDF-Java

CF 2.11

h4h5 tools

CF 2.2.3

www.hdfgroup.org41

Future HDF4 maintenance releases

• HDF 4.2.9 planned for February 2013• Close as many issues as possible

• HDF 4.2.10 planned for February 2014• Complete “self-configuration” work for library

maintainability and portability• ESDIS priorities?


www.hdfgroup.org42


• HDF5 1.8.11 May 2013• Features sponsored by European Synchrotron

Community (PSI/DECTRIS and DESY)• Writing compressed chunks directly to HDF5 file

bypassing library internals• Finished prototype; achieved order of magnitude boost in

I/O

• Support for dynamically loaded third party libraries (compression and other filters)

• Preliminary studies soon; several prototypes are already available


www.hdfgroup.org43


• How soon will MATLAB and IDL be able to support 1.8.11?

• ESDIS priorities?


www.hdfgroup.org44

Future h4h5tools maintenance releases

• 2.2.2 release March 2013• Use latest HDF4 and HDF5 releases• Address any ESDIS issues if arise

• 2.2.3 release March 2014• Possible improvements

• Conversion of dimension scales• Leveraging HDF5 1.8 features

• ESDIS priorities?


www.hdfgroup.org45

Future Java maintenance releases

• 2.10 release December 2013• We are gathering requirements (next slides)


www.hdfgroup.org46

TOOLS UPDATE


www.hdfgroup.org47

HDFVIEW USER SURVEY


www.hdfgroup.org48

Background

• HDFView was created for browsing and editing HDF files on a local machine

• The most popular HDF tool• August 2012 downloads

• HDFView – 2,805• HDF5 binaries for all platforms – 1,073

• Twelve releases since the first release in 1998• Time to reevaluate its capabilities and find out

users’ expectations


www.hdfgroup.org49

HDFView Survey Goals

• Understand users needs and future directions for of HDFView maintenance and evolution • New requirements• Current deficiencies• Most desired improvements

• Help with tasks prioritization


www.hdfgroup.org50

HDFView Survey

• Conducted a survey at https://www.surveymonkey.com/s/PLGRLJY

• Over 40 questions in 8 different areas• About 200 responses


https://www.surveymonkey.com/s/PLGRLJY

www.hdfgroup.org51

Requested Improvements Areas

• Import/export capabilities• Binary, text, Excel, tree-shape (XML, JSON),

GeoTIFF, TIFF, KML (Google Earth), PNG, netCDF

• Access• Cloud, OPeNDAP, mobile devices

• Data discovery/search• Values, metadata, file structure

• Data presentation• Plots, images, data with dimension scales


www.hdfgroup.org52

Requested Improvements Areas

• Usability• HDF5 functionality

• Performance with large files• Support for all HDF5 datatypes• Support for custom filters

• Data handling• Displaying, copying, selecting, exporting, importing

• GUI features• One working window• Java vs. new technologies (QT, C++, ?) • UNIX remote display

• Platform-specific requests • Mac OS X (installation, shortcuts)


www.hdfgroup.org53

HDFView Survey

• Report with raw data is available from http://www.hdfgroup.org/hdf-java-html/hdfview/


http://www.hdfgroup.org/hdf-java-html/hdfview/



www.hdfgroup.org54

HDF TOOLS PLAN


www.hdfgroup.org55

Plans for HDFView

• Plan for and address high-priority items for ESDIS from the survey

• Quality improvements• Clean issues database (37 right now)• Finish unit testing work (HDF4 JNI)• Set up GUI automatic testing

• Performance improvements• Address large files issues

• Evolution• Add support for dimension scales• Add support for misc. HDF5 features


www.hdfgroup.org56

HDFView Survey

• The ESDIS 2013 priority list as identified by the ESDSWG Executive Council available from

http://earthdata.nasa.gov/wiki/main/index.php/The_ESDIS_2013_Priority_List

• Topics relevant to HDFView• ESDIS standards: GeoTIFF and KML• Cloud and mobile computing• ESDS conventions for HDF5

• CF conventions• netCDF interoperability

• What are ESDIS priorities for HDFView?





www.hdfgroup.orgHDF Briefing to NASA 57

Plans for command-line tools

• Regular maintenance • Clean up issues database• Focus on quality and performance

• Focus on tools to close a gap between HDF5 and XML

• Settle on new XML schema• H5gen to generate HDF5 from XML• Enhance XML output generated by h5dump

• Possible open source experiment?• Small Apache project to learn to create tools that are

developed and maintained primarily by the open source community.

• ESDIS priorities?10/16/2012

www.hdfgroup.org58

Open Source Initiative Action PlanMission Statement• Conduct a small Apache project that will

enable us to learn to create tools for earth science data users that is developed and maintained primarily by the open source community.

Stakeholders• Key groups, organizations, individuals that will be

impacted • Earth science data users.• EOS projects, Decadal Survey Missions, other

NASA Earth Science data activities• All HDF communities

• Who needs to be involved in developing deliverables.• Developers and end users who work with NASA

earth science data.

Approach• Achieve approval from NASA to make this

project a part of the ESDIS project (EED Task 2 Rev 8).

• Identify a need that is likely to have community support. For example, a need for tools to convert HDF files to certain other formats.

• Learn to create and lead an Apache Incubator Project.

• Create an Apache Incubator project.• Develop tools.• Graduate the project from incubation.

Outcomes, Deliverables, Milestones • Specific outcome: the company learns to create an

open source project in which HDF software is developed and used by the community

• Work products• An Apache Incubator project • An Apache project supporting the software• Tools• Recommendation report

• Milestones• Periodic reports on steps in the approach.• Incubator project established.• Apache project achieved.• Final report


www.hdfgroup.org59

PLATFORM SUPPORT ISSUES


www.hdfgroup.org60

HDF4 platforms supported

• Systems• Linux 2.6 (CentOS-5)

Linux 2.6 x86_64• Linux PowerPC64• Solaris 5.10• 32/64bit Windows XP, 7

and Cygwin• Mac OS Intel 10.6.*

and 10.7.* 32/64-bit

• Compilers• IBM C and Fortran cmp• GNU C and Fortran up

to 4.6.2, g95 (4.0.3)• Intel C and Fortran v12• PGI C and Fortran v11• Sun C 5.10 and Fortran

95 8.4• VS 2008 Intel Fortran

10• VS 2010 Intel Fortran

10


www.hdfgroup.org61

HDF5 platforms supported

• Systems• Same as for HDF4• AIX 5.3

• Compilers• Same as for HDF4• IBM compilers


www.hdfgroup.org62

HDF4 and HDF5 platforms to drop

• Systems• Mac 10.6.* • Windows XP

• Last releases HDF 4.2.9 and HDF5 1.8.10

• Compilers• VS2008 with Intel 11

and MSVS build environment• We will use “CMake

only” for Windows for all HDF products starting in Spring 2013


www.hdfgroup.org63

HDF4 and HDF5 platforms to add

• Systems• Mac 10.8.*• How important is 32-bit

on 64-bit Mac OS X?• Windows 8

• Spring 2013

• Flavors of Linux?

• Compilers• Clang?• VS 2012

• Spring 2013

We are using virtualization very successfully.Can add any Linux or Windows flavors. Just let us know!


www.hdfgroup.org

The HDF Group

64

Integration with complementary

technologies and application domains


www.hdfgroup.org65

OPENDAP


www.hdfgroup.org66

OPeNDAP – current status

• HDF5-OPeNDAP handler• Re-engineered to support more NASA HDF5

and HDF-EOS5 products• HDF4-OPeNDAP handler

• Enhanced the support of many NASA HDF-EOS2 and HDF4 products

• More information in a separate talk


www.hdfgroup.org67

TOOLS TO IMPROVE INTEROPERABILITY BETWEEN HDF AND NETCDF


www.hdfgroup.org68

HDF and netCDF interoperabilty tools

• HDF-EOS5 augmentation tool• HDF-EOS2 dumper tool• HDF-EOS5 to netCDF-4 conversion tool• HDF4/HDF-EOS2 to CF conversion library• More information in a separate talk


www.hdfgroup.org

The HDF Group

69

HDF4 Layout Maps


www.hdfgroup.org70

HDF4 Layouts Map

Provide tools and support to ensure long-term access to EOS data via HDF4 File Content Maps.• H4mapwriter release 1.0.3 in June 2012• More information in a separate talk


www.hdfgroup.org

The HDF Group

71

Thank You!


www.hdfgroup.org72

Acknowledgements


This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.

Www.hdfgroup.org The HDF Group HDF Project Update Mike Folk, Elena Pourmal, Kent Yang The HDF Group...

Documents

Transcript of Www.hdfgroup.org The HDF Group HDF Project Update Mike Folk, Elena Pourmal, Kent Yang The HDF Group...