Post on 30-Mar-2015
www.hdfgroup.org
The HDF Group
1
HDF Project Update
Mike Folk, Elena Pourmal, Kent Yang The HDF Group
Briefing to ESDISOctober 16-17, 2012
10/16/2012 HDF Briefing to NASA
www.hdfgroup.orgHDF Briefing to NASA 2
Project Goals
• Support for NASA HDF data providers and data users, and the EOSDIS Core System (ECS)
• Maintenance and quality assurance for HDF4 and HDF5 libraries and utilities
• Evolving the HDF4 and HDF5 libraries and utilities to meet new NASA EOS requirement
• Integration with complementary technologies and application domains
• Prototype study of web technologies to access HDF data
10/16/2012
www.hdfgroup.org
The HDF Group
3
Support activities for NASA HDF data providers and
users and ECS
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org4
Activities
• Help Desk• Outreach
• Mailing lists, Web site• Conferences, Workshops, Other
• EOS support• netCDF support and collaboration• Later today:
• HDF OPeNDAP support• Support for NASA data products
• SMAP, ICESat-2, MEaSURES)
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org5
HELP DESK
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org6
The HDF Group Help Desk
• High-priority support for ESDIS users• Fast response• Problem investigation and resolution• Performance tuning• Access to HDF developers
• Basic support for entire HDF user community, as resources permit
• help@hdfgroup.org
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org7
Help Desk issues resolution time
< 4 hours
< 1 day
< 2 days
< 1 week
< 2 weeks
< 1 month
< 6 month
not closed
0
10
20
30
40
50
60
HDF Issues Resolution Time
201020112012*
Num
ber
of is
sues
* Number of messages up do August 31, 2012
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org8
Help Desk issues by years
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
*0
100
200
300
400
500
600
700
800
900
1000
HDF4 and HDF5 issues by years
HDF5HDF4
Nu
mb
er
of
issu
es
* Number of messages up do August 31, 2012
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org9
OUTREACH
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org10
Outreach
• Web pages improvements• FAQ support for HDF4 and HDF5
http://www.hdfgroup.org/HDF5-FAQ.html
http://www.hdfgroup.org/HDF-FAQ.html • Later today: Detailed talk on support for EOS
(Kent)• Organizing material based on the ESDIS
survey results and feedback from the users• Example: Tools by category (work in progress)
• How do I edit, copy, import/export data with HDF?
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org11
Tools by category
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org12
Outreach
• Mailing list and archivesnews@hdfgroup.org
http://www.hdfgroup.org/news/
hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org13
hdf-forum@hdfgroup.org
• Started in Summer 2007• Monitored by the members of the HDF group• Number of messages:
• Most discussed topics:• Parallel HDF5• Advanced HDF5 features (drivers, complex
types, performance)• .NET and C++ wrappers, HL libraries
2012 to Sep 30 2011 2010
703 1,334 1,721
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org14
hdf-forum@hdfgroup.org
• Members of the FORUM help with• Releases testing on a regular basis• Maintaining CMake build systems on platforms
beyond Windows• Answering questions
• The Help Desk focuses on ESDIS and other paying customers while referring others to the FORUM for topics that require domain knowledge or specific HDF5 usage
• Reporting defects• Securing funding for The HDF Group
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org15
Conferences, Workshops, Other
• 2011 IDV/THREDDS Workshop, July 2011• Learned how to use IDV and THREDDS
• 10th NASA ESDSWG Conference, Nov 2011• Poster on HDF and netCDF interoperability
• 2012 ESIP Summer Meeting, July 2012• Presentations
• HDF/HDF-EOS Data for Dummies• What does Open Source mean for HDF?
• Poster• What’s new at hdfgroup.org?
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org16
Conferences, Workshops, Other
• 15th HDF and HDF-EOS Workshop, April 2012• Five tutorials, nine presentations and lightning
talks, and one special session• Materials: http://www.hdfeos.org/workshops/ws15/
agenda.php
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org17
Conferences, Workshops, Other
• Advised ICESat-2, MOPITT and several MEaSURES projects on interoperability with netCDF-4
• Participated in the SMAP ISO-19115 work• Participated in ESDSWG SPG telecons• Joined several ESDSWG mailing lists related
to HDF
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org18
NETCDF SUPPORT AND COLLABORATION
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org19
NetCDF-4 support
• Help Unidata netCDF-4 developers• Frequent communications with netCDF-4
developers• Provided a fix for ncdump critical to the ESDIS
users• Test netCDF-4 daily with the HDF5
development code • Plan to participate in Unidata Advanced
netCDF Workshop on October 26, 2012
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org20
EOS SUPPORT
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org21
EOS Support
• Submitted several bugs and the corresponding fixes to the HDF-EOS team
• EOS2 and EOS5 are tested daily with HDF4 and HDF5 development code
• HDF-EOS website• Actively maintain the website• Improve the quality of the example codes and
plots to access NASA HDF and HDF-EOS data via MATLAB, IDL and NCL
• More information in a separate talk
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org
The HDF Group
22
Maintenance and quality assurance for HDF4 and
HDF5 libraries and utilities
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org23
Activities
• Manage and resolve issues• Perform maintenance releases of HDF software
• Support OSs and compilers identified by ESDIS• Address issues and software evolution requests
from ESDIS• Perform SQE and SQA activities
• Put in place missing processes• Improve existing processes• Build in software quality by performing testing
and using the best software development practices
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org24
HDF ISSUES
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org25
Issues and their priorities
• Must Fix• Data corruption• Portability• Backward and Forward
Compatibility• Funded Requests
• Fix after “Must Fix”• Power User Requests• Tools• Library issues• Build Infrastructure
• When resources permit• Wrappers• HL Libraries• Other
Need your input on priorities!
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org26
HDF4 opened and closed issues
10/16/2012
2003 2004 2005 2006 2007 2008 2009 2010 2011 20120
10
20
30
40
50
60
70
HDF4 open and closed issues
OpenedClosed
Num
ber
of I
ssue
s
HDF Briefing to NASA
www.hdfgroup.org27
HDF5 opened and closed issues
10/16/2012
20032004
20052006
20072008
20092010
20112012
0
100
200
300
400
500
600
HDF5 opened and closed issues
OpenedClosedClosed High Priority
Num
ber
of is
sues
HDF Briefing to NASA
www.hdfgroup.org28
SQE AND SQA AT THE HDF GROUP
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org29
Activities
• CMMI appraisal and process improvement action plan (Dec. 2011)
• Created SQE position• Focus on
• Implementing CMMI best practices and procedures
• Enforcing current practices and procedures
• Focused on improving software testing
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org30
Activities
• Worked on establishing a culture of code reviews for the HDF libraries and tools
• Improved releases planning and monitoring for HDF5
• Used static analysis tools to identify defects• Coverity (used at THG)• Klocwork (used at LLNL)
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org31
HDF MAINTENANCE RELEASES
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org32
General maintenance for 2011-2012
• Implemented CMake for all HDF software• Worked on closing reported issues• Focused on performance enhancements• Improved HDF documentation
• http://www.hdfgroup.org/HDF5/doc/Advanced.html • Explored the needs for internal documentation• Preventive measures:
• Coverity static analysis; Rational tools; valgrind• Have been exploring new testing tools to sustain
testing for HDF products (CDash, Bamboo, Cloud)
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org3310/16/2012 HDF Briefing to NASA
2011 May Jun Jul Aug Sep Oct Nov DecHDF4
HDF5 1.8.7 1.8.8
HDF-Java 2.8
h4h5 tools
CF – code freeze 6 weeks for HDF5 4 weeks for other products
2012 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
HDF4 4.2.7 patch1 4.2.8
HDF5 1.8.9 CF 1.8.10
HDF-Java
CF 2.9
h4h5 tools
2.2.1
Maintenance Releases 2011 – 2012
www.hdfgroup.org34
HDF4 maintenance releases
• HDF 4.2.7 released in February 2012• Extra functions to support H4 mapping project• Support for Linux PPC64 with IBM XL Fortran• Minor bug fixes and documentation improvements
• HDF 4.2.8 release in August 2012• Support for H4mapping project
• Information about releases available from http://www.hdfgroup.org/ftp/HDF/prev-releases/ReleaseFiles/
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org35
HDF5 maintenance releases
• HDF5 1.8.7-1.8.9• Many improvements in tools
• ESDIS request to handle “/” in attributes names• Fortran 2003 support• Fortran wrappers for Dimension Scales APIs• Information about releases available from
http://www.hdfgroup.org/ftp/HDF5/prev-releases/ReleaseFiles/
• HDF5 1.8.10 November 15, 2012• Many improvements in h5diff, minor bug fixe
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org36
H4h5tools maintenance releases
• 2.2.1 released February, 15th, 2012• Based on HDF 4.2.7 and HDF5 1.8.8• Fixed bugs reported by ESDIS users• Upgraded to HDF5 1.8 APIs• Updated documents
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org37
Java maintenance releases
• 2.8 release December 2011• Addressed all issues reported by ESDIS users
• 2.9 release December 2012• Based on HDF 4.2.8 and HDF5 1.8.10• ESDIS requests:
• Show groups/attributes in creation order• Export data to a binary/ascii file without having
to open the object in the TableView
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org38
FUTURE DIRECTIONS
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org39
General maintenance for 2013-2014
• Document and improve maintenance processes and procedures
• Focus on documentation for HDF4 and HDF5• Improve quality
• Establish code review procedures for all HDF products
• Expand static analysis to all HDF software packages
• Enhance testing framework
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org40
Future Maintenance Releases
10/16/2012 HDF Briefing to NASA
2013 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov DecHDF4 CF 4.2.9
HDF5 CF 1.8.11 CF 1.8.12
HDF-Java
CF 2.10
h4h5 tools
CF 2.2.2
CF – code freeze 6 weeks for HDF5 4 weeks for other products
2014 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
HDF4 CF 4.2.10
HDF5 CF 1.8.13 CF 1.8.14
HDF-Java
CF 2.11
h4h5 tools
CF 2.2.3
www.hdfgroup.org41
Future HDF4 maintenance releases
• HDF 4.2.9 planned for February 2013• Close as many issues as possible
• HDF 4.2.10 planned for February 2014• Complete “self-configuration” work for library
maintainability and portability• ESDIS priorities?
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org42
Future HDF5 maintenance releases
• HDF5 1.8.11 May 2013• Features sponsored by European Synchrotron
Community (PSI/DECTRIS and DESY)• Writing compressed chunks directly to HDF5 file
bypassing library internals• Finished prototype; achieved order of magnitude boost in
I/O
• Support for dynamically loaded third party libraries (compression and other filters)
• Preliminary studies soon; several prototypes are already available
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org43
Future HDF5 maintenance releases
• How soon will MATLAB and IDL be able to support 1.8.11?
• ESDIS priorities?
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org44
Future h4h5tools maintenance releases
• 2.2.2 release March 2013• Use latest HDF4 and HDF5 releases• Address any ESDIS issues if arise
• 2.2.3 release March 2014• Possible improvements
• Conversion of dimension scales• Leveraging HDF5 1.8 features
• ESDIS priorities?
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org45
Future Java maintenance releases
• 2.10 release December 2013• We are gathering requirements (next slides)
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org46
TOOLS UPDATE
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org47
HDFVIEW USER SURVEY
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org48
Background
• HDFView was created for browsing and editing HDF files on a local machine
• The most popular HDF tool• August 2012 downloads
• HDFView – 2,805• HDF5 binaries for all platforms – 1,073
• Twelve releases since the first release in 1998• Time to reevaluate its capabilities and find out
users’ expectations
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org49
HDFView Survey Goals
• Understand users needs and future directions for of HDFView maintenance and evolution • New requirements• Current deficiencies• Most desired improvements
• Help with tasks prioritization
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org50
HDFView Survey
• Conducted a survey at https://www.surveymonkey.com/s/PLGRLJY
• Over 40 questions in 8 different areas• About 200 responses
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org51
Requested Improvements Areas
• Import/export capabilities• Binary, text, Excel, tree-shape (XML, JSON),
GeoTIFF, TIFF, KML (Google Earth), PNG, netCDF
• Access• Cloud, OPeNDAP, mobile devices
• Data discovery/search• Values, metadata, file structure
• Data presentation• Plots, images, data with dimension scales
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org52
Requested Improvements Areas
• Usability• HDF5 functionality
• Performance with large files• Support for all HDF5 datatypes• Support for custom filters
• Data handling• Displaying, copying, selecting, exporting, importing
• GUI features• One working window• Java vs. new technologies (QT, C++, ?) • UNIX remote display
• Platform-specific requests • Mac OS X (installation, shortcuts)
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org53
HDFView Survey
• Report with raw data is available from http://www.hdfgroup.org/hdf-java-html/hdfview/
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org54
HDF TOOLS PLAN
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org55
Plans for HDFView
• Plan for and address high-priority items for ESDIS from the survey
• Quality improvements• Clean issues database (37 right now)• Finish unit testing work (HDF4 JNI)• Set up GUI automatic testing
• Performance improvements• Address large files issues
• Evolution• Add support for dimension scales• Add support for misc. HDF5 features
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org56
HDFView Survey
• The ESDIS 2013 priority list as identified by the ESDSWG Executive Council available from
http://earthdata.nasa.gov/wiki/main/index.php/The_ESDIS_2013_Priority_List
• Topics relevant to HDFView• ESDIS standards: GeoTIFF and KML• Cloud and mobile computing• ESDS conventions for HDF5
• CF conventions• netCDF interoperability
• What are ESDIS priorities for HDFView?
10/16/2012 HDF Briefing to NASA
www.hdfgroup.orgHDF Briefing to NASA 57
Plans for command-line tools
• Regular maintenance • Clean up issues database• Focus on quality and performance
• Focus on tools to close a gap between HDF5 and XML
• Settle on new XML schema• H5gen to generate HDF5 from XML• Enhance XML output generated by h5dump
• Possible open source experiment?• Small Apache project to learn to create tools that are
developed and maintained primarily by the open source community.
• ESDIS priorities?10/16/2012
www.hdfgroup.org58
Open Source Initiative Action PlanMission Statement• Conduct a small Apache project that will
enable us to learn to create tools for earth science data users that is developed and maintained primarily by the open source community.
Stakeholders• Key groups, organizations, individuals that will be
impacted • Earth science data users.• EOS projects, Decadal Survey Missions, other
NASA Earth Science data activities• All HDF communities
• Who needs to be involved in developing deliverables.• Developers and end users who work with NASA
earth science data.
Approach• Achieve approval from NASA to make this
project a part of the ESDIS project (EED Task 2 Rev 8).
• Identify a need that is likely to have community support. For example, a need for tools to convert HDF files to certain other formats.
• Learn to create and lead an Apache Incubator Project.
• Create an Apache Incubator project.• Develop tools.• Graduate the project from incubation.
Outcomes, Deliverables, Milestones • Specific outcome: the company learns to create an
open source project in which HDF software is developed and used by the community
• Work products• An Apache Incubator project • An Apache project supporting the software• Tools• Recommendation report
• Milestones• Periodic reports on steps in the approach.• Incubator project established.• Apache project achieved.• Final report
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org59
PLATFORM SUPPORT ISSUES
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org60
HDF4 platforms supported
• Systems• Linux 2.6 (CentOS-5)
Linux 2.6 x86_64• Linux PowerPC64• Solaris 5.10• 32/64bit Windows XP, 7
and Cygwin• Mac OS Intel 10.6.*
and 10.7.* 32/64-bit
• Compilers• IBM C and Fortran cmp• GNU C and Fortran up
to 4.6.2, g95 (4.0.3)• Intel C and Fortran v12• PGI C and Fortran v11• Sun C 5.10 and Fortran
95 8.4• VS 2008 Intel Fortran
10• VS 2010 Intel Fortran
10
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org61
HDF5 platforms supported
• Systems• Same as for HDF4• AIX 5.3
• Compilers• Same as for HDF4• IBM compilers
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org62
HDF4 and HDF5 platforms to drop
• Systems• Mac 10.6.* • Windows XP
• Last releases HDF 4.2.9 and HDF5 1.8.10
• Compilers• VS2008 with Intel 11
and MSVS build environment• We will use “CMake
only” for Windows for all HDF products starting in Spring 2013
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org63
HDF4 and HDF5 platforms to add
• Systems• Mac 10.8.*• How important is 32-bit
on 64-bit Mac OS X?• Windows 8
• Spring 2013
• Flavors of Linux?
• Compilers• Clang?• VS 2012
• Spring 2013
We are using virtualization very successfully.Can add any Linux or Windows flavors. Just let us know!
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org
The HDF Group
64
Integration with complementary
technologies and application domains
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org65
OPENDAP
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org66
OPeNDAP – current status
• HDF5-OPeNDAP handler• Re-engineered to support more NASA HDF5
and HDF-EOS5 products• HDF4-OPeNDAP handler
• Enhanced the support of many NASA HDF-EOS2 and HDF4 products
• More information in a separate talk
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org67
TOOLS TO IMPROVE INTEROPERABILITY BETWEEN HDF AND NETCDF
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org68
HDF and netCDF interoperabilty tools
• HDF-EOS5 augmentation tool• HDF-EOS2 dumper tool• HDF-EOS5 to netCDF-4 conversion tool• HDF4/HDF-EOS2 to CF conversion library• More information in a separate talk
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org
The HDF Group
69
HDF4 Layout Maps
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org70
HDF4 Layouts Map
Provide tools and support to ensure long-term access to EOS data via HDF4 File Content Maps.• H4mapwriter release 1.0.3 in June 2012• More information in a separate talk
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org
The HDF Group
71
Thank You!
10/16/2012 HDF Briefing to NASA
www.hdfgroup.org72
Acknowledgements
10/16/2012 HDF Briefing to NASA
This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.