HDF5 Software Process
description
Transcript of HDF5 Software Process
![Page 1: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/1.jpg)
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 1
HDF5 Software Process
MuQun Yang, Quincey Koziol, Elena Pourmal
The HDF Group
![Page 2: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/2.jpg)
Purposes
• Demonstrate how we maintain HDF5- Libraries and tools built on top of HDF5• HDF-EOS5, NetCDF4 and Pytables etc
• Hear your feedback
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 2
![Page 3: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/3.jpg)
Three pillars for robust software
• Correctness• Performance• Coding standard
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 3
![Page 4: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/4.jpg)
HDF5 software challenges - Portability
• Portability: IBM,SGI,windows, linux, Solaris, OSF1, cygwin, Cray,FreeBSD,Mac-OS
• Parallel IO: depends on MPI-IO, parallel File System and hardware
- MPI-IO: IBM AIX, MPICH, SGI Altix
- Parallel File System: GPFS, Lustre
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 4
![Page 5: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/5.jpg)
HDF5 software challenges - Features
• Programming languages - C, Fortran, C++
• External libraries: szlib encoder and decoder, zlib
• Comprehensive internal library test suite- time-consuming tests: fractral heap
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 5
![Page 6: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/6.jpg)
HDF5 software challenges - Others
• 34 configuration features --enable-cxx, --enable-fortran etc.
• THE TESTING CHALLENGEmachines x operating systems x compilers x languagesx Szip (encoder + no encoder) x (serial + parallel)= a very large number
• Coordination among developers- 3-4 core library developers 5-6 developers for tools and others- subversion not enough
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 6
![Page 7: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/7.jpg)
Solutions
• HDF5 Daily Test on main-stream UNIX platforms- Rob Matzke started around 1997- Albert Cheng took over
• More platforms, testing with more features
• Different version of HDF5 1.6, 1.8 • Other product: HDF4• Other platforms: Windows
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 7
![Page 8: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/8.jpg)
Daily automatic test procedure
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 8
Start the automatic job 1. Configuring2. Compiling library and tools3. Running tests for library and tools4. Installing the library5. Testing examples
Sending out the results to hdf5 library mailinglists
1. Platform watcher diagnoses the failure2. Inform the corresponding developer if the failure is real
The developer fixes the problem
![Page 9: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/9.jpg)
An example for daily test
Date: Tue, 6 Nov 2007 08:00:15 -0600 [08:00:15 AM CST]
From: HDF Tester [email protected]: [email protected]: kagiso
HDF5_Daily_Tests_1106Tue_FAILED!!!
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 9
Date: Tue, 6 Nov 2007 09:23:49 -0600 [09:23:49 AM CST]
From:
Quincey Koziol <[email protected]>
To: [email protected]:
Re: kagiso HDF5_Daily_Tests_1106Tue_FAILED!!!
![Page 10: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/10.jpg)
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 10
![Page 11: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/11.jpg)
Other helpers
• Committest script-automatically test a few platforms before checking in source code
• Save developers’ time
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 11
![Page 12: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/12.jpg)
Performance
• High IO performance is always a goal for THG
• Detect bad performance in time• Performance framework
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 12
![Page 13: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/13.jpg)
Performance framework
• Easy to Use for Various Benchmarks
• Multiple Platforms and Versions
• Long Term Regression Tests
• Help Debugging
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 13
![Page 14: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/14.jpg)
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 14
Background
• Backend: Cron job / DB Storage
• Core: Performance C/C++ Library
• Frontend: PHP / jpgraph
![Page 15: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/15.jpg)
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 15
Solution
Easy to Use
A User’s Benchmark
Performance Library
Database
Web Server
cron
www
HDF5 1.6 HDF5 1.8
PHP
Graph/Text
![Page 16: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/16.jpg)
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 16
| 178820 | 2007-08-17 21:51:14 | 10000 groups | creating 10000 empty groups | 1.8.0 | hdfdap | 0.670198 | 4384 |
for(i=0;i<1000 ;i++) { H5Gcreate(fileid,group_name,(size_t)0)); // Add groups}
H5Perf_endTimer(&time);
H5Perf_startTimer(&time);
H5Perf_addInstance(db_host, date, time);
Example Usage
00 21 * * * /home/local/hyoklee/src/chicago/test-perf-hdfdap-3.sh
Timestamp Instance Name Version Platform Time
![Page 17: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/17.jpg)
Demo
http://hdfdap.hdfgroup.uiuc.edu/h5perf/index.html
11/7/2007 17HDF and HDF-EOS Workshop XI, Landover,
MD
![Page 18: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/18.jpg)
Other Performance work
• Performance studiescompression, chunking and parallel IOhttp://www.hdfgroup.uiuc.edu/papers/papers/
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 18
![Page 19: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/19.jpg)
Coding standard
• Not much except seminars on HDF4/HDF5 coding standards – We definitely need to improve in this area
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 19
![Page 20: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/20.jpg)
Other work we have done to improve software process
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 20
![Page 21: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/21.jpg)
User involvements
• Public mailing [email protected]@[email protected]@hdfgroup.org
• Public RFCs• Solicit comments for new HDF5 features etc.
http://www.hdfgroup.uiuc.edu/RFC/HDF5/• Ask special groups to give us feedback
http://www.hdfgroup.uiuc.edu/RFC/HDF5/H5CHK/• Subversion repo
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 21
![Page 22: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/22.jpg)
Trainings for developers
• Internally Book reading: Programming Pearls
• Attending Dr. Dobb’s software conference
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 22
![Page 23: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/23.jpg)
Near-term plan
• Enhance daily correctness testsAPI compatibility tests: doneAPI Version tests: in the processJava wrapper tests: doneOpen source packages that use HDF
oEOS2 with HDF4oEOS5 with HDF5oNetCDF4 with HDF5
• Weekly “stable” code snapshots
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 23
![Page 24: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/24.jpg)
Long-term Plan
• Coding standard: code review• Standards :
- In the process of applying for ISO/ANSI standard for HDF5
• 500 random API tests to avoid ungraceful crash
• Collect existing HDF5 files such as EOS2, EOS5 files- Running all HDF4/HDF5 tools on these files
periodically
• Daily correlation regression tests on external machines
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 24
![Page 25: HDF5 Software Process](https://reader036.fdocuments.us/reader036/viewer/2022062422/56813d0a550346895da6b8c4/html5/thumbnails/25.jpg)
11/7/2007HDF and HDF-EOS Workshop XI, Landover,
MD 2525
Acknowledgement
This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.