An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS

Swath Format

Ken Stone, Center for Lower Atmospheric Studies -

University of Colorado, Boulder

with help from...

Acknowledgements

• Larry L. Gordley - GATS, Inc., Newport News, Virginia

• The HALogen Occultation Experiment (HALOE) Team

• Cheryl Craig, NCAR

• Joe McInerney, University of Colorado

Overview

• Overview of the toolkit– History– Motivation– Some examples

• HDF-EOS extensions– Problems & Limitations– Strengths– Lessons learned

• Summary

Motivation

• Problem: It takes considerable time and effort to make the many multiple comparisons, internal and external, needed during validation phase.

• Goal: A simple set of tools that allow validation staff to select, line-up, compare and plot profiles and cross sections.

• Latest twist: HDF-EOS files are now the archive format of choice.

History

• LIMS - Early 80’s– Fortran routine for comparisons and plotting– Fast, low memory usage– Input: multiple files– Output: text summaries and plots

• HALOE - Early 90’s– Perl Scripts and Standard I/O file format (bsel format)– Fortran routine ported to IDL– New functionality and visualization tools.

• HIRDLS– HDF-EOS Swath capability– Additional Tools (dot-plots, map differences)

Toolkit Overview

• IDL procedures with Perl scripts for control– Complex scenarios developed both with interactive use (storing results

in “journal” file) and stringing scripts together with Unix shell.

• Analytical Visualization tools– Mapping, cross sections, line plots.– Focus on quality and simplicity rather than complex visualizations.

• Analysis tools– Coincidence finder, statistical computations.

• Standardized file format(s) for I/O• “Profile” based format hinges on vectors of geo-referenced

quantities.

Toolkit Detail

• Prior to HDF-EOS extension the Toolkit used a single file type for input (BSEL format)– Similar in some ways to HDF 4 SDS but limited to sets of 1-d

arrays stored as records.

• User input for any given IDL scenario can be recorded in a journal (text) file for later use.

• Perl “wrapper” scripts create this file as output.– Keyword substitutions in a “mask” file.

– Depending on the mode (mood?) the user is in, they could modify the journal file directly, or change the inputs on the script.

Toolkit Script Methodology

Perl Script (e.g., 2dlat)

IDL procedureor C, or Fortran...

Mask File

Input File

User Input: -f filename -I id, -o outputfile -g -h, etc

Postscript

BSEL File

Perl helper lib

Journal file creation from IDL scripts Analysis routines

Toolkit Data Flow: Archive and Working Formats

Selection Analysis Display

Data

Source

HDF-EOS

Bsel

Toolkit Extension

“Standard” Data Format

Archive

“working” data format

Toolkit Data Model (BSEL Format)BSEL File

Event (1) Event (i)

Plot InfoRange, axis Label, legend Label, Scale, Logflag

Event HeaderReal(8)Long(8)

char*60(6)

Data (j,k)

. . .

Data HeaderData order (=2)J (Profile count)K (levels/event)

HIRDLS HDF-EOS Swath

Metadata

Data Fields

HDF-EOS Swath File

Inst. Name,Ranges, Labels, Scales, Missing Values, Profile count

Time

Global Attributes

Geolocation Fields

Pressure Ozone Mixing Ratio

nLev

els

nTimes

Dimensions:nTimes, nLevels etc.

HDF-EOS Extension Strengths

• Fast! Ten times faster than native (Bsel) format

• Easier to identify field names (and more flexible)

• Standard way of identifying missing data

• IDL/Fortran/C I/O interfaces

• Potentially more tools available

HDF-EOS Extension Difficulties

• Lots of different ways to store data and still meet the standard.– Aura standard format guidelines helps here.

• HDF-EOS Swath isn’t “packet-like” – You can’t just “cat” files together!

• IDL/Fortran/C I/O interfaces– Some trouble with stability of certain I/Fs.

• Point Interface (and raw Vdata) failed from the start due to lack of “id” resources.

Lessons Learned

• Developed general purpose swath reader in IDL – usage: data = get_sw_data(filename, swathname, varname)– Generally reads any variable or attribute.– http://www.eos.ucar.edu/hirdls/get_sw_data.pro

• Working out the best way to “easily select” variables from swaths is a work-in-progress.

• To map HDF to the internal data structure a small external data table was needed...

Mapping Table HDF-EOS - Internal

SwathName VarName Label ID Min Max Log Scale UnitHIRDLS H2O H2O Mixing Ratio 2 0.0 10.0 0 1.0e-06 ppmvHIRDLS NO2 NO2 Mixing Ratio 3 0.0 10.0 0 1.0e-09 ppbvHIRDLS O3 O3 Mixing Ratio 4 0.0 10.0 0 1.0e-06 ppmvHIRDLS CH4 CH4 Mixing Ratio 6 0.0 2.0 0 1.0e-06 ppmvHIRDLS CFCl3 CFCl3 Mixing Ratio 151 0.0 0.5 0 1.0e-09 ppbvCH4 L2gpValue CH4 Mixing Ratio 6 0.0 2.0 0 1.0e-06 ppmvO3 L2gpValue O3 Mixing Ratio 4 0.0 10.0 0 1.0e-06 ppmv

Internal to toolkitInternal to HDF-EOS

Designations for MLS Files

Summary

• HDF-EOS Swath format provides a good storage facility for atmospheric profile data.– Current swath implementation (a la Aura) provides solid

groundwork for future sophisticated software development.

• Interface instability can contribute to major headaches.

• Ancillary tools to “view and edit” or simply compute additional fields from existing ones could prove quite useful.

Future

• Extend data base table for mapping fields.

• Develop interface for Grid format.– Already tested in limited cases.

• Testing, testing, testing…

• Perhaps incorporate new IDL features– Improved color contouring

• Develop an easier to use interface for the scripting tools.