Climate Data Records and Science Data Stewardship:

Playing for Keeps

Bruce R. Barkstrom

National Climatic Data CenterNOAA

Outline

• What are CDRs– An Example– General Characteristics

• What’s Involved in SDS– Assuring that the data and context are valuable to the

future – Making sure data are ready to preserve– Making sure data and context will be useful– Making sure data and context will survive– Being cost effective

An Example CDR – Solar Constant

• Original data cover several decades

• Multiple data sources• Work needed:

– Physical model of causes of differences

– Development of homogeneous data set versions

– Estimation of detectable variability and trends

CDR Characteristics

• Covers long time period (decades or more if possible)• Likely to have multiple data sources• Every attempt to deal with errors on a physical basis• Every attempt to make errors homogeneous over record

– Software must have full configuration management– Input data sources should be as homogeneous as possible

• Intent is to provide– Quantified variability: Cumulative Distribution Functions (CDFs)

of parameter variations, not only for global averages, but also regional values and extreme value statistics

– Quantification of Change Detection: Ability to test observed CDFs against expected CDFs of potential changes

How Do We Assess the Value of a CDR?

• 3 Approaches:– Cost of Acquiring CDR– Cost of Reconstruction – if possible

Need to have original data, need to assemble hardware and software, need to run (maybe 2 or 3 million jobs)

– Present Value of Future Use Economists discount future benefits at 7%

Valuation is Tough

• OMB Question: Why do we need more than $2B/year for climate?

• CCSP and CEOS both have had trouble prioritizing

• Probably two scales of value– Scientific “Value” – represented by “Bretherton

Issues”– Societal Benefit – represented by reduction in

damage, lives saved, new industries created• Quantifying to OMB’s satisfaction is difficult• Question 1: Can CI help with justifying priorities?

Good Archival Practice

• ISO Standard for “What an Archive Should Do for Long-Term Preservation”– OAIS Reference Model

• Recommendation:– Prepare a Submission Agreement between an

Archive and a Data Provider– Evaluate condition and completeness of candidate

data and metadata– Plan work required to repair deficiencies

• SDS Preferred Approach – use “Maturity Model”

Maturity Model

• Evaluate Maturity 3 ways:– Scientific Maturity– Preservation Maturity– Societal Benefit

• For Each Axis:– Reduce evaluation to non-dimensional scaling of

attributes– Ask for evaluation from experts

• Question 2: Can CI help with evaluation of maturity?

Work Required to Produce CDRs

• Evaluation of Available Record for Gaps and Understandability– Gaps– Documentation

• Evaluation of Candidate CDR Uncertainties– Error Sources Considered– Calibration and Validation

• Evaluation of Record Repair Work– Gaps– Recalibration– Uncertainty Estimation

Roles of Satellite Data and In-Situ Data

• In-situ Data Complements Satellite Data– Satellites for coverage – although challenge is

getting adequate length of record– In-situ for calibration and validation

• For Data Stewardship– Need preservation of context: cal-val data

preservation, source code, documentation of procedures, metadata

– Results of intercomparisons should have measurable improvement in uncertainty

Some Thoughts on Quantifying Impact of In-Situ Data

• Errors in satellite measurements– Estimates should be based on physical causes– Stewardship needs way of making publically available – and accomodating changes in assessments by

community over time– Statistical in nature– Delimited by time interval and spatial region– Most rigorously specified as CDF of error– Might be simply specified in terms of std dev of error about “average” measured value

• Cal-Val efforts should improve “error bars”– Stringency: ratio of error dispersion about mean after cal-val to dispersion before

• 1 – no improvement; 2 to 5 – moderate improvement; >10 – really stringent requirement on cal-val• Related to number of independent samples in cal-val set

– Plausibility: significance of improvement• Unsuspicious – p of difference 20%; Somewhat convincing – p ~ 5%; Fairly confident – p ~ 1%

• Number of iterations in reprocessing– Inversely proportional to experience

– Increases with required stringency and plausibility

• Question 3: Can CI help evaluate proposed In-Situ Validation Data Sets for Error Reductions, Stringency, and Plausibility?

The Odds for Long-Term Preservation

• Preservation inclines one toward pessimism– If p is annual probability of survival and– N is number of years to survive– Probability of survival is (1 – p)**N– To have 99% probability of survival for 200 years,

requires p = 5. E -05

• Standard approach to reducing risk– Assess mechanisms of loss– Quantify annual probability of loss and probable value

of loss [note return to valuation issue]– Find affordable risk mitigation approach

Science Data Stewardship:What are the Odds

• Important Risks– IT Security Incidents

• 10% per year probability; maybe 10% of collection at risk of corruption (p = 1%/yr – need dispersion acrosss systems)

– Operator Error• 10% per year probability; loss depends on time operators

work and degree of automation (p = 1%/yr – need QA)– Hardware or Software Error

• 5% per year probability; loss as in operator error– Hardware or Software Obsolescence

• 100% probability of loss in 5 to 10 years (p = 20%/yr)• Suggests treating expenses of hardware and software

replacement as “insurance expenses” – not assets

Science Data Stewardship:How Do We Improve the Odds?

• SDS will require several new things:– Making the history and details of data provenance public

(anything proprietary dies)– Capturing now-tacit knowledge before it disappears (knowledge

not captured dies when the knower retires, gets sick, or dies)– Creating methods of tracing the evolution of data, metadata, and

assessments of same• Expectation: SDS grants program provides avenue for

bringing in ideas that – improve information survivability– reduce cost of archival– make data and context more useful for those that come after

• If we don’t succeed, we’ve all been publishing in The Journal of Irreproducable Results