Improving an Air Quality Decision Support System through the Integration of Satellite Data with

Ground-Based, Modeled, and Emissions Data

DSS Performance MetricsDSS Performance Metrics

Sarav Arunachalam Institute for the Environment

University of North Carolina at Chapel Hill

Presented at theNASA ROSES DSS Design Workshop

Cooperative Institute for Research in the AtmosphereFt Collins, CO

May 12-13, 2008

2

Background – NASA Reporting Requirements – (1 of 2)

• At least 2 Demonstrations and Prototypes– One demonstration should show the configuration and

performance of the prototype– One demonstration should show the use and performance of the

enhanced DSS in the end users’ operational environment

• Quarterly Reports: One-page chart with four sections– Major accomplishments and performance measures– Major milestones– Risks, risk mitigations, and project metrics– Issues and action items to address

3

Background – NASA Reporting Requirements – (2 of 2)

• Final Report– System configuration diagram– Results of feasibility evaluation(s), issues encountered and

resolved– Integration issues and interoperability issues encountered and

resolved– Robust documentation of procedures and guidelines describing the

steps to access, integrate, and utilize the Earth science research results

– Lessons learned– Resource estimate for the user organizations’ adoption and

sustained use of the Earth science products– Recommendations and remaining issues facing the sustained use

of the Earth science data in the enhanced DSS by the partner agency and end users

– Analysis of DSS performance with the Earth science results compared to the baseline

– Quantitative and qualitative enhancements to the DSS and related decision-making

– Quantitative and qualitative socio-economic benefits (actual or estimated) from the improved decision-making enabled by the project and enhanced DSS

4

Develop Performance Measures: Examples

• Focus on 2 broad types– Effectiveness– Achievement of end-user agency mission

5

Develop Performance Measures: Effectiveness

• Measures based upon Information Content – Content– Accuracy– Extensiveness– Usefulness– Search Capability– Use policy?

• Measures based upon Ease of Use – Overall Design– Ease of navigation

• Can the end-user find things needed in a timely manner?– Mechanism for end-user feedback– Accessibility– Speed– Ease in performing analyses

• Statistical, graphical, tabular reports

6

Develop Performance Measures: Achievability of Agency Mission

• Time taken to support decision-making– Performing analyses– Data downloads

• Others?

7

Develop Performance Measures: How to?

• Generate questionnaire to survey AQ Planners on defining baseline and probable performance measures– WRAP and VISTAS RPO mailing lists– EPA Offices– Other?

• Think of Emissions, Monitoring and Model Data

• Target all applications of air quality management– Develop SIPs, FIPS– Real-time forecasting– Regulatory Impact Analyses– Other?

• How to quantify expertise level of user– Distinguish between baseline and enhanced phases of DSS

• Others?

8

Develop Performance Measures: How often?

• How to define benchmarks of progress– Repeat survey, and see how things have improved / changed for

end-users

• Target other planned conference/meetings, and try to document progress– EPA organized State/Local Modelers Meeting?– RPO Modelers meetings

• Solicit feedback from end-users at these conferences

• Plan for Year 2 meeting of end-users – Funding constraints may mean teleconference instead of face-to-

face meeting

• Others?

9

Questions / Discussion