Post on 19-Dec-2015
Integrated Decision Support:A Tale of Two Systems
“It was the best of times, it was the worst of times…”
Charles Dickens, A Tale of Two Cities, 1859
“Actually, right now, I’d say the times are just OK…”
Shawn McClure, EPA Data Summit, 2008
“On second thought, the times must be pretty damn good. After all, we’re at a data summit convened by the EPA to talk about data systems. Woo-hoo!!…”
Shawn McClure, EPA Data Summit, 2008
VIEWSThe Visibility Information Exchange Web System
http://vista.cira.colostate.edu/views
Developed by the five Regional Planning Organizations (RPOs) in 2002
Provides easy online access to a wide variety of air quality data
Offers user-friendly tools for exploring, visualizing, and analyzing data
The primary source for IMPROVE Regional Haze Rule data
Over 1100 registered users
Over 300 organizations, institutions, and companies represented
Developed for RPOs, States, Tribes, Federal Land Managers, and local agencies
Used by researchers, analysts, planners, regulators, stakeholders, and students
TSSWRAP Technical Support System
http://vista.cira.colostate.edu/tss
Built upon the database and software infrastructure of VIEWS
Developed initially for Western States, Tribes, and air quality agencies
Provides a access point for regional technical data, guidance, and results
Offers tools and support for the development of SIPs and TIPs
Documents the technical methods used in implementation plans
Helps assess the impact of other areas on local Class I Areas
Provides ongoing tracking and assessment of emissions control strategies
Will adapt to and serve the future regional technical needs of WRAP members
DataCollection
PrimaryStorage
Aggregation, Integration, Processing,
Modeling
Analysis, Visualization,
Reporting
Decision Support
EndUsers
Draft Data Value Chain Diagram
Analysis, Visualization and ReportingApplications
Decision SupportApplications
Group A2: Data
Access
Group B2:Integration
Consolidation
Group C2:Data Visualization and
Analysis
Group A2: Data
Access
DataCollection
PrimaryStorage
Aggregation, Integration, Processing,
Modeling
Analysis, Visualization,
Reporting
Decision Support
EndUsers
Draft Data Value Chain Diagram
Analysis, Visualization and ReportingApplications
Decision SupportApplications
Group A2: Data
Access
Group B2:Integration
Consolidation
Group C2:Data Visualization and
Analysis
Group A2: Data
Access
Data
Data Collection
Standardization
Transformation
Aggregation
Normalization
Verification
Validation
Processing
Primary StorageAggregation, Integration,
Processing, Modeling
Modeling
Reanalysis
Analysis
Comparison
Assessment
Forecasting
Analysis
Analysis, Visualization, Reporting
Recommendations
Conclusions
Regulations
Decisions
Decision Support End Users
VIEWS/TSS Data Value Chain
Standardization
Transformation
Aggregation
Normalization
Modeling
Reanalysis
Analysis
Comparison
Assessment
Forecasting
Recommendations
ConclusionsVerification
Validation
Regulations
Data Processing Analysis Decisions
Data Collection Primary StorageAggregation, Integration,
Processing, ModelingAnalysis, Visualization, Reporting Decision Support End Users
Standardization
Transformation
Aggregation
Normalization
Modeling
Reanalysis
Analysis
Comparison
Assessment
Forecasting
Recommendations
ConclusionsVerification
Validation
Regulations
Data Processing Analysis Decisions
Data Collection Primary StorageAggregation, Integration,
Processing, ModelingAnalysis, Visualization, Reporting Decision Support End Users
VIEWS/TSS Data Processing: Import & Validation
AIRDATA_OLTP
Data Acquisition
System
AIRDATA_SOURCE
Metadata Import System
Data Import System
Validation
Validation
· Emailed updates from data providers· Automatic HTTP extraction of source
data files· Database replication· Imports data “as-is” into the source
database· Technologies: ASCII text files, Excel
spreadsheets, Access databases, etc.
· Facilitates the entry of new metadata
· Validates new metadata entries· Detects overlap with existing
metadata· Technologies: VB .Net, ASP .Net,
and SQL Server stored procedures
· Extracts data from the source database· Scrubs data and performs conversions· Maps source metadata to integrated metadata· Transforms the data into an integrated schema· Verifies and validates imported data· Loads data into the back-end OLTP system· Technologies: Microsoft SQL Server DTS,
stored procedures, SQL scripts, and Visual Basic routines
Metadata Import System Data Import SystemData Acquisition System
OLTP:OLTP:
• Functions as the “back-end” database• Fully relational and in 3rd normal form• Used for data import, validation, and
management• Technologies: Microsoft SQL Server
Data Warehouse Generation System:Data Warehouse Generation System:
• Extracts data from the OLTP• De-normalizes and transforms data• Loads data into the Data Warehouse• Builds table indexes• Archives “snapshots” of the database• Technologies: VB, stored procedures
Data Warehouse:Data Warehouse:
• Functions as the “front-end” database• Uses a de-normalized “star schema”• Used for querying and archiving data• Automatically generated from the OLTP• Technologies: Microsoft SQL Server
Standardization
Transformation
Aggregation
Normalization
Modeling
Reanalysis
Analysis
Comparison
Assessment
Forecasting
Recommendations
ConclusionsVerification
Validation
Regulations
Data Processing Analysis Decisions
Data Collection Primary StorageAggregation, Integration,
Processing, ModelingAnalysis, Visualization, Reporting Decision Support End Users
VIEWS/TSS Analysis Tools: Network Inter-comparison
Parameter: Nitrate Ion Concentrations
Location: Bondville, IL
Networks: IMPROVE, STN, and CASTNet
Graphs: Time Series and Scatter Plot
VIEWS/TSS Tools: Model Performance Evaluation
CMAQ Model Performance vs. Monitored Worst 20% Days in 2002
VIEWS/TSS Analysis Tools: Source Apportionment
Mass source apportionment by source category and region
From regional photochemical model with comprehensive emissions inputs
Species mass for various time periods – directly comparable to monitoring data
VIEWS/TSS Analysis Tools: Multidimensional Analysis
Multiple NAAQS indicators, multiple time periods, multiple locations
Displayed with consistent, comparable graphics in the same tool
VIEWS/TSS Analysis Tools: Emissions Review Tool
• Available Parameters:– Sulfur Dioxide– Sulfur Oxides (gas and
particulate)– Nitrogen Oxides (gas)– Nitrogen Oxides (gas and
particulate)– Other species from SMOKE
• Multiple dimensions:– Parameters– Emissions Scenarios– Source Categories– Regions of Interest
• Future plans:– Display regional summaries as
well as state-level emissions
VIEWS/TSS Interoperability
Import and output data in common formats, by standard protocols
K.I.S.S. – Because users like to do odd things with data
Practice a “Service Oriented Architecture” approach
Develop reusable, “component” tools
Make interoperability an important goal, while resisting the urge to develop the “Ultimate All-Things-to-All Users” type of system
Adhere to sound software and system architecture principles
Allow direct, read-only access to our database
Support and promote an open data and software architecture
Standardization
Transformation
Aggregation
Normalization
Modeling
Reanalysis
Analysis
Comparison
Assessment
Forecasting
Recommendations
ConclusionsVerification
Validation
Regulations
Data Processing Analysis Decisions
Data Collection Primary StorageAggregation, Integration,
Processing, ModelingAnalysis, Visualization, Reporting Decision Support End Users
VIEWS/TSS Decisions: Benefactors & Benefits
Federal Land Managers:
• Better air quality evaluation• More accurate source apportionment• More meaningful impact assessment• Informed permitting and regulation• Improved air quality in ecosystems
EPA and RPOs:
• Effective emissions control evaluation• Practical and equitable standards• Leveraging of existing efforts• Improved synergy with States and Tribes• Valuable case studies and precedents• More effective application of resources
States, Tribes, and Local Agencies:
• Effective Implementation Plans• Defensible and achievable control
strategies• Uniform and reasonable progress• Better air quality in Class I Areas
General Air Quality Community:
• Better insight into the decision process• Real life evidence of successes &
failures• Incorporation of feedback into next
generation products• Increasing expertise in developing
decision support tools
VIEWS/TSS Team: Future Plans
Add satellite data in conjunction with NASA ROSES contract
Add additional NAAQS (ozone, for example) and global data
Develop more tools that offer region- and locale-specific analyses
Develop appropriate new functionality as web services
Refine decision tools for making interpretations and recommendations
Expand and refine guidance documentation
Establish case studies from the user community as guidance
Allow users to upload and immediately work with their data
Develop modular, reusable, components for other developers
Establish reusable precedents for air quality decision support
Have a few beers every once (or twice) in a while…
Decisions(interpretations, conclusions, guidelines, regulations
Science(knowledge, examples,
expertise, theory)
Analysis(Tools, reanalysis, modeling,
inputs from other systems)
Processed data(calculated, RHR,
normalized, value-added)
Raw data(observations, methods)