3SAQS Technical WorkshopFebruary 25th, 2015

Data WarehouseCurrent and Future Development

Intermountain West Data Warehouse

Identification, Acquisition, Pre- and Post-processing, Extraction

Storage, Backup, Restore, Security, Summarizing, Statistics

Searching, Querying, Filtering, Aggregating, Formatting, Packaging

Charting, Graphing, Mapping, Analyzing

Architecture, Design, Implementation, Management, and Operation

WGANPS CIRA

Acquisition Integration Management Distribution Presentation

Verification, Validation, QA/QC, Mapping, Flagging, Tranformation

Tools

3SDW

Results

Documents

Raw Data

Modelers

Planners

Managers

Guidance, Requirements, Feedback, Funding

Monitored

Modeled

Aerosol

Deposition

Gaseous

Emissions

Met

Air Quality

AQS, VIEWS

WestJump, future modeling, etc

Data Warehouse System Overview

Ongoing Goals and Activities

Model Input Data:• Acquire and organize model input data for new and updated platforms

• Make model input data readily accessible

• Track and characterize download and request activity for input data

Model Results:• Enable uploading of model results

• Make model results readily accessible for download and interactive analysis

• Track and characterize download and request activity for model results

Ongoing Goals and Activities

General System Operation and Maintenance:• Backup, archive, and ensure the availability of cooperator data and resources

• Maintain and refine the data warehouse website

• Document the system and its tools

• Service approved cooperator requests for features and data

• Facilitate synergy and collaboration with other relevant projects

• Maintain the hardware and software infrastructure of the data warehouse

Data Warehouse Hardware - Current

Modeling Data• 36-bay Linux server with twenty four 64-bit, six-core CPUs and 24GB RAM

• 3 RAID 6 partitions:o Partition #1: Twelve 2TB disks (~17TB)

o Partition #2: Twelve 6TB disks (~55TB)

o Partition #3: Currently empty (12 6TB disks would give ~55TB)

o Potential total: 165TB (if 2TB disks are replaced with 6TB disks)

Monitoring Data• Windows server with two 64-bit, sixteen-core CPUs and 128GB RAM

• 1TB solid state drive (OS), 3 6TB drives (data)

• “Shared” with the FED project, which funded the purchase

Data Warehouse Hardware – Estimates for the Future

• Estimate the size of “representative” platforms to be maintained

• Estimate the number of representative platforms to maintain online

• Estimate the growth rate of new datasets

• Estimate volumes and rates for data delivery needs

• Estimate the archival rate for historical platforms

• Estimate the storage required for backup, redundancy, and recovery

• Estimate the aging and replacement rate for hardware components

Identification, Acquisition, Pre- and Post-processing, Extraction

Storage, Backup, Restore, Security, Summarizing, Statistics

Searching, Querying, Filtering, Aggregating, Formatting, Packaging

Charting, Graphing, Mapping, Analyzing

Architecture, Design, Implementation, Management, and Operation

WGANPS CIRA

Acquisition Integration Management Distribution Presentation

Verification, Validation, QA/QC, Mapping, Flagging, Tranformation

Tools

3SDW

Results

Documents

Raw Data

Modelers

Planners

Managers

Guidance, Requirements, Feedback, Funding

Monitored

Modeled

Aerosol

Deposition

Gaseous

Emissions

Met

Air Quality

AQS, VIEWS

WestJump, future modeling, etc

Data Warehouse System Overview – “Phase I”

Identification, Acquisition, Pre- and Post-processing, Extraction

Storage, Backup, Restore, Security, Summarizing, Statistics

Searching, Querying, Filtering, Aggregating, Formatting, Packaging

Charting, Graphing, Mapping, Analyzing

Architecture, Design, Implementation, Management, and Operation

WGANPS CIRA

Acquisition Integration Management Distribution Presentation

Verification, Validation, QA/QC, Mapping, Flagging, Tranformation

Tools

3SDW

Results

Documents

Raw Data

Modelers

Planners

Managers

Guidance, Requirements, Feedback, Funding

Monitored

Modeled

Aerosol

Deposition

Gaseous

Emissions

Met

Air Quality

AQS, VIEWS

WestJump, future modeling, etc

Data Warehouse System Overview – “Phase II”

Monitoring Data Tools – Example

Emissions Review Tool – Example Chart

Oil & Gas Emissions by State for CO, UT, WY, and surrounding states

Emissions Review Tool – Example Chart

Oil & Gas Emissions by County for Colorado

Emissions Review Tool – Example Report

Oil & Gas Emissions by SCC for Weld County, Colorado

Model-To-Obs Comparison Tool

Source Apportionment Tool

Gridded Data Visualization Tool

Future Goals and Activities

• Enable the uploading and cataloging of model results

• Refine and expand the capabilities of visualization and analysis tools

• Leverage GIS capabilities for data visualization

• Expand the selection of gridded-data visualization tools

• Enable “data drill-down” on relevant tools

• Link relevant tools to enable fluid data exploration

• Solicit and incorporate feedback from collaborators and users

Give Us Feedback:

http://views.cira.colostate.edu/tsdw/User/Feedback.aspx