3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.
-
Upload
merryl-cobb -
Category
Documents
-
view
212 -
download
0
Transcript of 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.
![Page 1: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/1.jpg)
3SAQS Data Warehouse Update
October 10, 2014CIRA - Fort Collins, CO
![Page 2: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/2.jpg)
3SAQS Steering Committee October 10, 2014
2
Outline
• Data Warehouse Team• System Overview• Phase 1 Operational State• Website Overview• Project & Request Workflow• Future Work
![Page 3: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/3.jpg)
3SAQS Steering Committee October 10, 2014
3
Data Warehouse Team
• Shawn McClure• Tom Moore• Dustin Schmidt• Rodger Ames• Cassie Archuleta (ARS)
![Page 4: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/4.jpg)
System Overview
![Page 5: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/5.jpg)
53SAQS Steering Committee October 10, 2014
![Page 6: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/6.jpg)
3SAQS Steering Committee October 10, 2014
6
Data Warehouse – Phase 1
• Project Management System– Project/inventory tracking, modeling platform
request form (Phase 1=NEPA), data manifest• Hardware Upgrades– New 55tb RAID– USB3 transfer to external disks
![Page 7: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/7.jpg)
3SAQS Steering Committee October 10, 2014
7
Data Warehouse – Phase 1 (cont.)
• Modeling Platforms– Base 2008b, Base 2011a CAMx (release pending
approval)• Data deliveries– Hiawatha (ENVIRON)– Converse County (AECOM)
• Website – Pages for studies, projects, documents, procedures
and results, meetings & events, literature, glossary, about (updates ongoing)
![Page 8: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/8.jpg)
Data Warehouse – Phase 1 (cont.)
• Inventory Management– Data is placed within the warehouse– Warehouse file system is represented in database
• Facilitates change control
– File system items are data• Files• Directories
– Categories are groups of File System Items• Files and Directories are grouped into Categories• Meta data is attached to Category• Categories may be hierarchically related• Data may belong in multiple categories
– Data Packages are groups of Categories• Collection of categories for intended use (NEPA)
![Page 9: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/9.jpg)
3SAQS Steering Committee October 10, 2014
9
Website overview
• Studies– Modeling platforms (completed and forward looking)– Monitoring data overview (pending)
• Documents– Modeling platform protocols, MPEs, etc.
• Procedures and Results– Specific how-to pages for DW web based tools (pending)
• About– General information about 3SAQS and the DW (pending)
• Projects– User initiated AQ modeling using 3SAQS platforms
![Page 10: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/10.jpg)
3SAQS Steering Committee October 10, 2014
10
Demo Website
http://views.cira.colostate.edu/TSDW/Demo/
![Page 11: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/11.jpg)
11
Data Warehouse Website – Phase 1
3SAQS Steering Committee October 10, 2014
![Page 12: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/12.jpg)
123SAQS Steering Committee October 10, 2014
![Page 13: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/13.jpg)
3SAQS Steering Committee October 10, 2014
13
Studies
3SAQS foundational modeling platforms• Phase 1 (done)– 2008b available– 2011a (release pending)
• Phase 2 (forward looking)– 2008_2020a– 2011_2020a (CAMx/CMAQ)– Preliminary 2014 (CAMX/CMAQ)– 2014_FutureCase (CAMX/CMAQ)
![Page 14: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/14.jpg)
3SAQS Steering Committee October 10, 2014
14
![Page 15: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/15.jpg)
3SAQS Steering Committee October 10, 2014
15
Documents• Memorialize work that is already done
– Network Assessment• Reference basis for modeling platforms that the DW is delivering
– Modeling protocols– MPE reports– Emissions reconciliations
• Reports and comments– 2008, 2011 Modeling Protocols & response to comments
• Calls for comments on draft documents– Base 2008b MPE– 2011 WRF MPE
• Technical memos– WestJump to 3SAQS – 3SAQS and 2011 NEI O&G Emission Inventories reconciliation– 3SAQS Methane inventory & response to comments
• Modeling Guidance – NEPA/non-NEPA guidance – SMOKE
• Work plans• Annual & Quarterly Reports
![Page 16: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/16.jpg)
3SAQS Steering Committee October 10, 2014
16
![Page 17: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/17.jpg)
3SAQS Steering Committee October 10, 2014
17
Procedures and Results
• “How to” pages for:– Data download and upload procedures– Application of visualization tools– Understanding modeling platforms– Interpreting emissions inventories– Accomplishments and future work of the 3SAQS
and DW
![Page 18: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/18.jpg)
3SAQS Steering Committee October 10, 2014
18
About
• Informational Flyer– Goals of 3SAQS and role of DW– Studies and project management overview– Modeling toolbox highlights
• Geospatial data selection
• 3SAQS and cooperators– BLM– NPS– Forest Service– States of CO, WY, UT
![Page 19: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/19.jpg)
3SAQS Steering Committee October 10, 2014
19
Projects
User initiated AQ modeling projects• NEPA Projects
– Hiawatha (ENVIRON)– Converse County (AECOM)– Others pending
• Non-NEPA (future work)– NPS N deposition– NPS Vertical Profiles of simulated ammonia– EPA QA 2011 Base A– State of CO RAQC O3 SIP– State on NM O3 nonattainment– Others…
![Page 20: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/20.jpg)
Project Workflow - High Level Goals
• Manage Data Access• Provide Project Managers an overview of data delivery
progress• Provide Users a concise view of available data• Provide Data Requestors with a detailed manifest (receipt)• Detailed data accounting (Inventory Management)
– Meta data– Who got what and when– Change control
• Track usage of supplied data• Capture communication threads (Future work)
![Page 21: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/21.jpg)
3SAQS Steering Committee October 10, 2014
21
Project Workflow - Overview
• 3SDW Administrators designate registered users as “Project Managers”– Cooperating Agency point of contact
• Project Manager creates new analysis project– Provides high level meta data– Designates project team members
• Contractors, Subordinates, Etc
• Project team member initiates a Data Request– Data Selection– Provide detailed meta-data
• 3SDW Team fulfills data request• Project team member signs-off request as completed
![Page 22: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/22.jpg)
Project Workflow – User Roles
![Page 23: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/23.jpg)
Project Workflow - User Roles
• Registered User– View projects (Read-only)– View Data Requests (Read-only)
![Page 24: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/24.jpg)
Project Workflow - User Roles
• Project Manager – A Registered user who is also a member of a
3SDW cooperating agency (BLM, EPA, NPS, etc). – Create new analysis projects– Assign project team members– Monitor data transfer progress– Issue data requests
![Page 25: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/25.jpg)
Project Workflow - User Roles
• Project Team Member– Assigned to team by Project Manager– Issue data requests for assigned projects
![Page 26: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/26.jpg)
Project Workflow
![Page 27: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/27.jpg)
Projects Overview
• Purpose– Present an overview of Data Request status– Entry point for Project process– Create a new project
![Page 28: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/28.jpg)
Projects Overview
![Page 29: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/29.jpg)
Projects Overview
![Page 30: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/30.jpg)
Project Configuration
• Purpose– Capture high level information about the project• High level description (Details provided by data
requestors)• Location• Timeframe
– Managing Data Access• Selected pool of Project Managers (3SDW Cooperators)• Project Managers assign 3SDW registered users to
project
![Page 31: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/31.jpg)
Project Configuration
![Page 32: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/32.jpg)
Project Configuration
![Page 33: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/33.jpg)
Data Request
• Allow users to browse available data• Capture optional information about – Intended use of data– Results/Conclusions from data usage
• Supply detailed manifest of requested data
![Page 34: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/34.jpg)
Data Request
![Page 35: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/35.jpg)
Data Request
![Page 36: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/36.jpg)
Data Request
![Page 37: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/37.jpg)
Data Request
![Page 38: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/38.jpg)
Future Work
• Provide finer granularity in data selection• Tune process for supplying supplemental data• Web interface for Data Cataloging & QA• Enhance performance of online data transfer• Capture e-mail communication traffic for
Project / Data Request• Data Submission Process (Return model
results to warehouse)
![Page 39: 3SAQS Data Warehouse Update October 10, 2014 CIRA - Fort Collins, CO.](https://reader030.fdocuments.us/reader030/viewer/2022032523/56649d785503460f94a5aea4/html5/thumbnails/39.jpg)
39
Data Warehouse Website – Phase 2
3SAQS Steering Committee October 10, 2014