PostgreSQL for Buoy Data
Transcript of PostgreSQL for Buoy Data
![Page 1: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/1.jpg)
PostgreSQL for Buoy DataSteve Ruberg, Steve Constant, Ron Muzzi, Russ
Miller, Joe Smith
![Page 2: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/2.jpg)
Utilization of PostgreSQL Database for Real-Time Western Lake Erie Data
Storage and DisseminationRUBERG, S.A., CONSTANT, S.A., MUZZI, R.W., MILLER, R.J., and SMITH, J.P.
![Page 3: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/3.jpg)
Many thanks!
![Page 4: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/4.jpg)
What are we talking about today?
1. Western Lake Erie (WLE) Real-time (RT) data collection2. Database3. Interface
![Page 5: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/5.jpg)
Map made by Kaye LaFond
![Page 6: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/6.jpg)
Map made by Kaye LaFond
![Page 7: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/7.jpg)
1. WLE RT-Data Collection
● Meteorological● Water Quality
○ Phosphorus○ Chlorophyll○ Blue-green Algae○ Nitrogen
![Page 8: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/8.jpg)
Format
● Flat text● ~50 columns● Split by day● Several MBytes per season● High temporal resolution (every 15 min)
1. WLE RT-Data Collection
![Page 9: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/9.jpg)
Problem
● Client-side computer variability● Data require intense processing to visualize● Tried complete client-side interface to this
○ Not great...
1. WLE RT-Data Collection
![Page 10: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/10.jpg)
2. Database
● SQL database○ Relational Database Management System (RDBMS)○ Tables connected by keys○ Through PHP, server power can be leveraged
■ Data delivered in smaller, more digestible chunks
![Page 11: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/11.jpg)
● Adhere to the LAMP stack?○ Linux, Apache, MySQL, PHP
2. Database - choices
![Page 12: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/12.jpg)
● Adhere to the LAMP stack?○ Linux, Apache, MySQL, PHP○ “If you don't really know or care much about databases, data validity,
ACID compliance, etc, but you do care about ease and speed, you tend
to go with MySQL.” - User Orthogonality on AskMetafilter■ http://ask.metafilter.com/92162/MySQL-vs-PostgreSQL#1349848
2. Database - choices
![Page 13: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/13.jpg)
● Adhere to the LAMP stack?○ Linux, Apache, MySQL, PHP○ “If you don't really know or care much about databases, data validity,
ACID compliance, etc, but you do care about ease and speed, you tend
to go with MySQL.” - User Orthogonality on AskMetafilter■ http://ask.metafilter.com/92162/MySQL-vs-PostgreSQL#1349848
● Deployment environment = Federal Laboratory
2. Database - choices
![Page 14: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/14.jpg)
● Adhere to the LAMP stack?○ Linux, Apache, MySQL, PHP○ “If you don't really know or care much about databases, data validity,
ACID compliance, etc, but you do care about ease and speed, you tend
to go with MySQL.” - User Orthogonality on AskMetafilter■ http://ask.metafilter.com/92162/MySQL-vs-PostgreSQL#1349848
● Deployment environment = Federal Laboratory● National Buoy Data Center uses Oracle
○ According to: Handbook of Automated Data Quality Control Checks and Procedures (2009)○ http://www.ndbc.noaa.gov/NDBCHandbookofAutomatedDataQualityControl2009.pdf
2. Database - choices
![Page 15: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/15.jpg)
● Adhere to the LAMP stack?○ Linux, Apache, MySQL, PHP○ “If you don't really know or care much about databases, data validity,
ACID compliance, etc, but you do care about ease and speed, you tend
to go with MySQL.” - User Orthogonality on AskMetafilter■ http://ask.metafilter.com/92162/MySQL-vs-PostgreSQL#1349848
● Deployment environment = Federal Laboratory● National Buoy Data Center uses Oracle
○ According to: Handbook of Automated Data Quality Control Checks and Procedures (2009)○ http://www.ndbc.noaa.gov/NDBCHandbookofAutomatedDataQualityControl2009.pdf
● Go for something a little more free
2. Database - choices
![Page 16: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/16.jpg)
https://www.postgresql.org/about/
○ 15+ years development○ Open-source○ Nearly unbounded scalability○ ACID Compliant
2. Database - PostgreSQL
![Page 17: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/17.jpg)
https://www.postgresql.org/about/
○ 15+ years development○ Open-source○ Nearly unbounded scalability○ ACID Compliant
■ Atomicity - all or nothing
2. Database - PostgreSQL
![Page 18: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/18.jpg)
https://www.postgresql.org/about/
○ 15+ years development○ Open-source○ Nearly unbounded scalability○ ACID Compliant
■ Atomicity - all or nothing■ Consistency - transactions done right
2. Database - PostgreSQL
![Page 19: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/19.jpg)
https://www.postgresql.org/about/
○ 15+ years development○ Open-source○ Nearly unbounded scalability○ ACID Compliant
■ Atomicity - all or nothing■ Consistency - transactions done right■ Isolation - result of concurrent transactions same
as if they were run separately
2. Database - PostgreSQL
![Page 20: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/20.jpg)
https://www.postgresql.org/about/
○ 15+ years development○ Open-source○ Nearly unbounded scalability○ ACID Compliant
■ Atomicity - all or nothing■ Consistency - transactions done right■ Isolation - result of concurrent transactions same
as if they were run separately■ Durability - transaction permanence
2. Database - PostgreSQL
![Page 21: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/21.jpg)
![Page 22: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/22.jpg)
● 2 parallel tables○ Configurations○ Data and Quality Flags
● R used to process text data, Unix for automation
2. Database - Structure
![Page 23: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/23.jpg)
● Configuration ID (primary key)● Start Date● End Date● Notes● GMT Offset● Matrix with the configurations, 7 columns
2. Database - Configuration table
![Page 24: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/24.jpg)
● Matrix with the configurations, 7 columns○ Variable name universal across tables○ Variable name unique to the data under config○ Data units (as applicable)○ Statistic of data (min, max, mean, etc.)○ Data type○ QARTOD inspired gross/climatological range limits
■ Min■ Max
2. Database - Configuration table
![Page 25: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/25.jpg)
● Data and Quality Flags○ Data Label (primary key)○ Corresponding configuration ID○ Measure Date○ Data matrix○ Quality flag matrix
2. Database - Data table
![Page 26: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/26.jpg)
● Configuration Table○ Configuration ID ○ Start Date○ End Date○ Notes○ GMT Offset○ Config matrix
● Data and Quality Flags○ Data Label○ Configuration ID○ Measure Date○ Data matrix○ Quality flag matrix
2. Database
![Page 27: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/27.jpg)
● Configuration Table○ Configuration ID ○ Start Date○ End Date○ Notes○ GMT Offset○ Config matrix
● Data and Quality Flags○ Data Label○ Configuration ID○ Measure Date○ Data matrix○ Quality flag matrix
2. Database
![Page 28: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/28.jpg)
● Configuration Table○ Configuration ID ○ Start Date○ End Date○ Notes○ GMT Offset○ Config matrix
● Data and Quality Flags○ Data Label○ Configuration ID○ Measure Date○ Data matrix○ Quality flag matrix
2. Database
![Page 29: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/29.jpg)
● Configuration Table○ Configuration ID ○ Start Date○ End Date○ Notes○ GMT Offset○ Config matrix
● Data and Quality Flags○ Data Label○ Configuration ID○ Measure Date○ Data matrix○ Quality flag matrix
2. Database
![Page 30: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/30.jpg)
● Configuration Table○ Configuration ID ○ Start Date○ End Date○ Notes○ GMT Offset○ Config matrix
● Data and Quality Flags○ Data Label○ Configuration ID○ Measure Date○ Data matrix○ Quality flag matrix
Rows here match with... columns here.
2. Database
![Page 31: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/31.jpg)
https://www.glerl.noaa.gov/res/HABs_and_Hypoxia/rtMonSQL.php
![Page 32: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/32.jpg)
https://www.glerl.noaa.gov/res/HABs_and_Hypoxia/rtMonSQL.php
![Page 33: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/33.jpg)
https://www.glerl.noaa.gov/res/HABs_and_Hypoxia/rtMonSQL.php
![Page 34: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/34.jpg)
Features
● FAST
![Page 35: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/35.jpg)
Features
● FAST● Click location on map to
view by location○ Use the drop down too
![Page 36: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/36.jpg)
Features
● FAST● Click location on map to
view by location○ Use the drop down too
● OR Select handful of variables to view across all sampling locations
![Page 37: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/37.jpg)
Features
● FAST● Click location on map to
view by location○ Use the drop down too
● OR Select handful of variables to view across all sampling locations
● Pick central date and day range of interest, then click ‘Go!’ to refresh the page and view that time’s data
![Page 38: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/38.jpg)
![Page 39: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/39.jpg)
![Page 40: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/40.jpg)
![Page 41: PostgreSQL for Buoy Data](https://reader031.fdocuments.us/reader031/viewer/2022022522/621802043dcb9b45710c5219/html5/thumbnails/41.jpg)
jpthesmithe.com/iaglr.html