The Shipboard Automated Meteorological and Oceanographic System (SAMOS) Initiative Data Management...
1
The Shipboard Automated Meteorological and Oceanographic System (SAMOS) Initiative Data Management System http://rvdata.us/ http:// samos.coaps.fsu.edu/ Shawn R. Smith 1 , Robert A. Arko 3 , Mark A. Bourassa 1,2 , Jiangyi Hu 1 , Michael McDonald 1 , Jacob Rettig 1 , and Jeremy Rolph 1 Poster 3-25 IMDIS 2010 29-31 March 2010 Ship Ema il SAMOS Process ing Analys t Feedba ck Satellit e Broadban d (e.g., HiSeasNe t) Chec k & Veri fy Quali ty Contr ol ASCII to NetCD F Prelimin ary Data Posted Problem OK Analyst Notific ation Cron/ Manual Trigger Merg e Data File SASSI: Statist ical Quality Control Analyst Notific ation Visua l Quali ty Contr ol Save & Post Monthl y Data NODC Archi ve Inter net RSYNC Overall, the data management system is a series of automated processes coupled to an SQL database. The database stores vessel metadata profiles, ship-specific processing parameters, file tracking and version control tags, and data quality flags. SAMOS Data Management System SAMOS Data Management System Currently, recruited SAMOS vessels use an email protocol to transmit 1-min. averaged data to the DAC. Transmissions nominally sent at 0000UTC and include all observations for the previous day. Email receipt by the DAC triggers automated SAMOS processing (whitelist controlled). After verifying source and format of incoming message, data are converted to network common data form and merged with vessel specific metadata from database. Automated QC checks data for valid ranges, ship speed, and location; agreement with a climatology; temporal sequence, and physical consistency. Preliminary files posted via web, ftp, and THREDDS. Analyst notifies operators at sea when problems are detected. On a 10 day delay, all files for a single ship and observation day are combined (allowing for late file receipt). Merge uses preliminary QC to remove temporal duplicates. Secondary automated QC locates spikes, steps, and highly variable observations, resulting in an intermediate product. NOAA supports visual QC for select vessels. Trained data quality analyst uses a graphical user interface to add, modify, or remove QC flags. Resulting research quality files are posted. The final stage of the SAMOS DMS is to submit the data and metadata to the National Oceanographic Data Center. On a monthly basis, all original, preliminary, intermediate, and research-quality SAMOS data files are uploaded to NODC. NODC develops FGDC metadata records and provides long-term access and stewardship. SAMOS-R2R Partnership SAMOS-R2R Partnership In 2009, the SAMOS DAC forged a partnership with the R2R to expand SAMOS data stewardship to all vessels in the U. S. academic fleet. Objectives 1.Initiate transfers of relevant parameters (Table 1) from all U. S. academic fleet vessels 2.Increase data transfer frequency to meet needs of operational weather forecasting community (at least every 6 hours) 3.Conduct fully automated quality control (QC) 4.Provide operators with QC feedback and recommendations for sensor deployment 5.Develop automated protocol for ship- to-shore metadata transfers Standards The SAMOS DAC will move to adopt R2R vocabularies for vessel names, ports, etc. The DAC continues to move towards the Climate and Forecast (CF) network common data form Plan to utilize R2R cruise inventory and cross populate vessel profiles Proposed Data Transfer Mechanism Two options: 1.Large operators with sufficient technical staff will continue to use current SAMOS data transfer protocol. 2.SAMOS-R2R exploring development of software/hardware solution for other vessels SAMOS Objective: SAMOS Objective: To collect, quality evaluate, distribute, and ensure future access (via national archives) to underway meteorological and near-surface ocean data collected on research vessels. SAMOS Data SAMOS Data The SAMOS data assembly center (DAC) receives voluntary contributions from 26 research vessels (as of March 2010; Fig. 1). Observations are from automated instrument systems operated and maintained by the research vessel’s home institution. Parameters collected (Table 1) vary from vessel to vessel, but must include observation time, position, and units. SAMOS observations are one-minute average values derived from higher frequency (~1 Hz) instrument samples. Fig. 1: SAMOS ship tracks for FY2008 and FY2009 Fig. 1: SAMOS ship tracks for FY2008 and FY2009 color coded by operator. The Australians provide color coded by operator. The Australians provide the only international data via the IMOS project. the only international data via the IMOS project. 1 Center for Ocean-Atmospheric Prediction Studies, Florida State University 2 Department of Meteorology, Florida State University 3 Lamont-Doherty Earth Observatory of Columbia University The Rolling Deck to Repository (R2R) project was recently launched with the ambitious goal of documenting “routine underway data” from the U. S. academic research fleet and delivering those data to established national archives. Data distributions will be submitted by 18 operating institutions for 30 vessels. SAMOS/R2R Partnership In 2003, NOAA established a data assembly center (DAC) at the Florida State University to provide data stewardship for underway meteorological and near- surface oceanographic data collected by research vessels. A partnership has been established between the R2R project and the DAC to extend SAMOS data stewardship to the U. S. academic fleet. The SAMOS Data Center The DAC has developed and implemented an automated data management system (DMS) that collects, formats, quality controls, distributes, and archives near real-time surface marine data from research vessels. A SAMOS is a computerized data logging system that continuously records navigational (ship’s position, course, speed, and heading), meteorological (winds, air temperature, pressure, moisture, rainfall, and radiation), and near- surface oceanographic (sea temperature, salinity, conductivity, fluorescence) parameters while the vessel is at sea. The SAMOS initiative relies on the high-quality instrumentation purchased and deployed by the research vessel operators and does not provide instrumentation to the vessels. Currently, 26 research vessels are providing routine SAMOS observations to the DAC. The Data Management System Presently, SAMOS data are acquired directly from research vessels at sea via a daily email transfer protocol. The DMS automatically tracks progress of the daily data acquisition and quality control (QC), stores metadata on instrumentation and ships, and provides data monitoring capability via a user-friendly web interface. An SQL database stores essential parameters to support tracking, data QC, and version control throughout the process. R2R Real-time Data Protocol The DAC is developing new protocols to transfer SAMOS data from the academic fleet to shore. Ideally, the protocol will either allow operators to transmit high- frequency (~1 Hz) SAMOS data from participating vessels to real-time servers at R2R or the DAC to develop Preliminary processing is fully automated. Fig. 3: Fig. 3: Original MS Original MS Word Word (above) and (above) and interactive interactive web-based web-based (right) (right) SAMOS SAMOS instrument instrument metadata metadata forms. forms. Fig. 2: Examples of digital Fig. 2: Examples of digital imagery from select SAMOS vessels. imagery from select SAMOS vessels. (far left) metadata collage from (far left) metadata collage from NOAA’s NOAA’s Okeanos Explorer Okeanos Explorer ; (left) ; (left) Frozen anemometer on USCG Frozen anemometer on USCG Healy Healy ; ; (right) instrument mast from the (right) instrument mast from the L. L. M. Gould M. Gould . . Metadata Challenge Metadata Challenge The SAMOS DAC endeavors to collect extensive ship and instrument metadata, including digital imagery (Fig. 2), to meet the scientific goals of our user community. Initially, metadata was requested using forms that could be transmitted via email (Fig. 3). Although many operators fill out forms initially, tedious entry into our database was required and updates were difficult to obtain. New web-based forms (Fig. 3) were developed to allow operators to directly enter and update their metadata in our database. Web forms eased data entry, but are mostly used by DAC staff. In partnership with the R2R, the DAC will move to develop a protocol to automatically transfer necessary metadata along with the physical observations from ship to shore. The Rolling Deck to Repository Project acknowledges support from the National Science Foundation, Oceanographic Instrumentation and Technical Services (OITS) Program. Base support for the SAMOS data center is provided by NOAA’s Office of Climate Observation.
-
Upload
destiny-powell -
Category
Documents
-
view
217 -
download
0
Transcript of The Shipboard Automated Meteorological and Oceanographic System (SAMOS) Initiative Data Management...
The Shipboard Automated Meteorological and Oceanographic System (SAMOS) Initiative Data Management System
The Shipboard Automated Meteorological and Oceanographic System (SAMOS) Initiative Data Management System
http://rvdata.us/http://samos.coaps.fsu.edu/
Shawn R. Smith1, Robert A. Arko3, Mark A. Bourassa1,2, Jiangyi Hu1, Michael McDonald1, Jacob Rettig1, and Jeremy Rolph1
Poster 3-25IMDIS 2010
29-31 March 2010
ShipShipEmailEmail
SAMOSProcessing
SAMOSProcessing
AnalystFeedbackAnalyst
FeedbackSatellite Broadband
(e.g., HiSeasNet)
Satellite Broadband
(e.g., HiSeasNet)
Check &
Verify
Check &
Verify
QualityControlQualityControl
ASCII toNetCDFASCII toNetCDF
PreliminaryData PostedPreliminaryData Posted
Problem
OK AnalystNotification
AnalystNotification
Cron/ManualTrigger
Cron/ManualTrigger
MergeDataFile
MergeDataFile
SASSI:Statistical
Quality Control
SASSI:Statistical
Quality Control
AnalystNotification
AnalystNotification
Visual Quality Control
Visual Quality Control
Save &Post
Save &Post
MonthlyData
MonthlyData
NODCArchiveNODC
ArchiveInternetRSYNC
InternetRSYNC
Overall, the data management system is a series of automated processes coupled to an SQL database. The database stores vessel metadata profiles, ship-specific processing parameters, file tracking and version control
tags, and data quality flags.
Overall, the data management system is a series of automated processes coupled to an SQL database. The database stores vessel metadata profiles, ship-specific processing parameters, file tracking and version control
tags, and data quality flags.
SAMOS Data Management SystemSAMOS Data Management SystemSAMOS Data Management SystemSAMOS Data Management System
Currently, recruited SAMOS vessels use an email protocol to transmit 1-min. averaged data to the DAC.
Transmissions nominally sent at 0000UTC and include all observations for the previous day.
Email receipt by the DAC triggers automated SAMOS processing (whitelist controlled).
Currently, recruited SAMOS vessels use an email protocol to transmit 1-min. averaged data to the DAC.
Transmissions nominally sent at 0000UTC and include all observations for the previous day.
Email receipt by the DAC triggers automated SAMOS processing (whitelist controlled).
After verifying source and format of incoming message, data are converted to network common data form and merged with vessel specific metadata from database.
Automated QC checks data for valid ranges, ship speed, and location; agreement with a climatology;
temporal sequence, and physical consistency.
Preliminary files posted via web, ftp, and THREDDS.
Analyst notifies operators at sea when problems are detected.
After verifying source and format of incoming message, data are converted to network common data form and merged with vessel specific metadata from database.
Automated QC checks data for valid ranges, ship speed, and location; agreement with a climatology;
temporal sequence, and physical consistency.
Preliminary files posted via web, ftp, and THREDDS.
Analyst notifies operators at sea when problems are detected.
On a 10 day delay, all files for a single ship and observation day are combined (allowing for late file receipt).
Merge uses preliminary QC to remove temporal duplicates.
Secondary automated QC locates spikes, steps, and highly variable observations, resulting in an
intermediate product.
On a 10 day delay, all files for a single ship and observation day are combined (allowing for late file receipt).
Merge uses preliminary QC to remove temporal duplicates.
Secondary automated QC locates spikes, steps, and highly variable observations, resulting in an
intermediate product.
NOAA supports visual QC for select vessels.
Trained data quality analyst uses a graphical user interface to add, modify, or remove QC flags.
Resulting research quality files are posted.
NOAA supports visual QC for select vessels.
Trained data quality analyst uses a graphical user interface to add, modify, or remove QC flags.
Resulting research quality files are posted.
The final stage of the SAMOS DMS is to submit the data and metadata to the National
Oceanographic Data Center.
On a monthly basis, all original, preliminary, intermediate, and research-quality SAMOS data
files are uploaded to NODC.
NODC develops FGDC metadata records and provides long-term access and stewardship.
The final stage of the SAMOS DMS is to submit the data and metadata to the National
Oceanographic Data Center.
On a monthly basis, all original, preliminary, intermediate, and research-quality SAMOS data
files are uploaded to NODC.
NODC develops FGDC metadata records and provides long-term access and stewardship.
SAMOS-R2R PartnershipSAMOS-R2R Partnership
In 2009, the SAMOS DAC forged a partnership with the R2R to expand SAMOS data stewardship to all
vessels in the U. S. academic fleet.
Objectives1.Initiate transfers of relevant parameters (Table 1) from all U. S. academic fleet vessels2.Increase data transfer frequency to meet needs of operational weather forecasting community (at least every 6 hours)3.Conduct fully automated quality control (QC)4.Provide operators with QC feedback and recommendations for sensor deployment5.Develop automated protocol for ship-to-shore metadata transfers
StandardsThe SAMOS DAC will move to adopt R2R vocabularies for vessel names, ports, etc.
The DAC continues to move towards the Climate and Forecast (CF) network common data form
Plan to utilize R2R cruise inventory and cross populate vessel profiles
Proposed Data Transfer MechanismTwo options:
1.Large operators with sufficient technical staff will continue to use current SAMOS data transfer protocol.2.SAMOS-R2R exploring development of software/hardware solution for other vessels
• This protocol will support smaller operators• Recommended by UNOLS Research Vessel
Technical Enhancement Committee• Wecoma and Endeavor are test vessels• Input from technical community welcome.
SAMOS-R2R PartnershipSAMOS-R2R Partnership
In 2009, the SAMOS DAC forged a partnership with the R2R to expand SAMOS data stewardship to all
vessels in the U. S. academic fleet.
Objectives1.Initiate transfers of relevant parameters (Table 1) from all U. S. academic fleet vessels2.Increase data transfer frequency to meet needs of operational weather forecasting community (at least every 6 hours)3.Conduct fully automated quality control (QC)4.Provide operators with QC feedback and recommendations for sensor deployment5.Develop automated protocol for ship-to-shore metadata transfers
StandardsThe SAMOS DAC will move to adopt R2R vocabularies for vessel names, ports, etc.
The DAC continues to move towards the Climate and Forecast (CF) network common data form
Plan to utilize R2R cruise inventory and cross populate vessel profiles
Proposed Data Transfer MechanismTwo options:
1.Large operators with sufficient technical staff will continue to use current SAMOS data transfer protocol.2.SAMOS-R2R exploring development of software/hardware solution for other vessels
• This protocol will support smaller operators• Recommended by UNOLS Research Vessel
Technical Enhancement Committee• Wecoma and Endeavor are test vessels• Input from technical community welcome.
SAMOS Objective: SAMOS Objective: To collect, quality evaluate, distribute, and ensure future access (via national archives) to underway meteorological and near-surface ocean data collected on research vessels.
SAMOS Objective: SAMOS Objective: To collect, quality evaluate, distribute, and ensure future access (via national archives) to underway meteorological and near-surface ocean data collected on research vessels.
SAMOS DataSAMOS Data
The SAMOS data assembly center (DAC) receives voluntary contributions from 26 research vessels
(as of March 2010; Fig. 1).
Observations are from automated instrument systems operated and maintained by the research vessel’s
home institution.
Parameters collected (Table 1) vary from vessel to vessel, but must include observation time,
position, and units.
SAMOS observations are one-minute average values derived from higher frequency (~1 Hz) instrument
samples.
SAMOS DataSAMOS Data
The SAMOS data assembly center (DAC) receives voluntary contributions from 26 research vessels
(as of March 2010; Fig. 1).
Observations are from automated instrument systems operated and maintained by the research vessel’s
home institution.
Parameters collected (Table 1) vary from vessel to vessel, but must include observation time,
position, and units.
SAMOS observations are one-minute average values derived from higher frequency (~1 Hz) instrument
samples.
Fig. 1: SAMOS ship tracks for FY2008 and FY2009 color coded by operator. Fig. 1: SAMOS ship tracks for FY2008 and FY2009 color coded by operator. The Australians provide the only international data via the IMOS project. The Australians provide the only international data via the IMOS project. Fig. 1: SAMOS ship tracks for FY2008 and FY2009 color coded by operator. Fig. 1: SAMOS ship tracks for FY2008 and FY2009 color coded by operator. The Australians provide the only international data via the IMOS project. The Australians provide the only international data via the IMOS project.
1 Center for Ocean-Atmospheric Prediction Studies, Florida State University2 Department of Meteorology, Florida State University3 Lamont-Doherty Earth Observatory of Columbia University
1 Center for Ocean-Atmospheric Prediction Studies, Florida State University2 Department of Meteorology, Florida State University3 Lamont-Doherty Earth Observatory of Columbia University
The Rolling Deck to Repository (R2R) project was recently launched with the ambitious goal of documenting “routine underway data” from the U. S. academic research fleet and delivering those data to established national archives. Data distributions will be submitted by 18 operating institutions for 30 vessels.
SAMOS/R2R PartnershipIn 2003, NOAA established a data assembly center (DAC) at the Florida State University to provide data stewardship for underway meteorological and near-surface oceanographic data collected by research vessels. A partnership has been established between the R2R project and the DAC to extend SAMOS data stewardship to the U. S. academic fleet.
The SAMOS Data CenterThe DAC has developed and implemented an automated data management system (DMS) that collects, formats, quality controls, distributes, and archives near real-time surface marine data from research vessels. A SAMOS is a computerized data logging system that continuously records navigational (ship’s position, course, speed, and heading), meteorological (winds, air temperature, pressure, moisture, rainfall, and radiation), and near-surface oceanographic (sea temperature, salinity, conductivity, fluorescence) parameters while the vessel is at sea. The SAMOS initiative relies on the high-quality instrumentation purchased and deployed by the research vessel operators and does not provide instrumentation to the vessels. Currently, 26 research vessels are providing routine SAMOS observations to the DAC.
The Data Management SystemPresently, SAMOS data are acquired directly from research vessels at sea via a daily email transfer protocol. The DMS automatically tracks progress of the daily data acquisition and quality control (QC), stores metadata on instrumentation and ships, and provides data monitoring capability via a user-friendly web interface. An SQL database stores essential parameters to support tracking, data QC, and version control throughout the process.
R2R Real-time Data ProtocolThe DAC is developing new protocols to transfer SAMOS data from the academic fleet to shore. Ideally, the protocol will either allow operators to transmit high- frequency (~1 Hz) SAMOS data from participating vessels to real-time servers at R2R or the DAC to develop software/hardware that can be installed on each vessel. Under either protocol, the SAMOS DAC would support data reduction (averaging), shore-side monitoring, quality control, metadata acquisition, data distribution, and archival at a national data center.
Preliminary processing is fully automated.Preliminary processing is fully automated.
Fig. 3: Original Fig. 3: Original MS Word MS Word
(above) and (above) and interactive web-interactive web-
based (right) based (right) SAMOS SAMOS
instrument instrument metadata forms.metadata forms.
Fig. 3: Original Fig. 3: Original MS Word MS Word
(above) and (above) and interactive web-interactive web-
based (right) based (right) SAMOS SAMOS
instrument instrument metadata forms.metadata forms.
Fig. 2: Examples of digital imagery from select Fig. 2: Examples of digital imagery from select SAMOS vessels. (far left) metadata collage from SAMOS vessels. (far left) metadata collage from NOAA’s NOAA’s Okeanos ExplorerOkeanos Explorer; (left) Frozen ; (left) Frozen anemometer on USCG anemometer on USCG HealyHealy; (right) instrument ; (right) instrument mast from the mast from the L. M. GouldL. M. Gould..
Fig. 2: Examples of digital imagery from select Fig. 2: Examples of digital imagery from select SAMOS vessels. (far left) metadata collage from SAMOS vessels. (far left) metadata collage from NOAA’s NOAA’s Okeanos ExplorerOkeanos Explorer; (left) Frozen ; (left) Frozen anemometer on USCG anemometer on USCG HealyHealy; (right) instrument ; (right) instrument mast from the mast from the L. M. GouldL. M. Gould..
Metadata ChallengeMetadata Challenge
The SAMOS DAC endeavors to collect extensive ship and instrument metadata,
including digital imagery (Fig. 2), to meet the scientific goals of our user community.
Initially, metadata was requested using forms that could be transmitted via email (Fig. 3).
Although many operators fill out forms initially, tedious entry into our database was required
and updates were difficult to obtain.
New web-based forms (Fig. 3) were developed to allow operators to directly enter and update
their metadata in our database.
Web forms eased data entry, but are mostly used by DAC staff.
In partnership with the R2R, the DAC will move to develop a protocol to automatically transfer necessary metadata along with the physical observations from ship to shore.
Metadata ChallengeMetadata Challenge
The SAMOS DAC endeavors to collect extensive ship and instrument metadata,
including digital imagery (Fig. 2), to meet the scientific goals of our user community.
Initially, metadata was requested using forms that could be transmitted via email (Fig. 3).
Although many operators fill out forms initially, tedious entry into our database was required
and updates were difficult to obtain.
New web-based forms (Fig. 3) were developed to allow operators to directly enter and update
their metadata in our database.
Web forms eased data entry, but are mostly used by DAC staff.
In partnership with the R2R, the DAC will move to develop a protocol to automatically transfer necessary metadata along with the physical observations from ship to shore.
The Rolling Deck to Repository Project acknowledges support from the National Science Foundation, Oceanographic Instrumentation and Technical Services (OITS) Program. Base support for the SAMOS data center is provided by NOAA’s Office of Climate Observation.
