CUAHSI WaterML
description
Transcript of CUAHSI WaterML
![Page 1: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/1.jpg)
CUAHSI WaterML
Ilya Zaslavsky (SDSC),David Valentine (SDSC), Tim Whiteaker (UT-Austin)
/editors/
CUAHSI = Consortium of Universities for the Advancement of Hydrologic Sciences, Inc.;
![Page 2: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/2.jpg)
Background
• CUAHSI HIS: NSF-supported collaborative project: UT Austin + SDSC + Drexel + Duke + Utah State (www.cuahsi.org/his/). PI: David R. Maidment (UT-Austin)
• A cyberinfrastructure project• Current focus: providing uniform access to
heterogeneous observations data, from different agencies• Plus an easy way to publish hydrologic observations data• And to assemble comprehensive observations databases for your
watershed, catchment, etc.
• From ad hoc SOAP wrappers for agency data to a consistent schema, agency buy-in, and support for multiple clients => WaterML
![Page 3: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/3.jpg)
WaterOneFlow Web Services
Data access through web
services
Data storage through web
services
Dow
nlo
ads
Upl
oa
ds
Observatory servers
Workgroup HIS
SDSC HIS servers
3rd party servers
e.g. USGS, NCDC
GIS
Matlab
IDL
Splus, R
D2K, I2K
Programming (Fortran, C, VB)
Web services interface
Web portal Interface (HDAS)
Information input, display, query and output services
Preliminary data exploration and discovery. See what is available and perform exploratory analyses
HTML -XML WS
DL
- SO
AP
Hydrologic Information System Service Oriented Architecture
![Page 4: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/4.jpg)
SupercomputerCenters:NCSA,TACC
Domain Sciences:
Unidata, NCARLTER, GEON
Government:USGS, EPA,
NCDC, USDA
Industry:ESRI, Kisters,
OpenMI
HISTeam
WATERSTestbed
WATERS Network Information System
CUAHSI HIS
The CUAHSI Community, HIS and WATERS
CUAHSI: 116 Universities (Nov. 2006)
HIS Team:Texas, SDSC,Utah, Drexel,
Duke
![Page 5: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/5.jpg)
CUAHSI HIS as a mediator across multiple agency and PI data
• Keeps identifiers for sites, variables, etc. across observation networks
• Manages and publishes controlled vocabularies (in the Annex to the WaterML paper), and provides vocabulary/ontology management and update tools
• Provides common structural definitions for data interchange
• Provides a sample protocol implementation• Governance framework: a consortium of universities,
MOUs with federal agencies, collaboration with key commercial partners, led by renowned hydrologists, and NSF support for core development and test beds
![Page 6: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/6.jpg)
WaterML design principles
• Driven largely by hydrologists; the goal is to capture semantics of hydrologic observations discovery and retrieval
• Relies to a large extent on the information model as in ODM (Observations Data Model), and terms are aligned as much as possible• Several community reviews since 2005
• Driven by data served by USGS NWIS, EPA STORET, multiple individual PI-collected observations
• Is no more than an exchange schema for CUAHSI web services
• The least barrier for adoption by hydrologists• A fairly simple and rigid schema tuned to the current
implementation• Conformance with OGC specs not in the initial scope
![Page 7: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/7.jpg)
Point Observations Information Model
• A data source operates an observation network• A network is a set of observation sites• A site is a point location where one or more variables are measured• A variable is a property describing the flow or quality of water• An observation series is an array of observations at a given site, for a given variable, with start time and end time• A value is an observation of a variable at a particular time• A qualifier is a symbol that provides additional information about the value
Data Source
Network
Sites
ObservationSeries
Values
{Value, Time, Qualifier}
USGS
Streamflow gages
Neuse River near Clayton, NC
Discharge, stage, start, end (Daily or instantaneous)
206 cfs, 13 August 2006
Return network information, and variable information within the network
Return site information, including a series catalog of variables measured at a site with their periods of record
Return time series of values
![Page 8: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/8.jpg)
Observations Data Model Schema (version 4.0)
Data Source and Network Sites Variables Values Metadata
Depth of snow pack
Streamflow
Landuse, Vegetation
Windspeed, Precipitation
Controlled Vocabulary Tables
e.g. mg/kg, cfs
e.g. depth
e.g. Non-detect,Estimated,
A site is a point location where one or more variables are measured
A data source operates an observation network A network is a set of observation sites
Metadata provide information about the context of the observation.A variable is a property describing the flow or quality of water
A value is an observation of a variable at a particular time
From Ernest To, David Maidment, CRWR
![Page 9: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/9.jpg)
Challenges… (1/2)
• Sites• STORET has stations, and measurement points, at various offsets…• Site metadata lacking and inconsistent (e.g. 2/3 no HUC info, 1/3 no state/county info);
agency site files need to be upgraded to ODM…• A groundwater site is different than a stream gauge…
• Censored values• Values have qualifiers, such as “less than”, “censored”, etc. – per value. Sometimes
mixed data types.. • Units
• There are multiple renditions of the same units, even within one repository• There may be several units for the same parameter code (STORET)• If no value recorded – there are no units??• Unit multipliers
• E.g. NCDC ASOS keeps measurements as integers, and provides a multiplier for each variable
• Sources• STORET requires organization IDs (which collected data for STORET) in addition to site
IDs• Time stamps: ISO 8601
• A service to determine UTC offsets given lat/lon and date??
![Page 10: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/10.jpg)
Challenges… (2/2)
• Values retrieval• USGS: by site, variable, time range• EPA: by organization-site, variable, medium, units, time range• NCDC: fewer variables, period of record applies to site, not to
seriesCatalog• Variable semantics
• Variable names and measurement methods don’t match• E.g. NWIS parameter # 625 is labeled ‘ammonia + organic nitrogen‘,
Kjeldahl method is used for determination but not mentioned in parameter description. In STORET this parameter is referred to as Kjeldahl Nitrogen.
• One-to-one mapping not always possible• E.g. NWIS: ‘bed sediment’ and ‘suspended sediment’ medium types vs.
STORET’s ‘sediment’.
Ontology tagging, semantic mediationContolled vocabularies are in Annex A
![Page 11: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/11.jpg)
- From different database structures, data collection procedures, quality control, access mechanisms to uniform signatures … Water Markup Language- Tested in different environments- Standards-based- Can support advanced interfaces via harvested catalogs- Accessible to community- Templates for development of new services- Optimized, error handling, memory management, versioning, run from fast servers- Working with agencies on setting up services and updating site files
NWIS Daily Values (discharge), NWIS Ground Water, NWIS Unit Values (real time), NWIS Instantaneous Irregular Data, EPA STORET, NCDC ASOS, DAYMET, MODIS, NAM12K, ODM
![Page 12: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/12.jpg)
WaterML key elements
• Response Types
– SiteInfo
– Variables
– TimeSeries
• Key Elements– site– sourceInfo– seriesCatalog– variable– timeSeries
• values
– queryInfo
GetValues
GetVariableInfo
GetSiteInfo
![Page 13: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/13.jpg)
variables
variablesResponse
variable
1
many
timeSeriesqueryInfo
criteria
timeSeriesResponse
variable
sourceInfo
queryURL
values
site queryInfo
criteria
sitesResponse
seriesCatalog siteInfo
queryURL
variable
series
variableTimeInterval
1
many
Structure of responses
![Page 14: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/14.jpg)
DatasetInfoTypeSiteInfoType
Elements Defining Spatial Location
for observation sites for continuous surfaces
(other site information)
SourceInfoType
GeogLocationType
(other dataset information)
LatLonPointType
childelements
GeogLocationType
LatLonBoxType
LatLonPointType
![Page 15: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/15.jpg)
SiteInfoResponseType
• Namespaces
• queryInfo
• site
Network
Sites
Variables
![Page 16: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/16.jpg)
queryInfo example
userparameters
query URL
• Parameters sent to service
• URLs called (if external resource)
![Page 17: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/17.jpg)
siteInfo
• Name
• Site Code
• Location
![Page 18: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/18.jpg)
geoLocation
• geogLocation – geographic coordinates– LatLon point– LatLon box
• localSiteXY – projected coordinates
![Page 19: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/19.jpg)
series
• variable – what is measured• valueCount – how many measurements• variableTimeInterval – when is it measured
TimePeriodType
![Page 20: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/20.jpg)
variable
• variableCode – global identifier
• variableName
• units
Sites
Variables
Values
TimePeriodType
![Page 21: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/21.jpg)
variableTimeInterval• TimePeriodType – date range (including
“last n days”
• TimeInstantType – single measurement
Compare with…
![Page 22: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/22.jpg)
SiteInfo responsequeryInfo
site
name
code
location
seriesCatalog
variables
what
how many
when
TimePeriodType
![Page 23: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/23.jpg)
VariablesResponseType
• variable – same as in series element
• Code, name, units Sites
Variables
Values
![Page 24: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/24.jpg)
TimeSeriesResponseType
• queryInfo
• timeSeries– sourceInfo – “where”– variable – “what”– values
Sites
Variables
Values
![Page 25: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/25.jpg)
sourceInfo• SiteInfoType
– Same as siteInfo element– code, name, location
• DataSetInfoType– For data continuous in space– LatLonPointType– LatLonBoxType
![Page 26: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/26.jpg)
values
• Each time series value recorded in value element
• Timestamp, plus metadata for the value, recorded in element’s attributes
ISO Time
valuequalifier
Compare with…
![Page 27: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/27.jpg)
value metadata examples
• qualifiers• censorCode (lt, gt, nc)• qualityControlLevel (Raw, QC’d, etc.)• methodID• offset
– offsetValue– offsetUnitsAbbreviation– offsetDescription– offsetUnitsCode
![Page 28: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/28.jpg)
TimeSeries responsequeryInfo
location
variable
values
![Page 29: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/29.jpg)
Clients
• Tested with .Net and Java• Desktop clients:
Excel, Matlab, ArcGIS, VB.NET,more beingwritten
• Web client: DASH (Data Access System for Hydrology): http://river.sdsc.edu/DASH (beta)
![Page 30: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/30.jpg)
DA
SH
AGS Server
IIS
Windows 2003 Server4 GB Ram1 TB Disk
Quad Core CPU
SQLServer
VS 2
005
WaterOneFlow Web Services
ArcGIS 9.2
GIS Data Mxd Service
OD
M L
oad
er
OD
M t
ools
OD
M
Current Deployment Current Deployment ArchitectureArchitecture
Direct DB connection
![Page 31: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/31.jpg)
SQL Server
ODMs and catalogs. All instancesexposed as ODM (i.e. have standard ODM tables or views: Sites, Variables, SeriesCatalog, etc.)
NWIS-IID
NWIS-DV
ASOS
STORET
TCEQ
BearRiver
. . .
Spatial store
Geodatabase or collection of shapefilesor both
NWIS-IID points
NWIS-DV points
ASOS points
STORET points
TCEQ points
BearRiver points
. . .
My new ODM
My new points
More databases
More synced layers
DASH Web Application
Background layers
(can be in the same or separate spatial store)
WOF services
Web services from a common template
NWIS-IID WS
NWIS-DV WS
ASOS WS
STORET WS
TCEQ WS
BearRiver WS
. . .My new WS
More WS fromODM-WS template
USGS
NCDC
EPA
TCEQ
Web Configuration fileStores information about registered networks
MXDStores information about layers
WSDLs
, web
serv
ice U
RLs Connection
strings
Layer info,
symbology, etc.
ODMDataLoader
2
6
5
3
1
4
WORKGROUPHISSERVERORGANIZATION
STEPS FORREGISTERINGOBSERVATIO
NDATA
![Page 32: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/32.jpg)
New network registration steps
Using the ODM DataLoader, load your data into a blank ODM instance (this will create all ODM tables, including Sites, Variables and SeriesCatalog that HIS application relies on)
Copy Web Services template to a new folder, edit the template web.config file to point to the new ODM, test to make sure the new service works as expected
Create a point layer (a feature class in GDB, or a shapefile) from the new ODM’s Sites table or from GetSites web service (using GetSitesTool – this will also test the service)
Add the point layer to the MXD document, specify symbology, scale-dependent rendering, etc.
Add information about the new ODM, the associated web service, and the associated point layer, to HIS configuration file (see the first slide for the exact content)
Restart the HIS service6
5
4
3
2
1
![Page 33: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/33.jpg)
Near future
• Need further reviewed, based on initial implementation• Within and beyond OGC membership• Further engage with hydrologic observation groups at agencies• Internationalization (with CSIRO WRON, European WISE, others?)
• CUAHSI O&M profile?• Carry CUAHSI WaterML messages over O&M - need to talk with
Simon about it… • Test it within an initiative (e.g. Oceans IE, or Water Data
Interoperability Testbed, OWS-5? )• Have a list of suggestions for versions after 1.0 (including linear
referencing, GetCapabilities, WFS integration, handling multiple siteCodes and variableCodes in a single call) – more suggestions are welcome
• Divorcing from implementation? Looking for golden middle in standard complexity…
![Page 34: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/34.jpg)
OGC Harmonization Best Practices
• WaterML text includes steps for harmonizing with GML/O&M
• Align spatial feature descriptions (e.g. using gml:Point, gml:Envelope)
• Align service signatures (getCapabilities)• Align terminology with O&M
• Provides guidance to other communities on harmonization steps
![Page 35: CUAHSI WaterML](https://reader035.fdocuments.us/reader035/viewer/2022062808/56815235550346895dc07a44/html5/thumbnails/35.jpg)
Motion
• EO/NRE WG recommends to the TC that the CUAHSI WaterML (document 07-041) be released as an OGC Discussion paper• Pending minor editorial changes• Proposed• Second: Ben Domenico• Unanimous
• Roadmap:• Reporting initial implementation experience• CUAHSI WaterML 1.1, aligning with O&M• Testing within an OGC initiative