CUAHSI WaterOneFlowWeb Services

By Tim Whiteaker

CE 394K.2 Hydrology

1 February 2007

Outline

• What are Web Services

• What is WaterOneFlow

• What is WaterML

• How do we use HIS Analyst to access WaterOneFlow

Programs Use Functions

• Functions are pieces of code that perform a specific task

Program

Mathlibrary

Add

Multiply

SquareRoot

Functions:

What’s the square root

of 16?

16

4

result = Math.SquareRoot(16)

Web Services Let Computers Share Functions

Program

Mathlibrary

Add

Multiply

SquareRoot

Functions:What’s the square root of 16

The answer is 4

• Local programs take advantage of remote resources

• Works just like local function

result = Service.SquareRoot(16)

ServiceClient

How Do Web Services Work

• SOAP (Simple Object Access Protocol) is a protocol for exchanging messages over a network

• WSDL (Web Services Description Language) is a language for describing what a web service can do

How Do Web Services Work

Client

Server

SOAP WSDL(This is what I

can do)

SOAP is like speaking the same languageWSDL is like a contract

CUAHSI Web Services (WaterOneFLow)

• Standard mechanism for flow of hydrologic data between hydrologic data servers (databases) and users.

• Provides data access to– USGS NWIS– EPA STORET– NCEP North American Model (NAM)– NASA MODIS– Daymet– NWS ASOS

http://water.sdsc.edu/waterOneFlow/

Water Data Web Sites

NWISWeb site output# agency_cd Agency Code# site_no USGS station number# dv_dt date of daily mean streamflow# dv_va daily mean streamflow value, in cubic-feet per-second# dv_cd daily mean streamflow value qualification code## Sites in this file include:# USGS 02087500 NEUSE RIVER NEAR CLAYTON, NC#agency_cd site_no dv_dt dv_va dv_cdUSGS 02087500 2003-09-01 1190USGS 02087500 2003-09-02 649USGS 02087500 2003-09-03 525USGS 02087500 2003-09-04 486USGS 02087500 2003-09-05 733USGS 02087500 2003-09-06 585USGS 02087500 2003-09-07 485USGS 02087500 2003-09-08 463USGS 02087500 2003-09-09 673USGS 02087500 2003-09-10 517USGS 02087500 2003-09-11 454

Time series of streamflow at a gaging station

Observation Stations

Ameriflux Towers (NASA & DOE) NOAA Automated Surface Observing System

USGS National Water Information System NOAA Climate Reference Network

Map for the US

WaterOneFlow• Consistent Query Format • Consistent Data Format

WaterOneFlow Methods

• (Almost) the same for all networks

• Methods– GetSiteInfo – Give me info about a site– GetVariableInfo – Give me info about a

variable– GetValues – Give me a time series for a

variable at a site

GetSiteInfo

• Input– Network name and

site code– Authorization token

• Output– Site name– Site location– Variables measured at

the site• Variable code• Period of record

response = service.GetSiteInfo("NWIS:08158000", "")

where

GetVariableInfo

• Input– Network name and

variable code– Authorization token

• Output– Variable name– Variable description– Variable units

response = service.GetVariableInfo("NWIS:00060", "")

what

GetValues

• Input– Network name and

location (e.g., site code)

– Network name and variable code

– Start datetime– End datetime– Authorization token

• Output– Site info– Variable info– Time series

response = service.GetValues("NWIS:08158000", "NWIS:00060", _ "2006-12-31T01:00:00", "2006-12-31T08:00:00", "")

where what

when

The “Where” in GetValues

• Works for sites, or geographic locations• NetworkName:SiteCode

– NWIS:08158000

• GEOM:POINT(Longitude Latitude)– GEOM:POINT(-113 35)

• GEOM:BOX(WLon SLat,ELon NLat)– GEOM:BOX(-108 45,-107 46)

Locations

Variable Codes

Date Ranges

Extract – Transform – Load

GetSiteInfoGetVariableInfoGetValues

WaterOneFlowWeb Service

Client

STORET

NAMNWIS

DataRepositories

Data

DataData

EXTRACTTRANSFORMLOAD

WaterML

WaterML – The Output of WaterOneFlow

• Standardized for all data sources

• Formats– Object– XML (for programs that can’t handle objects)

• Defined in XML

XML – A Primer

• The Extensible Markup Language (XML) is a W3C-recommended general-purpose markup language that supports a wide variety of applications. – Wikipedia

• XML represents data• XML is both human and machine readable

<site> <name>Mansfield Dam</name></site>

XML Structure

<site></site>

<site></site>

<site />

OR

OR

element

start tag end tag

• Example – Let’s describe a streamflow site

element name = “site”

Elements can have Children

<site> <name>Mansfield Dam</name></site>

element

child element

element text

Elements can have Attributes

<site> <name>Mansfield Dam</name> <siteCode network=“NWIS”>08154510</siteCode></site>

attribute name attribute value

XML Nesting

<site> <name>Mansfield Dam</name> <siteCode network=“NWIS”>08154510</siteCode> <location> <latitude>30.39</latitude> <longitude>97.91</longitude> </location></site>

WaterML

• XML for describing water data

• Defined at

• WaterOneFlow returns data in WaterML format

http://water.sdsc.edu/waterOneFlow/documentation/schema/

Point Observations Information Model

Data Source

Network

{Value, Time, Qualifier}

USGS

Streamflow gages

Neuse River near Clayton, NC

Discharge, stage (Daily or instantaneous)

206 cfs, 13 August 2006

• A data source operates an observation network• A network is a set of observation sites• A site is a point location where one or more variables are measured• A variable is a property describing the flow or quality of water• A value is an observation of a variable at a particular time• A qualifier is a symbol that provides additional information about the value

Sites

Variables

Values

http://www.cuahsi.org/his/webservices.html

WaterML Response

SiteInfoResponseType

VariablesResponseType

TimeSeriesResponseType

WaterML Key Elements

• Response Types

– SiteInfo

– Variables

– TimeSeries

• Key Elements– site– sourceInfo– seriesCatalog– variable– timeSeries

• values

– queryInfo

GetValues

GetVariableInfo

GetSiteInfo

SiteInfoResponseType

• Namespaces

• queryInfo

• site

Network

Sites

Variables

queryInfo

• Parameters sent to service

• URLs called (if external resource)

queryInfo – Another Example

userparameters

query URL

site

• siteInfo – info about the site

• seriesCatalog – catalogs of variables measured

siteInfo

• Name

• Site Code

• Location

geoLocation

• geogLocation – geographic coordinates– LatLon point– LatLon box

• localSiteXY – projected coorindates

European Petroleum Survey Groupi.e., NAD 83, etc.

seriesCatalog

name web service location

original data sourceseries (variables)

series

• variable – what is measured• valueCount – how many measurements• variableTimeInterval – when is it measured

variable

• variableCode – global identifier

• variableName

• units

Sites

Variables

Values

variableTimeInterval

• TimeIntervalType – date range

• TimeSingleType – single measurement

• TimePeriodRealTimeType – last n days

ISO Time

• International Organization for Standardization• 1998-03-01T14:30:00 =

March 1, 1998, at 2:30 in the afternoon

SiteInfo Response, RecapqueryInfo

site

name

code

location

seriesCatalog

variables

what

how many

when

VariablesResponseType

• variable – same as in series element

• Code, name, units Sites

Variables

Values

Additional variable Children

• variableDescription

• valueType (enumeration)– Field Observation– Sample– Model Simulation Result– Derived Value

• generalCategory – e.g., “water quality”

• sampleMedium – e.g., “water column”

-- Many are optional

TimeSeriesResponseType

• queryInfo

• timeSeries– sourceInfo – “where”– variable – “what”– values

Sites

Variables

Values

sourceInfo• SiteInfoType

– Same as siteInfo element– code, name, location

• DataSetInfoType– For data continuous in space– LatLonPointType– LatLonBoxType

variable

• Same as previous variable element

• code, name, units, etc.

values

• Each time series value recorded in value element

• Timestamp, plus metadata for the value, recorded in element’s attributes

ISO Time

valuequalifier

value Metadata Examples

• qualifiers• censorCode (lt, gt, nc)• qualityControlLevel (Raw, QC’d, etc.)• methodID• offset

– offsetValue– offsetUnitsAbbreviation– offsetDescription– offsetUnitsCode

TimeSeries Response, RecapqueryInfo

location

variable

values

Conclusions about WaterML

• Consistent Format

• Includes lots of information

• How do I use it?

HIS Analyst

HIS Analyst

• Desktop apps that use WaterOneFlow

• Examples for various application environments

• HydroObjects – for apps that aren’t web service enabled

• Workbook available at:

http://www.cuahsi.org/his/docs/HIS-workbook-20061130.pdf

HIS Workbook

• Ingesting NWIS Data into Excel

• Ingesting STORET Data into Excel

• Ingesting Weather and Streamflow Data into ArcGIS

• Plotting MODIS Data with Matlab

• Ingesting NWIS Data using VB.Net

• Ingesting NWIS Data Using Java

Excel Demo

Weather Downloader

Historical Weather Data

Forecasted Weather Data

Streamflow

Other

Recap

• Computers share functionality through Web Services

HIS Analyst

Client WaterOneFlowWeb Service

GetSiteInfoGetVariableInfoGetValues

WaterMLTime SeriesRepository