CUAHSI and CLEANER CUAHSI and CLEANER Overview:Overview:

Infrastructure and InformaticsInfrastructure and Informatics

Jon DuncanProgram Manager

22

Collaborative Large Scale Collaborative Large Scale Engineering Analysis Network Engineering Analysis Network for Environmental Researchfor Environmental Research

(CLEANER)(CLEANER) Project Office funded, based in Project Office funded, based in

Arlington, VAArlington, VA Science, Cyberinfrastructure, Social Science, Cyberinfrastructure, Social

Science, Sensors, Organization Science, Sensors, Organization committees committees

Major infrastructure planning process Major infrastructure planning process and moreand more

33

Consortium of Universities for Consortium of Universities for the Advancement of Hydrologic the Advancement of Hydrologic

Science, Inc.Science, Inc. (CUAHSI) (CUAHSI) A consortium of A consortium of

105 research 105 research universities, 6 universities, 6 affiliate members, affiliate members, and 8 int’l affiliatesand 8 int’l affiliates

Incorporated June, Incorporated June, 2001 as a non-2001 as a non-profit corporation profit corporation in Washington, DCin Washington, DC

44

What is Hydrologic Science?What is Hydrologic Science?

Expands beyond traditional hydrologyExpands beyond traditional hydrology Focus on “why” the earth works as it does, Focus on “why” the earth works as it does,

like other earth sciences, moving beyond like other earth sciences, moving beyond traditional problem-solving orientationtraditional problem-solving orientation

Embraces parts of hydrology, Embraces parts of hydrology, geomorphology, hydrogeology, geomorphology, hydrogeology, biogeochemistry, …biogeochemistry, …

Hydrologic cycle is central organizing Hydrologic cycle is central organizing principleprinciple

55

PurposePurpose Science Objective:Science Objective: To further predictive To further predictive

understanding of the terrestrial hydrologic understanding of the terrestrial hydrologic cycle and its linkages with climate and cycle and its linkages with climate and biogeochemical cyclesbiogeochemical cycles

Societal Need:Societal Need: Will there be enough water Will there be enough water for the next century?for the next century? ……of appropriate qualityof appropriate quality ……to meet society’s needsto meet society’s needs ……to maintain the integrity of our ecosystemsto maintain the integrity of our ecosystems

66

Need for CUAHSINeed for CUAHSI

Larger-scale, longer-term research to Larger-scale, longer-term research to advance scienceadvance science

Enable research at disciplinary Enable research at disciplinary boundariesboundaries

Support of larger research teamsSupport of larger research teams Improve efficiency and effectiveness Improve efficiency and effectiveness

of data collection and disseminationof data collection and dissemination

77

What CUAHSI is doing?What CUAHSI is doing?

Hydrologic Information SystemsHydrologic Information Systems ObservatoriesObservatories Hydrologic Measurement FacilitiesHydrologic Measurement Facilities Education and OutreachEducation and Outreach Hydrologic Synthesis CenterHydrologic Synthesis Center

8

-Mathematical Formulae-Solution Techniques

Abstractions in Modeling

Physical World

ConceptualFrameworks

DataRepresentation

ModelRepresentations

“Digital Observatory”Real World

Measurements

•Theory/Process Knowledge•Perceptions of this place•Intuition

Water quantity and quality

MeteorologyRemote sensing

GeographicallyReferenced

Mapping

Validation

DNA SequencesVegetation SurveyHydrologist

Q, Gradient, Roughness?

GroundwaterContribution?

SnowmeltProcesses?

Biogeochemist

Hyporheic exchange?

Mineralogy? Chemistry?

Redox Zones?

DOC Quality?

Geomorphologist

Glaciated Valley

Perifluvial

Well sorted?Thalweg?

Aquatic Ecologist

Backwater habitat

Substrate Size, Stability?Benthic Community

Oligotrophic?Carbon source?

9

Data Representation

• Four-dimensional {x,y,z,t}

• Continental scope• Multi-scale, multi-

resolution• Points, coverages,

dynamic fieldsSpace, L

Time, T

Variable, V

DA data value

1:1200 scale

United States1:500,000 scale

River Basin1:100,000 scale

Watershed1:24,000 scale

River reach

Point scaleA plot

1:1,000,000 scale North American and Global

1010

Digital ContinentDigital Continent

Integrating monitoring and research data Integrating monitoring and research data yields a single body of information for the yields a single body of information for the countrycountry

Observatories contribute intensive information Observatories contribute intensive information to this bodyto this body

Observatories are placed within context of Observatories are placed within context of climate, geology, soils, etc. but are not climate, geology, soils, etc. but are not assumed to be representative of an area.assumed to be representative of an area.

Digital observatories may be watersheds, Digital observatories may be watersheds, aquifers, river reaches, or any region that is aquifers, river reaches, or any region that is part of the continentpart of the continent

11

Observation Stations

Ameriflux Towers (NASA & DOE) NOAA Automated Surface Observing System

USGS National Water Information System NOAA Climate Reference Network

Map for the US

1212

Inference SpaceInference Space

““Transcending place” means testing Transcending place” means testing hypotheses in areas thought to be similar (in hypotheses in areas thought to be similar (in some attributes).some attributes).

Digital continent will enable identification of Digital continent will enable identification of “similar” areas and (some) data about that “similar” areas and (some) data about that spot.spot.

Observatories will enable inference about Observatories will enable inference about similar regions (e.g., presumably one can infer similar regions (e.g., presumably one can infer more about Delaware R. from Potomac than more about Delaware R. from Potomac than about Rio Grande).about Rio Grande).

1313

DOs are the foundation of DOs are the foundation of EOsEOs

Collaboration of Mission and Science AgenciesCollaboration of Mission and Science Agencies DO contains both monitoring and research dataDO contains both monitoring and research data DO supports hypothesis test, decision support DO supports hypothesis test, decision support

systems, mgmt modelssystems, mgmt models Interdisciplinary communicationInterdisciplinary communication

Scientists can access multiple conceptualizations to Scientists can access multiple conceptualizations to improve understandingimprove understanding

Everyone benefits from context providedEveryone benefits from context provided Incentives must exist for people and agencies Incentives must exist for people and agencies

to to wantwant to contribute (and they do!) to contribute (and they do!)

Water Resource Regions and HUC’s

1515

NHDPlus for Region 17ENHDPlus for Region 17E

1616

NHDPlus Reach Catchments ~ NHDPlus Reach Catchments ~ 3km3km22

About 1000 reach catchments in each 8-digit HUC

Average reach length = 2km 2.3 million reaches for continental US

1717

Reach AttributesReach Attributes

SlopeSlope ElevationElevation Mean annual flowMean annual flow

Corresponding velocityCorresponding velocity Drainage areaDrainage area % of upstream % of upstream

drainage area in drainage area in different land usesdifferent land uses

Stream orderStream order

1818

HIS 1.0 (1 Nov 2006)HIS 1.0 (1 Nov 2006)

Point Time SeriesPoint Time Series Discovery and PublicationDiscovery and Publication AgenciesAgencies

USGS NWIS USGS NWIS NCDCNCDC EPA StoretEPA Storet

[LTER Trends][LTER Trends] Static Federation to Observatory Test Static Federation to Observatory Test

BedsBeds

19

CUAHSI Hydrologic Information System Architecture

National HIS – San Diego Supercomputer Center

Workgroup HIS – river authority, research centre or univ.

Personal HIS – an individual scientist or manager

HIS Server

HIS Analyst

Map interface, observations catalogs and web services for national data sources

Map interface, observations catalogs and web services for regional data sources; observations databases and web services for individual investigator data

Application templates and HydroObjects for direct ingestion of data into analysis environments: Excel, ArcGIS, Matlab, programming languages; MyDB for storage of analysis data

2020

HIS ServerHIS Server Supports data Supports data

discovery, delivery and discovery, delivery and publicationpublication Data discoveryData discovery – how do – how do

I find the data I want?I find the data I want? Map interface and Map interface and

observations catalogsobservations catalogs Data deliveryData delivery – how do I – how do I

acquire the data I want?acquire the data I want? Use web services or Use web services or

retrieve from local retrieve from local databasedatabase

Data PublicationData Publication – how – how do I publish my do I publish my observation data?observation data?

Use Observations Data Use Observations Data ModelModel

2121

National and Workgroup National and Workgroup HISHIS

National HIS has a polygon in it marking the region of coverage of a workgroup HISserver

Workgroup HIS has local observations catalogs for coverage of national data sources in its region. These local catalogs are partitioned from the national observations catalogs.

For HIS 1.0 the National and Workgroup HIS servers will not be dynamically connected.

National HIS Workgroup HIS

2222

NWISNWIS

ArcGISArcGIS

ExcelExcel

NCARNCAR

UnidataUnidata

NASANASAStoretStoret

NCDCNCDC

AmerifluxAmeriflux

MatlabMatlab

AccessAccess JavaJava

FortranFortran

Visual BasicVisual Basic

C/C++C/C++

Some operational services

CUAHSI Web ServicesCUAHSI Web Services

Data SourcesData Sources

ApplicationsApplications

Extract

Transform

Load

http://www.cuahsi.org/his/

23

Excel CUAHSI Web service

How Excel connects to ODM

• Obtains inputs for CUAHSI web methods from relevant cells.

• Available Web methods are GetSiteInfo, GetVariableInfo GetValues methods.

converts standardized request to SQLquery.

imports VB object into Excel and graphs it

converts response to a standardized XML.

Observations Data

Model

SQL query

Response

HydroObjects

converts XML to VB object

parses user inputs into a standardized CUAHSI web method request.

2424

Data TypesData TypesHydrologic Observation

Data

GeospatialData

Weather and ClimateData

Remote SensingData

(NetCDF)

(GIS)(Relational database)

(EOS-HDF)

Digital Watershed

2525

HIS ExtensionsHIS Extensions Integration of Weather DataIntegration of Weather Data

Work with NCAR; prototype on OhioWork with NCAR; prototype on Ohio Move from gridded to watershed-based delivery of dataMove from gridded to watershed-based delivery of data

HydrogeologyHydrogeology Constructing stratigraphy for continentConstructing stratigraphy for continent

Geologic FrameworkGeologic Framework Geomorphic and geologic historyGeomorphic and geologic history

Incorporation of human dimensionIncorporation of human dimension Transportation; structuresTransportation; structures Permits, Toxic Release Inventory, etc. Permits, Toxic Release Inventory, etc. Flood plain (contribution from real estate sector?)Flood plain (contribution from real estate sector?) FEMA Lidar productsFEMA Lidar products

Explicit development of AK, HI, PR beyond CONUSExplicit development of AK, HI, PR beyond CONUS

2626

Digital ObservatoriesDigital Observatories Data RepresentationData Representation

Need to develop integrated 4-dimensional Need to develop integrated 4-dimensional data basedata base

Add coverages (easy) and fields (more Add coverages (easy) and fields (more complicated)complicated)

Conceptual Frameworks and ModelingConceptual Frameworks and Modeling Can single data representation fulfill diverse Can single data representation fulfill diverse

set of science needs?set of science needs? Data assimilation techniques to guide sensor Data assimilation techniques to guide sensor

deployment and operationdeployment and operation

2727

Environmental Environmental ObservatoriesObservatories

Two Paths:Two Paths: Bottom-up- Critical Zone Bottom-up- Critical Zone

Observatories (CZO). EAR- Hydrologic Observatories (CZO). EAR- Hydrologic Science, Science, Geochemistry/Biogeochemistry, Geochemistry/Biogeochemistry, GeomorphologyGeomorphology

Top-Down- WATERS Network- Top-Down- WATERS Network- CLEANER/CUAHSICLEANER/CUAHSI

2828

Observatory Development: Observatory Development: Bottom-upBottom-up

EAR Research funds: CZO SolicitationEAR Research funds: CZO Solicitation Two sites selected in 2007Two sites selected in 2007 CUAHSI will push for public data CUAHSI will push for public data

accessaccess If successful, these first two hopefully If successful, these first two hopefully

initial pieces of a prototype initial pieces of a prototype observatory networkobservatory network

2929

Observatory Development: Top Observatory Development: Top DownDown

Major Research Equipment and Facility Major Research Equipment and Facility Construction (MREFC)Construction (MREFC)

WATERS Network-WATERS Network- Currently 11 testbeds have been Currently 11 testbeds have been

either awarded or recommended for either awarded or recommended for awardaward

An initiative of the U. S. National Science Foundation Engineering and Geosciences Directorates

HydroVieHydroVieww

WATERS NetworkWATERS Network

Presenter Name, Affiliation

Presented at ….Date

3131

Environmental Observatories: a new approach for integrated, field-oriented collaborative research at regional to continental scales

Rely on advances in: sensors and sensor networks at intensively instrumented sites shared by the research community cyberinfrastructure with high bandwidth to connect the sites, data repositories, and researchers into collaboratories distributed modeling platforms

From USGS

3232

WATERS NetworkWATERS NetworkWATWATer and er and EEnvironmental nvironmental RResearch esearch

SSystems Networkystems Network

Joint collaboration between the CLEANER Project Office Joint collaboration between the CLEANER Project Office and CUAHSI, Inc, sponsored by ENG & GEO and CUAHSI, Inc, sponsored by ENG & GEO Directorates at the National Science Foundation (NSF)Directorates at the National Science Foundation (NSF)

CLEANER = Collaborative Large Scale Engineering Analysis CLEANER = Collaborative Large Scale Engineering Analysis Network for Environmental ResearchNetwork for Environmental Research

CUAHSI = Consortium of Universities for the Advancement of CUAHSI = Consortium of Universities for the Advancement of Hydrologic ScienceHydrologic Science

Planning underway to build a nationwide environmental Planning underway to build a nationwide environmental observatory network using NSF’s Major Research observatory network using NSF’s Major Research Equipment and Facility Construction (MREFC) fundingEquipment and Facility Construction (MREFC) funding

33

The Need…and Why Now?

Nothing is more fundamental to life than water. Not only is water a basic need, but adequate safe water underpins the nation’s health, economy, security, and ecology.

NRC (2004) Confronting the nation’s water problems: the role of research.

● Water use globally will triple in the next two decades, leading to increases in erosion, pollution, dewatering, and salinization.

● Only ~55% of US river and stream miles and acres of lakes and estuaries fully meet their intended uses; ~45% are polluted, mostly from diffuse-source runoff.

● Polluted runoff caused more than 16,000 beach closings and swimming advisories in the US in 2001

● Floods cause an average of $5.2 billion/year in damage and 80 deaths/year.

● Of 45,000 U.S. wells tested for pesticides, 5,500 had harmful levels of at least one.

● Fish consumption advisories are common in 30 states because of elevated mercury levels.

34

Four critical deficiencies in current abilities:

1. Basic data about water-related processes at the needed spatial and temporal resolution. 2. The means to integrate data across scales from different media and sources (observations, experiments, simulations).

3. Sufficiently accurate modeling and decision-support tools to predict underlying processes and forecast effects of different engineering management strategies.

4. The understanding of fundamental processes needed to transfer knowledge and predictions across spatial and temporal scales—from the scale of measurements to the scale of a desired management action

Courtesy of Tom Harmon

35

DRAFT Vision for the WATERS Network:

WATERS is an integrated observing system that will transform our ability to predict the quality and quantity of the nation’s waters in real time.

36

CLEANER GRAND CHALLENGE

How do we detect, predict, and manage the effects of human activities and natural perturbations on the quantity, distribution and quality of water in near real time?

• Develop capacity to detect effects on water of key drivers:• Population and land use shifts• Energy, water and material resource use• Climate change

37

CUAHSI GRAND CHALLENGE

Develop a predictive understanding of continental water dynamics and its interaction with climate, landscape, ecology and civilizationThree themes in CUAHSI science plan:

•Patterns and variability•Climate and human development•Prediction and adaptation to change

3838

The WATERS Network will: 1. Consist of: (a) a national network of interacting field sites; across a range of spatial scales, climate and land-use/cover conditions (b) teams of investigators studying human-stressed

landscapes, with an emphasis on water problems; (c) specialized support personnel, facilities, and technology;

and (d) integrative cyberinfrastructure to provide a shared-use

network as the framework for collaborative analysis

2. Transform environmental engineering and hydrologic science research

and education

3. Enable more effective management of human-dominated environments

based on observation, experimentation, modeling, engineering

analysis, and design

39

Common Vision: WATERS Network

Sensors and Measurement Facility

CA

dx)x(r)x(Q

Synthesis

Informatics

Server

Workstation

domainDNS

Disk array

Observatories/ Environmental Field Facilities

40

Network Design Principles:

Enable multi-scale, dynamic predictive modeling for water, sediment, and water quality (flux, flow paths, rates), including:

Near-real-time assimilation of data Feedback for observatory design Point- to national-scale prediction

Network provides data sets and framework to test: Sufficiency of the data Alternative model conceptualizations

Master Design Variables: Scale Climate (arid vs humid) Coastal vs inland Land use, land cover, population density Energy and materials/industry Land form and geology

Nested (where appropriate) Observatories over Range of Scales:Point Plot (100 m2)Subcatchment (2 km2)Catchment (10 km2) – single land useWatershed (100–10,000 km2) – mixed useBasin (10,000–100,000 km2)Continental

41

I

III II I

I

Simplified schema of a potential national WATERS Network based on three hydrologicregions: (I) coastal, (II) humid-continental, and (III) arid-continental; large blue circles: regional, watershed-based observatories; small circles: intensively instrumented field sites

42

PrecipitationPrecipitationDepositionDeposition

Agricultural FieldsAgricultural FieldsCAFOsCAFOs

ReservoirReservoir Water TreatmentWater Treatment

Retention BasinRetention Basin

EvaporationEvaporation

Humid Continental Watershed:Humid Continental Watershed:Potential WATERS EFF sitePotential WATERS EFF site

SensorSensor

SensorsSensors

4343

Sensor NetworkSensor Network

Distributed network of Distributed network of interchangeable arrays of interchangeable arrays of remote and embedded remote and embedded stationary and mobile stationary and mobile sensors sensors

Satellite and Remote Satellite and Remote SensingSensing

Individual Deployed SensorsIndividual Deployed Sensors

44

Potential Observatory

WATERS Network Cyberinfrastructure

4545

WATERS Network CI PlanningWATERS Network CI Planning

CLEANER Cyberinfrastructure Committee is CLEANER Cyberinfrastructure Committee is creating a CI program plan with CUAHSIcreating a CI program plan with CUAHSI

Two major projects are creating CI prototypes for Two major projects are creating CI prototypes for WATERS (with more to come):WATERS (with more to come):

• CUAHSI Hydrologic Information System (HIS) project (David CUAHSI Hydrologic Information System (HIS) project (David Maidment, PI)Maidment, PI)

• NCSA Environmental CI Demonstration (ECID) project NCSA Environmental CI Demonstration (ECID) project (Barbara Minsker and Jim Myers co-leads)(Barbara Minsker and Jim Myers co-leads)

Have jointly proposed a draft common environmental Have jointly proposed a draft common environmental CI architectureCI architecture

Are demonstrating how observatories can enable Are demonstrating how observatories can enable adaptive forecasting and managementadaptive forecasting and management

46

Environmental CI Architecture: Research Services

Create Hypo-thesis

Obtain Data

Analyze Data &/or Assimilate into Model(s)

Link &/or Run Analyses &/or Model(s)

Discuss Results

Publish

Knowledge Services

Data Services

Workflows & Model Services

Meta-Workflows

Collaboration Services

Digital Library

Research Process

Supporting TechnologyIntegrated CIECID Project Focus: Cyberenvironments

HIS Project Focus

47

Collaboration Services: CyberCollaboratory

• A web portal to allow sharing of information and ideas across the community.

• Currently being used by CLEANER Project Office team

• Public CI demo space shows numerous cyberinfrastructure demonstrations

http://cleaner.ncsa.uiuc.edu/cybercollab

48

CUAHSI Data Services

Web ServicesLibrary

Web application: Data PortalYour application• Excel, ArcGIS, Matlab• Fortran, C/C++, Visual Basic• Hydrologic model• …………….

Your operating system• Windows, Unix, Linux, Mac

Internet Simple Object Access Protocol

Source: David Maidment, Univ. of Texas

49

Path Forward: Timeline

2007 Joint workshop on integrating social sciences into EOs (January) Public comment period on draft program plan and conceptual design (science, sensor, organization, social science, education, and

cyberinfrastructure-February) Completion of program plan and conceptual design (August)

2008 Consortium arrangements established Conceptual Design Review - move to MREFC Readiness Status

2009 Continue funding development of enabling technologies, test-beds, and prototypes

FY 2012 Start of MREFC funding and construction phase

FY 2016 Project completion; start of full-scale operations

5050

ConclusionsConclusions Networked observatories will create a new paradigm for Networked observatories will create a new paradigm for

environmental researchenvironmental research Shared infrastructure at large scalesShared infrastructure at large scales Interdisciplinary teams collaborating remotely to address complex Interdisciplinary teams collaborating remotely to address complex

environmental issuesenvironmental issues Enabling improved understanding & management of large-scale Enabling improved understanding & management of large-scale

natural environmental systemsnatural environmental systems

Cyberinfrastructure will be critical to the observatory’s Cyberinfrastructure will be critical to the observatory’s goalsgoals Enabling real-time collaboration, data services, modeling, & Enabling real-time collaboration, data services, modeling, &

managementmanagement Creating a national knowledge network, not a set of individual Creating a national knowledge network, not a set of individual

field sites and investigatorsfield sites and investigators

5151

AcknowledgmentsAcknowledgments

100s of researchers and educators from 100s of researchers and educators from across the US are contributing to across the US are contributing to WATERS’ planning processWATERS’ planning process http://cleaner.ncsa.uiuc.eduhttp://cleaner.ncsa.uiuc.edu http://www.cuahsi.orghttp://www.cuahsi.org

Funding sources:Funding sources: NSF grants BES-0414259, BES-0533513, NSF grants BES-0414259, BES-0533513,

EAR-0413265, and SCI-0525308EAR-0413265, and SCI-0525308 Office of Naval Research grant N00014-04-1-Office of Naval Research grant N00014-04-1-

04370437

5252

Questions?Questions?

Contact info:Contact info:

Jon Duncan, Project ManagerJon Duncan, Project Managerjduncan@cuahsi.orgjduncan@cuahsi.org202 777 7305202 777 7305

How can CUAHSI and CLEANER make How can CUAHSI and CLEANER make better linkages with NWQMC?better linkages with NWQMC?