Pacific Northwest Water Quality Data

Exchange

Oregon Department of Environmental Quality Washington Department of Ecology

Idaho Department of Environmental QualityAlaska Department of Environmental Conservation

Environmental Protection Agency, Region X

The Pacific Northwest Water Quality Data Exchange

Pacific Northwest Water Quality Data

Exchange

Challenge Grant Objectives

Provide access to a comprehensive collection of Water Quality data for the Pacific Northwest

Apply Exchange Network principles

Enable participation from a wide range of data sources

Design data exchange flow to support partner needs and eventual upload to EPA STORET

Pacific Northwest Water Quality Data

Exchange

Collaboration is Key to Success

Active Participants

Environmental Protection Agency Region 10 Nestucca-Neskowin Watersheds Council (Oregon) Northwest Indian Fisheries Commission State of Alaska Department of Environmental Conservation State of Idaho Department of Environmental Quality State of Idaho Soil Conservation Commission State of Oregon Department of Environmental Quality State of Washington Department of Ecology University of Idaho - Water Resources Research Institute Windsor Solutions, Inc.

Pacific Northwest Water Quality Data

Exchange

Collaboration is Key to Success

Affiliates Coeur d’Alene Tribe Confederated Tribes of the Siletz Indians Idaho National Engineering and Environmental Laboratories King County (Washington) Department of Natural Resources Long Tom Watershed Council (Oregon) National Park Service Nez Perce Tribe Pacific States Marine Fisheries Commission – StreamNet Rogue Valley Council of Governments (Oregon) South Coast Watersheds Council (Oregon) State of Oregon Watershed Enhancement Board Surfrider Foundation Tanana Chiefs Council

Pacific Northwest Water Quality Data

Exchange

Collaboration is Key to Success

Related, Collaborative Project Contacts

Environmental Data Standards Council – Environmental Sampling, Analysis, and Results Standard

National Water Quality Monitoring Council – Water Quality Data Elements

USGS -NWIS / EPA –STORET Data Integration

Environmental Protection Agency – OEI / CDX

Environmental Protection Agency – OWOW / STORET

Pacific Northwest Water Quality Data

Exchange

Vision for the Exchange Data Flow

INTERNET

Washington

Oregon

DataCatalog

EPA

HostDatabase

Data AccessApplication

NetworkNode

NetworkNodeData

Source

DataSource

NetworkNode

DataSource

Alaska

Idaho

CDX STORETData

Warehouse

.xml

.xml

.xml.xml

.xml

NetworkNode

DataSource

Pacific Northwest Water Quality Data

Exchange

Components of the Exchange Flow

DET

FCD TPA

- Data Exchange Template - data elements- relationships and conditions- derive XML schema

Pacific Northwest Water Quality Data

Exchange

Data Exchange Template

Pacific Northwest Water Quality Data

Exchange

XML Schema

Pacific Northwest Water Quality Data

Exchange

Components of the Exchange Flow

DET

FCD TPA

- Data Exchange Template - data elements- relationships and conditions- derive XML schema

- Flow Configuration Document - method definition, parameters- query flow: filtering, wildcards,

paging, ordering, errors- load flow: process sequence, security

Pacific Northwest Water Quality Data

Exchange

Flow Configuration Document

Pacific Northwest Water Quality Data

Exchange

Components of the Exchange Flow

DET

FCD TPA

- Data Exchange Template - data elements- relationships and conditions- derive XML schema

- Trading Partner Agreement - roles and responsibilities - data stewardship - administration

- Flow Configuration Document - method definition, parameters- query flow: filtering, wildcards,

paging, ordering, errors- load flow: process sequence, security

Pacific Northwest Water Quality Data

Exchange

Trading Partner Agreement

Pacific Northwest Water Quality Data

Exchange

http://deq12.deq.state.or.us/pnwwqx/

Accessing the Exchange

Pacific Northwest Water Quality Data

Exchange

First step: query the data sources

1. Select Data Sources

2. Go to Map

1. Select Data Sources

2. Go to Map

Pacific Northwest Water Quality Data

Exchange

First step: query the data sources

3. Select a Region3. Select a Region

Pacific Northwest Water Quality Data

Exchange

First step: query the data sources

4. Coordinates Returned from map

5. Choose More Parameters

6. Search initiates real-time Query.

4. Coordinates Returned from map

5. Choose More Parameters

6. Search initiates real-time Query.

Pacific Northwest Water Quality Data

Exchange

Second step: browse the data and the details

7. Click on any field in a row to see the details

Pacific Northwest Water Quality Data

Exchange

Second step: browse the data and the details

Pacific Northwest Water Quality Data

Exchange

Second step: browse the data and the details

8. Download initiates off-line Solicit method

Pacific Northwest Water Quality Data

Exchange

Third step: download the data

10. Solicit is processed later, maybe overnight

9. Email used to return Download Status page link to user

Pacific Northwest Water Quality Data

Exchange

Third step: download the data

11. All processed files are zipped together with a Readme file.

Pacific Northwest Water Quality Data

Exchange

Third step: download the data

12. Each file comes in a common, tilde-delimited format with a standardized header.

Pacific Northwest Water Quality Data

Exchange

Third step: download the data

13. These files readily parse into spreadsheet or database formats.

Pacific Northwest Water Quality Data

Exchange

Next Steps

Engage additional Exchange partners

Municipalities, other agencies

Watershed councils, volunteer groups

Pilot ambient WQ data flow with EPA-STORET

Pacific Northwest Water Quality Data

Exchange

Challenges

Challenges: Inventing a regional-scale model of the NEIEN Communications with EPA; code lists Allow for a wider range of (non-node) data sources Long term funding for administration

Pacific Northwest Water Quality Data

Exchange

Lessons Learned

Here is your heads-up from our experience:

You are going to learn. (You will walk through a cloud) New Technology New Acronyms New Ways of Doing Business

You will understand better what Data Quality means. Everyone you let in the door will be able to see your data in its

current state. Acceptable data o your organization may not be acceptable data

to your trading partners. Old out-of-date data is seen by others as valid new data.

Pacific Northwest Water Quality Data

Exchange

Lessons Learned cont.

You will appreciate Data Standards and other standards. If your organization does not have some form of formal data

standards, develop them now. Name things what they are. Later in mapping data elements to a

schema, this is gold. Do not make your standards too hard to use or understand. Avoid complex abbreviation schemes to ease readability and

understanding. Use names for elements like Person_Last_NM instead of PLastName, Address_ID not AID, or Last_Update_DT not LUpDay.

Collaborate with related data standards and network projects.

Pacific Northwest Water Quality Data

Exchange

Lessons Learned cont.

You will redevelop a new meaning for change control Things change as you progress. Things change rapidly at first and more rapidly later. Many things may need to be changed at the same time. Your critical path will become confused if you are not actively

managing it. What appears to be an insignificant request may require a

significant amount of effort. You will learn that this is not just a simple thing that

anyone can do. This type of project requires dedicated staff to manage project. The learning curve on these projects is not quick. Ensure broad user participation.

Pacific Northwest Water Quality Data

Exchange

Lessons Learned cont.

You will see the importance of the 3 flow components. (The DET and FCD are your roadmap to success.)

– Data Exchange Template - DET/Schema– Flow Configuration Document - FCD– Trading Partner Agreement - TPA

Use existing work if you can find it. (Borrowing beats building any day).

– Need to reconcile published schemas– Reuse available XML schemas (FRS, Beaches and others

will help in the future)

Pacific Northwest Water Quality Data

Exchange

Lessons Learned - Summary

We have learned: that the goals we set were achievable, that we built more then a flow or software, we built a functioning team and a deeper

understanding of our business, and we built an access point to data so that

others can use it to study and help us as a result of that research, to provide a more environmentally sound collection of states and a cleaner, safer future.

Pacific Northwest Water Quality Data

Exchange

Thank you!

Questions?????

Contact Miles Neale for details

mnea461@ecy.wa.gov

360-407-6592