7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS...

7 Nov 2007 2007 Geospatial Interoperability Summit

Iowa Environmental Mesonet: Using Open Source GIS Tools and Web

Services to Disseminate Environmental Data

Daryl HerzmannIowa StateUniversity


Data, data, everywhere data


The Desperate Challenge

• Disparate Formats– Grib– METAR– Text CSV– Free form text– ADAS III– SHEF– Excel (gasp!)

• Disparate Network Protocols– HTTP– FTP– LDM IDD (tcp)– TCP stream– Modems

The data comprising the IEM is made available to us in the following formats and network protocols.


Why an Open Source solution?

1. They are the best tools out there. UNIX mentality -> Do something small and do it well.

2. Transparency. We are a public institution doing public funded research.

3. Excellent Support• Public bug trackers• Community forums• IRC• Motivated developers


Open Source SolutionPostGIS / PostgreSQLSpatial Database

Map ServerMapping Engine +Scripting

Red Hat Linux Operating System

PythonData processing andnetwork services

GDALSwiss Army Knifeof all things GIS

PHPDynamic Website


Key Component: PostGIS

• Early IEM code (2001) was a series of flat files in Shapefile format and database of observations. This was a disaster.

• PostGIS provided the opportunity to manage data, metadata, and the GIS data in one location.– Network scalability– Transaction / stored procedure support– Speed / functionality with merge of GEOS library


Look at all of my data!

• 1 billion observations• 1 terabyte of data

– Archives of NEXRAD, GOES Satellite, precipitation datasets, web cameras!, Weather Service products, …….

• Now what? Swim in all of my data?


Here’s an idea, lets create a webpage where they can download the data!

• Daryl:– Creates webpage

• End user:– Finds my page– Fills out form– Saves data, attempts to

figure out format– Attempts to find geo-

referencing for their GIS.– Hopefully loads data into

app– Commits numerous errors

along the way


The Open Geospatial Consortium (OGC) Way


In 2002, we built OGC services, but they didn’t come….

• Why?– Where were the cool

clients?– ESRI’s OGC support

wasn’t there yet– It all felt slow as

compared with local disk access

– A bit ahead of its time


In 2007, they have come!

• Thanks to cool clients– Google Maps– OpenLayers– ESRI Interop Extension– Map Server and others

• ~ 2 million hits per day (~20 “maps” per second)


Supporting the Load

• 12 systems comprise IEM web farm

• Various levels of redundancy, but no spatial database redundancy yet

• Roughly 30 GB in , 60 GB out each day

• Linux IPVS provides redundant entry point and load balancing


Available OGC Services


http://mesonet.agron.iastate.edu/ogc/

• CONUS GOES (Infrared, Visible)

• CONUS NEXRAD Base Reflectivity, Storm Total Precip, 1 Hour Precip

• NWS issued advisories,watches, and warnings

• Iowa Rainfall Estimates

• Iowa Road Conditions

• IEM Data Services (temp, winds, etc)


Raw NEXRAD Composites

Have a large amount of noise due to ground clutter. A simple GDAL scriptcomparing radar echos to its elevation quickly gets rid of most noise.


Cleaned Composite


Some IEM developed GIS webapps


4 May 2003, heavy rains on freshly tilled soils

25 Sep 2005, heavy rains on mature crops


GIS + webcams

IEM developed software to have the cameras automatically track the sun and the moon. Can track NEXRAD identified storms as well!Done using FOSS GIS tools.


What others have done.


Lessons Learned

• Adopting OGC services can greatly reduce the complexity of serving out your data and using your own data!

• Performance hasn’t been an issue, most of the overhead is with HTTP.

• For Map Server, try to limit the number of mapfiles you maintain!


Future Work

• Increase Metadata!

• Help support NetCDF in GDAL (?)

• Work with end users on getting IEM data into their platform.


Time for Questions?

Daryl [email protected]

7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS...

Documents

Transcript of 7 Nov 20072007 Geospatial Interoperability Summit Iowa Environmental Mesonet: Using Open Source GIS...