GeoServer and Architectures of GeoServer and Architectures of Participation for Geospatial Participation for Geospatial

InformationInformation by by

Chris HolmesChris Holmes

The Open Planning Project The Open Planning Project

OverviewOverview

• Introduction to GeoServer• Geospatial Open Standards• GeoServer and Google Earth• Future - towards collaborative geodata

The Open Planning Project The Open Planning Project (TOPP)(TOPP)

• http://topp.openplans.org• High Tech Non-Profit based in New York City• Founded by Mark Gorton, part of the Lime Group• TOPP builds technology to enhance the role of the

citizen in democratic society.• Projects

– OpenPlans– New York City Streets Renaissance Campaign– GeoServer

TOPP's approachTOPP's approach

• Hybrid between a traditional non-profit and a business

• Socially conscious mission, and non-profit tax status

• But run more like a start-up, big initial investment, to incubate ambitious projects that help make the world a better place

• If popular, find socially conscious business models for maturing projects - exist in the market, not just in a donor's checkbook

• Re-invest money from successful projects in to more innovative approaches

• Find and recruit top developers, treat them very well, and trust them to do great work

TOPP's goalsTOPP's goals

• Make tools to enhance government transparency and provide opportunity for citizen participation in shaping society.

• Bring the root philosophy behind open source to other domains – Geospatial– Politics– Urban Planning– More coming soon

• Open Source for government, help solve chicken and egg problem

GeoServerGeoServer

• An open source server to publish and edit geospatial data

• Strong support for Open Geospatial Consortium (OGC) standards, including WMS, WFS, Filter and SLD

• Translates from existing databases to a variety of formats– Inputs: PostGIS, Oracle Spatial, DB2, ArcSDE,

MySQL, Shapefile, WFS– Outputs: GML, Shapefile, KML/KMZ, JPEG, PNG, GIF,

SVG, PDF

TOPP's GeoServer MotivationsTOPP's GeoServer Motivations

• TOPP's motivation behind funding GeoServer is to help make geospatial information more accessible to civil society - to increase government transparency and to provide a substrate for intelligent community organizing.

• Instead of harassing public officials to spend time and money to open up data, we provide freely available tools to make it easier to make existing data more accessible

• Focus on WFS, the 'source code' of a map, opening the data for traffic modeling and the like, not just making pretty pictures

• Prove that we can build a true open source community around a project started by a non-profit

Original GeoServer GoalsOriginal GeoServer Goals

• TOPP's goals in a GIS server were:– support (and indeed advocate for) open standards– be extremely easy to set up and use

• No special training costs to make data accessible– connect to a wide variety of legacy formats.– Work efficiently with large datasets, must scale.– attract outside contributors

• for a stronger ecology, not a monoculture driven by TOPP

• Being able to meet others' needs makes a stronger overall project

GeoServer NowGeoServer Now

• Reference Implementation for WFS 1.0 and soon 1.1– WFS-Transactional and Locking

• Fully certified WMS 1.1.1 compliant– Full SLD and Filter implementations

• Installers, great documentation, support available through email lists and commercial companies

• Connect to all the most popular spatial databases

• Very good performance, and getting faster all the time.

• Works with databases gigabytes in size

GeoServer CommunityGeoServer Community

• The successful open source ecology around GeoServer is what TOPP is most proud of

• Choice to build on GeoTools, an existing community, has paid off as much code has flowed to GeoServer– Helped GeoTools become a founding OSGeo project

• GeoServer contributors come from a variety of companies, universities and government agencies

• Recently formed Project Steering Committee

• Modular plug-in architecture with 'community' section of svn. Can easily build commercial services on top of base.

GeoServer FutureGeoServer Future

• The Future is in the hands of the community– WCS and Raster Support in WMS

– Complex Features

– Modular, open platform for new, innovative GIS web services

– Join us and help determine the future

• As a strong member of the community, TOPP will continue to push on its original motivations

– More accessible geospatial information, focus on the data

• Enable user contributions and participation in maintenance and creation of geospatial information - 'cvs for the geospatial web'

• Continue to scale and work with new formats - do the unglamorous work

– Work on the bigger picture of an open, true geospatial web

• Citizens demand open geodata, just like governments now have to be online

Open StandardsOpen Standards

Why Open Standards?Why Open Standards?

• Interoperability

• Lower barrier to entry - level playing field.

• Builds something bigger than any one organization can alone– Netscape vs. Microsoft Web - both had servers and

browsers, and started putting extra extensions in.– Would have had to decide which web you wanted to

surf.– Apache httpd server prevented the split, gathering of

information became far more valuable than a microsoft or netscape web would have been.

Open Geospatial ConsortiumOpen Geospatial Consortium

• Leading standards organization for web based GIS

• Overall does good work, though sometimes in need of a reality check

• WMS is most successful standards, WFS is the next major one emerging. SLD and Filter are both great supplemental specs

• GML is a xml language for geographic features, focused on the geometry definitions that are useful, leaves the rest to implementors to define.

• WFS-Transactional enables user collaboration on geospatial information

GeoServer and Google EarthGeoServer and Google Earth

MotivationsMotivations

• To provide a way to link existing databases of geospatial information directly to Google Earth.– Passing around KML files doesn't scale to many

geospatial datasets– Network links are a powerful construct, but leave a

heavy burden on the implementor.

• To give GeoServer users another option to visualize their data– Google Earth is an extremely accessible tool, so

allowing existing and future GeoServer users to have their data automatically available on google earth is a big win.

Use CasesUse Cases

• The primary use case for GeoServer is an organization that already has a large amount of geospatial data and wants to enable users to visualize it using Google Earth.

• An organization can also make use of the transaction capabilities, to collaborate on a shared resource.

– Passing around edited KML files stops scaling with more than a couple users.

• With SLD allow users to choose a new style to visualize the data, perhaps emphasizing a different aspect of the data set.

• Additionally enables access to raw data, can explore in Google Earth, then download a shapefile of the area.

• Connect from remote WFS's and style them in to GE.

Implementation OverviewImplementation Overview

• Leverage open standards and existing code as much as possible– Network Link uses WMS directly, with KML as output

format– KMLReflector makes WMS more accessible, provides

defaults for confusing options– Use SLD, an existing open standard of a language for

how to style a map– KMScore leverages WMS even more for large datasets– KMLWriter uses GML writer from WFS for geometries– AJAX Style wizard works with SLD and PutStyle

Web Map Service (WMS)Web Map Service (WMS)

• RESTful requests to get an map from a server

• GetCapabilities - describe what layers are on the server, and the styles I can request them in

• GetMap - get a map with parameters• http://example.com:8080/geoserver/wms?

• request=GetMap&

• layers=states&

• styles=population&

• srs=EPSG:4326&

• width=600&height=600&

• format=image/png&

• bbox=126.3,12.8,-63.7,62.9

WMS CompatibilityWMS Compatibility

• Supported natively by lots of software– uDig, WorldWind, mapbuilder, mapbender, openlayers (with tiling), ArcGIS, gvSig,

cadcorp, intergraph, etc.

• Very compatible with other services– For Google Maps API v2, many have used the custom tile layer to connect directly

to a WMS, as long as it supports Mercator and Lat/long projections

• WMS google maps + WMS javascript library for GeoServer now available.

• Can combine with Squid to do caching of tiles after initial dynamic generation

– KML WMS reflectors were among the first network links, they took bbox request of a network link and appended it to the rest of a WMS request, returning it wrapped in a Ground Overlay

• Allows different output formats– PNG, GIF, SVG, PDF, SVG

– Why not KML/KMZ?

WMS with KML/KMZ outputWMS with KML/KMZ output

• No additional configuration needed for users of GeoServer, utilizes the same connections already made with GeoServer web admin tool

• Network links can connect directly using the WMS protocol, just request a KML or KMZ output format

• For an active, region based network link, just leave off the BBOX parameter and let Google Earth send the refresh requests

• From our development perspective we just had to write a new output plug-in, instead of a whole connection stack

KML ReflectorKML Reflector

• Goal was to make WMS more accessible, provide defaults for the more confusing options

• http://sigma.openplans.org/geoserver/wms/kml_reflect?layers=states

• VS.• http://sigma.openplans.org/geoserver/wms?styles=population&Format=application/vnd.google-

earth.kmz&request=GetMap&layers=topp:states&width=550&height=250&srs=EPSG:4326&transparency=true

• Also divides up layers in to their own network links, so one can turn them on and off individually in Google Earth

• Sets defaults for network links reloading, 3 seconds after camera stops.

• returns correct mime type so putting it in to a browser automatically fires up google earth if installed

SLD for StylingSLD for Styling

• SLD is Styled Layer Descriptor, an OGC standard for styling mapping data

• 'CSS for GeoSpatial information' - a vendor independent 'stylesheet' that works against geospatial data

• Powerful when combined with KML output since it allows one to define 'rules' for symbology, based on attributes of the dataset, instead of just basing it on feature IDs

• Already used natively in GeoServer configuration, all data formats are styled with SLD

– You define your style once, view it on GE, desktop GIS, 2d web map, PDF, etc.

Basic SLDBasic SLD

Advanced SLDAdvanced SLD

• Based on 'rules', filters and min/max scales to limit number of features rendered

– 'color all streams blue'.

– 'color all major highways orange, minor highways yellow, and normal streets white, with black outlines'

– 'draw 3 pixels wide when zoomed out, 7 when zoomed in, and 13 when zoomed way in'

– Thematic maps, color based on quantiles or statistical analysis

• User Defined SLD– Administrator does not have final say on how a map looks

– Users can pass in their own SLD definitions, and the WMS will render it for them

– Can include XML directly in WMS request, or point to an SLD file online

SLD WizardSLD Wizard

• Very Basic, purpose was to replicate Google Earth Pro shapefile import functionality

• Previously you had to actually edit SLD XML files by hand

• AJAX based, as its own utility, so that it's independent of the GeoServer web admin system

• Uses standardized PutStyle operation

• Hope to expand the functionality for min/maxScale, Filter, etc. Along with a preview window and advanced gui

– AJAX would allow us to mashup with Google Maps API, do preview there and then persist to GeoServer for use in Google Earth and more.

KMScore parameterKMScore parameter

• It's often more efficient to process the data on the server side and just send an image back (rather than a whole KML dump).

• KMScore allows a user to have GeoServer return a 'Ground Overlay' when there are too many features for Google Earth to render.

• Does this with a PNG WMS request, but puts a ground overlay wrapper around it and zips in to a KMZ

– Uses same SLD definitions for both, so they look the same, though not as dynamic

• KML 2.1 Regions is another solution to help with this, but it did not exist when we were implementing. And indeed this is compatible with KML 2.1 Regions

KMLWriter implementationKMLWriter implementation

• Leverages GML writer for geometries, since the two are very similar

• Reuses existing code for rendering geometries with SLD, but instead of rendering it writes out the KML for the images.

• Streams from database directly to output stream, so there is very little memory overhead, even less than a normal WMS request– Database streaming is geotools default, needed for

huge GIS datasets

FutureFuture

• 2.5D, allow users to define an attribute to be used as the altitude for the feature in Google Earth

– Could combine with Filter functions so you could do interesting statistical analysis portrayed visually as the height of features

• KML 2.1 Regions and SuperOverlays– Work with KMScore and min/max SLD scales to return the right amount of

information at different zoom levels

• expanded SLD ajax editor– Nice visual editor to set styles for different regions, preview window, etc.

• Native Caching– Tiles and KML regions would be cached with no additional configuration

• What do you want? Open Source and Open Community

Geospatial Architectures of Geospatial Architectures of ParticipationParticipation

Architectures of ParticipationArchitectures of Participation

• Term I use for 'the open source process'

• Coined by Tim O'Reilly initially to describe open source process, currently used more with passive contributions like napster

• Strong roots in Yochai Benkler's 'commons-based peer production' of Coase's Penguin - I take an architecture of participation to be the 'thing' that results from a commons based peer production

• Stephen Weber also hits on main ideas in The Success of Open Source Software

My definitionMy definition

• An ‘Architecture of Participation’ is both social and technical, leveraging the skills and energy of users as much as possible to cooperate in building something bigger than any single person could alone.– Pre-requisite is that the results must be available to all users.

• This is accomplished in Open Source Software with Licensing– Complex interaction of effort versus results, but the chances for

success are highest by maximizing three aspects• Working on something useful (or at least have the promise of being

useful relatively soon).• Viewing users as ‘co-developers’. Give them responsibility, see them

as the most valuable resource, and they will become it.• Lowering barriers to contributing. First time contributors won’t come

back if it is difficult to have their time incorporated to the commons.

Geospatial WebGeospatial Web

• A World Wide Web for geospatial information (WWW is one of the great examples of an architecture of participation)

• In contrast to Spatial Data Infrastructures set up and run by government agencies - it's bottom up.

• Google is leading the way, with API's for mashups and KML files (it now becomes much easier for me to explain what I do - shows off the value of opening up data and what people can do on top of it)

Architectures of Participation for Architectures of Participation for GeoDataGeoData

• Goal is to create a movement similar to the Open Source Software movement around open GeoData

• User able to correct base data (streets, parks, etc.), collaborate on new datasets of value (wifi spots, scenic views, bike trails, etc.)

• Where all TOPP's 'reinvestment' with GeoServer goes, main issue we care about.

• Is starting to happen, like early days of Free Software, hobbyists and hardcore 'freedom' people.

• Will evolve in to making economic sense for large organizations - collaborate on something open, instead of continuing to pay a monopolist or an oligarchy of control.

PrerequisitePrerequisite

• 'Commons-based' - from Benkler, the thing people are contributing to must be guaranteed to remain open.

• Needs to be a set of clear licenses for geodata that are well marketed and/or used.– Creative Commons would be a great ally, bootstrap on

their marketing effort– Customized for geodata - does a viral 'share-alike'

license mean other layers need to be shared? Or just the data of the one layer? Copyright data or produced map? Or styles + data?

– Used as defaults, or at least options in GeoServer, in KML creation tools, etc.

SocialSocial

• Needs innovation in workflow. Two existing models are Wikipedia and FOSS

– Open Street Map is a wikipedia model, anyone can commit, rely on rollbacks

– FOSS model gives commit rights to few, but patches allow more contributions

– I believe it will end up somewhere between FOSS and wikipedia, perhaps with more automation

• automatic rejection of features that don't meet certain pre-defined requirements, perhaps a peer review process, commit rights to an area, etc.

• will be many different projects, instead of one atlas to rule them all– though some projects like streets would have more contributors

• Incentives - dual licensing similar to MySQL? People could choose to buy it outright, or agree to contribute a certain amount.

TechnicalTechnical

• Base on open standards - WFS-Transactional– Existing clients can be used, strength in diversity

– Extend on standards, for rollbacks, for notifications, for limited commit access, etc.

– Notion of a 'patch file', perhaps using WFS-Transactional XML or KML <Update>

• Allow pluggable workflows built in to WFS-T– choose from automatic validation, peer review, hierarchical review, limited commit

rights, sandboxes, rollbacks, link to issue tracker, area based notifications (like wiki 'watch this page)

• Easier to use desktop and webapps that work with workflow, aren't full hog GIS systems

– uDig - already has WFS-T, scale down and work with GPS

– Google Earth - submit directly to WFS-T instead of creating KML file

• Perhaps send a KML <Update> through WFS-T

– AJAX on top of Google Maps/OpenLayers, submit and edit points

How to encourageHow to encourage

• Make available high quality base layer satellite data under Share Alike terms

• Hold copyright in a neutral organization (like FOSS or wikimedia foundations)

• Start with public domain data, like Tiger or VMAP

• Build applications on top, integrate with others, have them expose 'fix me' button.

• Integrate with handheld GPS devices, location aware mobile phones/PDAs

• Hosted service of geodata commons

Geodata CommonsGeodata Commons

• Harlan Onsrud et al. wrote a paper on these ideas– Described a system where people could upload geodata under open access

licenses, fill out metadata and make it available for others

• Extend to a web services environment– Anyone can upload their data and pick a license.

– Instantly available as WMS, WFS, on Google Earth

– Gives people a compelling reason to add existing data

• Add user contributions– WFS-T and easy to use editing tools that follow the defined workflows

– Let them accept patches, allow others to have commit rights

• Becomes 'Sourceforge for GeoSpatial Web'– Many of the pieces to do this are already in GeoServer

Geospatial Web FutureGeospatial Web Future

• A substrate for other collaborative webs of information– Traffic modeling– Urban planning– Environmental modeling

• More effective collaboration between citizens, companies and government for efficient maintenance and creation of geospatial data.

• Ability to search and find geospatial datasets, overlay and analyze them, and even contribute back.

• TOPP is helping to build GeoServer to make this future a reality. Join us.

Questions?Questions?

LinksLinks

• GeoServer homepage - http://geoserver.org

– Google Earth info: http://docs.codehaus.org/display/GEOSDOC/Google+Earth

– Google Maps info: http://docs.codehaus.org/display/GEOSDOC/Google+Maps

• My blog - http://cholmes.wordpress.com

• The Open Planning Project - http://topp.openplans.org

• Coase's Penguin, Yochai Benkler - http://www.benkler.org/CoasesPenguin.html

• Public Commons of Geographic data - http://www.dpi.inpe.br/gilberto/papers/commons_giscience2004.pdf

• Open Geospatial Consortium - http://opengeospatial.org

• Architecture of Participation by Tim O'Reilly- http://www.oreillynet.com/pub/wlg/3017

• Notice: these slides are all released under a Creative Commons attribution license: http://creativecommons.org/licenses/by/2.0/ Reuse and remix in anyway you please.

Unused slides that were used to Unused slides that were used to prep for talk but cut for time prep for talk but cut for time

follow, in case you're interested. follow, in case you're interested. I will write about these topics in I will write about these topics in

my blog soonmy blog soon

Next Steps for GeoSpatial WebNext Steps for GeoSpatial Web

• Go Beyond Mash-ups, to allowing anyone to search and combine layers of information– Users search for Chicago Crime Map + Chicago

Craigslist and display on one map– One way is to have maps API hooks to let other users

to get the data displayed on an existing mash-up– KML files on google maps is another way, if they

become preferred means for mash-ups– Or base on web servers of geospatial information

• Which would be needed regardless for big KML files

Next Steps for GeoSpatial WebNext Steps for GeoSpatial Web

• Geospatial Search– 'Catalogs' are the traditional/SDI way of doing things,

register your node yourself• Stuck in library metaphor, few good examples

– Mapdex and others are crawling services, often bootstrapping Google

• looking for request=GetCapabilities as it means an OGC service

– Crawling and then searching KML files + WMS/WFS services within a bounding box on

– Incentivize registering existing services/files

Next Steps for GeoSpatial WebNext Steps for GeoSpatial Web

• Scaling WMS– WMS is a dynamic protocol, but can be easily cached

in to tiles (in debt to google for the innovation)– We've experimented with GeoServer WMS + Squid +

Google Maps/OpenLayers– Wide group of interest at:

http://wiki.osgeo.org/index.php/WMS_Tile_Caching– An ambitious project could cache WMS servers at the

zoom levels and tiles set by a tile caching standard.

Next Steps for GeoSpatial WebNext Steps for GeoSpatial Web

• More on a Caching Catalog for Search– Could combine with WFS-Transactional, have

transactions send notifications, perhaps as GeoRSS, when something has been updated.

– Cache would listen and expire areas that were recently updated (could use KML expiration time)

– Already done with KML files for Google Maps– Solves one of the biggest problems of WMS - as a

dynamic protocol you can hose servers quite easily when they get popular

• Would incentivize registering on a spatial search engine since it would take off server load

GeoServer Advanced FeaturesGeoServer Advanced Features

• Support for Transactions (insert, update, delete) and Locking of backend data stores through open standards

• Validation engine, automatically reject transactions that don't meet set criteria

• Combine with Squid to do caching, for performance with tiling clients

• New modular, plug-in architecture based on Spring to easily add new services

• Easy to add new input and output formats with pure GeoTools API

Web Feature Service (WFS)Web Feature Service (WFS)

• Access to the raw data behind the map, the 'source code'.

• Can do analysis on the data, remix and recombine it with other sources, etc.

• Operations– GetCapabilities - what's on the server

– DescribeFeatureType - what properties does this featureType have

– GetFeature - get me the actual features, like a SQL select statement.

• Uses 'Filters' to delimit the features returned

• Default format is GML, but can be most anything, we return shapefiles

– Could return KML, but it'd be very boring since there's no style in WFS

WFS-Transactional (WFS-T)WFS-Transactional (WFS-T)

• Transactions - insert, update, and delete. Allows you to modify the backend database. Opens the door for collaborative mapping.

• Locking - LockFeature and GetFeatureWithLock. Prevents others from editing features that you are working on.

• Can build extensions on it– GeoServer's validation engine, to automatically reject

features if they don't pass rules.– Email and GeoRSS notifications– Limit access control with security modules

FilterFilter

• Think SQL 'Where' clauses - Like, Greater Than, Less Than or Equal To, Between, AND, OR, NOT, etc.

• Also has spatial predicates - Intersects, BBOX, Crosses, Touches, Disjoint, etc.

• FeatureID filter allows requesting a feature by ID

• Used in WFS GetFeature and in SLD. Extremely useful for limiting the number of features returned.

• Latest version has the ability to do 'joins' between different datasets

Geographic Markup Language Geographic Markup Language (GML)(GML)

• Default return type from WFS

• GML 2 very nice, indeed KML uses just about the same constructs

• GML 3 actually not so bad, but OGC did a poor job by not providing a clear migration path. Finally have that with Simple Features for GML. – New geometry types, to deal with 3d coordinates. – Modular design, so one can safely ignore most of the

spec, only import the needed XML Schemas– Meant to be combined with other schemas

SLD and FiltersSLD and Filters

• Filter is an independent specification, used in SLD and also in GetFeature WFS requests

– Think SQL 'Where' clauses - Like, Greater Than, Less Than or Equal To, Between, AND, OR, NOT, etc.

– Also has spatial predicates - Intersects, BBOX, Crosses, Touches, Disjoint, etc.

• With SLD allows one to make rules to render against properties of the dataset

• Includes 'Functions', divide, sqrt, cos, min, max, etc.– If number of unemployed divided by total population is less than .2 then render red

– Derive vectors of dataset, make an arrow with the strength of the wind

• Lots of power in SLD, just like CSS does for HTML. – GeoServer combines SLD plus native formats for Google Earth compatibility