Collaborative Open Access to Geospatial Analysis

Andrew TurnerChief Technology Officer & Neogeographer

@ajturnerandrew@geoiq.com

Hurricanes Katrina and RitaGeospatial Analysis with Long Production Cycles

flickr: gregorio

in 2005 we were active in the response to Hurricanes Katrina and Rita. Back then the request came to perform a flood analysis. Analysts worked in DC for a few days and sent their results in powerpoint slides to the operations center. They then asked “where are our shelters and response centers?” Churn away a few days and send that back down. By then they were knee deep in water. How do you change the paradigm to give the tools to access data and ask questions into the hands of the people that need it. Go from question to answer in under 5 minutes.

Crisis Reporting

Realtime emergent information sources

at the same time we saw the emergence of new forms of citizen communication and reporting. No voice and no data meant people sitting on rooftops would text family in Michigan to then call the Red Cross back in Louisiana. The typical organizations had little capability to quickly adapt to these new streams and incorporate it into their response.

Our Mission: On-demand realtime interactive maps

in the hands of everyone, exploration and collaboration

Today this is dramatically different. We’ve spent the past 6 years building tools and capability for anyone to quickly and effectively access their data and devise actionable information. By leveraging the web and modern, emergent technology and an open-architecture it’s now possible to provide advanced tools to everyone - opening the conversation to easily enable collaboration when it’s needed most.

Collaboration

flickr: chadcat

Access

Visualize

Analyze

we focus on three overall capabilitiesData: networked, unbounded, standards-based, community driven, open data.Visualization: We need ways to get through this data. What do I have? What is the shape and idea of it? How is it changing? What can I do with it?Analysis: In the end, I want to make decisions. To do that, I need to ask questions. I need tools to help me easily ask questions and find answers without having to worry about the “how” of the question asking. Just let me explore.

Cross-DomainShare KnowledgeMake Smarter Decisions... Collaboratively

Organizations

Research

MediaDevelopers

Enterprise

Citizens

a few GeoIQ features...• Data upload• Data Parsing• Web Feeds• Dataset Aggregation• Geocoding• Geojoin• Data Permissions• Group Management• Data Cleanup• Metadata management• Tagging• Data export• Dataset caching• External Relational Database integration• User Authentication• LDAP Integration• Thematic Visualization• Spatial, Tabular, Statistical Analysis• User-defined analysis

• Stream updating• Mobile web interface• Native mobile applications• Drawing and annotation• Geographic Maps• User defined basemaps• Charts (Pie, Line, etc.)• REST and Javascript API Integration• Facebook integration• OGC Web Services• PNG and PDF Map Export• Multisite Catalog Federation• White Label Customization• Featured Maps• External Analysis Integration• Stats and Tracking• Graceful Map Degradation

“GIS” in all but name

Easy explorationQuick data publication

Data management and search

Metadata extraction, statisticsOne-click downloadsBaked into the web: durable URL, link to sources,

Advanced analysis through easy, online interfaces

http://geocommons.com

15m+ maps served

100k+ unique datasetsbillions of features.

4m+ downloads

1m+ geographic attributes

Start now by joining our GeoCommons community of more than 25k GeoIQ users. You can readily find, visualize, and very soon analyze the public repository of 1m+ geographic attributes, billions of features.

Collaborative Data

= 499,999,500,000

Possible Comparative Analysis Combinations?

Collaborative Data

well... close enough

Mapping Health http://geocommons.com/maps/77282

Mapping Traffic + Caffeine http://geocommons.com/maps/82402

Mapping Zombie Outbreaks

Mapping Socially

An Open Stack

flickr: alfr3do

Streaming Services

Streaming Analysis

GeoiQ Social

REST and Javascript API

GeoiQ Explorer GeoiQ Mobile

Tabular

Spatial

Image

Custom Analysis Workers

Spatial Data

ETL

Spatial Data

Managem

ent

User

Managem

ent

Distributed Object Store GeoiQ Connect

MySQ

L

PostgreSQL

Oracle

Mongo

HB

ase

Geospatial

Search Index

Hadoop

Distributed Job Queues

External Analysis

HDFS

Federated Data Network

OpenStreetMap

Map R

endering Engine

Open-Source Stack

Streaming Services

Streaming Analysis

GeoiQ Social

REST and Javascript API

GeoiQ Explorer GeoiQ Mobile

Tabular

Spatial

Image

Custom Analysis Workers

Spatial Data

ETL

Spatial Data

Managem

ent

User

Managem

ent

Distributed Object Store GeoiQ Connect

MySQ

L

PostgreSQL

Oracle

Mongo

HB

ase

Geospatial

Search Index

Hadoop

Distributed Job Queues

External Analysis

HDFS

Federated Data Network

OpenStreetMap

Map R

endering Engine

ModestMaps))&)Polymaps)

jQuery'&''Backbone.js'

Ac#onScript+&+Appcelerator+

AMQP%&%Redis%

WebSockets*Socket.io*

Ruby%on%Rails%

Lucene&&&Solr& Mapnik'

Ruby%&%OSGeo%

HBase&

PostgreSQL+

Node.js(&(MongoDB(

Open-Source Stack

Open-ArchitectureOpen-Standards

•REST, GeoJSON •OpenSearch-Geo•GeoRSS, Atom, KML•OAuth, LDAP•OGC WFS, WMS, CSW

•Usable Interface•Web Services•Download button

flickr: dolinski

Open-Data

•OpenStreetMap•TIGER/Line•GeoNames•World Bank•UN•IADB•US Government •100k+ open-source

datasets

flickr: ajturner

GeoCommons GeoCoder

GeoCommons GeoCoder

We worked with the FCC to make this even easier. Enable Staffers and anyone to go from a Spreadsheet to a Map as quickly as possible. Then download or publish it for use in meeting, publications, and social networks.

We worked with the FCC to make this even easier. Enable Staffers and anyone to go from a Spreadsheet to a Map as quickly as possible. Then download or publish it for use in meeting, publications, and social networks.

We worked with the FCC to make this even easier. Enable Staffers and anyone to go from a Spreadsheet to a Map as quickly as possible. Then download or publish it for use in meeting, publications, and social networks.

issuemap.orggithub.com/geoiq/issuemap

We worked with the FCC to make this even easier. Enable Staffers and anyone to go from a Spreadsheet to a Map as quickly as possible. Then download or publish it for use in meeting, publications, and social networks.

Open Visualization Modest mapsPolymaps

Full user choice for quickly growing number of basemaps and providers. Each vary by cartographic style, data quality, language.

MapnikOpenStreetMapTileStache

developer.geoiq.com/tools/acetate/

We wanted to provide a basemap uniquely capable of visually attractive and understandable quantitative thematic visualizations. We also wanted to bring back a traditional concept of cartography by separating the layers to “Sandwich” it with the data. Together users can see the data and also the context of roads, places and other geographic data. Our UI makes this easy without having to think about it.

MapnikOpenStreetMapTileStache

developer.geoiq.com/tools/acetate/terrain

areas

roads

points

labels

We wanted to provide a basemap uniquely capable of visually attractive and understandable quantitative thematic visualizations. We also wanted to bring back a traditional concept of cartography by separating the layers to “Sandwich” it with the data. Together users can see the data and also the context of roads, places and other geographic data. Our UI makes this easy without having to think about it.

Spatial Operations

Buffer, Intersect, AggregationCorrelation, Prediction, Custom

Tools: geos, JTS, PostGIS

We want to go beyond simple visualization and allow people to ask deep questions of their data. So we built a number of common tabular, statistical, and geospatial analysis modules that are easy to use with your data.

Spa$al&Aggrega$on:+Group+data+by+arbitrary+geographic+boundaries

(Proprietary+and+confiden0al)

Spa$al&Correla$on:+Rela0onship+between+mul0ple+datasets+by+geography

(Proprietary+and+confiden0al)

Spa$al&Buffer&and&Distance&Filter:+Radius+buffer+from+a+point,+line+or+polygon

(Proprietary+and+confiden0al)

Intersec$on:+Overlap+between+two+polygonal+areas,+lines+or+points

Use&Cases:• Locate+that+a+social+media+men0ons+happened+within+a+geoCfence

• Track+mobile+traffic+entering+a+zip+code

(Proprietary+and+confiden0al)

User;Defined&Analysis:+Custom+equa0ons+via+intui0ve+wizard+interface

(Proprietary+and+confiden0al)

Traceable&Analyses:+Analysis+and+Filter+versions+can+be+tracked+back

All data and analyses are tracked and linked to their history and sources. Think “Github for Data”.

Realtime Social Media Indicatorshttp://geocommons.com/maps/51058

And we’re working on more than just traditional, static, analysis. We can analyze emergent and realtime streams of data.

Deploy to the Edgeflickr: oddboy

The “cloud” offers amazing capability to publish and share across the web. But many users need to work in remote and disconnected locations. Because we use open-source stack and our own technology we can easily deploy to any online or offline systems.

We’ve deployed servers to work offline in Jalalabad, Afghanistan as part of the #beer4data program that enables field collaboration of data sharing. The full offline stack can geocode, visualize and analyze data locally and optionally push the data up to the public web when there is connectivity.

Deploy to the Edges

GeoIQ Enterprise

GeoCommons

Enterprise Databases

Business Operations

GeoIQ Cloud Web!

Intranet!

Open-Source Deployment

Workers Data Store

Deployed as a Single VM

App Servers

Data Stores

Clustered Cloud Deployment

We use the same installers and virtual machine to on-demand deploy a new front-end server, or back-end analysis worker or data server.

Publish(to(mobile(

Make(a(map(in(GeoIQ(

Layer(Control( Filter((by(Loca;on( Field(Annota;ons(

Track(Field(Annota;ons((

Real(;me(upda;ng(

SelfCService(Mobile(App(Crea;on(in(Minutes(

And beyond scaling horizontally, we can push to the real edge of mobile devices. Users can create and publish a map in minutes that enables mobile access to maps and even realtime mobile collaboration.

Distributed Operations Search and Rescuehttps://vimeo.com/30435757

Combined together, these tools allow for on-demand access to emergent information, realtime streams, analysis, and field reporting.

Organizations

Research

MediaDevelopers

Enterprise

Citizens

By providing a common open platform for collaboration, members of many communities can contribute data and insight to reach common consensus.

Development Aid http://maps.worldbank.org

The World Bank is sharing the locations and impact of their projects throughout the world. The data is explorable and downloadable.

IADB MapAmericas http://www.iadb.org/mapamericas

The Inter-American Development Bank has launched a similar initiative linking project funding with on the ground outputs, photos, videos, and stories.

Climate Science WWF Climascope

The information that has traditionally been limited to expert only access is now in the hands of anyone to ask detailed important questions of quality scientific data. Citizens can ask about the impact of global phenomenon on their local environments.

Expert & Citizen Sourced Science Data

http://marine.tokelau.grida.no/

Organizations such as the United Nations Environmental Programme Grid-Arendal are gathering more scientific data and opening the tools for visualization and analysis to the world.

Social Governance

Beyond traditional science, government and organizations can ask relevant questions on urban planning and society by combining social media information. For example, investigating the crowding and ‘popularity’ of metro transit locations compared with location checkins.

Join the Communityhttp://geocommons.com

GeoCommons is the public repository to explore this data - or find out how to use GeoIQ within your organization to get better insight and information.

Collaborative Open Access to Geospatial Analysis

Andrew TurnerChief Technology Officer & Neogeographer

@ajturnerandrew@geoiq.com

Collaborative Open Access to Geospatial Analysis

Andrew TurnerChief Technology Officer & Neogeographer

@ajturnerandrew@geoiq.com

slideshare.net/ajturner github.com/geoiqgithub.com/geocommons