Analysing OpenStreetMap Data with QGIS

47
Analysing Analysing OpenStreetMap Data OpenStreetMap Data with QGIS with QGIS Jerry Jerry Clough Clough SK53 on OpenStreetMap @SK53onOSM [email protected]

description

Presentation to UK QGIS South East User Group, 2nd April 2014 at Imperial College London

Transcript of Analysing OpenStreetMap Data with QGIS

Page 1: Analysing OpenStreetMap Data with QGIS

Analysing Analysing OpenStreetMap Data OpenStreetMap Data

with QGISwith QGIS

JerryJerry CloughCloughSK53 on OpenStreetMap

@[email protected]

Page 2: Analysing OpenStreetMap Data with QGIS

My BackgroundMy Background

● Biologist, Computer Scientist, Management ConsultantNaturalist

● GIS--, DB++– OLAP platforms since late 1980s

● OSM since Dec 2008● QGIS since Jan 2011 (1.1 => 2.0)● Mainly analytical uses● Interests: landuse, landcover, biotopes, local

government open data, (pubs)

Page 3: Analysing OpenStreetMap Data with QGIS

OSM Need to KnowOSM Need to Know

● Open Vector Data● 3 Geo-primitives

– Node (= point)– Way (= linestring)

● Closed ways may represent areas

– Relations● More complex geothings

– Multipolygons– Geo-relations

● NO layers

● Volunteer Sourced– “Wiki map of the

world”

● Free Tagging– aka Folksonomy

● Variable Coverage–

Page 4: Analysing OpenStreetMap Data with QGIS

Some 'Interesting' Stats for GBSome 'Interesting' Stats for GB(with apologies to Ordnance Survey)

● Pylons: 58,487 (OSGB: 80,517)● Post Boxes: 42,742 (93.728) ● Camp sites: 3,192 (8,908) ● Buildings: 1,890,835 (35,397,754)● Bus Stops: 215,720 (354,099) ● Petrol Stations: (7,702)● Addresses: 27,341,262 (OSGB);

532,886● Electricity Poles: 94,199 (183, 987)● Road length: 522,627 km

(407,532 km)

● 5 post boxes with Edward VIII cypher

● Only 110 War Memorials● 847 Fire Hydrants● 1,378 Real Ale pubs

– 82 with Real Fires

● 4771 Cycle Parking● 300 Wildlife Hides● 5,552 Stiles● 1,774 Canal Locks● 2 Knitting Shops

Ordnance Survey figures: /www.ordnancesurvey.co.uk/blog/2013/04/10-fascinating-facts-from-ordnance-survey/OSM figures (April '13): /taginfo.openstreetmap.org.uk/

Page 5: Analysing OpenStreetMap Data with QGIS

How I use QGISHow I use QGIS

● OSM data => PostGIS DB● Initial analysis in QGIS● PostGIS routines for more complex data

manipulation● R and other tools for stats/segmentation● Visualisation in QGIS

Page 6: Analysing OpenStreetMap Data with QGIS

Case Study 1 : PubsCase Study 1 : Pubs

Page 7: Analysing OpenStreetMap Data with QGIS

Pub Density in Great BritainPub Density in Great Britain

Page 8: Analysing OpenStreetMap Data with QGIS

Cartograms based on PubsCartograms based on Pubs

Page 9: Analysing OpenStreetMap Data with QGIS

Cartograms based on PubsCartograms based on Pubs

Page 10: Analysing OpenStreetMap Data with QGIS

Case Study 2:Case Study 2:Simulating Urban AtlasSimulating Urban Atlas

● 300+ EU cities population >100k– 119 in April 2010– 228 in Sept. 2010

● Baseline date 2006-7● Used 2.5 m imagery● 5-6 year refresh cycle● Minimum Map Unit (MMU) 0.25 ha

urban / 1 ha rural

http://sia.eionet.europa.eu/Land Monitoring Core Service/Urban Atlas

Page 11: Analysing OpenStreetMap Data with QGIS

Examples of mapping OSM Tags Examples of mapping OSM Tags to Urban Atlas Categoriesto Urban Atlas Categories

UA Code

UA Description OSM Tags Comments

14100 Parks, Urban Green Space amenity=graveyardlanduse=cemeteryleisure=parkleisure=village_green

14200 Sports Areas landuse=allotmentslanduse=recreation_groundleisure=golf_courseleisure=pitchleisure=stadium

20000 Agricultural Land landuse=farmlanduse=farmlandlanduse=pasturelanduse=orchardlanduse=vineyardleisure=nature_reservenatural=scrub,natural=heathnatural=wetlandnatural=rock,natural=scree

Additional OSM tags are also valid for this code (e.g., natural=glacier)

30000 Woods & Forest natural=woodlanduse=forest

50000 Water landuse=reservoirwaterway=riverbanknatural=water

Page 12: Analysing OpenStreetMap Data with QGIS

Painter’s Algorithm in QGISPainter’s Algorithm in QGIS

Page 13: Analysing OpenStreetMap Data with QGIS
Page 14: Analysing OpenStreetMap Data with QGIS
Page 15: Analysing OpenStreetMap Data with QGIS

Case Study 3:Case Study 3:

Retail in OSMRetail in OSM

Page 16: Analysing OpenStreetMap Data with QGIS

Retail Geo-dataRetail Geo-data

DriversDrivers

–Personal interest• Used to consult to large retail chains & FMCG firm

–Article in Directions about Geolytix• Featured Nottingham, my main mapping location

– Availability of Food Hygiene Open Data

QuestionsQuestions– How difficult was it to systematically get retail landuse and retail

sites into OSM?

– Was OSM data good enough for segmentation of landuse?

Source: Geolytix in Directions Magazine

Page 17: Analysing OpenStreetMap Data with QGIS

FHRS 1

(local) Government Open Data

• Addresses

• Partial geolocation

– postcode

• Business Type

– Pub/Bar/Nightclub

– Supermarket

– Café/Restaurant

– Other Retail

• Covers at least 50-60% of retail outlets

• Usually current

– Typical inspection interval 6-12 months

Page 18: Analysing OpenStreetMap Data with QGIS

Tracking my ownTracking my ownOSM MappingOSM Mapping● Plot premises by postcode centroid

● OpenLayers plugin for background

● Track areas visited and added to OSM in Excel Spreadsheet

● S/s linked in as layer

● Update to show places to map

● Push un-surveyed postcodes out as a GPX

● Load GPX on Garmin

Page 19: Analysing OpenStreetMap Data with QGIS

Conclusions Nottingham Retail 2

Page 20: Analysing OpenStreetMap Data with QGIS

Conclusions Nottingham Retail 3

Page 21: Analysing OpenStreetMap Data with QGIS

Classifying Retail Areas

Page 22: Analysing OpenStreetMap Data with QGIS

Case Study 4 : Street LightsCase Study 4 : Street Lights

Page 23: Analysing OpenStreetMap Data with QGIS

Street Lights and OSM QualityStreet Lights and OSM Quality

Page 24: Analysing OpenStreetMap Data with QGIS

Street Lights and OSM QualityStreet Lights and OSM Quality

Page 25: Analysing OpenStreetMap Data with QGIS

Maps for DogsMaps for Dogs

Page 26: Analysing OpenStreetMap Data with QGIS

Approaches to using OSM DataApproaches to using OSM Data● Direct from OSM (API/ XML

files)– Earlier Plugin (deprecated)

– 2.0 method

– ogr2ogr

● via Postgres DB– osm2pgsql

– osmosis

– imposm

– osm2postgresql

– osm2pgrouting

● via Shapefiles– Geofabrik

● Limited number of layers

● Limited sets of attributes

– Roll your own

http://wiki.openstreetmap.org/wiki/Osmosis http://wiki.openstreetmap.org/wiki/Osm2postgresqlhttp://sourceforge.net/projects/osm2postgresql/http://download.geofabrik.de/

Page 27: Analysing OpenStreetMap Data with QGIS

Postgre-SQL/GIS and osm2pgsqlPostgre-SQL/GIS and osm2pgsql● osm2pgsql converts osm

data to postgres/postgis– Slightly lossy

● Relationship between members of multipolygons

● Road and other network topologies

– Can choose projection ● default 3087

– Can tweak import rules● Style files● LUA

– Fiddly under Windows

● osmconvert & osmfilter– Very useful tools to preprocess

data for particular purposes● Filter on OSM tag values● Convert polygons to centroids

● ALWAYS USE -k option – Stores less widely used tags as

an hstore column– Maximises flexibility– Throws away coastline by

default (sometimes useful to keep it)

http://wiki.openstreetmap.org/wiki/Osm2pgsqlhttp://wiki.openstreetmap.org/wiki/Osmconverthttp://wiki.openstreetmap.org/wiki/Osmfilter

Page 28: Analysing OpenStreetMap Data with QGIS

ProblemsProblems

● Polygon Handling● Generalisation● Missing data● Free-form Tagging

Page 29: Analysing OpenStreetMap Data with QGIS

The Problem with PolygonsThe Problem with Polygons

• No Area primitive in OSM• Overlapping polygons• OSM

– Broken polygons– Intersecting polygons

– osm2pgsql

• In QGIS

– Render OK– Geometry Operations fail

• Essential tool: cleangeometry PostGIS function (SOGIS)

http://www.sogis1.so.ch/sogis/dl/postgis/cleanGeometry.sql

Page 30: Analysing OpenStreetMap Data with QGIS

GeneralisationGeneralisation

• Multiple Ways– Most objects will be formed

from many OSM ways (e.g, Thames, M4)

• No simplified data– Dual carriageways– Roundabouts and flares– Built-up areas – Over noded for many uses

• Fine-grain tagging

• May require elaborate pre-processing

Page 31: Analysing OpenStreetMap Data with QGIS

Tagging IssuesTagging Issues• Synonymy

– natural=wood

– landuse=forest

• Variable Semantics

– highway=path

– place=hamlet

– highway=trunk (gets changed every now & then)

• Tagging for the Render

– natural=sand for Golf bunker

– landuse=grass Everywhere

• Semantic Degradation

– Tag with accepted semantics being used for something else

– landuse=recreation_ground for Ski areas in US

• Odd names

– shop=mall Shopping Centre

Page 32: Analysing OpenStreetMap Data with QGIS

Incomplete DataIncomplete Data

Page 33: Analysing OpenStreetMap Data with QGIS

Other things I do in QGISOther things I do in QGIS

● Vice County maps using OSGB Open Data– Plan to investigate Atlas module now

● Distribution Maps of Trees in N. Hemisphere● Attempts to analyse suburban structure based

on building dates– Used Portland Oregon data

– Huge Delauney triangulation

Page 34: Analysing OpenStreetMap Data with QGIS

ConclusionsConclusions● QGIS fantastic tool for a wide range of manipulations of

OpenStreetMap data– Particularly well suited for

● Prototyping & visualisation● Combining with other Open Data sources

● Recommend use with PostGIS– Maximises flexibility

– Reduces complexity of potential learning curve for the OSM toolchain

– Ability to manipulate data in PostGIS may be important

● Be aware of limitations and gotchas of OSM data

Page 35: Analysing OpenStreetMap Data with QGIS

Supplementary SlidesSupplementary Slides

● Managing polygons for detailed analysis (Urban Atlas)

Page 36: Analysing OpenStreetMap Data with QGIS

PostGIS ProcessingOSM

Polygons

OSMLines

Painter'sAlgorithm

Rules

ClippedPolygons

ClippedLines

Cleaned &Clipped

Polygons

UA ShapePolygons

Clean GeometryGridded UA

ClassesFilter on Tags & Grid

Gridded &Buffered

UA ClassesTag Filter, Grid & Buffer

Clip to Area

Clip to Area

Piecewise Union Union Step 1

Un

ion

Union Step 2

Me

rge

Class GriddedPolygons

Merge

GridGridded UAPolygons

UnionClipping areasby UA Class

Clip

pin

g R

eg

ion

FinalPolygons

CompareUA/OSM

Union/Intersect/Difference

Page 37: Analysing OpenStreetMap Data with QGIS

Comparison 1

No OSM Data

Residential

Disagreement

Agreement

Nottingham Area

Page 38: Analysing OpenStreetMap Data with QGIS

Comparison 2

No OSM Data

Residential

Disagreement

Agreement

Page 39: Analysing OpenStreetMap Data with QGIS
Page 40: Analysing OpenStreetMap Data with QGIS
Page 41: Analysing OpenStreetMap Data with QGIS

Agreement

Page 42: Analysing OpenStreetMap Data with QGIS

Supplementary SlidesSupplementary Slides

● Examples of OSM Mapping from Port-au-Prince January 2010

Page 43: Analysing OpenStreetMap Data with QGIS
Page 44: Analysing OpenStreetMap Data with QGIS
Page 45: Analysing OpenStreetMap Data with QGIS
Page 46: Analysing OpenStreetMap Data with QGIS
Page 47: Analysing OpenStreetMap Data with QGIS