Beyond arcpy, Python for GIS, Grant Miller-Francisco

Beyond arcpyGIS in ActionMarch 29-30, 2011

Grant Miller-FranciscoGIS AnalystSky Research Ashland, OR

arcpyArcGIS Python

{‘H3574’: ‘C:\path\H3574.jpg’,

‘H3575’: ‘C:\path\H3575.jpg’,

‘H3623’: ‘C:\path\H3623.jpg’,

. . . }

{‘H3574’: ‘C:\path\H3574.jpg’,

‘H3575’: ‘C:\path\H3575.jpg’,

‘H3623’: ‘C:\path\H3623.jpg’,

. . . }

Yak shaving

pyscripter (http://code.google.com/p/pyscripter)

Wingware (https://wingware.com/)

The Standard Library

# handling paths with os.pathimport os

f = r‘c:\data\somefile.txt’

base = os.path.basename(f)> ‘somefile.txt’

directory = os.path.dirname(f)> ‘c:\\data’

full_path = os.path.join(directory, base)> ‘c:\\data\\somefile.txt’

# get list of files with glob

from glob import glob

shps = glob(r‘c:\somedir\*.shp’)

> [‘C:\\somedir\\shp1.shp’, ‘C:\\somedir\\shp2.shp’, …]

Other useful standard library modules:

• csv• shutil• datetime• math• decimal• re• itertools

Python Module of the Weekhttp://www.doughellmann.com/PyMOTW/

Installing 3rd Party Python Packages

The Python Package Index(PyPI)

http://pypi.python.org/pypi

setuptools / easy_install

>c:\Python26\Scripts> easy_install pip

pip>pip install antigravity

>pip install antigravity==1.0.4

# use ==, >=, >, <, <=

>pip install ––upgrade antigravity

>pip uninstall antigravity

>pip search “kml”

>pip freeze > c:\requirements.txt

>pip install –r c:\requirements.txt

virtualenv – creates isolated Python environments

Yak shaving

That wasn’t so bad!

Public Art near PSU

(http://www.civicapps.org/datasets)

tablib

import arcpy, csv

src = r'C:\GIS_in_Action.gdb\public_art_selected'dest = r'C:\data\public_art_selected.csv‘

with open(dest, 'w') as f: w = csv.writer(f, quoting=1, lineterminator='\n') fieldlist = arcpy.ListFields(src) ignorefields = ['OBJECTID', 'Shape', 'detail_url'] fieldnames = [field.name for field in fieldlist if field.name not

in ignorefields]

w.writerow(fieldnames) rows = arcpy.SearchCursor(src) for row in rows: outputrow = [] for fieldname in fieldnames: outputrow.append(str(row.getValue(fieldname))) w.writerow(outputrow)

import tablib

src = r'C:\data\public_art_selected.csv'

contents = []

with open(src, 'r') as f: for line in f: contents.append(tuple(line.rstrip('\n').split(',')))

data = tablib.Dataset(*contents[1:], headers=tuple(contents[0]))

import tablib

src = r'C:\data\public_art_selected.csv'

contents = []

with open(src, 'r') as f: for line in f: contents.append(tuple(line.rstrip('\n').split(',')))

data = tablib.Dataset(*contents[1:], headers=tuple(contents[0]))

print data.csv

artist,title,street,lat,lng,image_url,record_id

Steven Gillman,Peace Chant,SW Park Ave. and SW Columbia St.,45.514946,-122.683125,http://data.racc.org/pa_inventory/0657/0657.jpg,789

Paul Sutinen,In the Shadow of the Elm,SW Park Ave. and SW Market St.,45.513751,122.684352,http://data.racc.org/pa_inventory/0658/06 . . .

Oliver Barrett,Rebecca at the Well (Shemanski Fountain),SW Park Ave. and SW Main St.,45.516939,-122.682039,http://data.racc.org/pa_in. . .

James Carpenter,Spectral Light Dome,1111 SW Broadway,45.516623,-122.681327,http://data.racc.org/pa_inventory/0898/0898.jpg,966

Edo Period,Kano School Japanese Screen,1111 SW Broadway,45.516623,-122.681327,http://data.racc.org/pa_inventory/0906/0906.jpg,972

Linda Ethier,Muse Maze Mirror Gaze,1111 SW Broadway,45.516623,-122.681327,http://data.racc.org/pa_inventory/0989/0989.jpg,1065

Henk Pander,Portland Town,1111 SW Broadway,45.516623,-122.681327,http://data.racc.org/pa_inventory/0990/0990.jpg,1066

Tom Hardy,Running Horses,SW 6th Ave. and SW Madison St.,45.515729,-122.680413,http://data.racc.org/pa_inventory/1071/1071.jpg,1185

Alexander Phimister Proctor,Theodore Roosevelt Rough Rider,SW Park Ave. and SW Jefferson St.,45.515738,-122.683275,http://data.r . . .

George Fite Waters,Abraham Lincoln,SW Park Ave. and SW Madison St.,45.516406,-122.682923,http://data.racc.org/pa_inventory/0660/066 . . .

Juan Alonso,Diva,1900 SW 4th Ave,45.509748,-122.68105,http://data.racc.org/pa_inventory/1261/1261.jpg,1674

Robert Calvo,Rose City Labyrinth,1900 SW 4th Ave,45.509748,-122.68105,http://data.racc.org/pa_inventory/1264/1264.jpg,1701

Zhao Suikang,Portland Pamphlets,1900 SW 4th Ave,45.509748,-122.68105,http://data.racc.org/pa_inventory/1266/1266.jpg,1705

Alejandro Colunga,Mago Hermano (Brother Wizard or Magician),1111 SW Broadway,45.516623,-122.681327,http://data.racc.org/pa_invento . . .

Donald Wilson,Holon,1525 SW Park Ave.,45.514043,-122.684217,http://data.racc.org/pa_inventory/1492/1492.jpg,2176

Cynthia Lahti,Station 4,511 SW College Street,45.509734,-122.682872,http://data.racc.org/pa_inventory/1550/1550.jpg,2278

Robert Yoder,Sweet Air Bend,1900 SW 4th Ave,45.509748,-122.68105,http://data.racc.org/pa_inventory/1262b/1262b.jpg,2518

Robert Yoder,Sweet Air Bend,1900 SW 4th Ave,45.509748,-122.68105,http://data.racc.org/pa_inventory/1262c/1262c.jpg,2519

Robert Yoder,Sweet Air Bend,1900 SW 4th Ave,45.509748,-122.68105,http://data.racc.org/pa_inventory/1262d/1262d.jpg,2520

Patti Warashina,City Reflections,SW 6th Ave. and SW Main St.,45.516409,-122.680049,http://data.racc.org/pa_inventory/1872/1872.jpg,2881

Jack Archibald,Icarus Triumphant,1510 SW 13th Avenue,45.515271,-122.688128,http://data.racc.org/pa_inventory/1837/1837.jpg,2937

Keiko Hara,Verses - Reflected and Reflecting,1800 SW 6th Ave Suite 550,45.511733,-122.682559,http://data.racc.org/pa_inventory/18 . . .

artist title street lat lng

Steven Gillman Peace Chant SW Park Ave. and SW Columbia St. 45.514946 -122.683125

Paul Sutinen In the Shadow of the Elm SW Park Ave. and SW Market St. 45.513751 -122.684352

Oliver Barrett Rebecca at the Well (Shemanski Fountain) SW Park Ave. and SW Main St. 45.516939 -122.682039

James Carpenter Spectral Light Dome 1111 SW Broadway 45.516623 -122.681327

Edo Period Kano School Japanese Screen 1111 SW Broadway 45.516623 -122.681327

Linda Ethier Muse Maze Mirror Gaze 1111 SW Broadway 45.516623 -122.681327

Henk Pander Portland Town 1111 SW Broadway 45.516623 -122.681327

Tom Hardy Running Horses SW 6th Ave. and SW Madison St. 45.515729 -122.680413

Alexander Phimister Proctor Theodore Roosevelt Rough Rider SW Park Ave. and SW Jefferson St. 45.515738-122.683275

George Fite Waters Abraham Lincoln SW Park Ave. and SW Madison St. 45.516406 -122.682923

Juan Alonso Diva 1900 SW 4th Ave 45.509748 -122.68105

Robert Calvo Rose City Labyrinth 1900 SW 4th Ave 45.509748 -122.68105

Zhao Suikang Portland Pamphlets 1900 SW 4th Ave 45.509748 -122.68105

Alejandro Colunga Mago Hermano (Brother Wizard or Magician) 1111 SW Broadway 45.516623 -122.681327

Donald Wilson Holon 1525 SW Park Ave. 45.514043 -122.684217

Cynthia Lahti Station 4 511 SW College Street 45.509734 -122.682872

Robert Yoder Sweet Air Bend 1900 SW 4th Ave 45.509748 -122.68105

Patti Warashina City Reflections SW 6th Ave. and SW Main St. 45.516409 -122.680049

Jack Archibald Icarus Triumphant 1510 SW 13th Avenue 45.515271 -122.688128

Keiko Hara Verses - Reflected and Reflecting 1800 SW 6th Ave Suite 550 45.511733 -122.682559

print data.tsv

print data.json

[{"artist": "Steven Gillman", "title": "Peace Chant", "street": "SW Park Ave. and SW Columbia St.", "lat": "45.514946", "lng": "-122.683125", "image_url": "http://data.racc.org/pa_inventory/0657/0657.jpg", "record_id": "789"}, {"artist": "Paul Sutinen", "title": "In the Shadow of the Elm", "street": "SW Park Ave. and SW Market St.", "lat": "45.513751", "lng": "-122.684352", "image_url": "http://data.racc.org/pa_inventory/0658/0658.jpg", "record_id": "790"}, {"artist": "Oliver Barrett", "title": "Rebecca at the Well (Shemanski Fountain)", "street": "SW Park Ave. and SW Main St.", "lat": "45.516939", "lng": "-122.682039", "image_url": "http://data.racc.org/pa_inventory/0661/0661.jpg", "record_id": "791"}, {"artist": "James Carpenter", "title": "Spectral Light Dome", "street": "1111 SW Broadway", "lat": "45.516623", "lng": "-122.681327", "image_url": "http://data.racc.org/pa_inventory/0898/0898.jpg", "record_id": "966"}, {"artist": "Edo Period", "title": "Kano School Japanese Screen", "street": "1111 SW Broadway", "lat": "45.516623", "lng": "-122.681327", "image_url": "http://data.racc.org/pa_inventory/0906/0906.jpg", "record_id": "972"}, {"artist": "Linda Ethier", "title": "Muse Maze Mirror Gaze", "street": "1111 SW Broadway", "lat": "45.516623", "lng": "-122.681327", "image_url": "http://data.racc.org/pa_inventory/0989/0989.jpg", "record_id": "1065"}, {"artist": "Henk Pander", "title": "Portland Town", "street": "1111 SW Broadway", "lat": "45.516623", "lng": "-122.681327", "image_url": "http://data.racc.org/pa_inventory/0990/0990.jpg", "record_id": "1066"}, {"artist": "Tom Hardy", "title": "Running Horses", "street": "SW 6th Ave. and SW Madison St.", "lat": "45.515729", "lng": "-122.680413", "image_url": "http://data.racc.org/pa_inventory/1071/1071.jpg", "record_id": "1185"}, {"artist": "Alexander Phimister Proctor", "title": "Theodore Roosevelt Rough Rider", "street": "SW Park Ave. and SW Jefferson St.", "lat": "45.515738", "lng": "-122.683275", "image_url": "http://data.racc.org/pa_inventory/0659/0659.jpg", "record_id": "1192"}, {"artist": "George Fite Waters", "title": "Abraham Lincoln", "street": "SW Park Ave. and SW Madison St.", "lat": "45.516406", "lng": "-122.682923", "image_url": "http://data.racc.org/pa_inventory/0660/0660.jpg", "record_id": "1323"}, {"artist": "Juan Alonso", "title": "Diva", "street": "1900 SW 4th Ave", "lat": "45.509748", "lng": "-122.68105", "image_url": "http://data.racc.org/pa_inventory/1261/1261.jpg", "record_id": "1674"}, {"artist": "Robert Calvo", "title": "Rose City Labyrinth", "street": "1900 SW 4th Ave", "lat": "45.509748", "lng": "-122.68105", "image_url": "http://data.racc.org/pa_inventory/1264/1264.jpg", "record_id": "1701"}, {"artist": "Zhao Suikang", "title": "Portland Pamphlets", "street": "1900 SW 4th Ave", "lat": "45.509748", "lng": "-122.68105", "image_url": "http://data.racc.org/pa_inventory/1266/1266.jpg", "record_id": "1705"}, {"artist": "Alejandro Colunga", "title": "Mago Hermano (Brother Wizard or Magician)", "street": "1111 SW Broadway", "lat": "45.516623", "lng": "-122.681327", "image_url": "http://data.racc.org/pa_inventory/1481/1481.jpg", "record_id": "2152"}, {"artist": "Donald Wilson", "title": "Holon", "street": "1525 SW Park Ave.", "lat": "45.514043", "lng": "-122.684217", "image_url": "http://data.racc.org/pa_inventory/1492/1492.jpg", "record_id": "2176"}, {"artist": "Cynthia Lahti", "title": "Station 4", "street": "511 SW College Street", "lat": "45.509734", "lng": "-122.682872", "image_url": "http://data.racc.org/pa_inventory/1550/1550.jpg", "record_id": "2278"}, {"artist": "Robert Yoder", "title": "Sweet Air Bend", "street": "1900 SW 4th Ave", "lat": "45.509748", "lng": "-122.68105", "image_url": "http://data.racc.org/pa_inventory/1262b/1262b.jpg", "record_id": "2518"}, {"artist": "Robert Yoder", "title": "Sweet Air Bend", "street": "1900 SW 4th Ave", "lat": "45.509748", "lng": "-122.68105", "image_url": "http://data.racc.org/pa_inventory/1262c/1262c.jpg", "record_id": "2519"}, {"artist": "Robert Yoder", "title": "Sweet Air Bend", "street": "1900 SW 4th Ave", "lat": "45.509748", "lng": "-122.68105", "image_url": "http://data.racc.org/pa_inventory/1262d/1262d.jpg", "record_id": "2520"}, {"artist": "Patti Warashina", "title": "City Reflections", "street": "SW 6th Ave. and SW Main St.", "lat": "45.516409", "lng": "-122.680049", "image_url": "http://data.racc.org/pa_inventory/1872/1872.jpg", "record_id": "2881"}, {"artist": "Jack Archibald", "title": "Icarus Triumphant", "street": "1510 SW 13th Avenue", "lat": "45.515271", "lng": "-122.688128", "image_url": "http://data.racc.org/pa_inventory/1837/1837.jpg", "record_id": "2937"}, {"artist": "Keiko Hara", "title": "Verses - Reflected and Reflecting", "street": "1800 SW 6th Ave Suite 550", "lat": "45.511733", "lng": "-122.682559", "image_url": "http://data.racc.org/pa_inventory/1838/1838.jpg", "record_id": "3018"}]

open(r'C:\data\public_art_selected.xls', 'wb').write(data.xls)

artist title street lat lngSteven Gillman Peace Chant SW Park Ave. and SW Columbia St. 45.514946 -122.683125Paul Sutinen In the Shadow of the Elm SW Park Ave. and SW Market St. 45.513751 -122.684352Oliver Barrett Rebecca at the Well (Shemanski Fountain) SW Park Ave. and SW Main St. 45.516939 -122.682039James Carpenter Spectral Light Dome 1111 SW Broadway 45.516623 -122.681327Edo Period Kano School Japanese Screen 1111 SW Broadway 45.516623 -122.681327Linda Ethier Muse Maze Mirror Gaze 1111 SW Broadway 45.516623 -122.681327Henk Pander Portland Town 1111 SW Broadway 45.516623 -122.681327Tom Hardy Running Horses SW 6th Ave. and SW Madison St. 45.515729 -122.680413Alexander Phimister Proctor Theodore Roosevelt Rough Rider SW Park Ave. and SW Jefferson St. 45.515738 -122.683275George Fite Waters Abraham Lincoln SW Park Ave. and SW Madison St. 45.516406 -122.682923Juan Alonso Diva 1900 SW 4th Ave 45.509748 -122.68105Robert Calvo Rose City Labyrinth 1900 SW 4th Ave 45.509748 -122.68105Zhao Suikang Portland Pamphlets 1900 SW 4th Ave 45.509748 -122.68105Alejandro Colunga Mago Hermano (Brother Wizard or Magician) 1111 SW Broadway 45.516623 -122.681327Donald Wilson Holon 1525 SW Park Ave. 45.514043 -122.684217Cynthia Lahti Station 4 511 SW College Street 45.509734 -122.682872Robert Yoder Sweet Air Bend 1900 SW 4th Ave 45.509748 -122.68105Robert Yoder Sweet Air Bend 1900 SW 4th Ave 45.509748 -122.68105Robert Yoder Sweet Air Bend 1900 SW 4th Ave 45.509748 -122.68105Patti Warashina City Reflections SW 6th Ave. and SW Main St. 45.516409 -122.680049Jack Archibald Icarus Triumphant 1510 SW 13th Avenue 45.515271 -122.688128Keiko Hara Verses - Reflected and Reflecting 1800 SW 6th Ave Suite 550 45.511733 -122.682559

tabular

import tabular

src = r'C:\data\public_art_selected.csv‘table = tabular.tabarray(SVfile=src)

table.sort(order=['street'])

print table[table['artist'] == 'Robert Yoder']

pivot_table = table.pivot('street', 'record_id')['artist']

pivot_table.saveSV(r'C:\data\public_art_selected_pivot.csv')

Shapely

“set-theoretic analysis and manipulation of planar features”

Create: Point, LineString, LinearRing, PolygonsMultiPoint, MultiLineString, MultiPolygon

General Attributes/Methods.area, .length, .bounds, geom_type, .distance (to another object)

Binary Predicatescontains, crosses, disjoint, equals, intersects, touches, within

set-theoretic methods.boundary, .centroid, .difference, .intersection, .symmetric_difference, .union

constructive method.buffer, .convex_hull, .envelope, .simplify

othermerging, cascading union

InteroperationWKT, numpy arrays, geo interface (GeoJSON)

import arcpyfrom shapely.geometry import MultiPoint, asShape, mapping

fc = r'C:\GIS_in_Action.gdb\public_art_selected‘

g = arcpy.Geometry()geometryList = arcpy.CopyFeatures_management(fc, g)mp = []

for item in geometryList: mp.append(asShape(item.__geo_interface__))points = MultiPoint(mp)

out_fc = r'C:\GIS_in_Action.gdb\public_art_selected_convex_hull‘

ch = points.convex_hullbuffered_ch = ch.buffer(100)

points_convex_hull = arcpy.AsShape(mapping(buffered_ch))arcpy.CopyFeatures_management(points_convex_hull, ch_fc)

prjFile = ‘path_to_prj_file’spatialRef = arcpy.SpatialReference(prjFile) arcpy.DefineProjection_management(out_fc, spatialRef)

Use Shapely in conjunction with pyproj for projected coordinates

gdal/ogr

from osgeo import gdal

dataset = gdal.Open(r'C:\data\rasters\LiDAR_be.tif')

cols, rows = dataset.RasterXSize, dataset.RasterYSizebands = dataset.RasterCountprint 'Columns: %i, Rows: %i, Bands: %i' % (cols, rows, bands)

Columns: 1914, Rows: 1858, Bands: 1

band = dataset.GetRasterBand(1)

print 'Band Type =',gdal.GetDataTypeName(band.DataType)min, max = band.GetMinimum(), band.GetMaximum()print 'Min=%.3f, Max=%.3f' % (min, max)

Band Type = Float32Min=10.094, Max=1000.430

from osgeo import ogr

shapefile = ogr.Open(r'C:\data\shps\public_art_selected.shp')layer = shapefile.GetLayer()

for i in range(layer.GetFeatureCount()): feature = layer.GetFeature(i) title = feature.GetField('title') geometry = feature.GetGeometryRef() print i, title, geometry.GetGeometryName()

0, Peace Chant ,POINT1, In the Shadow of the Elm ,POINT2, Rebecca at the Well (Shemanski Fountain) ,POINT3, Spectral Light Dome ,POINT4, Kano School Japanese Screen ,POINT5, Muse, Maze, Mirror, Gaze ,POINT6, Portland Town ,POINT7, Running Horses ,POINT

0, Peace Chant ,POINT1, In the Shadow of the Elm ,POINT2, Rebecca at the Well (Shemanski Fountain) ,POINT3, Spectral Light Dome ,POINT4, Kano School Japanese Screen ,POINT5, Muse, Maze, Mirror, Gaze ,POINT6, Portland Town ,POINT7, Running Horses ,POINT. . .

reportlab

public_art.csv

tablib

for row in data: # put in PDF

# Setting up the PDF

from reportlab.pdfgen.canvas import Canvasfrom reportlab.lib.pagesizes import letterfrom reportlab.lib.styles import getSampleStyleSheet

pdfFile = r'somedir/somefile.pdf'pdf = Canvas(pdfFile, pagesize=letter)pdf.showPage()pdf.save()

styles = getSampleStyleSheet()styleMRA = styles['Normal']pdf.setFont("Helvetica",24)

# Drawing boxes, images, and text

from reportlab.lib.units import inchfrom reportlab.lib.utils import ImageReader

pdf.rect(x, y, width, height, stroke=1, fill=0)# measurements are from lower left corner# use dist*inch

pdf.drawAlignedString(x, y, text)pdf.drawCentredString(x, y, text) #note British spellingphoto = ImageReader(url)pdf.drawImage(image, x, y, width, height, mask, preserveAspectRatio)

# Flowables

from reportlab.platypus import Paragraph, Framefrom reportlab.platypus.flowables import Spacer

text = []text.append(Paragraph(text, styleMRA))text.append(Spacer(0.1*inch, 0.1*inch))

f = Frame(x, y, width, height, leftPadding, bottomPadding, rightPadding, topPadding)

f.addFromList(text, pdf)