GEOGRAPHIC SCRIPTING IN GVSIGHALFWAY BETWEEN USER AND DEVELOPER

Geoinformation Research Group,Department of Geography

University of Potsdam

21-25 November 2016

Andrea Antonello

PART 3: THE SCRIPTING COMPOSER AND PYTHON

THE SCRIPTING COMPOSER

Scripting is one of the powerful things in GIS.

Non developers have the possibility to create great automatisms throughit.

gvSIG's scripting composer is based on the (through jython).

python scripting language

THE SCRIPTING COMPOSERTo open the scripting composer access the Tools -> Scripting menu:

THE SCRIPTING COMPOSEROnce opened, it shows up like:

new script

THE SCRIPTING COMPOSEROnce opened, it shows up like:

newscript

savescript

cutcopypaste

localscriptslibrary

editor

output consoleand error view

run or saveand run

syntax check*

*if Composer Tools installed

THE SCRIPTING COMPOSERRunning and checking errors

run

output

errorconsole

If you run the script by hitting the run button, the editor will not save thescript for you (CTRL-S). It does if you instead use the save+run button.

THE BASE LANGUAGE: PYTHON!In order to be able exploit 100% the scripting engine, it is mandatory toget a good insight of the base language that can be used in the editor.

Before we start with the geo part, we will investigate a bit python.

Let's create a new script to do so. Let's call it python_essentials.py and letit print out a simple message to make sure it works.

def main(*args):

print "Python essentials" print "*****************"

PYTHON: FORMATTINGIn python formatting is mandatory. It is the formatting that defines thelogical blocks.

For example this will not work:

def main(*args):

print "Python essentials" print "*****************"

and produce an error like:

IndentationError: unindent does not match any outer indentation level in<script> at line number 4 at column number 1 python_essentials 4:1

PYTHON: WRITING STRINGSWe have seen writing of strings already:

# write a string print "This is a string"

lines that begin with a hash at the begin are ignored -> comments

Comments can be placed without formatting.

Also more complex strings can be used, by surrounding text with triplequotes:

# write a complex string print """Inside triple quotes you can do almost anything.It is kind of an advanced string!"""

And also kind of defeat the laws of formatting.

PYTHON: VARIABLESVariables definition is very simple. Just write a name and assign a value:

# define variables # a string name = "Luis" # a number age = 25 # insert the variables into a string template # through the use of %s, %f, %i myTemplate1 = "Hi, my name is %s and I'm %i." % (name, age) print myTemplate1 # strings and numbers can also be concatenated through the + sign myTemplate2 = "Hi, my name is " + name + " and I'm " + str(age) + "." print myTemplate2

Note that string concatenation with the + (plus) sign works only withstring. Hence, the age variable has first to be converted to a stringthrough the str(...) function.

PYTHON: WORKING WITH STRINGPython allows for a whole pile of operations on string. Let's focus on

those we will use later:

# strip whitespaces string = " a line of text with spaces " print string.lstrip() print string.rstrip() print string.strip()

# split a string string = "a, comma, separated, text" print string.split(",")

# starts/ends with string = "a b c" print string.startswith("a") print string.endswith("a")

# to upper/lowercase string = "A b C" print string.upper() print string.lower()

# substrings string = "123456" print string[2:] print string[:-2] print string[2:-2]

# find in strings (0 indexed) print string.find("2") print string.find("22") print string.find("345")

# find after certain index print string.find("345", index)

# replace s = "Hola world" print s.replace("world", "mundo")

PYTHON: FORMATTING NUMBERSWhen printing out numbers, they can be formatted as follows

print "Formatting numbers" oneThird = 1.0/3.0 print "%f" % oneThird print "%i" % oneThird print "%2.2f" % oneThird print "%5.2f" % oneThird print "%2.1f" % oneThird print "%+2.2f" % oneThird

%[width][.precision]type

EXTRACT DATA FROM THE WEBIn python it is very simple to get data from the web, all that is necessary isthe import of the urllib and a few lines of code.

For example, let's print out the tiobe index that defines the ranking of theprogramming languages:

import urllib

def main(*args): connection = urllib.urlopen("http://www.tiobe.com/tiobe-index/") htmlString = connection.read() print htmlString

EXTRACT DATA FROM THE WEBAs an exercise let's extract the table of the ranking of the programming

languages using what we learned until now:

index = htmlString.find('TIOBE Index for November 2016')

# extract the header startHeader = htmlString.find('<thead', index) endHeader = htmlString.find('</thead', startHeader) tableHeader = htmlString[startHeader:endHeader] tableHeader = tableHeader.replace("<thead>","").replace("</thead>","") \ .replace("<th>","").replace("</th>","").replace("<tr>","") \ .replace("</tr>","").replace("\n"," | ").strip()

print tableHeader

startData = htmlString.find('<tbody', index) endData = htmlString.find('</tbody', startData) tableData = htmlString[startData:endData] tableData = tableData.replace("\n"," | ").replace("<tbody>","").replace("</tbody>","") \ .replace("<td>","").replace("</td>","|").replace("<tr>","") \ .replace("</tr>","\n|").strip() print tableData

PYTHON: TYPE AND DIR

type and dir can be two valuable friends in moments ofdesperation:

type: returns the type of the passed objectdir: returns the list of valid attributes and methodsof the object passed

string = "A string" print type(string) print dir(string)

PYTHON: LISTSLists are containers for objects and are defined through square brackets:

# working with lists # create a list of strings list = ["Merano", "Bolzano", "Trento"] print list # how to access the elements? print "The elements start at position 0: " + list[0] # what if you use negative indexes? print "Element at position -1: " + list[-1] # how to add an element to the end of the list? list.append("Potsdam") print list # how to add an element as first element? list.insert(0, "Berlin") print list # how to remove an element from the list? # by object? list.remove("Potsdam") print list # or by index through the del statement del list[0] print list # how many elements are in the list? print "Elements count: %s" % len(list)

PYTHON: LOOPING THROUGH LISTS

# looping through lists list = ["Merano", "Bolzano", "Trento"] # to loop over a list a for loop can be used for value in list: print value # if you also need an index for index, value in enumerate(list): print "%s is town N.%i" % (value, index)

PYTHON: SORTING LISTS # sorting lists list = ["Merano", "Bolzano", "Trento"] print "This is the original list: " + str(list)

# lists can be sorted list.sort() print "This is the sorted list: " + str(list)

# and reverse sorted list.reverse() print "This is reverse list: " + str(list)

To print lists concatenated with strings we use the str(...) function.

PYTHON: FILTERING LISTS # lists and numbers list = [12, 5, 7, 1, 2, 3] print "This is the original list: " + str(list)

# calculate min, max and sum print "max is %i, min is %i, sum is %i" % (max(list), min(list), sum(list))

# filter based on conditions # create the filter myFilter = lambda x: x > 5.5

# apply it filteredData = filter(myFilter, list) print filteredData

The filter applies the lambda expression to each element of the list,assigning it as x variable of the lambda expression.

PYTHON: CONCATENATE LISTS # concatenate and print lists nicely abc = ["a", "b", "c"] cde = ["c", "d", "e"] newabcde = abc + cde print newabcde

# through 'join' the list can be # concatenated to one single string print ",".join(newabcde) print " | ".join(newabcde)

PYTHON: MODIFY LISTS # modify each element of the list with # a lambda and map abc = ["a", "b", "c"] print "Original abc: " + str(abc)

# create a function to apply to the elements lambdaFunction = lambda x: x.upper()

# and apply it through the map statmente modifiedAbc = map(lambdaFunction, abc) print "Modified abc: " + str(modifiedAbc)

# what if you want an index in the lambda? lambdaFunction = lambda (i, x): "%i) %s" % (i, x.upper()) modifiedAbc = map(lambdaFunction, enumerate(abc)) print "Modified abc with index: " + str(modifiedAbc)

PYTHON: DICTIONARIES # working with dictionaries

# create a dictionary from key1:value1, key2:value2, ... townsProvinceMap = {'merano':'BZ', 'bolzano':'BZ', 'trento':'TN'} print townsProvinceMap

# ways of accessing the values of a map # the "array" way print townsProvinceMap['merano'] # through the "getter" print townsProvinceMap.get('merano')

# add an element townsProvinceMap['St. Walburg'] = "BZ" print townsProvinceMap

# remove an element del townsProvinceMap['St. Walburg'] print townsProvinceMap

PYTHON: LOOPING THROUGH DICTIONARIES # check if a key is present in the dict print "has key 'merano'? %s" % townsProvinceMap.has_key("merano")

# get a list of keys, values, items print "keys: %s" % townsProvinceMap.keys() print "values: %s" % townsProvinceMap.values() print "items: %s" % townsProvinceMap.items()

# how do we check the type of a variable print type(townsProvinceMap.keys())

# loop through a dictionary for key, value in townsProvinceMap.items(): print key + " = " + value

# or through the list of keys for key in townsProvinceMap.keys(): print key + " = " + townsProvinceMap[key]

PYTHON: LOOP WITH RANGES

# working with ranges

# they are used for iterations for i in range(0, 3): print "Iteration: %i" % i

# using xrange for i in xrange(0, 3): print "Xrange iteration: %i" % i

# xrange vs. range print type(xrange(0, 3)) print type(range(0, 3)) # xrange creates a number generator, not a list

PYTHON: WORKING WITH THE FILESYSTEM

import osimport getpass

def main(*args): # get the user home folder home = os.path.expanduser("~") # get the user name user = getpass.getuser() print "home: " + home print "user: " + user

# use the file separator newDir = home + os.sep + "tmp_folder_" + user print "create folder: " + newDir # create a folder os.mkdir(newDir) print "folder exists: %s" % os.path.exists(newDir)

To work with the filesystem it is necessary to first understand imports,that are statements used to make modules available. That way anyexternal library and script can be used from within the code.

Import statements can be places anywhere in the code, but it is best practice to addthem at the beginof the script.

One importantmodule is theos module.

PYTHON: LISTING FILES # simple folder listing homeList = os.listdir(home) homeList.sort() for i in homeList: if not i.startswith('.') and not i.startswith('_'): path = os.path.join(home, i) isFile = os.path.isfile(path) typeString = "file" if isFile else "folder" print i , " -> " , typeString

# recursive file listing count = 0 for path,dirsInPath,filesInPath in os.walk(home): count = count + 1 print "\nFiles & folders in: " + path print "----------------------------" for foldername in dirsInPath: print "FOLDER:", os.path.join(path,foldername) for filename in filesInPath: print "FILE:", os.path.join(path,filename) if count > 1: break

PYTHON: WRITE/READ FILES newFile = home + os.sep + "testFileToBeRemoved.txt" # open a new file in write mode f = open(newFile, "w") f.write("""A new header. A new testline. A second testline. \n""") lines = ["line1\n", "line2\n", "line3\n"] f.writelines(lines) f.close()

# open file in read mode f = open(newFile, "r") # read everything into a list lines = f.readlines() for line in lines: print line f.close()

# read line by line f = open(newFile, "r") line = f.readline() while line: print line.rstrip() line = f.readline() f.close() # delete the file os.remove(newFile)

PYTHON: PARSE A CSV FILES - PART 1

# first create a sample csv file f = open(home + os.sep + "sample.csv", "w"); f.write("""@id, lat, lon, elev 1, 11.0, 46.0, 250 2, 11.1, 46.1, 251 # this is a comment 3, 11.2, 46.2, 252 """) f.close()

PYTHON: PARSE A CSV FILES - PART 2 # now read and parse the csv values header = [] dataList = [] f = open(home + os.sep + "sample.csv", "r"); for line in iter(f): line = line.strip() if len(line) == 0: continue elif line.startswith('#'): print "Comment line hit: " + line continue elif line.startswith('@'): print "Header line hit: " + line line = line[1:] lineSplit = line.split(",") for i in lineSplit: header.append(i.strip()) continue else: lineDict = {} lineSplit = line.split(",") for index,value in enumerate(lineSplit): lineDict[header[index]] = value dataList.append(lineDict) f.close() # print out the result of the parsing for index, value in enumerate(dataList): print "\nline: " + str(index) print "----------------------" for k, v in value.items(): print k + "=" + v

We parse the file and check for:

empty linescomments (starts with #)the header (starts with @)

After the parsing we want tohave the complete information ofeach line (field name + value).

PYTHON: PARSE A REAL CSV FILE

minPath = "/home/hydrologis/data/potsdam_data/22yr_T10MN" dataFile = open(minPath, "r") lines = dataFile.readlines() dataFile.close() data = {} dataStarted = False for line in lines: if dataStarted or line.startswith("-90"): dataStarted = True lineSplit = line.split(" ") lat = lineSplit[0].strip() lon = lineSplit[1].strip() point = (lon, lat) temperatures = [] for i in range(2, len(lineSplit)-1): # last is avg temperatures.append(float(lineSplit[i])) data[point] = temperatures # Potsdam, house 24: lat=52.40669, lon=12.97619 p1 = ("13", "52") p2 = ("13", "53") t1List = data[p1] t1 = t1List[4]; t2 = data[p2][4]; t = (t2+t1)/2.0 print "Temperature in May in Potsdam: ", t

Parse one of the temperatures files from the dataset and create adictionary containing the tuple of the lat/lon and the list of thetemperature in each month.

CONDITIONS & LOOPS CHEATSHEET x = False y = False if x: print "x = true" elif y: print "x = true" else: print "x and y false"

if - elif -else

y = 5 x = "worked" if (y > 1) else "failed" print x

the ternary operator

x = 5 while x > 0: print x x = x - 1

while loop

for v in "123456789": print v

for i,v in enumerate("123456789"): print str(i) + ") " + v

for loop

<license> This work is released under Creative Commons Attribution Share Alike (CC-BY-SA).</license>

<sources> Much of the knowledge needed to create this training material has been produced by the sparkling knights of the <a href="http:www.osgeo.org">Osgeo</a>, <a href="http://tsusiatsoftware.net/">JTS</a>, <a href="http://www.jgrasstools.org">JGrasstools</a> and <a href="http:www.gvsig.org">gvSIG</a> communities. Their websites are filled up with learning material that can be use to grow knowledge beyond the boundaries of this set of tutorials.

Another essential source has been the Wikipedia project.</sources>

<acknowledgments> Particular thanks go to those friends that directly or indirectly helped out in the creation and review of this series of handbooks. Thanks to Antonio Falciano for proofreading the course and Oscar Martinez for the documentation about gvSIG scripting.</acknowledgments>

<footer> This tutorial is brought to you by <a href="http:www.hydrologis.com">HydroloGIS</a>.<footer>