Chapter 1€¦ · Chapter 1 Introducing advanced Python scripting PythonAdv.indb 1 4/28/20 9:15 AM....

1.1 Introduction

Python has become one of the most widely used programming languages, and this increase includes geospatial applications. Python is employed for many different tasks, from automat-ing data processing using desktop software, to web scraping for downloading structured data, to developing machine-learning algorithms for classifying imagery hosted in the cloud. Python is a versatile, open-source programming language supported on different platforms. These features contribute to its growing popularity in the geospatial community. Python is also the preferred scripting language for working with ArcGIS Pro.

This book represents the logical follow-up to Python Scripting for ArcGIS Pro, also published by Esri Press (2020), which introduces the fundamentals of Python and teaches you how to write basic scripts to automate workflows. Advanced Python Scripting for ArcGIS Pro picks up where Python Scripting for ArcGIS Pro left off by focusing on more advanced scripting techniques and the development of tools and notebooks to be shared with others. This book also includes working with third-party packages and the ArcGIS API for Python, which opens new and exciting pos-sibilities to use Python for geospatial applications.

This book is written for ArcGIS Pro and Python 3. The topics covered in this book require substantial previous experience in writing Python scripts for ArcGIS. The fundamentals of Python and ArcPy, including setting up a Python editor and writing basic scripts for data processing using ArcPy, are covered in Python Scripting for ArcGIS Pro.

1.2 Python scripting in ArcGIS Pro using ArcPy

ArcGIS Pro provides support for the use of Python as a scripting language, including the ArcPy package installed as part of ArcGIS Pro. ArcPy provides access to all the tools available in ArcGIS Pro, including those that are part of ArcGIS Pro extensions. This feature makes Python scripting an attractive and efficient method for automating tasks in ArcGIS Pro.

Chapter 1Introducing advanced Python scripting

PythonAdv.indb 1PythonAdv.indb 1 4/28/20 9:15 AM4/28/20 9:15 AM

2 Chapter 1: Introducing advanced Python scriptingAdvanced Python Scripting for ArcGIS Pro

Python scripting has become a fundamental tool for GIS professionals to extend the function-ality of ArcGIS Pro and automate workflows. Python is the scripting language of choice to work with ArcGIS Pro and is included in every ArcGIS Pro installation. Python is also directly embed-ded in many tools in ArcGIS Pro. For example, Python is one of the standard expression types for field calculations. As another example, several geoprocessing tools in ArcGIS Pro consist of Python scripts, even though the casual user does not necessarily notice it (or need to).

One of the goals for using the current book is to learn how to develop new geoprocessing tools that expand the functionality of ArcGIS Pro but that look and feel like regular tools that are part of the software. This is accomplished using Python script tools and Python toolboxes. A secondary goal is to become familiar with the ArcGIS API for Python to expand the use of Python to working with web GIS. This is accomplished using notebooks.

1.3 Python versions and ArcGIS

Compared with other programming languages, Python has gone through a limited number of versions, reflecting a philosophy of incremental change and backward compatibility. Python 3 was released in 2008 as a major overhaul, with the primary goal to clean up the code base and remove redundancy. The most recent version, at the time of writing, is 3.8, with 3.9 under development.

Some of the changes in Python 3 are fundamental, which result in breaking with the backward compatibility philosophy of Python. As a result, not all code written in Python 3 works in Python 2. Some of the new functionality added in Python 3 was also added to Python 2, a process known as backporting. With careful attention to detail, it is therefore possible to write code that works in both versions.

The two versions of Python will continue to coexist for some time, but officially Python 2 will no longer be maintained past 2020. This means that any existing code will continue to work, but there will be no further improvements to version 2.

ArcGIS Desktop 10.x uses Python 2 whereas ArcGIS Pro uses Python 3, which has several implications. If you are going to write scripts for both versions of ArcGIS or are planning to migrate scripts and tools from ArcGIS Desktop 10.x to ArcGIS Pro, you must learn some of the differences between the two versions of Python. Resources and utilities exist to assist with this conversion, which are covered in chapter 8.

The purpose of this book is to focus on writing scripts and developing tools for ArcGIS Pro using Python 3. Although Python code is not 100 percent backward compatible between versions 3 and 2, it is, in principle, possible to write Python code that works for both versions. However, because of fundamental differences between ArcGIS Desktop 10.x and ArcGIS Pro, many scripts and tools written for one version are unlikely to work in the other. Nonetheless, sometimes the differences are small, and strategies to identify and correct for these differences are covered in chapter 8.


3Chapter 1: Introducing advanced Python scripting Advanced Python Scripting for ArcGIS Pro

Note: Many GIS users will continue to use both ArcGIS Desktop 10.x and ArcGIS Pro for some time. At the time of writing, the most current versions are ArcMap 10.7.1 and ArcGIS Pro 2.5. The installation of ArcMap 10.7.1 includes the installation of Python 2.7.16, and the installation of ArcGIS Pro 2.5 includes the installation of Python 3.6.9. These two versions can run on the same computer. When working with ArcGIS Pro 2.5, you should use only version 3.6.9.

1.4 ArcGIS API for Python and Jupyter Notebook

Python and ArcPy make it possible to extend the functionality of ArcGIS Pro using scripts and tools. ArcGIS Pro is a software application that runs on desktop computers and is primarily designed to work with local datasets. Increasingly, however, geospatial data and their applications reside on the web, referred to as web GIS. Web GIS is a type of distributed information system that allows you to store, manage, visualize, and analyze geographic data. ArcPy has limited functional-ity to work directly with web GIS. The ArcGIS API for Python is a different Python package from Esri to work directly with web GIS. This API complements the use of ArcPy for desktop GIS.

Code that uses the ArcGIS API for Python is typically written in Jupyter Notebook, an open-source web application that works like a Python editor and provides built-in visualization capabilities. Notebooks can also be used directly within ArcGIS Pro. Details on using the ArcGIS API for Python are covered in chapter 9.

1.5 The structure of this book

Advanced Python Scripting for ArcGIS Pro consists of nine chapters that focus on developing tools for ArcGIS Pro and writing more advanced scripts. Sample code is provided throughout the text.

Chapter 1 introduces Python scripting for ArcGIS Pro and illustrates several example scripts, tools, and notebooks that were developed using Python.

Chapter 2 demonstrates how to create custom functions and classes in Python. Custom functions and classes make it easier to organize more complex code and use parts of your code in multiple scripts. Custom functions and classes are widely used in script tools and Python toolboxes.

Chapter 3 explains how to create custom script tools, which make Python scripts available as regular geoprocessing tools with a familiar tool dialog box. Script tools are one of the preferred methods for sharing Python scripts with other users and make it easier to add a Python script as a tool to a larger sequence of operations.

Chapter 4 covers how to create Python toolboxes as an alternative to Python script tools. In a Python toolbox, the tool dialog box is written in Python itself, which is often more robust.

Chapter 5 outlines strategies for sharing tools with others, including how to organize your files, work with paths, and provide documentation for tools.

Chapter 6 covers the use of managing packages using conda. Packages allow you to add func-tionality to Python, and conda is a convenient way to install and manage these packages as well as Python environments, which control which packages are available.



to work with the default environment arcgispro-py3 or a cloned environment. Chapter 6 provides details on using conda to manage environments, but the configuration of Python editors is covered in Python Scripting for ArcGIS Pro.

You can also use the Python window in ArcGIS Pro to write and test Python code. However, the Python window is most suited to running short snippets of code for testing purposes. The more complicated and longer scripts developed in this book require the use of a dedicated Python editor, such as IDLE, Spyder, or PyCharm.

1.8 Exploring example scripts, tools, and notebooks

This section uses several examples to illustrate how Python is used to create scripts, tools, and notebooks. The examples were obtained from Esri and the ArcGIS user community. One of the reasons for presenting these examples is for you to become more familiar with looking at Python code and tools developed by others. One of the best ways to learn how to write code and develop tools is to work with existing examples. You are not expected to fully understand all the code at this point, but the examples will give you a flavor of what is to come in this book.

Example 1: Terrain ToolsThe Terrain Tools were developed by Esri and extend what is available in ArcGIS Pro by

providing capabilities for creating alternative terrain representations. These representations include different types of hillshade surfaces and contours, which can greatly enhance the cartographic display of terrain data.

The tools are made available as a collection of tools in a toolbox. Each tool consists of a tool dialog box and has a corresponding Python script. Although these scripts are written in Python, the functionality of the script can be accessed the same way as any other geoprocessing tools. The figure illustrates what the toolbox looks like in ArcGIS Pro.

Chapter 7 describes the use of selected built-in modules and third-party packages other than ArcPy, which can greatly enhance the functionality of your scripts. The modules and packages include ftplib, urllib, openpyxl, json, NumPy, Pandas, and Matplotlib.

Chapter 8 explains the key steps in migrating scripts and tools from ArcGIS Desktop 10.x to ArcGIS Pro, including the use of several utilities to facilitate this process.

Chapter 9 introduces ArcGIS API for Python, which makes it possible to use Python to work with web GIS. This chapter also introduces Jupyter Notebook as the preferred way to write and document Python code using this API. The resulting notebooks can also be shared with others.

1.6 A note about code in this book

Most of the code in this book is written for ArcGIS Pro 2.5, which uses Python 3.6.9. Most of the code will work in earlier versions of ArcGIS Pro, except for the most recently added function-ality. As new functionality is added to future releases of ArcGIS Pro, the code in this book will continue to work for the foreseeable future. However, much of the code will not work in ArcGIS Desktop 10.x. Some of the code in this book also uses the ArcGIS API for Python version 1.7.0. This is the version that is installed with ArcGIS Pro 2.5, but the ArcGIS API for Python can also be installed separately. If installed separately, Python 3.5 is required to use the ArcGIS API for Python.

Note: The update cycle of the ArcGIS API for Python does not follow the same schedule as ArcGIS Pro. For example, at the time of writing, version 1.8.0 of the ArcGIS API for Python has been released, whereas ArcGIS Pro 2.5 installs with version 1.7.0. This version will be updated with future releases of the ArcGIS Pro software. The differences in these versions are typically small.

The code in this book employs the coding conventions of the official Style Guide for Python Code, also referred to as PEP 8. The complete style guide can be found at http://www.python.org/dev/peps/pep-0008/. Although not required, following coding guidelines improves the consis-tency and readability of your code.

1.7 Working with Python editors

Writing scripts and developing tools requires a Python editor. You are expected to be already familiar with using a Python editor and configuring it to use the correct environment. Details on working with Python editors are covered in Python Scripting for ArcGIS Pro.

The code is this book is not specific to one Python editor. IDLE is installed by default with every Python installation, and therefore most code illustrations in this book use IDLE as the Python editor of choice. Other recommended editors include PyCharm and Spyder, and some code illustrations use these editors as well. You are free to use the Python editor of your choice. Regardless of which editor is used for code illustrations, the Python code is the same for any Python editor. To use a Python editor with the code in this book, however, it must be configured



to work with the default environment arcgispro-py3 or a cloned environment. Chapter 6 provides details on using conda to manage environments, but the configuration of Python editors is covered in Python Scripting for ArcGIS Pro.

You can also use the Python window in ArcGIS Pro to write and test Python code. However, the Python window is most suited to running short snippets of code for testing purposes. The more complicated and longer scripts developed in this book require the use of a dedicated Python editor, such as IDLE, Spyder, or PyCharm.

1.8 Exploring example scripts, tools, and notebooks

This section uses several examples to illustrate how Python is used to create scripts, tools, and notebooks. The examples were obtained from Esri and the ArcGIS user community. One of the reasons for presenting these examples is for you to become more familiar with looking at Python code and tools developed by others. One of the best ways to learn how to write code and develop tools is to work with existing examples. You are not expected to fully understand all the code at this point, but the examples will give you a flavor of what is to come in this book.

Example 1: Terrain ToolsThe Terrain Tools were developed by Esri and extend what is available in ArcGIS Pro by

providing capabilities for creating alternative terrain representations. These representations include different types of hillshade surfaces and contours, which can greatly enhance the cartographic display of terrain data.

The tools are made available as a collection of tools in a toolbox. Each tool consists of a tool dialog box and has a corresponding Python script. Although these scripts are written in Python, the functionality of the script can be accessed the same way as any other geoprocessing tools. The figure illustrates what the toolbox looks like in ArcGIS Pro.



The “scroll” icon indicates that these tools are written in Python, also referred to as Python script tools.

The tool dialog boxes look like those of regular geoprocessing tools in ArcGIS Pro. As an example, consider the Illuminated Contours tool. The tool provides an analytical version of the hand-drawn Tanaka method of symbolizing contours that includes coloring and varying the thickness of contour lines. Assuming a certain lighting direction, contours are drawn lighter on parts of the terrain that are illuminated and darker on parts of the terrain that are not illuminated.

The tool dialog box looks much like the regular Contour tool available in ArcGIS Pro.

The Illuminated Contours tool has five parameters, two of which are optional. The required parameters include the input raster, which is a digital elevation model or DEM, as well as the contour interval to be used and the output contour feature class. The optional parameters include the base contour and z-factor to be used. The result of the tool is a new polyline feature class, in which each contour is broken up into segments with new attributes for the color (grayscale, from white to black) and the appropriate thickness.

An example of the resulting illuminated contours is shown, with the contours overlaid on top of the regular DEM shown in grayscale from dark (low elevation) to light (high elevation). The assumed lighting direction is from the northwest, as revealed in the different levels of illumination of the contours.

The Illuminated Contours tool effectively carries out a series of steps, which can be accom-plished by running regular geoprocessing tools and applying symbology. Some of these steps include creating contours from a DEM, creating a default hillshade, converting hillshade bright-ness values, reclassifying this grid of values into five-degree intervals, converting the reclassified grid to polygons, intersecting the contour polylines with these polygons, and assigning symbology on the basis of the new attributes of the polylines. The purpose of the script tool is to automate these steps and provide a user-friendly interface.



Example 2: Random SampleThe Random Sample tool was developed by the author and is discussed in more detail in chapter 3. This tool creates a random sample based on an input feature class and a user-defined number of features. The output is saved as a new feature class. The tool is created as a Python script tool. The tool dialog box is shown in the figure.

The tool provides functionality not available in ArcGIS Pro. Several online resources such as Stack Exchange (http://stackexchange.com) and Esri Support (http://support.esri.com) provide various code solutions to select features at random from a feature class, but employing these solu-tions requires substantial coding skills. By developing a Python script tool, the script becomes more user-friendly. As a Python developer, you can write this type of script, develop and test the Python script tool, and then make the tool available to other users who can use the tool without having to learn Python.

A single Python script is used in this tool, and the script can be opened to get an inside look at what the tool does. When you open the script in a Python editor, it looks like the figure.

When scrolling through the script, you will find the equivalent of the tasks you would need to carry out in ArcGIS Pro using existing tools. For example, making sure you have a license for the Spatial Analyst extension; running geoprocessing tools such as Hillshade, Contour, Reclassify, and Intersect; and applying symbology using a layer file. You could complete these steps using existing tools and a few manual manipulations, but the Python script tool includes all of them in a single easy-to-use tool dialog box.

One of the nice things about working with Python script tools is that you can view the under-lying code. Not only can you learn from the code from others, you can also copy it and make a modified version of it for your own work.

Detailed documentation, all the source code, and example datasets to experiment with these tools can be found in ArcGIS Online at www.arcgis.com by searching for Terrain Tools Sample.


Chapter 1€¦ · Chapter 1 Introducing advanced Python scripting PythonAdv.indb 1 4/28/20 9:15 AM....

Documents

Transcript of Chapter 1€¦ · Chapter 1 Introducing advanced Python scripting PythonAdv.indb 1 4/28/20 9:15 AM....