Victor Bhattacharyya

High Performance Batch Geocoding with ArcGIS

Goal: Get the best possible batch geocoding performance from your system, be it Desktop or Enterprise

• Batch geocoding overview

• Desktop geocoding recommendations

• ArcGIS Enterprise geocoding recommendations

• Working with Clients for Enterprise

Outline

• Batch geocoding overview

• Desktop geocoding recommendations

• ArcGIS Enterprise geocoding recommendations

• Working with Clients for Enterprise

Outline

Batch GeocodingBatch Geocoding Overview

• Turns addresses into points on the map

Addresses: your dataBatch Geocoding Overview

• Addresses hard to control as a data type

• Addresses change

• Reference data changes

• Approaches must fit any need for scale

Geocoding in ArcGIS ProBatch Geocoding Overview

• Pro ships with default locators- World Geocoding Service (requires internet &

credits)- XY provider for coordinate and grid handling

• Locate pane defines available locators- Add more to your project

Locator options in ArcGIS ProBatch Geocoding Overview

• Build your own locators using the CreateLocator tool

• Use Esri-provided locators- World Geocoding Service- World Geocoder for ArcGIS- Street Map Premium locators

• Batch geocoding overview

• Desktop geocoding recommendations

• ArcGIS Enterprise geocoding recommendations

• Working with Clients for Enterprise

Outline

Desktop Geocoding RecommendationsBest practices for local geocoding

• Use the latest version of ArcGIS Pro (2.3)

• Use new locators

- Most recent release of SMP locators

- Build your own locators with the CreateLocator tool in ArcGIS Pro 2.3

• Set the number of threads on your locator to “Auto”

- This uses 1 less than the number of cores on your machine

• Put the locator and input table on a solid state drive

• Use the Geocode Table pane in ArcGIS Pro

• Automate the process with the GeocodeAddresses GP Tool

Automating Desktop Geocoding: Geocode Addresses Best practices for local geocoding

• Batch geocoding overview

• Desktop geocoding recommendations

• ArcGIS Enterprise geocoding recommendations

• Working with Clients for Enterprise

Outline

ArcGIS Enterprise Recommended Batch Geocoding SystemArchitecture

• Get the most performance out of your enterprise geocoding services

• Manage backend resources

• Perform batch geocoding behind your organizational firewall

• Has chunking and retry logic built-in

ArcGIS Portal

GIS Servers

Geocode Server

World Locator ServiceSMP Locator Service(s)

GeocodingToolsGeoprocessing Service

Hosting Server

System RecommendationsBest practices for Enterprise geocoding

• Use the latest version of Asynchronous Batch Geocoding enabled clients- Geocode Locations from Table Mapviewer tool -> Web- Geocode Table and Geocode Locations from Table in ArcGIS Pro 2.3 -> Desktop- arcpy.geocoding.GeoocdeLocationsFromTable() and GeocodeFile API -> Scripting

• ArcGIS Server- Multiple cores (the more the better)- Enough memory for those cores (locator size + 500MB per core)- SSD Drive- Fast processors

How the system works

GIS Servers

Geocode Server

World Locator ServiceSMP Locator Service(s)Custom Locator Service

Main Process

Hosting Server

Asynchronous Batch Geocoding

Input Table

1. Chunks requests2. Makes geocodeAddresses API calls to the geocoding service3. Uses retry logic to retry failed requests4. Receives geocoding results5. Stitches together results

How the system works

GIS Servers

Geocode Server

World Locator ServiceSMP Locator Service(s)Custom Locator Service

Geocoding ToolsGeoprocessing Service

Hosting Server

Asynchronous Batch Geocoding

Input Table

1. Chunks requests2. Makes geocodeAddresses API calls to the geocoding service3. Uses retry logic to retry failed requests4. Receives geocoding results5. Stitches together results

How the system works

GIS Servers

Geocode Server

World Locator ServiceSMP Locator Service(s)Custom Locator Service

Geocoding ToolsGeoprocessing Service

Hosting Server

Asynchronous Batch Geocoding

Input Table

“numBatchThreads”: 4

1. Chunks requests2. Makes geocodeAddresses API calls to the geocoding service3. Uses retry logic to retry failed requests4. Receives geocoding results5. Stitches together results

How the system works

GIS Servers

Geocode Server Geocoding ToolsGeoprocessing Service

Hosting Server

Asynchronous Batch Geocoding

Input Table

Output Feature Layer

“numBatchThreads”: 4

World Locator ServiceSMP Locator Service(s)Custom Locator Service

1. Chunks requests2. Makes geocodeAddresses API calls to the geocoding service3. Uses retry logic to retry failed requests4. Receives geocoding results5. Stitches together results6. Makes output available for download

NumBatchThreads: A Closer LookArchitecture

“numBatchThreads”: 4

Geocoding Services

GeocodingTools Geoprocessing Service

NumBatchThreads with Multiple UsersArchitecture

“numBatchThreads”: 2

Geocoding Services

GeocodingTools Geoprocessing Service

Configure Portal Batch Geocoding

• How to Publish locators and add them as Utility Services

• Set the number of threads per geocoder via the Sharing API

• Scale the GP Service based on server resources via Server Manager

• Learn more at the Configure Portal to Geocode Addresses online help

https://bit.ly/2UspGqa; https://bit.ly/2CbVBnt; https://bit.ly/2C65UJF; https://bit.ly/2NR4WWv;

Configure Threads

Configuration

Configuration Demo

• Batch geocoding overview

• Desktop geocoding recommendations

• ArcGIS Enterprise geocoding recommendations

• Working with Clients for Enterprise

Outline

Clients for geocoding data on the Portal

Demo in Pro

Demo in MapViewer

Automating Enterprise Geocoding: Geocode Locations from Table Python

API for geocoding data on a local machine

New Approach: GeocodeFile

GIS Servers

Geocode Server

World Locator ServiceSMP Locator Service(s)Custom Locator Service

Hosting Server

• Input needs to be uploaded to the server as a zipped gdb, or a zipped CSV or XLSX file

• Output CSV, XLSX or Feature Class is available for download from the server when the job is completed

• Geocode File API available in ArcGIS Enterprise 10.7

Geocoding Output

Client uploads table to server

Client downloadsgeocoding results

- Discover available geocoding services- Authenticate requests

GeocodingTools GP Service

ArcGIS Server

ArcGIS Portal

1

2

4

Chunking and geocoding3

Best practices for Enterprise geocoding

Automating Enterprise GeocodingGeocode File API

Automating Enterprise GeocodingGeocode File API (Part 2)

Clients for geocoding data on a Big Data File Share

(BDFS Input)

GeoAnalytics: Geocode Locations from Table Data in HDFS -> Geocode Locations from Table GeoAnalytics tool

• If you have the GeoAnalytics Extension enabled in ArcGIS Enterprise this tool appears in the map viewer

• Has chunking and retry logic built-in

• Takes input table from a BDFS in ArcGIS Enterprise

GeoAnalytics: Geocoding via the Python APIData in HDFS -> Geocode Locations from Table GeoAnalytics tool

• GeoAnalytics tool exposed via the “geoanalytics” module in the Python API

• Has chunking and retry logic built-in

• Takes input table from a BDFS in ArcGIS Enterprise

Configuring Geocode Locations GeoAnalytics ToolData in HDFS -> Geocode Locations from Table GeoAnalytics tool

• Add the geocoding service as a utility service in your organization

• Set the “percentageMaxAllowedComputeCores” and “percentageMaxAllowedComputeMemory” system properties

• Read more at: http://enterprise.arcgis.com/en/server/latest/get-started/windows/geoanalytics-settings.htm

https://<machine_name>/server/admin/system/properties/update

Complete answersand select “Submit”

Scroll down to find the feedback section

Select the session you attended

Download the Esri Events app and find your event

Please Take Our Survey on the App

Questions?