Deploying Distributed Raster AnalyticsJie Zhang, Nirupam Dey

Aerial Photos

High-Resolution SatellitesSmall Sats

ArcGIS Has Five Core Capabilities for Imagery and Remote Sensing

Integrated and Massively Scalable

Indexes

AI / Deep Learning

Classification

Change Analysis

Raster

Processing

Functions

Analysis

Image

Analyst

Management

& Dissemination

Image Server

Temporal

Profile

Spectral Profile

Charting

Visualization &

Exploration

Full-Motion Video

Image Space

and Mensuration

Street Views

Oriented Imagery

Drone2Map

Drone Mapping

Map Production

DTM

Generation

Seamless

Orthophotos

Ortho Mapping

Stereo

Ortho Maker

Content

All Major Sensors

Drones

Analysis

Image Server

Indexes

AI / Deep Learning

Classification

Change Analysis

Raster

Processing

Functions

Analysis

Image AnalystOrtho Mapping

Image AnalystAnalytics

Apps(Excalibur)

ArcGIS Pro

AnalysisIntegrated and Massively Scalable

Exploitation & Visualization

What is Raster Analytics?

• The ArcGIS way to create and execute spatial analysis

models and image processing chains which leverage

distributed storage and analytics

• Requires: ArcGIS Enterprise + ArcGIS Image Server

ArcGIS ProWeb map

Viewer

Hosting server Raster Analysis Image Hosting

ArcGIS

Server

Image Server Image Server

Distributed Raster Data StoreArcGIS

Data StoreImagery and GIS Data

ArcGIS

ServicesGdb

Portal

Raster Analytics System Configuration

ArcGIS Data Store Setup and Hosting Server

• ArcGIS Data Store = a setup to configure relational

database on hosting server

• Hosting server is used to store hosted feature service

• Some raster analysis workflows generate hosted feature

service output

Federated Image Servers in Enterprise

• Raster Analysis

- Processing engine of the Raster Analysis tasks

- Must have a registered raster data store of file share or

cloud store.

• Image Hosting

- Dedicate to host the published image services

- Raster analytic result will be shared as image service on

this server

- Must have at least one file share type raster store

- And must have the same raster store as Raster Analysis

image server if they are no the same server

Raster Analysis and Image Hosting Server Function

• “Roles” assigned to federated image server site

• Split functions of server site to avoid resource

competition

• Server function can be assigned to any federated image

server but only one server is allowed to have the same

function

• You can have federated Image Server without “role”

Configuring ArcGIS Image server

Portal

Webmap

Cloud

storage

(Azure, AWS

…)

File share

storage

Raster

Data Store

for storing

CRFs

Federation

Federated Server 1

Hosting Server

Image Hosting Server

and

Raster Analysis Server

All-in-one

Configuring ArcGIS Image server

Portal

Webmap

Cloud

storage

(Azure, AWS

…)

File share

storage

Raster

Data Store

for storing

CRFs

Federation

Federated Server 1

Hosting Server

Federated Server 2

Image Hosting Server

and

Raster Analysis Server

Separate Image Server

from Hosting Server

Configuring ArcGIS Image server

Portal

Webmap

Cloud

storage

(Azure, AWS

…)

File share

storage

Raster

Data Store

for storing

CRFs

Federation

Federated Server 1

Hosting Server

Federated Server 2

Raster Analysis Server

Federated Server 3

Image Hosting Server

Further separation of

Raster Analysis Server

and Image Hosting Server

Raster Store

• Raster Store registered to Image Server is for storing

output imagery of Raster Analysis tasks

- Shared file system storage

- Cloud Object Storage

- Amazon S3

- Azure Blob Storage

- Alibaba Cloud

Note: register a Cloud Store first then register a Raster Store

reference to the Cloud Store

Optimized Cloud Raster Format (CRF)

• New ArcGIS Raster Dataset format for distributed read/write

- Default output imagery format for Raster Analysis tasks

- Image is split to “bundles” with optimized schema

- Support cloud storage and file system

- Published as image service

- Rendered through Raster Rendering service

Raster Rendering service

Image Service

CRF Imagery

Rendering request

readredirect

reference

Distributed Raster Analytics System Services

Geo Processing service as master

Image Service as worker

Image Service for rendering output

Raster Analysis Tools System Services

Note: Generate Raster

tool is the generic

service tool takes raster

function as input to

support various analytic

workflows

ref: http://esriurl.com/rasterfunc

Nirupam Dey

ArcGIS

Portal/Server

configuration

Nirupam Dey

Deployment in the Cloud

Deployment Platforms

• On-premise + network file shared storage

appliance

• Cloud platform + Cloud Object Storage

- Amazon Cloud Formation template

or ArcGIS Enterprise Cloud Builder CLI for AWS

- ArcGIS Cloud Builder for Azure

https://arcgisstore1071.s3.amazonaws.com/11595/docs/index.html

Nirupam Dey

ArcGIS Enterprise

Configuration and

Enterprise Builder

CLI

Federated with

Portal

S3 Storage for

Portal/Server

configuration

S3 Raster Storage

Professional Imagery /

Geospatial Analysts

Client consuming

Imagery ItemsBase Enterprise

EC2

Elastic IP for Portal

VPC

Elastic Load

Balancer

EC2

Configure Store

Client consuming

Hosted Image

Services

Image Services

Image Server

Image Services

Image Server

EC2 EC2

Image Services

Image Server

EC2

Dynamic Image

ServicesRaster Analytics

Elastic Load

Balancer

Client consuming

Dynamic Image

ServicesEnterprise + ArcGIS Server + Image server + RDS in AWS

• Used when we need to do raster analytics

• Mosaic datasets sit in Postgres ( RDS)

• All server machines are in cluster

• Config store is on a different ec2 machine

• Different Stack for Dynamic services and raster

analytic

RDS

Postgres RDS

ArcGIS Enterprise Cloud Builder for Azure

• Stand-alone app for both Desktop and Enterprise deployment

• Deployment template for Image Server

• Premium storage account recommended

https://enterprise.arcgis.com/en/server/latest/cloud/azure/install-azure-cloud-builder.htm

Special Enterprise Raster Analytics Configuration

• Ortho Maker

- Focus on drone imagery orthomapping

- Configure the elevation utility service

- Must have file share type of raster store

- Enable Ortho Maker app in your Enterprise

• Deep Learning

- Install additional deep learning python modules

- Parallelization using GPU server nodes

Deployment Best Practice

• Split Raster Analysis and Image Hosting server functions

• Adjust maximum number of processing service instances based on machine specs

• Shutdown unused services

• Configure local cache folder for each server node for

fast read/write of Cloud Raster Format (CRF)

• In cloud deployment, have data store close to your server machine

• 10:00 AM – 10:45 AM, Thursday, July 11

Image Processing Using Raster Functions

• 10:00 AM – 11:00 AM, Thursday, July 11

Using Machine Learning and Deep Learning with Imagery in ArcGIS (2nd)

• 2:30 PM - 3:30 PM, Thursday, July 11

ArcGIS Enterprise: Raster Analytics in ArcGIS Image Server (2nd)

Recommended Imagery Enterprise Demo Theatres/Workshops

Please Share Your Feedback in the App

Download the Esri

Events app and find

your event

Select the session

you attended

Scroll down to

“Survey”

Log in to access the

survey

Complete the survey

and select “Submit”