Best Practices for Data Store Management in an active Hosted Feature Services

EnvironmentBill Major

Agenda

• Review – On-premises Web GIS• ArcGIS Data Store and the various stores• Configuring a Standby Data Store for Availability• Command Line Utilities - Demo• Move ArcGIS Data Store to another server• Upgrading ArcGIS Data Store• ArcGIS Manager and Administrator API• Summary

A

BasicData Store knowledge

A

Web GIS | Anatomy of the Portal

Desktop Web Device

Server Online Content and Services

portal

The portal in ArcGIS Online just is:hosted by Esri and no need for installation or maintenance nor breaking apart intothe backend components..

For a deployment using Portal for ArcGIS there arethree main components that make up a Web GIS:

Portal for ArcGIS

Hosting server(ArcGIS Server)

ArcGIS Data Store

Understanding Data Stores

• GIS Server “data store” concept- Valid location that contains data used for web services- Read/write by the arcgis account

• ArcGIS Data Store- Separate software install included with ArcGIS for Server (since 10.3)- Used to configure and deploy a “hosting server” configuration with

Portal for ArcGIS

GIS Server “data store” Concept

• Valid data source locations accessible by GIS site• 2 types

1. Databases2. Folders

• 2 ways to define- ArcGIS Server Manager- ArcCatalog

ArcGIS Data Store Types

A

• Relational Data Store- Hosted Feature Services, managed by Portal and ArcGIS Server

• Tile cache Data Store- Publish Scene Layers, not Hosted Tile Services

• Spatiotemporal Big Data Store- Archive observation data streamed when using ArcGIS GeoEvent Extension for Server.- We will not be discussing in this session

What does the Relational ArcGIS Data Store provide?

A

• With an ArcGIS Data Store configured with ArcGIS Server and federated/hosted with Portal for ArcGIS, several key capabilities become enabled in your Web GIS environment.

- Publish thousands of hosted feature layers (including empty hosted feature layers)- Use spatial analysis tools in Portal or Insights for ArcGIS- Analyze imagery and store output data- Enable or disable attachments via a feature layer’s item details- Enable ability to track feature creation and updates, and enforce ownership-based access

control via the feature layer’s item details- Create feature templates in Portal- Calculate values for a field in Portal- Apps: Survey123 and GeoPlanner require your Portal to use ArcGIS Data Store

ArcGIS Data Store Installation and Configuration Notes

A

• Starting at 10.4, ArcGIS Data Store installs using a OS or Domain service account.• In a distributed server environment, the ArcGIS Data Store should be installed

“closest” to ArcGIS Server. Portal for ArcGIS does not communicate directly with the ArcGIS Data Store, only ArcGIS for Server does.

• ArcGIS Data Store Ports: 2443 (HTTPS), 9876 (TCP)- Tile cache data store: 29080 (HTTP), 29081 (HTTPS)

• Configuration with ArcGIS Server can be done via web browser or a command line utility

• Supports availability via a Primary and Standby node

Installing a Standby ArcGIS Data Store for Availability

A

• Validate ports are open to second server• Install ArcGIS Data Store on second server using same Domain Account as Primary

Data Store • Configure ArcgIS Data Store to same ArcGIS Server site• Verify the Configuration Summary knows that this is a Standby Data Store

Sample Highly Available ArcGIS for Server Hosting Site

A

ArcGIS Server Site

Web Servers withArcGIS Web Adaptor

Network Load Balancer

Standby ArcGISData Store

Primary ArcGISData Store

Network File Share for site Config-Store

Network File Share for Data Store Backups

Demo

Configuring a Stand by Data Store

Overview of Command Line Utilities for Data Store Management

A

• Data Store provides a number of command line utilities for relational data store administration.

• Installed at <ArcGIS Data Store installation directory>\datastore\tools• Important commands that we will explore further:

- describedatastore- changebackuplocation- updatebackupschedule & updatebackupretaindays- changedbproperites- changenosqldslocation- importmanageddb & exportmanageddb

Describe the Data Store

A

• Useful utility to provides an overall status check on your Data Store

Command line utilities: changebackuplocation

A

• Data store backup files are stored on the same machine as the data store by default• You should move your backup files to a separate machine to ensure you can access

them if the data store is inaccessible.• Set the data store service to run using a domain account, grant that account read

and write access to a shared network directory, and use the changebackuplocation utility to move your data store backup files to the shared directory on a different machine

• Syntax: changebackuplocation <new directory path> [--is-shared-folder <true|false>] [--keep-old-backups <true|false>]

Command line utilities: updatebackupschedule & updatebackupretaindays

A

• By default, ArcGIS Data Store creates a full backup every four days. • You can change how often a full backup is created by running the

updatebackupschedule utility.• Syntax: updatebackupschedule [--starttime <local server time>] --frequency <number of

days>• Specify a start time using 24-hour clock notation, for example, 00:00:00 for midnight

and 13:00:00 for 1 p.m. • Use the frequency option to specify the number of days between backups. To

disable automatic backups, set the frequency to 0. • Additionally, use updatebackupretaindays to change how often backup files are

purged from the backup directory (default is 7 days).

Changing Data Store Database Properties

A

• The changedbproperties utility allows you to change different properties depending on the type of data store you run it against.

• In particular for the relational data store, there is a property to change the number of connections allowed.

• Syntax: changedbproperties --store relational --max-connections 100• Depending on the server resources available to you (especially RAM) and how

active users are with Hosted Feature Services, this number can be changed for tuning.

Changing the Scene Layer Tile Cache Location

A

• If your Web GIS becomes active for Scene Layer caches, tile cache sizes can add up over time.

• The location of the tile cache can be changed using this command line utility.• Syntax: changenosqldslocation <path> [--prompt {yes | no}• If you move the data to a shared directory, you must grant read and write

permissions on the directory to the user running the ArcGIS Data Store process (Linux) or service (Windows).

Command line utilities: exportmanageddb & importmanageddb

A

• The exportmanageddb utility creates a dump file of the database that stores hosted feature layer data, metadata about the data store, and all databases that store hosted scene layer caches.

• Export the data store if you need to make a backup of the data store to be restored to an ArcGIS Data Store installation that is on a machine with a different operating system or is a different ArcGIS Data Store release.

• exportmanageddb <backup name> --destination <output location> [--include-tilecache <true|false>] [--prompt {yes|no}]

• You can restore to an upgraded ArcGIS Data Store machine or to an ArcGIS Data Store installation on a machine with a different operating system than the source ArcGIS Data Store using the importmanageddb utility.

• importmanageddb <backup name> --source-loc <source backup location> --data-dir <data store data directory> --server-admin <user name of ArcGIS Server admin> --server-password <password of ArcGIS Server admin> [--include-tilecache <true|false>] [--bound {true|false}] [--server-url <ArcGIS Server URL registered with data store>] [--prompt {yes|no}]

Demo

ArcGIS Data StoreCommand Line Utilities

Moving ArcGIS Data Store to a different server

• There are times when organizations may need to move the ArcGIS Data Store content to a different server, such as a server hardware upgrade.

• Steps- Install Data Store on new server- Add machine to ArcGIS Server site as standby Data Store- Once server is added and data is fully replicated, promote the standby Data Store to

primary- Remove the standby data store from the site (the original primary)- Uninstall Data Store software from original server; decommission.

A

Upgrading ArcGIS Data Store

• Run exportmanageddb to create a full backup of relational and tile cache data stores; store in a safe location.

• Upgrade Portal for ArcGIS and ArcGIS Server before upgrading the ArcGIS Data Store.

• Run the upgrade version installation.• Configure the ArcGIS Data Store with the same ArcGIS Server site you were using

before upgrading; use the web Configuration Wizard or command line utility.• If you have a Primary and Standby ArcGIS Data Store, upgrade the Primary and

repeat the steps to upgrade the Standby.

A

Migrating from a traditional Managed Geodatabase to ArcGIS Data Store

• If you have an existing managed database and hosted feature layers, follow the below steps to migrate to the Relational ArcGIS Data Store.

- Export each Hosted Feature Service to a File Geodatabase via Item Details.- Delete the existing Hosted Feature Service from Portal.- Remove existing Managed Database from ArcGIS Server and replace with the ArcGIS Data

Store- Install ArcGIS Data Store and configure to the ArcGIS for Server site

- Republished Hosted Feature Services from the File Database backups.- Update WebMaps and republish apps

• AUTOMATION!! - Python and REST – Wednesday 5:00pm Tech Theater 15 (Hall A)- ArcGIS Online: Administering Your ArcGIS Organization through Scripting – Thursday

1:30pm Room 03A

Using ArcGIS Server Manager and Administrators Directory

• ArcGIS Manager and Administrator API can be used to provide some administration of the Data Store.

A

Using ArcGIS Server Manager and Administrators Directory

A

Demo

Data Store Administrator API

Summary

• Review – On-premises Web GIS• ArcGIS Data Store and the various stores• Configuring a Standby Data Store for Availability• Command Line Utilities - Demo• Move ArcGIS Data Store to another server• Upgrading ArcGIS Data Store• ArcGIS Manager and Administrator API

A

Want to learn more?

• Visit the Server GIS area and find the ArcGIS Data Store product stand

• Workshops- Building your first Web GIS – Tuesday 4:30pm Demo Theater 6- Web GIS: Architectural Patterns and Practices – Wednesday 8:30am Room 15B- Setting up a Basic High Availability Web GIS – Wednesday 4:30pm Demo Theater 6- Deploying ArcGIS for Server in the Cloud – Thursday 3:15pm Room 03- Python and REST – Wednesday 5:00pm Tech Theater 15 (Hall A)- ArcGIS Online: Administering Your ArcGIS Organization through Scripting – Thursday

1:30pm Room 03

A

Please take our SurveyYour feedback allows us to help maintain high standards and to help presenters

Find the session you want to review

Find your event in the Esri Events App

Scroll down to the bottom of the session

Answer survey questions and submit