Overcoming Challenges While

Overcoming Challenges While

Migrating SDE Versions

Jameel Al-Khalaf

June 28, 2016

ESRI User Conference

© Copyright 2016, Saudi Aramco. All rights reserved.

2

Saudi Aramco: Company General Use

• Project Overview

• Project Objectives

• Migration Methods

• Challenges

• Benefits

• Lessons Learned

Agenda

3


• Meeting IT Demand

Why Did We Initiate This Project?

4


Project Objectives

System Upgrade

Hardware & Software

Data Migration

Raster &Vector

5


System Upgrade

Database

Server 1

ArcSDE 9.3.1

Database

Server 2

ArcSDE 9.3.1

Old Cluster Environment

Source Geodatabase

(Our Data Center)

Cluster_Server_1 Cluster_Server_2

SAN

7 TB

Database

Upgraded Server 1

ArcSDE 10.2

Database

Upgraded Server 2

ArcSDE 10.2

New Cluster Environment

Target Geodatabase

(IT Data Center)


SAN

15 TB

Upgrading

H/W & S/W

6


Data Migration

Database

Server 1

ArcSDE 9.3.1

Database

Server 2

ArcSDE 9.3.1

Old Cluster Environment

Source Geodatabase

(Our Data Center)


SAN

7 TB

Database

Upgraded Server 1

ArcSDE 10.2

Database

Upgraded Server 2

ArcSDE 10.2

New Cluster Environment

Target Geodatabase

(IT Data Center)


SAN

15 TB

Raster & Vector

(SDE Versions)

1

QC

2

QC

3

QC

4

QC

5

QC

6

QC

SDE

1 2 3 4 5 6

SDE

7


Clearing The Path

U p g r a d e D e p e n d e n c e s U p g ra d e & D a t a M i g ra t i o n

S D E V e r s i o n s M i g r a t i o n P r o j e c t

System Upgrade

SDE Versions Migration

Raster

Updating TNS File

Upgrading Clients

App. Compatibility

8


Data Migration Methods

Aramco

Develop tool to Export all SDE versions & Import it

on the target

Compress GDB to trim state tree

back to zero

Back up the entire DB & restore it on

the new server

Works only at one version below SDE

ESRI Oracle FGDB• No Documented Procedure

9


• Listing FCs’ feature count

• Exporting DSs/FCs/Tables

• Listing Versions

• Listing Versions changes count

Features of Aramco Versions Migration Tool

• Creating Versions

• Exporting output logfile

• Exporting Deltas

• Importing Deltas

10


1. Listing

Versions

2. Export

Datasets_DataOwner

3. Data

Statistics_DataOwner

4. Listing Versions

Changes Count

5. Create Version on

Target DB

6. Exporting

Versions

Deltas

11


ChallengesIT •Lack of Admin

Privilege (TNS)

Technic

al

•DownloadCenter (ArcGIS & Ext.)

12


Download Center

13



Privilege (TNS)

Technic

al

•DownloadCenter(ArcGIS & Ext.)

•New Tool (Lots of Issues)

14


Aramco Geodatabase Versioning Tree

15


Version Name4. Listing Versions changes

count

5. Create Version on

Target

6. Exporting Versions

Deltas

DCs_Duba_Corr 0:05:37 Created 00:09:24

DCS_NorthWest_PipeLine 0:08:39 Created 00:10:08

DCS_Shaybah Wildlife 0:00:37 Created 0:04:45

EWG-1 PL Corr 0:05:25 Created 0:12:04

Ghinah_Reservation 00:13:44 Created

Harmaliyah_Reservation 00:05:11 Created 00:08:00

Jaladi_Reservation 01:15:59 Created

Jauf_Reservation 00:17:17 Created

Jizan_Refinery 00:03:17 Created 00:03:27

KAPSARC 00:04:19 Created 00:04:29

MNFG-1 00:08:09 Created

Riyadh Refinery Mapping 00:14:43 Created

Wasea_Bulk_Plant 00:12:35 Created

Exporting Main SDE Versions

16



Privilege (TNS)

Technic

al

•DownloadCenter(ArcGIS & Ext.)

•New Tool (Lots of Issues)

• Finding Alternative Solutions (Free Tools)

17


Section Tools Source Action / Alternative Solution

LSS

LSS Tools (As-Built, Proposed Pipeline, RTP) Developed In-House

Recompile for ArcGIS 10.2

Extraction (GDB to DGN)WDMS

EditorID Developed In-House

Z-Management

Free ESRI “Production Mapping” Extension3D Snapping

G&PSS

3D Snapping

PLTS ESRI Extension ESRI “Data Reviewer” Extension

Raster Clipping Free ESRI “Clip Data Management” Tool

Text2GDB In House Developed TBC/new developed procedure

Geodetic Manager Developed In-House Compatible with ArcGIS 10.2

Adding Ortho SWO Metadata

Developed In-House Recompile for ArcGIS 10.2

ASCOT Loader

GSITU

GDB Data RefreshSSD Data Synchronization ToolsAerial Photo PublisherAutomate Reconcile & Post Job

In-House Tools & Apps

18


Project’s BenefitsH

ard

ware

• Better Performance

• Increased storage capacity

Soft

ware

• Bug-Fixes

• Cope with new Technology

• Enhanced editing functionalities

• New geo-processing tools

• Upgrade legacy tools

• House Cleanup (Versions)

• Fixed R_PipelineDataset

Data

19


1. Corporate IT

2. Capitalize on the Upgrade Opportunity

3. Creating Checklists

4. Practice makes perfection

Lessons Learned…

20


• Corporate IT …

- Identify the group/individual

- Meet ahead of time & make any necessary requests

- Lookout for any future IT policies/procedures to be

implemented

Lessons Learned [ 1 ]

21


• Capitalize on the Upgrade Opportunity

- Get rid of free/unsupported tools & use maintained S/W

- Standardized our installation process (ArcGIS & TNSNames.ora)

- Updated our inventory lists (users, workstations, S/W licenses)


22


• Creating Checklists …

- Important to keep track of all implementation

procedures and details

• Example: Data Migration Checklist


23


Preparation:

1 • Re-compiling G&PSS and LSS existing 9.3 tools

2 • Availabily of ESRI Extensions

3 • Installing ArcGIS 10.2.2 on all G&PSS and LSS client machines

4 • Availability of SDE Media, SDE License, new cluster environment

5 • Develop the (Versions Migration Tool) to Read & Write other versions

6 • List of all headcounts (Schemas to be migrated, FC, FDS, Versions)

7 • Prepare MXD for QCing migrated data for each version

8 • Create User.Test Version Under Default on Source and make few changes (Add, Modify a name)

9 • Create MyUser.JDK Version Under Default on Source and make few changes (Add, Modify a name)

10 • Synch the databases (Replication)

11 • Stop and break the Replication process

12 • Send the announcement

13 • Make an Oracle dump (a restorable backup)

14 • Reconcile, Post, and Delete all child versions to the DEFAULT version. (Up to the Project Level)

DATA MIGRATION CHECKLIST

Source Geodatabase:

15 • Disconnect all users

16 • Make a backup of SDE (Using XML recordset)

17 • Analyze to update GDB statistics

18 • Compress the GDB

19 • Export Pipeline dataset as XML

Target Geodatabase:

20 • Prepare the server (Operating System, Database, ArcSDE 10.2.2)

21 • Copy Database Users, Roles, Profiles, Tablespaces from Source to Target

22 • Verify the count of imported (Users, Roles, Profiles, Tablespaces)

23 • Copy the DEFAULT version ( Source ) schema (Excluding Pipeline DS)

24 • Import the Pipeline DS XML Workspace file to target DB schema with its data

25 • Copy other schemas ( all data owners )

26 • Default data QC

27 • Role Assignments (grant Oracle Roles access permission to the FDS)

28 • Register Datasets As-Versioned

29 • Import MyUser.JDK version (using the Version Migration Tool)

QC (Data on Target):

37 • Verify the head count of (Feature Classes, Datasets, Versions, delta changes)

38 • MXD: Check the imported versions by selecting random areas on each version (source vs. target)

39 • Connect QCUser (G&PSS) to check migrated versions and data

40 • QCUser will edit on QC.Test_Version to test the new environment (Add, Delete)

41 • QCUser (G&PSS): Reconcile and Post QC.Test_Version to SDE.Default

42 • Other QC Users to test by creating new versions, editing, reconciling and posting sample data

43 • Send the announcement to connect all users

24


• Practice makes perfection

- Once decided on the migration method:

• Conducted many tests

• Making different scenarios

• Utilizing different environments

• Developed validation procedures


25


DCs_Duba_Corr

Version Comparison

SOURCE TARGET MatchingR_AVT_AIRPORT_PT - (11) R_AVT_AIRPORT_PT - (11) TRUER_AVT_AVIATON_PL - (3) R_AVT_AVIATON_PL - (3) TRUER_BMC_POST_PT - (197) R_BMC_POST_PT - (197) TRUER_HYD_HYDRO_LN - (6555) R_HYD_HYDRO_LN - (6555) TRUER_HYD_HYDRO_PT - (66) R_HYD_HYDRO_PT - (66) TRUER_LDF_EXCAVATION_PL - (112) R_LDF_EXCAVATION_PL - (112) TRUER_LDF_SURFACE_PL - (171) R_LDF_SURFACE_PL - (171) TRUER_OAG_OILGAS_PL - (9) R_OAG_OILGAS_PL - (9) TRUER_OAG_TANK_PL - (12) R_OAG_TANK_PL - (12) TRUER_OAG_WELLPAD_PL - (6) R_OAG_WELLPAD_PL - (6) TRUER_OAG_WELL_PT - (1) R_OAG_WELL_PT - (1) TRUER_TRN_HIGHWAYSIGN_PT - (180) R_TRN_HIGHWAYSIGN_PT - (180) TRUER_TRN_ROADCNTLINE_LN - (13) R_TRN_ROADCNTLINE_LN - (13) TRUER_TRN_ROAD_LN - (1741) R_TRN_ROAD_LN - (1741) TRUER_TRN_TRANSPORTSTRUCTURE_PL - (8) R_TRN_TRANSPORTSTRUCTURE_PL - (8) TRUER_UDF_UNDEFINED_LN - (16) R_UDF_UNDEFINED_LN - (16) TRUER_UDF_UNDEFINED_PL - (94) R_UDF_UNDEFINED_PL - (94) TRUER_UTL_SUPPORTSTRUCTURE_PT - (2937) R_UTL_SUPPORTSTRUCTURE_PT - (2937) TRUER_UTL_TRANSMISSIONTOWER_PL - (371) R_UTL_TRANSMISSIONTOWER_PL - (371) TRUER_UTL_UNDGRNDCBLMARKER_PT - (868) R_UTL_UNDGRNDCBLMARKER_PT - (868) TRUER_UTL_UNDGRNDCBL_LN - (13) R_UTL_UNDGRNDCBL_LN - (13) TRUER_UTL_UTILITY_LN - (17) R_UTL_UTILITY_LN - (17) TRUER_UTL_UTILITY_PT - (427) R_UTL_UTILITY_PT - (427) TRUER_UTL_UTILITY_PL - (12) R_UTL_UTILITY_PL - (12) TRUE

26


Source Target Diff.

B_MAPPINGBOUNDARY_PL - (9) B_MAPPINGBOUNDARY_PL - (9)B_PLACE_PT - (2) B_PLACE_PT - (2)

R_AVT_AVIATON_PL - (4) R_AVT_AVIATON_PL - (4)

R_BMC_POST_PT - (221) R_BMC_POST_PT - (221)R_HYD_HYDRO_LBL - (1) R_HYD_HYDRO_LBL - (1)

R_HYD_HYDRO_LN - (1342) R_HYD_HYDRO_LN - (1341) 1R_HYD_HYDRO_PT - (35) R_HYD_HYDRO_PT - (35)

R_TRN_CAMELCROSSING_PL - (82) R_TRN_CAMELCROSSING_PL - (82)R_TRN_GASOLINE_PT - (13) R_TRN_GASOLINE_PT - (13)R_TRN_HIGHWAYSIGN_LN - (2) R_TRN_HIGHWAYSIGN_LN - (2)R_TRN_HIGHWAYSIGN_PT - (24) R_TRN_HIGHWAYSIGN_PT - (24)R_TRN_ROADCNTLINE_LN - (4489) R_TRN_ROADCNTLINE_LN - (4489)

R_TRN_ROAD_LN - (3159) R_TRN_ROAD_LN - (3157) 2R_TRN_TRANSPORTSTRUCTURE_LN - (172) R_TRN_TRANSPORTSTRUCTURE_LN - (172)

R_TRN_TRANSPORTSTRUCTURE_PL - (77) R_TRN_TRANSPORTSTRUCTURE_PL - (77)R_UTL_UTILITY_PL - (186) R_UTL_UTILITY_PL - (186)R_UTL_WATERTANK_PL - (346) R_UTL_WATERTANK_PL - (346)

R_VEG_VEGETATION_PT - (889) R_VEG_VEGETATION_PT - (889)R_VEG_VEGETATION_PL - (121) R_VEG_VEGETATION_PL - (121)

3

Wasea_Bulk_Plant

27


Wasea_Bulk_Plant

Source Geodatabase Target Geodatabase

R_TRN_ROAD_LN

28


IT

CAD

• Geodatabase

• QC

• Data Captures

• Rapid Update

Team

• Infrastructure

• IT Support

• App. Development

• CSL

29


Overcoming Challenges While

Documents

Transcript of Overcoming Challenges While