Overcoming Challenges While
Migrating SDE Versions
Jameel Al-Khalaf
June 28, 2016
ESRI User Conference
© Copyright 2016, Saudi Aramco. All rights reserved.
2
Saudi Aramco: Company General Use
• Project Overview
• Project Objectives
• Migration Methods
• Challenges
• Benefits
• Lessons Learned
Agenda
3
Saudi Aramco: Company General Use
• Meeting IT Demand
Why Did We Initiate This Project?
4
Saudi Aramco: Company General Use
Project Objectives
System Upgrade
Hardware & Software
Data Migration
Raster &Vector
5
Saudi Aramco: Company General Use
System Upgrade
Database
Server 1
ArcSDE 9.3.1
Database
Server 2
ArcSDE 9.3.1
Old Cluster Environment
Source Geodatabase
(Our Data Center)
Cluster_Server_1 Cluster_Server_2
SAN
7 TB
Database
Upgraded Server 1
ArcSDE 10.2
Database
Upgraded Server 2
ArcSDE 10.2
New Cluster Environment
Target Geodatabase
(IT Data Center)
Cluster_Server_1 Cluster_Server_2
SAN
15 TB
Upgrading
H/W & S/W
6
Saudi Aramco: Company General Use
Data Migration
Database
Server 1
ArcSDE 9.3.1
Database
Server 2
ArcSDE 9.3.1
Old Cluster Environment
Source Geodatabase
(Our Data Center)
Cluster_Server_1 Cluster_Server_2
SAN
7 TB
Database
Upgraded Server 1
ArcSDE 10.2
Database
Upgraded Server 2
ArcSDE 10.2
New Cluster Environment
Target Geodatabase
(IT Data Center)
Cluster_Server_1 Cluster_Server_2
SAN
15 TB
Raster & Vector
(SDE Versions)
1
QC
2
QC
3
QC
4
QC
5
QC
6
QC
SDE
1 2 3 4 5 6
SDE
7
Saudi Aramco: Company General Use
Clearing The Path
U p g r a d e D e p e n d e n c e s U p g ra d e & D a t a M i g ra t i o n
S D E V e r s i o n s M i g r a t i o n P r o j e c t
System Upgrade
SDE Versions Migration
Raster
Updating TNS File
Upgrading Clients
App. Compatibility
8
Saudi Aramco: Company General Use
Data Migration Methods
Aramco
Develop tool to Export all SDE versions & Import it
on the target
Compress GDB to trim state tree
back to zero
Back up the entire DB & restore it on
the new server
Works only at one version below SDE
ESRI Oracle FGDB• No Documented Procedure
9
Saudi Aramco: Company General Use
• Listing FCs’ feature count
• Exporting DSs/FCs/Tables
• Listing Versions
• Listing Versions changes count
Features of Aramco Versions Migration Tool
• Creating Versions
• Exporting output logfile
• Exporting Deltas
• Importing Deltas
10
Saudi Aramco: Company General Use
1. Listing
Versions
2. Export
Datasets_DataOwner
3. Data
Statistics_DataOwner
4. Listing Versions
Changes Count
5. Create Version on
Target DB
6. Exporting
Versions
Deltas
11
Saudi Aramco: Company General Use
ChallengesIT •Lack of Admin
Privilege (TNS)
Technic
al
•DownloadCenter (ArcGIS & Ext.)
12
Saudi Aramco: Company General Use
Download Center
13
Saudi Aramco: Company General Use
ChallengesIT •Lack of Admin
Privilege (TNS)
Technic
al
•DownloadCenter(ArcGIS & Ext.)
•New Tool (Lots of Issues)
14
Saudi Aramco: Company General Use
Aramco Geodatabase Versioning Tree
15
Saudi Aramco: Company General Use
Version Name4. Listing Versions changes
count
5. Create Version on
Target
6. Exporting Versions
Deltas
DCs_Duba_Corr 0:05:37 Created 00:09:24
DCS_NorthWest_PipeLine 0:08:39 Created 00:10:08
DCS_Shaybah Wildlife 0:00:37 Created 0:04:45
EWG-1 PL Corr 0:05:25 Created 0:12:04
Ghinah_Reservation 00:13:44 Created
Harmaliyah_Reservation 00:05:11 Created 00:08:00
Jaladi_Reservation 01:15:59 Created
Jauf_Reservation 00:17:17 Created
Jizan_Refinery 00:03:17 Created 00:03:27
KAPSARC 00:04:19 Created 00:04:29
MNFG-1 00:08:09 Created
Riyadh Refinery Mapping 00:14:43 Created
Wasea_Bulk_Plant 00:12:35 Created
Exporting Main SDE Versions
16
Saudi Aramco: Company General Use
ChallengesIT •Lack of Admin
Privilege (TNS)
Technic
al
•DownloadCenter(ArcGIS & Ext.)
•New Tool (Lots of Issues)
• Finding Alternative Solutions (Free Tools)
17
Saudi Aramco: Company General Use
Section Tools Source Action / Alternative Solution
LSS
LSS Tools (As-Built, Proposed Pipeline, RTP) Developed In-House
Recompile for ArcGIS 10.2
Extraction (GDB to DGN)WDMS
EditorID Developed In-House
Z-Management
Free ESRI “Production Mapping” Extension3D Snapping
G&PSS
3D Snapping
PLTS ESRI Extension ESRI “Data Reviewer” Extension
Raster Clipping Free ESRI “Clip Data Management” Tool
Text2GDB In House Developed TBC/new developed procedure
Geodetic Manager Developed In-House Compatible with ArcGIS 10.2
Adding Ortho SWO Metadata
Developed In-House Recompile for ArcGIS 10.2
ASCOT Loader
GSITU
GDB Data RefreshSSD Data Synchronization ToolsAerial Photo PublisherAutomate Reconcile & Post Job
In-House Tools & Apps
18
Saudi Aramco: Company General Use
Project’s BenefitsH
ard
ware
• Better Performance
• Increased storage capacity
Soft
ware
• Bug-Fixes
• Cope with new Technology
• Enhanced editing functionalities
• New geo-processing tools
• Upgrade legacy tools
• House Cleanup (Versions)
• Fixed R_PipelineDataset
Data
19
Saudi Aramco: Company General Use
1. Corporate IT
2. Capitalize on the Upgrade Opportunity
3. Creating Checklists
4. Practice makes perfection
Lessons Learned…
20
Saudi Aramco: Company General Use
• Corporate IT …
- Identify the group/individual
- Meet ahead of time & make any necessary requests
- Lookout for any future IT policies/procedures to be
implemented
Lessons Learned [ 1 ]
21
Saudi Aramco: Company General Use
• Capitalize on the Upgrade Opportunity
- Get rid of free/unsupported tools & use maintained S/W
- Standardized our installation process (ArcGIS & TNSNames.ora)
- Updated our inventory lists (users, workstations, S/W licenses)
Lessons Learned [ 2 ]
22
Saudi Aramco: Company General Use
• Creating Checklists …
- Important to keep track of all implementation
procedures and details
• Example: Data Migration Checklist
Lessons Learned [ 3 ]
23
Saudi Aramco: Company General Use
Preparation:
1 • Re-compiling G&PSS and LSS existing 9.3 tools
2 • Availabily of ESRI Extensions
3 • Installing ArcGIS 10.2.2 on all G&PSS and LSS client machines
4 • Availability of SDE Media, SDE License, new cluster environment
5 • Develop the (Versions Migration Tool) to Read & Write other versions
6 • List of all headcounts (Schemas to be migrated, FC, FDS, Versions)
7 • Prepare MXD for QCing migrated data for each version
8 • Create User.Test Version Under Default on Source and make few changes (Add, Modify a name)
9 • Create MyUser.JDK Version Under Default on Source and make few changes (Add, Modify a name)
10 • Synch the databases (Replication)
11 • Stop and break the Replication process
12 • Send the announcement
13 • Make an Oracle dump (a restorable backup)
14 • Reconcile, Post, and Delete all child versions to the DEFAULT version. (Up to the Project Level)
DATA MIGRATION CHECKLIST
Source Geodatabase:
15 • Disconnect all users
16 • Make a backup of SDE (Using XML recordset)
17 • Analyze to update GDB statistics
18 • Compress the GDB
19 • Export Pipeline dataset as XML
Target Geodatabase:
20 • Prepare the server (Operating System, Database, ArcSDE 10.2.2)
21 • Copy Database Users, Roles, Profiles, Tablespaces from Source to Target
22 • Verify the count of imported (Users, Roles, Profiles, Tablespaces)
23 • Copy the DEFAULT version ( Source ) schema (Excluding Pipeline DS)
24 • Import the Pipeline DS XML Workspace file to target DB schema with its data
25 • Copy other schemas ( all data owners )
26 • Default data QC
27 • Role Assignments (grant Oracle Roles access permission to the FDS)
28 • Register Datasets As-Versioned
29 • Import MyUser.JDK version (using the Version Migration Tool)
QC (Data on Target):
37 • Verify the head count of (Feature Classes, Datasets, Versions, delta changes)
38 • MXD: Check the imported versions by selecting random areas on each version (source vs. target)
39 • Connect QCUser (G&PSS) to check migrated versions and data
40 • QCUser will edit on QC.Test_Version to test the new environment (Add, Delete)
41 • QCUser (G&PSS): Reconcile and Post QC.Test_Version to SDE.Default
42 • Other QC Users to test by creating new versions, editing, reconciling and posting sample data
43 • Send the announcement to connect all users
24
Saudi Aramco: Company General Use
• Practice makes perfection
- Once decided on the migration method:
• Conducted many tests
• Making different scenarios
• Utilizing different environments
• Developed validation procedures
Lessons Learned [ 4 ]
25
Saudi Aramco: Company General Use
DCs_Duba_Corr
Version Comparison
SOURCE TARGET MatchingR_AVT_AIRPORT_PT - (11) R_AVT_AIRPORT_PT - (11) TRUER_AVT_AVIATON_PL - (3) R_AVT_AVIATON_PL - (3) TRUER_BMC_POST_PT - (197) R_BMC_POST_PT - (197) TRUER_HYD_HYDRO_LN - (6555) R_HYD_HYDRO_LN - (6555) TRUER_HYD_HYDRO_PT - (66) R_HYD_HYDRO_PT - (66) TRUER_LDF_EXCAVATION_PL - (112) R_LDF_EXCAVATION_PL - (112) TRUER_LDF_SURFACE_PL - (171) R_LDF_SURFACE_PL - (171) TRUER_OAG_OILGAS_PL - (9) R_OAG_OILGAS_PL - (9) TRUER_OAG_TANK_PL - (12) R_OAG_TANK_PL - (12) TRUER_OAG_WELLPAD_PL - (6) R_OAG_WELLPAD_PL - (6) TRUER_OAG_WELL_PT - (1) R_OAG_WELL_PT - (1) TRUER_TRN_HIGHWAYSIGN_PT - (180) R_TRN_HIGHWAYSIGN_PT - (180) TRUER_TRN_ROADCNTLINE_LN - (13) R_TRN_ROADCNTLINE_LN - (13) TRUER_TRN_ROAD_LN - (1741) R_TRN_ROAD_LN - (1741) TRUER_TRN_TRANSPORTSTRUCTURE_PL - (8) R_TRN_TRANSPORTSTRUCTURE_PL - (8) TRUER_UDF_UNDEFINED_LN - (16) R_UDF_UNDEFINED_LN - (16) TRUER_UDF_UNDEFINED_PL - (94) R_UDF_UNDEFINED_PL - (94) TRUER_UTL_SUPPORTSTRUCTURE_PT - (2937) R_UTL_SUPPORTSTRUCTURE_PT - (2937) TRUER_UTL_TRANSMISSIONTOWER_PL - (371) R_UTL_TRANSMISSIONTOWER_PL - (371) TRUER_UTL_UNDGRNDCBLMARKER_PT - (868) R_UTL_UNDGRNDCBLMARKER_PT - (868) TRUER_UTL_UNDGRNDCBL_LN - (13) R_UTL_UNDGRNDCBL_LN - (13) TRUER_UTL_UTILITY_LN - (17) R_UTL_UTILITY_LN - (17) TRUER_UTL_UTILITY_PT - (427) R_UTL_UTILITY_PT - (427) TRUER_UTL_UTILITY_PL - (12) R_UTL_UTILITY_PL - (12) TRUE
26
Saudi Aramco: Company General Use
Source Target Diff.
B_MAPPINGBOUNDARY_PL - (9) B_MAPPINGBOUNDARY_PL - (9)B_PLACE_PT - (2) B_PLACE_PT - (2)
R_AVT_AVIATON_PL - (4) R_AVT_AVIATON_PL - (4)
R_BMC_POST_PT - (221) R_BMC_POST_PT - (221)R_HYD_HYDRO_LBL - (1) R_HYD_HYDRO_LBL - (1)
R_HYD_HYDRO_LN - (1342) R_HYD_HYDRO_LN - (1341) 1R_HYD_HYDRO_PT - (35) R_HYD_HYDRO_PT - (35)
R_TRN_CAMELCROSSING_PL - (82) R_TRN_CAMELCROSSING_PL - (82)R_TRN_GASOLINE_PT - (13) R_TRN_GASOLINE_PT - (13)R_TRN_HIGHWAYSIGN_LN - (2) R_TRN_HIGHWAYSIGN_LN - (2)R_TRN_HIGHWAYSIGN_PT - (24) R_TRN_HIGHWAYSIGN_PT - (24)R_TRN_ROADCNTLINE_LN - (4489) R_TRN_ROADCNTLINE_LN - (4489)
R_TRN_ROAD_LN - (3159) R_TRN_ROAD_LN - (3157) 2R_TRN_TRANSPORTSTRUCTURE_LN - (172) R_TRN_TRANSPORTSTRUCTURE_LN - (172)
R_TRN_TRANSPORTSTRUCTURE_PL - (77) R_TRN_TRANSPORTSTRUCTURE_PL - (77)R_UTL_UTILITY_PL - (186) R_UTL_UTILITY_PL - (186)R_UTL_WATERTANK_PL - (346) R_UTL_WATERTANK_PL - (346)
R_VEG_VEGETATION_PT - (889) R_VEG_VEGETATION_PT - (889)R_VEG_VEGETATION_PL - (121) R_VEG_VEGETATION_PL - (121)
3
Wasea_Bulk_Plant
27
Saudi Aramco: Company General Use
Wasea_Bulk_Plant
Source Geodatabase Target Geodatabase
R_TRN_ROAD_LN
28
Saudi Aramco: Company General Use
IT
CAD
• Geodatabase
• QC
• Data Captures
• Rapid Update
Team
• Infrastructure
• IT Support
• App. Development
• CSL
29
Saudi Aramco: Company General Use
Top Related