Implementing the Change Detector transformer to process data in an SDE

June 12, 2014

Brooks Wilson – Northrop GrummanJeff Safran – Bureau of Land Management

Steve Gregonis – Bureau of Land Management

Abstract

This project involved collecting spatial data from 10 different sources and creating a single compilation dataset. To avoid having to rebuild the compilation every week to keep it current, FME and the change detector were employed. An SDE for each source is updated through a replica. A version in the database is used to keep the status of the data before replication. The change detector transformer finds the differences between the default and the version. Features that are unchanged are ignored, added features are passed to the compilation SDE and deleted features are removed from the compilation SDE. Added features insert a new attribute to maintain the original Global ID so it can be identified for deletion in the future. This workflow greatly reduces processing time of large changing datasets.

Our Goal Create a compiled dataset from many office submissions. Track the changes from each office.

To achieve these goals, we created a custom transformer that processed only the changes from each office and ignored the unchanged features.

How it works

Each office provides their data through a replica to a local SDE for that office.

The replica synchronizes with a version that is a child of the Default version.

When the replica is synchronized, Default contains the original features and the Replica Version contains the updated features.

The FeatureChangeDetector looks at the changes between Default and the Replica Version.

Added features are added to the compilation. Deleted features are removed from the compilation. SDE treats updated features as an Add and a Delete.

Inputs and Outputs

INPUTS OUTPUTS

Original Feature Class Unchanged Features

Updated Feature Class Added Features

Compilation Features Deleted Features

The Feature Change Detector

Change Detector

The first step is comparing the original features (Default) to the changed features (Replica Version) in the Change Detector transformer.

Unchanged features are passed out of the transformer with no further processing.

Added Features

As features are written to the Compilation SDE they are given a new GlobalID.

An attribute,Original_GlobalID, is added. The Original_GlobalID field allows us to find the office

features in the compilation feature class. The ability to differentiate between office features is

important during deletion of features

Deleted Features

Deleted features are passed to FeatureMerger transformer. The feature is compared by GlobalID to the compilation SDE

using the Original GlobalID. Features that match are passed out the referenced port and

are marked for deletion.

Output

Added and Deleted outputs are directed to the compilation writer.

Updated features are treated as an Add and a Delete by SDE

Unchanged features are already a part of the compilation, they are ignored or written out to a inspector.

Initialization

The first time the transformer runs, there will not be any differences between Default and the Replica Version.

For the initialization of the compilation, all features from the unchanged are routed as changed features.

Problems Encountered

An office would break replication, which would make it difficult to remove their work from the compilation.

One office did not use replication. If the version was accidentally posted, it was difficult

to find the changes that were missed. It was eventually determined that it was not

necessary to track all changes.

Currently, the compilation tables are dropped and recreated each time the data is compiled as a way to ensure our datasets are complete. However, it does take much longer to process in this manner.

Review

Process works and is useable. There should be a version per replica you are creating. The data needs to be clean. The data structure needs to be strong and unchanging. All data suppliers need to use SDE. There should be a requirement of speed or edit tracking

to justify the work necessary to set up this process.

Contact Information

Brooks WilsonNational ESRI Software Support EngineerNational Operations Center (NOC)Office - (775) 861-6545Email - bhwilson@blm.govContractor - Northrop Grumman