ENOVIA SmarTeamData Loading Strategies

Marc Young, CEO

xLM Solutions, LLC

Strategies to ease the migration pain…Loading, or migrating, data into ENOVIA SmarTeam can be a daunting task…

With diligent planning and thorough testing, the process can go smoothly

• Data Clean-up and Process Review• Data Formats• Data Loading Rules• Migration Cost Estimation

1. Pre-Migration Data Analysis

• Bulk Loader• Import Tools• Custom Scripts

2. Data Loading Options

• Testing the Migration Process• Data Verification• Pre-Data Load Planning• Production Data Load

3. Execution

Data AnalysisAnalyze the Data to be Loaded

Determine legacy data nature and

location• File-/Folder-based• Legacy systems

(databases, vaults, etc.)

• Hard copy (e.g., aperture cards)

• Type of data: CAD (with internal links), office, etc.

Anticipate data access issues

• All sources available?

• Third-party systems accesses?

• Additional experts?• Additional

resources?

Is CAD data conversion needed?

• e.g., ProE → CATIA• V4 → V5

Data volume and estimated time

to load• Will/Can downtime

be scheduled for data loading?

Analysis: Data Clean-Up

Typically

involves

human intellige

nce

May be time consuming

May not be worthwhile

Opportunity for Data

Clean-Up

Analysis: Data Clean-Up

What types of data clean-up are required?• Duplicated part numbers per file• Duplicated file names• Missing or incorrect file attributes• Missing or incorrect file associations or links• File revision not current• Non-current parts in current assemblies• Non-compliance or old revision/versioning

scheme

Validate data accuracy

Analysis: Process Reviews and Updates

• An opportunity to review business processes

• Opportunity to evaluate and update user experiences

• Important: new changes to the underlying data model must be verified with the migration processes

Analysis: Data FormatsCAD Data• File-based• Typically has relationships among files• Attributes and properties contained within the file

Meta Data to be Loaded or Added• Text description of data external to the associated (CAD) file• Users

• Import from existing Active Directory• Lookup Lists• Projects• File location pointers

Combined Data• CAD data with embedded relationships; but…• Attributes and properties reside outside of the CAD file

• e.g., in an Excel spreadsheet• Construct a link between the CAD file and attribute data

CAD Data

Meta Data to be Loaded or Added

Combined Data

Analysis: Determine the Data Loading RulesWhat are the rules to determine

where the data goes?

How will data fields be mapped from legacy system to SmarTeam?

What state(s) will the data be loaded to?

Will the data be linked to other SmarTeam data?

How to handle the cases with missing data?

Which data revision to load: latest or full history?

• Should the data be located in a folder?• Should a folder be automatically

created for the data?

• Text, Lookup, Reference To Class, URL

• If so, what is the ruleset?

• If a full data history is desired, determine how the revision history is maintained in the legacy system

Analysis: Effort and Cost Estimations

Data Load Project Cost Quantification

Time (data loading tools development

and testing)

Price

Resources involved

Impact of downtime for

users

• Data Clean-up and Process Review• Data Formats• Data Loading Rules• Migration Cost Estimation

1. Pre-Migration Data Analysis

• Manual Processing• SmarTeam Import and Export Tools• Custom Scripts (executables, add-ins, etc)• Combined Manual Processing and Custom Scripting• Pre-Developed Data Migration Programs

2. Data Loading Options

• Testing the Migration Process• Data Verification• Pre-Data Load Planning• Production Data Load

3. Execution

2. Data Loading Options

3. Execution

Overview SmarTeam Bulk Loading Tool• Load CAD File data that

SmarTeam has integration for• CATIA, SolidWorks, SolidEdge,

Inventor, AutoCAD• Highlight a set of files that should

be loaded into SmarTeam• The bulk loader will open each file

in the CAD application one at time and perform a SmarTeam Save

• Recommended way to bulk load standard parts

Setup for SmarTeam Bulk Loading Tool• CAD Integration should be configured

• Associated CAD File Type to SmarTeam Class• Map CAD Attributes to SmarTeam field

• Sequence needs to be assigned to the primary Id• Batch Mode Save Option should be turned on

• Otherwise, you will have to hit okay for each file to be loaded.

• Default Class to be specified

Issues with SmarTeam Bulk Loading Tool• Speed – an open and save performed on all files being loaded• CAD data integrity – If links are bad/missing in the CAD data:

• the files will not be saved• may require user intervention per occurrence

• Attribute data needs to exist in CAD properties* • No log created for data load*• Difficult if CAD data is to be associated with more than one class*

• An Engineering Part, Manufacturing Part, Standard Part• All data is linked to only one project*• May require copying CAD to data from various directories to one

directory• Does not handle loading of multiple revisions

*Avoidable with scripting

Overview of SmarTeam Import Tool• Out of the box tool provided by

SmarTeam to load meta data• The ability to pick a class to load data

• Leaf Classes• Link Classes i.e. Document Project

Links• The ability to map data from:

• Database – SmarTeam DB or non SmarTeam DB

• Text File – Delimited or Fixed Width• Default Value – Defined in SmarTeam• Default Value – Specific for the data

load• Populate field by a Sequence

• Log created for data that is imported• Import Tool can run in an Add or Update

mode

Issues with SmarTeam Import Tool• Limited to 32,000 records per load

• Greater than 32,000, data needs to be split and multiple loads conducted

• Tool can be “buggy” • Crashing from time to time• Data Mappings may get lost

• Tool is manual• For each class to be imported mappings

should be configured• Can be tedious to do for many classes

• Tool does not handle certain condition (with out Scripting)• Loading multiple revisions and specifying

the latest revision• Loading data to internal SmarTeam fields or

Reference to Class fields

Overview Custom Scripts / Programs• Best suited for larger data

migrations• Almost any type of data loading

rule can be created• Determine what class to load

the data • Modify data to be loaded

based on data in the system• Re-formatted before going

into SmarTeam• Need to create or build link

relationships in the system• Various complex Data load

scenarios will be conducted• Need custom logging an error

handling

Custom Script Data Loading - Simple Example• Data exists in a legacy Folder• File Name contains meta data attributes such as• ID_Descritpion_Rev• .doc files will go into an Engineering folder• .xls files will go into an Manufacturing folder• Project will need to be specified• Maintain latest revison

SmarTeam Custom Script Tool Example

Combination Bulk Load and Custom script / Import Utility• Bulk Load CAD data into

SmarTeam• Hook scripts to the SmarTeam

“Before Add” Event to retrieve information • i.e. from the file name to

update fields on the Profile Cards

• Write custom code that will query the database based on the file name• Update information from

another source where the file name is used to link the file data /object already in SmarTeam and the other external source for data

• Data Clean-up and Process Review• Data Formats• Data Loading Rules• Migration Cost Estimation

1. Pre-Migration Data Analysis

• Bulk Loader• SmarTeam Import and Export Tools• Custom Scripts (executables, add-ins, etc)• Combined Manual Processing and Custom Scripting• Pre-Developed Data Migration Programs

2. Data Loading Options

• Testing the Migration Process• Data Verification• Pre-Data Load Planning• Production Data Load

3. Execution

2. Data Loading Options

3. Execution

Execution:Testing the Migration Process

• Test the migration process!!!• Mitigate all discovered issues

• Legacy data issues» Missing data files» Orphan links and/or blank fields» Inconsistent legacy data such as revision schema» Bad formats

– E.g., Attempts to load characters into integer fields– Class determination rules fail

» Repairing the legacy data– Manual clean-up– Change data load rules (i.e., change the code)– Modify the SmarTeam data model to better accommodate the

legacy data

• Bug(s) in the migration tool code

• Data loading procedural issues» e.g., failure to delete all files with a creation date before 12/31/1999

• Document the migration procedure or update the tool to automate the migration process

• Test again!!!

Crucial to implement a data verification strategy

Analyze migration process log files Spot-check data

Randomly verify the data objects loaded Check at least 10% of data

User validation (done through testing with the pilot system) – an important part of the verification process

User validations >> IT/CAD administrators sign-offs

Develop a tool to compare and analyze the data in SmarTeam and legacy system(s) Verify the number of records between systems Ensure pertinent fields and data are preserved

in both systems

Execution: Pre-Data Load Planning• When will the legacy system be turned off?• How long will it take to load the data?• Will the data loading occur during non-

business hours?• Specify the data load process:

o What are the required pre-data loading activities/events?

• e.g., legacy data clean-up

o Document the data load sequence• Populate the lookup tables• Import users before documents• Import projects and documents, followed by the

associated links• Create references to class data

• Identify personnel responsible for the various parts of the data load process

Execution: Final Data Load• With proper planning and testing, the

final data load process “should” go smoothly

• Advices:

• Follow the documented procedures

• Allocate buffer time for unexpected issues

• Prepare to support users in the new system

Thank You!

Questions?

Marc YoungxLM Solutions, LLC.248-926-5932myoung@xlmsolutions.com

Script example shown is available at: http://www.xlmsolutions.com/COE2009/example.zip