Introduction to the Data Migration Framework in Microsoft...
-
Upload
hoangthien -
Category
Documents
-
view
227 -
download
1
Transcript of Introduction to the Data Migration Framework in Microsoft...
1 AX 2012 Data Migration Framework
Introduction to the Data Migration Framework in Microsoft Dynamics
by Ruben Barron
Senior Consultant - Blue Horseshoe Solutions.
2 AX 2012 Data Migration Framework
Table of Contents Intro to the Data Migration Framework (DMF) ........................................................................................ 3
DMF Process Overview ............................................................................................................................. 3
Prepare Source Data ............................................................................................................................. 3
Define type of AX entity to be used ...................................................................................................... 3
Fields in the Source file are mapped from the source file to the staging table .................................... 4
Transferring Data to the Staging Table ................................................................................................. 4
Data transferred from the Staging tables to the final AX Target destination. ...................................... 4
DMF Components ..................................................................................................................................... 4
Accessing the DMF tool. ........................................................................................................................ 4
Set Up – Target Entities ......................................................................................................................... 5
Set Up - Source Data Format ................................................................................................................ 6
Set Up – File Location (data migration framework parameters) .......................................................... 8
Processing Groups ..................................................................................................................................... 9
Entities .................................................................................................................................................. 9
Moving Data from Source Files to Staging Tables............................................................................... 14
Moving Data from staging table to the Final AX destination. ............................................................. 16
3 AX 2012 Data Migration Framework
Intro to the Data Migration Framework (DMF) For any organization embarking into a new MS Dynamics AX 2012(AX 2012) implementation loading
legacy data from difference sources into AX2012 is a critical component to have a good start and a
successful implementation. Microsoft introduced the new Data Migration Framework (DMF) as an
extension of AX 2012 to address the challenge of loading data to AX 2012. We consider that the DMF
tool represents a great step forward for data upload and with great potential that can be used to
complete data migration for the most common data entities needed for an implementation as well as a
framework that can be customized to load other required entities.
DMF Process Overview The DMF process can be summarized in the following diagram and includes preparing source data,
defining the type and order of AX entities to load, mapping the source file to the staging table,
transferring from the source file to the staging table and finally transferring the data to the final
destination in AX.
Data Migration Process
Prepare Source Data
Define type of AX entity to be used
Mapping and Validation of
Source File fields to Staging AX fields
Transferring Data from Source file to the Staging Table
Transfer of data from Staging to
target AX destination.
· Prepare source files to be uploaded.
· Define AX entity(entities) to be used for upload.
· Example of entities include Customers, Vendors, Sales Orders, Journal Entries, etc.
· Mapping and validation before data is transferred from source file to staging
· Transfer data from source file to Staging table.
· Data can be validated at this stage too.
· Transfer data from Staging to final target AX destination.
· AX records are created or updated
DMF Process Overview
Prepare Source Data
During this step, source data files are created and prepared to be uploaded to the AX application. A
typical source file will have a header row with field names identifying each data column. The number of
fields for each file will depend on the requirements for data upload. Thus it is important to note that
some of these field names are required for each entity. The DMF version only supports flat files. Future
versions of the DMF tool will support data from ODBC and AX destinations.
Define type of AX entity to be used
During this step we define the entity or entities that will be loaded to AX. These DMF entities are
predefined relationships that come in the application and include common AX entities such as
customers, vendors, products, etc. and available AX fields.
4 AX 2012 Data Migration Framework
Fields in the Source file are mapped from the source file to the staging table
During the mapping stage, the DMF tool will allow you to map each field from the source file to an AX
field in the in the staging table for the corresponding DMF entity. The DMF tool uses the field name to
map them against the available fields in the AX Staging table. Users have the ability to review and
manually map these fields.
Transferring Data to the Staging Table
Once fields from the source file have been mapped to AX fields in the staging table data should be ready
to be transferred to the staging table. This staging table acts as a temporary area and intermediate
location before data is transferred to the final AX destination. While the data resides on the staging
table, the DMF can apply some basic validation for referential data integrity.
Data transferred from the Staging tables to the final AX Target destination.
Once data has been reviewed and validated in the stating table, the data is then moved to the target AX
entity and AX records are created. If data is reloaded the DMF tool will try to update the AX fields.
DMF Components
Accessing the DMF tool.
After installation, the DMF tool can be accessed from the AX client and shows as a module in the AX
application. You will see a common, setup and periodic areas as part of the DMF module.
Data Migration Framework in AX 2012
5 AX 2012 Data Migration Framework
Set Up – Target Entities
There is some setup that needs to be configured prior to use the DMF tool.
Target Entities - The DMF tool comes with a set of predefined entities that can be used to load the most
common entities in AX. The following list includes the entities supported out of the box:
· Bill of materials (BOM)
· Contacts
· Contact addresses
· Customer
· Customer addresses
· Dimension
· Employees
· Employee addresses
· Inventory journal
· Ledger balances
· Main account
· Open invoices (accounts payable)
· Open invoices (accounts receivable)
· Open sales orders
· Open purchase orders
· Price disc (agreements)
· Product
· Project
· Unit of measure
· Vendor
· Vendor address
The list of available entities can be accessed from the DMF> setup > Entities area. These DMF entities
come already preconfigured and can be defined as a mapping relationship between the fields in the
staging table and the target destination. These DMF entities also include procedures to move data
from the staging table to the final destination creating objects and fields in AX. It is noteworthy to say
that these entities can be used to create new custom entities with additional fields or logic. The DMF
tool also comes with a custom entity wizard to create a DMF entity from scratch.
6 AX 2012 Data Migration Framework
Set up – Data Entity Sample
Set Up - Source Data Format
Source format allow the DMF tool to identity the type of source data that will be imported into AX. In
the source format, a user specifies the different parameters that define the source file to be imported.
Some of the parameters that can be specified include the type of source(file, ODBC, AX table), file
format, first row header, row column delimiters and few other described in the following table. A user
can define more than one source format to be used in the DMF upload process based on specific needs.
Example a source format can be created for flat files that are comma delimited. Another source format
can be created for flat files with fixed length and a different source format can be created for data that
is planned to be imported from an ODBC connection.
7 AX 2012 Data Migration Framework
Setup - Source Data Formats
FIELD DESCRIPTION
GENERAL TAB
Source Name Identifies the different source that can be configured in the application.
Type Types include File, AX and ODBC. DMF currently supports only Flat files. Future versions of the DMF will support AX tables and ODBC connections.
File Format Options include character delimited or fixed length. Character delimited includes characters such as commas, semicolon, pipes, etc.
First Row Header Flag to indicate if firs row of the file includes a header row.
Row Column Delimiter Indicates which characters will be used to determine end of the line for the source file. Defaults to {CR}{LF)
Text Qualifier Indicates character to encase text and commas. Example “8230 E Main Ave, Suite A“
Regional Settings Regional settings for file. We normally default this to code 1252 for English. A very large selection of regional setting come with the application to cover all world regions/languages depending on customer requirements. It also includes list for Unicode and other specific formats.
8 AX 2012 Data Migration Framework
Multiple Value Separator Delimited used to indicate multiple entries. Example multiple addresses, multiple email, multiple phones 317-509-90900 ; 508-999-1212
Table 1.1 Setup – Source Format General Setup
There is an Application tab that is part of the Source Format configuration and that include several
setting to configure dimensions and how data is going to be processed.
Setup – Source format – Application tab
Set Up – File Location (data migration framework parameters)
There is a required configuration that includes defining a shared folder in which files will be located for
the DMF tool. The DMF admin user needs to have admin access to this folder in order for the DMF to
properly work.
Setup – DMF Parameters (shared directory)
9 AX 2012 Data Migration Framework
Processing Groups
Processing Groups are used in DMF as the main driver that will allow users to configure entities, move
data to the stating table, validate data and copy data from the staging to the final AX destination. These
processing groups can be reused to reimport data and also keep a history log of transactions and errors.
Processing Group – Creating a Processing Group
Entities
After creating a processing group, the first thing to configure is an entity. These entities are used to
indicates what DMF entity to be used( example Vendor, customer) for processing, order in which
entities will be processed if more than one is selected, data source to use, source file to use for
processing and if AX business logic will be taking into consideration when uploading. The user will also
use this entity to map and validate the source fields against the staging AX fields.
Settings for Entity configuration
New Entity: A user will select the New menu from the processing group to create a new entity or
entities.
Sequence: Indicates the order in which entity or entities will be processed to load data from source to
staging area. You can have more than one entity to be processed. An example could be to load
vendors, products and customers in one single load. While this is option is available, we prefer to keep
it simple and only run one entity per processing group to keep things simple to debug and also allow the
processing group to be reused for future loads.
Entity Name: The entity name is used to select the DMF entity that will be used for processing data.
This entity contains the mapping relationship with all the available fields that will be mapped from the
source file to the staging tables.
10 AX 2012 Data Migration Framework
Source Format: This source data formats are configured in the setup and specifies the type of source
data that will be loaded.
Run Business Logic: There are two setting that can be used to enable the check for AX business logic.
Sample File Path: This path is used to indicate the location of the source file to be used for the mapping
and validation of source data. Note: If a large file is used you can use just a sample of data to do
perform the mapping and validation.
Processing Groups – Entity Configuration
Generate Source Mapping: The next step is to map the source file against the staging table. If you use
AX column names as your header the DMF tool will automatically try to do the mapping. If a different
name is used, a user will have to manually configure the mappings. It is probably recommended to
generate the source template file using the DMF tool tp speed up the mapping process. Still the user
will need to review and approve the mapping. Use the “Generate Source File” to create a template with
AX fields.
11 AX 2012 Data Migration Framework
Modify Source Mapping: During this stage the user can review and modify the mappings from the
source to the staging area. If any of the fields on the Source file is not mapped, a user has the option to
manually link the field to desired field on the staging table using the graphical interface or on the table
view.
Processing Group – Entity Mapping Source to Staging
12 AX 2012 Data Migration Framework
Validate: After the mappings have been reviewed, the user can then validate. The validation process
ensures that all fields in the source file are correctly mapped to the staging table. If an error is
encountered a user can go back to the mappings and make corrections in the mappings.
Processing Group – Validation
View Target Mapping: This option can be used to validate the mappings from the staging to final Ax
target destination. Please note that the staging only contains fields from the source file. Fields that
are not used are nor displayed and will not be used in your migration.
13 AX 2012 Data Migration Framework
Processing Group – View Target Mapping
Preview Source File: Once mappings have been validated, a user has the option to preview the source
file before the file is copied to the Staging Table. This is a good check that show how data aligns with
each AX field after mapping. Please note that is not data validation during this stage..
Processing Group – Preview data source file.
14 AX 2012 Data Migration Framework
Moving Data from Source Files to Staging Tables
Once data has been mapped and validated for the Entities, the data is ready to be transferred from the
source file to the staging table.
Get Staging Tables: Data is ready to be copied to the staging table. A user selects the Get Staging Table
menu option to start the process. During this stage the user has the option to run the job on demand or
batch it. This batch option is probably
useful when processing large files.
Processing Groups – Moving data to Staging table
15 AX 2012 Data Migration Framework
You will see a message indicating that the data has been copied to the staging table. If errors were
encountered during the load, these errors can be viewed in the log file in the execution history of the
processing group menu.
Processing Group – Copying data to Staging
16 AX 2012 Data Migration Framework
Moving Data from staging table to the Final AX destination.
Once data has been transferred to the staging table and validated, the data is ready to be transferred to
the final destination in AX by selecting the “Copy data to target.” A message will be displayed informing
of the processing and creating the AX records if the processing was successful. If records are reloaded,
the AX records will be updated.
Processing Group – Copying Data from Staging Table to Target
17 AX 2012 Data Migration Framework
Processing Group – Created Vendor Records.