Data Migration Strategy and Approach

13
DATA MIGRATION STRATEGY AND APPROACH PROJECT IDENTIFICATION Project Name CPI/Project Number Project Type (Business Consulting, Implementation, Upgrade, Internal, other) Customer Name Customer Number Planned Start/Finish Project Sponsor Program Manager Project Manager (Customer) Project Manager (SAP) SAP Service Partner(s) Project Manager (Service Partner) 1 of 13

description

Data Migration Strategy and Approach

Transcript of Data Migration Strategy and Approach

Page 1: Data Migration Strategy and Approach

DATA MIGRATION STRATEGY AND APPROACH

PROJECT IDENTIFICATION

Project Name CPI/Project NumberProject Type(Business Consulting, Implementation, Upgrade, Internal, other)

Customer Name Customer Number Planned Start/Finish

Project Sponsor Program Manager Project Manager (Customer)

Project Manager (SAP) SAP Service Partner(s) Project Manager (Service Partner)

1 of 11

Page 2: Data Migration Strategy and Approach

ASAP Implementation Methodology

TABLE OF CONTENT

1. PURPOSE OF THIS DOCUMENT 4

2. STRATEGY AND APPROACH – SCOPE OF THIS DOCUMENT 4

3. DATA ASSESSMENT PHASE 5

4. DATA QUALITY AND CLEANSING6

5. ETL METHODOLOGY 7

5.1 Analysis 7

5.2 Design7

5.3 Develop 8

5.4 Production 8

6. GAP ANALYSIS – DEFINITION AND ADDRESSING OF GAP 8

7. MIGRATION TESTING METHODOLOGIES 9

7.1 Analysis 9

7.2 Design10

7.3 Develop 10

7.4 Production 10

8. DATA EXPORT – SAP TESTING 10

9. DOCUMENTATION AND SIGN OFF PROCESS 11

9.1 Business Requirements 11

9.2 Data acquisition 11

9.3 Data quality 11

9.4 Data Access 11

9.5 Administration11

9.6 Metadata 12

9.7 Testing 12

10. BACKUP PROCEDURE 12

10.1 Databases 12

10.2 Documentation 12

2 of 11

Page 3: Data Migration Strategy and Approach

ASAP Implementation Methodology

1. Purpose of this Document

The purpose of this document is to plan for, and address, the high level methodologies and approach to be employed in the Migration Project for AIM Company. The manner of addressing this is via definition of the different elements that encompass the project, defining individual methodologies and approaches to be employed in each aspect of the project.

It is used :

to confirm the commitment to the project of the organization. as an aid to communication both within and outside the project. as a basis for project definition.

2. Strategy and Approach – Scope of this document

SAP AG and Business Objects have been contracted by AIM Company to perform various aspects of data migration, data conversion and data quality within the AIM Company environment. This covers a wide potential scope of deliverables that may be required by the AIM Company, so it is important to place into perspective the intended scope of this document.

Business Objects hold the belief that there the potential deliverables need to be matched against priorities and timescales as defined by AIM Company. Therefore, three different timescales are being identified here. Different deliverables, and therefore different methodologies and approaches shall be utilized, dependant on the requirements associated with the timescale mentioned.

Immediate timescale (late October 2008 till late December2008):

The most immediate timescale is associated with the following required deliverables: Data Assessment of Legacy data , this will include Profiling the Data Development of Data Migration strategies to complement the requirements relating to the deployment of SAP. For the

intent of this document, SAP solution encompasses Retail functions. Necessary Data Cleansing of existing data as to ensure suitable data quality of data provided into the SAP system. Identification of gaps between desired and existing data deliverables, as well as, where possible, suggested solutions

to assist in closing or removing the identified gaps. Identification of areas where existing processes / data collection may be more complete than those associated with

the new SAP solution, and documentation of these gaps.

Medium-Term timescales (1 January 2009 till 31 March 2009)

The medium-term timescale is currently associated with the following required deliverables: Migration of data from Legacy into a suitable environment for SAP purposes. Necessary Data Transformation of existing data into suitable formats as defined by the SAP Project Team.

3 of 11

Page 4: Data Migration Strategy and Approach

ASAP Implementation Methodology

Development of Data Migration strategies to complement the requirements relating to the deployment of SAP. For the intent of this document, SAP solution encompasses Retail functions.

Necessary Data Cleansing of existing data as to ensure suitable data quality of data provided into the SAP system. Identification of gaps between desired and existing data deliverables, as well as, where possible, suggested solutions

to assist in closing or removing the identified gaps. Identification of areas where existing processes / data collection may be more complete than those associated with

the new SAP solutions, and documentation of these gaps. The analysis and development of appropriate reconciliations

Long-Term timescales (Remainder of the 2009 year)

The long-term timescale is currently associated with the following required deliverables: Go-Live of SAP Retail Identification and development of internal AIM Company business practices that compliment and encourage

excellence in ongoing data-quality and completeness.

3. Data Assessment Phase

The intent behind the Data Assessment Phase is to conduct all consulting requirements associated with the identification of all data migration requirements associated with the intended AIM Company deployment of SAP for the following Retail streams: Article Vendor Conditions Prices Sites

As part of this process, a review shall be conducted of Legacy environment, identifying the function, source and type of data held. This shall be documented and delivered by Business Objects as a Project Deliverable.

A Top-Down / Bottom –Up Analysis shall be conducted of the existing Legacy environment, detailing the flow of capturing of data, through to table and field level. In addition, analysis shall be conducted from individual field level within Legacy.

From this function, GAP Analysis shall be conducted to identify where the following situations exist: Data exists in Legacy environment which does not relate to any current data capturing / process requirements

existent in the AIM Company. Data exists in Legacy environment which is aged in nature, and may be logical to archive.

Development of Cross Reference Tables from the Top-Down / Bottom-up Analysis which identifies or meets in the middle and provides a set of cross-references between the users sphere of knowledge and the technical table bases view of the system. From this it will be possible to perform detailed analysis on the data required for the migration and that which can be archived.

As part of the deliverable, an Entity Relationship Model shall be developed and documented to help map the sources to the target database in later phases of the project. Also, to build of an entity life history to better understand how data is created, maintained and deleted/archived – this will be a high level process/data subject CRUD matrix.

4 of 11

Page 5: Data Migration Strategy and Approach

ASAP Implementation Methodology

One of the key deliverables of the Data Assessment Phase shall be the building of a data dictionary or catalogue to include information such as;

a. name, including synonyms and homonyms b. description c. source reference (i.e. table/column(s)) d. type e. length f. data type g. validation rules h. business rules i. foreign key j. primary key k. trigger, constraint or equivalent l. also to include information about the relationship with other entities

The final part of this phase will be a specification of the data to be migrated together with a checklist of reasons why not to be migrated for other data items. All of the above shall be documented thoroughly and presented to Project Management Office / Sponsor for buy-off in a number of meetings.

All findings shall be collated for review by the Audit Office Function, to allow for independent correlation of findings against those identified by Audit Function.

4. Data Quality and Cleansing

Data quality relates to the correctness, consistency and completeness of AIM Company data. Data quality needs to be understood as an iterative process within the AIM Company environment. There are a number of factors impacting on ongoing data quality, with signs of poor data quality normally include the following: Inability to compare data from different sources Data entered in wrong fields Lack of consistent data definitions Inability to consolidate data from multiple sources Inability to track data across time Inability to comply with government regulations

For the purposes of this Data Migration Project, the Data Transformation Stream will be delivering an exercise aimed at: improving the quality of existing data in the AIM Company, rather than Developing internal procedures focused at ongoing quality within AIM Company. This is perceived as being part of

longer-term deliverables to the business.

Data shall be drawn from the Legacy database. These samples shall consist of 20%, 60%, and 100% respectively of the data available from the Legacy database. These shall be reviewed to examine for improvements in Data Quality during the iterations.

5 of 11

Page 6: Data Migration Strategy and Approach

ASAP Implementation Methodology

In order to improve data quality within AIM Company environment prior to the migration of data to SAP solution, the Data Transformation Stream shall be focusing on delivering the following processes within the existing database environment: Quantifying the number and types of defects Assessing the nature and cause of the defects Identifying and isolating data elements in data structures. Standardising and normalising the data values, units of measure and formats according to business rules Verifying data values. Enhancing data via appending additional data to records where required Matching records and identifying duplicate records Consolidating multiple records into one master record where required

The Data Transformation Stream shall report back on Data Quality on a regular basis during the Data Migration Project. Assessments on missing or inconsistent data may result in the Data Transformation Stream making recommendations on the procurement of additional data sources from external parties to assist in improving the overall quality of the data.

5. ETL Methodology

The Extraction Transformation and Load methodology refers to the procedures envisaged in the analysis, design, development and implementation of the actual data migration processes.

5.1 AnalysisAnalysis deals with the discovery, analysis and distillation of business requirements into a scope of work and in the ETL analysis the business analysis results will be used to match business requirements to available data. The process will deal with the following aspects: Source identification and analysis Target identification and analysis Business rules identification Data format analysis

5.2 DesignOnce the ETL analysis has been completed the results will utilized to design the actual data migration processes. This phase will entail the following:Product installation design (Job server/s, work stations and administration servers).Database design (Development, production and central repositories as well as staging database)High level extraction design (source, target, procedure)ETL process design (source to target field mapping, transforms, rules, formats)

5.3 DevelopThe development translates the ETL design into physical data extraction, transformation and load processes. The processes are made up of the following: Table and field definitions – to ensure accurate source to target transforms Projects and jobs – to logically group the physical ETL processes

6 of 11

Page 7: Data Migration Strategy and Approach

ASAP Implementation Methodology

Workflows and data flows – to perform the physical data movement Central repository version control to manage a multi user development environment Iterative development - o Verification by target expertso Modification of processes

5.4 ProductionCompleted and signed off ETL jobs will placed into production and will provide data to the SAP team to ensure that the Retail Master can go live. The jobs will also be available for continuous update as data quality improvements take place.

The ideal methodology for a success data migration project is an iterative and interconnected four-stage approach: analyzing the data, extracting and transforming the data, validating the data, and loading it into targets with testing, and repeating the same process until the migration is successfully completed. This approach enables target-driven analysis, validating assumptions, refining designs and applying best practices as the project progresses.Data Migration is the only phase which data is actually moved from the legacy application (IBM) to SAP and effective Data Migration directly affects business user acceptance rates of the new SAP system.

6. Gap Analysis – Definition and Addressing of Gap

This Step of the Data Migration Project relates to the identification and understanding of all data requirements relating to the AIM Company deployment of SAP Retail.

An initial deliverable to be received from SAP Retail Stream: formats of fields required lengths of fields relationship of fields order of fields, amongst others.

Data Transformation Stream will then develop a Match / GAP Analysis Document, which documents the following details: Source Field Source Field format Source Field Length Matching to Target field Target Field Length Target field Format Transformation necessary.

From this analysis, it is possible to identify three deliverables:1. Matched items2. Items found in source but not in target location3. Items found in target but not in source location.

7 of 11

Page 8: Data Migration Strategy and Approach

ASAP Implementation Methodology

With regards to the items found in target location but not in source, the following processes shall be completed: Communication informally with SAP Team to further understand requirement, suggest alternative if possible and

resolve if possible Generation of a spreadsheet identifying and documenting all gaps. Scheduling and attendance at a meeting with SAP Retail stream, to review and address next steps. Creation and distribution of a Request for Information Document (RFI), requesting clarification as to the requirement,

detail and source of data to allow resolution of all outstanding issues in GAP Analysis.

With regards to the items found in the source but not in the target location, the following processes shall be completed: Communication informally with SAP Team to further understand the lack of requirement for data / process within the

new Retail SAP solution. Generation of a spreadsheet identifying and documenting all gaps. Scheduling and attendance at a meeting with SAP Retail as well as the Data Migration Sponsor, to review and

address next steps. Creation and distribution of a Request for Information Document (RFI), requesting clarification as to the requirement,

detail and source of data to allow resolution of all outstanding issues in GAP Analysis.

7. Migration Testing Methodologies

The testing methodology follows the analysis, design, development and production phases of the rest of the project. The following practices will be followed during the build/test phase:

Developers should often switch to different areas of work. This ensures that the knowledge for a particular unit of work is not restricted to the one developer. It also means that many heads are looking at a particular task and looking for ways to improve a particular task. Moving from task to task also has the benefit of keeping developers more motivated and interested in their work.

Developers will not test their own code – No matter how good a developer is, they should not test their own code. This is for the simple reason that developers will want the tests to be successful, whereas others will want the tests to actually test the code.

Standards will be reviewed, shared and enforced – This includes naming standards, best practices, environments etc.

The testing specific aspects of the project are the following:

7.1 AnalysisThe testing standards and procedures will be identified during the analysis phase. Deliverables from this phase are:

Testing standards and procedures Testing requirements

7.2 DesignThe testing of the implementation must be planned and the deliverables from this phase are:

Test plans for integration and systems testing

8 of 11

Page 9: Data Migration Strategy and Approach

ASAP Implementation Methodology

Test plans for regression and volume testing Test plans for data quality testing

7.3 DevelopDuring the development phase the testing plans will be implemented and the following actions will be taken: Perform integration and systems testing Perform regression and volume testing Perform data quality testing

The results of the test will also be documented and signed off

7.4 ProductionOnce the data migration jobs have gone into production further data quality and performance tests will be carried out to ensure that that all requirements are met.

8. Data Export – SAP Testing

The Data Migration Team will provide a significant volume of data for testing purposes .

The methodology and approach employed for the deliverable is as follows: Work in parallel to the delivery of the rest of the project, with a separate team person responsible. Receive required, finalized target definitions from the SAP Retail Team. Plan and Develop a Target Repository structure, based on the criteria and definitions documented by the SAP Retail

Teams. Creation and Development of necessary ETL schemas relating to a source of Legacy or other source data

repositories, to plan for the data migration.

Necessary data transformation functions shall be developed to accommodate alterations in data format / structure from the source to target environments.

It is expected that gaps in data shall exist between the requirements provided from SAP Retail and Legacy. For every gap identified, the following processes shall be adopted:

1. Communication informally with SAP Team to further understand requirement, suggest alternative if possible and resolve if possible.

2. Generation of a spreadsheet identifying and documenting all gaps.3. Scheduling and attendance at a meeting with SAP Retail stream, to review and address next

steps.

Finally, the provision of a TXT format file(s) with all migrated and transformed data to allow for loading and testing function within SAP streams.

It is not anticipated that Audit Function shall have extensive auditing function within this Phase, nor need to sign off as the output developed is for testing purposes only.

9 of 11

Page 10: Data Migration Strategy and Approach

ASAP Implementation Methodology

9. Documentation and Sign off Process

Managing the project requires the generation of multiple documents for sign off and audit purposes. The list of documents to be created is the following:

9.1 Business Requirements Methodology

This document provides the processes and procedures required to deliver the project and must be signed off by the project sponsor.

Scope documentThis document details the business requirements and the work that needs to be done to meet the business requirements and must be signed off by the project sponsor

9.2 Data acquisition Source identification document to be signed off by the designated business representative. Source to target mappings and business rules

Document mapping source system data to SAP target requirements, inclusive of transformation and business rules. This document will be signed off by the designated business representative, as well as SAP development team.

Source Gap documentDocument listing data required by SAP but not available in the sources to be actioned by AIM Company and SAP.

Target Gap document to be actioned by AIM Company and SAP.Document listing data available in the source systems but not required by SAP to be actioned by AIM Company and SAP.

9.3 Data quality Data profiling documents

The data profiling documents will provide business with a view of the content and patterns within their source data. Data quality documents

These documents will identify data quality issues and will advise business as to the best solution.

9.4 Data Access Document detailing contact person and access procedures.

9.5 Administration Project planThe project plan will contain various documents such as:o Scope definitiono Organizational structureo Work breakdown structureo Change management plano Risk management plano Communication management plan

9.6 Metadata Data lineage document which provides graphical lineage information

10 of 11

Page 11: Data Migration Strategy and Approach

ASAP Implementation Methodology

9.7 Testing ETL test document detailing results of ETL tester’s findings. Internal audit results document Training procedure document

10. Backup Procedure

The methodology must make provision for backup and recovery procedures and in the case of the AIM Company data migration project these procedures are as follows:

10.1DatabasesThe following databases must be backed up: Development repositories Pre-Staging Staging database Target database

These databases will reside on the \\xxxxxxxxxxxxxxxxxxxx instance and will be backed up daily.

10.2DocumentationThe following documentation must be backed up: Project documentation SAP file format sheets Source to target documentation CSV data files Timesheet documents Analysis sheets Profiling documents Data quality documents

These documents will all reside on the \\xxxxxxxxxxxxxxxx Server and will be backed up daily.

11 of 11