Data Migration Plan 2.1 – UNOFFICIAL · Data Migration Plan 2.1 – UNOFFICIAL 15.4.2020 Finnish...

Data Migration Plan 2.1 – UNOFFICIAL

15.4.2020


15.4.2020

Finnish version updated

27 March 2020 2 (43)

Fingrid Datahub Oy

Street address Postal address Telephone Fax Business ID 2745543-5, VAT

reg.

Läkkisepäntie 21 P.O. Box 530 [email protected]

00620 HELSINKI 00101 HELSINKI +358 (0)30 395 5000 +358 (0)30 395 5196 www.fingrid.fi

Table of contents

1 Executive Summary .............................................................................................................................. 5

2 Parties and responsibilities .................................................................................................................. 7

2.1 Market parties’ responsibilities for delivery of source data .............................................................. 7

2.1.1 Master data delivery responsibility and rules concerning data selection .................................... 8

2.1.2 Metering data delivery responsibility and rules concerning data selection ............................... 10

2.2 Responsibilities of Fingrid Datahub Oy ......................................................................................... 12

2.3 Responsibilities of the data migration service provider ................................................................. 13

2.4 Responsibilities of the Datahub system supplier ........................................................................... 13

2.5 RACI matrix ................................................................................................................................... 13

3 Data migration project ........................................................................................................................ 15

3.1 General description and schedule of the data migration project ................................................... 15

3.2 Requirements for source data quality ............................................................................................ 16

3.3 Requirements for the duration of migration file production and delivery ....................................... 19

3.4 Targets of the project stages and acceptance criteria for checkpoints ......................................... 20

3.5 Detailed schedule of Stage 3 ......................................................................................................... 22

3.6 Detailed preliminary timetable for stages 4 and 5 ......................................................................... 28

4 Instructions for data migration work ................................................................................................. 32

4.1 Titta service ................................................................................................................................... 32

4.2 Description of data migration process ........................................................................................... 33

4.3 Market party tasks during the data migration process ................................................................... 34

4.3.1 Production of data migration files .............................................................................................. 35

4.3.2 Import and checking of master data .......................................................................................... 35

4.3.2.1 File check ............................................................................................................................... 36

4.3.2.2 Integrity check ........................................................................................................................ 37

4.3.2.3 Source data consistency check ............................................................................................. 37

4.3.3 Processing of observations produced in the Titta service ......................................................... 37

4.3.4 Import and checking of metering data ....................................................................................... 38

4.3.5 The transfer of data to the Datahub system .............................................................................. 38

4.3.6 Data migration check ................................................................................................................. 38

4.4 Data check against external data sources .................................................................................... 39


15.4.2020


27 March 2020 3 (43)

Fingrid Datahub Oy


reg.



4.5 The deployment of new IDs ........................................................................................................... 39

5 Data security for the data migration service .................................................................................... 40

5.1 General information about data security for server rooms ............................................................ 40

5.2 Data ownership .............................................................................................................................. 40

5.3 Maintenance of data security ......................................................................................................... 40

6 Data protection for the data migration service ................................................................................ 41

7 Definitions ............................................................................................................................................ 42

8 Appendices .......................................................................................................................................... 43


15.4.2020


27 March 2020 4 (43)

Fingrid Datahub Oy


reg.



Change history

Date Version Change

27 September 2018

1.0 First official version

17 December 2018

1.1

Added a link to appendix 3

In table 6, the word “preliminary” has been removed and the time limit for the submission of measurement data has been specified as 120 hours.

22 February 2019

1.2 The rules for selecting customer information data are specified in Table 2.

30 April 2019 1.3 Applied validation checks for data quality in every stage added to Table 5

10 May 2019 1.4

Responsibilities for gird operators have been specified in Table 2 and in section 2.1.2.

Market party definitions have been specified in Table 1

11 December 2019

2.0

Stages 2 and 3 of the data migration in Chapter 1 updated

Figure 2 updated to reflect the new data migration schedule

Updated data migration checkpoints T2–T5 in tables 5, 6, and 7

Added chapter “3.5 Detailed schedule of Stage 3”

Added chapter “3.6 Detailed schedule of Stage 4”

Removed obsolete appendices:

Appendix 2: Preparation by the industry.

Appendix 7: Tasks of distribution system operators in the different stages of the data migration project

Appendix 8: Tasks of the electricity supply company in different stages of the data migration project

Appendix 9: Tasks of the service providers in different stages of the data migration project

Appendix 10: Tasks of the TSO and regional grid operators in the different stages of the data migration project

Updated appendix numbers and links.

15 April 2020 2.1 Chapter 3.5: Data migration Stage 3 schedule updated to match Datahub Table of Milestones version 1.1.

Chapter 3.6: Data migration Stage 4 and 5 preliminary schedules

updated to match Datahub Table of Milestones version 1.1.


15.4.2020


27 March 2020 5 (43)

Fingrid Datahub Oy


reg.



1 Executive Summary

Datahub data migration means quality assurance and initial download of master data and

metering data of electricity retail market business processes from source systems to the

new Datahub system before its commissioning.

The source systems are primarily the business applications of distribution system

operators and electricity suppliers, in which customer, accounting point, contract and

metering data are kept. In addition, data is also required from the systems of transmission

system and regional grid operators.

The data migration work has two main goals:

The harmonisation of data supplied from source systems

Implementing the initial load for the datahub system.

The datahub system operations require consistency of the data used in the system.

Fingrid Datahub Oy offers electricity retail market parties and service providers a data

migration service (Titta service), which checks data consistency and via which the data

will be transferred to the Datahub system. The market parties can implement the Titta

service by signing a service agreement with Fingrid Datahub Oy.

The Datahub data migration project is divided into five stages, and the following

preliminary schedules and objectives have been specified for the first three stages. The

schedules will be updated if necessary once the new Electricity Market Act takes effect:

Stage 1: 01/2019 - 05/2019

Targets:

The market parties can deliver all up-to-date master data in the format

specified in the migration file instructions.

o All master data has been delivered to Titta in the migration files.

Metering data is not master data.

The quality of master data that is critical to Datahub is at a sufficient level for

data integrity checks and checking the accuracy of customer information and

addresses.


15.4.2020


27 March 2020 6 (43)

Fingrid Datahub Oy


reg.



Stage 2: 6/2019 - 12/2019

Targets:

The quality of data critical to market functionality is at a sufficient level for

consistency checks that cover the entire market.

Mass checks of the accuracy of customer information and addresses can be

performed against external registers.

Stage 3: 01/2020 - 12/2020

Targets:

The use of new GS1 and EIC IDs will begin in the data migration.

o The GLN party IDs and GSRN accounting point IDs being deployed

are compliant with the GS1 system

The quality of data transferred in the data migration is sufficient to allow a

transition to preparations for Datahub commissioning.

The data migration performance meets the requirements for Datahub

commissioning.

After stage 3, the data migration work will progress to support preparations for

commissioning and the actual commissioning of the datahub system, and the quality of

data will continue to be maintained and improved.

The data migration plan helps the electricity retail market parties and their system

suppliers to prepare for commissioning of the datahub system, and supports the parties

during possible system changes and improvement of the quality of data being delivered

from the source systems. The plan has been compiled in cooperation with the Data

migration work group, which consists of representatives from companies in the industry.

The Electricity Market Act that became valid in 2019 will oblige distribution system

operators and electricity suppliers to participate in the Datahub data migration project and

implement the migration of their company’s data to Datahub in accordance with this plan.

In addition, regional grid operators and the transmission system operator are obliged to

transfer data related to exchange point metering to Datahub in accordance with this plan.


15.4.2020


27 March 2020 7 (43)

Fingrid Datahub Oy


reg.



The parties participating in the data migration project and their responsibilities are

described in chapter 2. The stages of the Datahub data migration project and the

acceptance criteria associated with them are described in chapter 3. Chapter 4 includes

instructions for the practical data migration work.

2 Parties and responsibilities

2.1 Market parties’ responsibilities for delivery of source data

In conjunction with the Datahub data migration, the term ‘market party’ refers to all

companies that have singed a data migration service agreement with Fingrid Datahub

Oy, including companies that do not have an electricity retail market ID. Signature of the

agreement is a prerequisite for use of the data migration system. A market party can

operate in one or more of the roles described in Table 1.

TABLE 1 MARKET PARTIES

Party Definition

Distribution system operator Distribution system operators are companies that practise electricity grid operations in a 0.4 - 70 kV distribution grid and have an electricity grid permit granted by the Finnish Energy Authority.

Closed distribution system operator

Closed distribution system operators are companies that practise electricity grid operations in a closed distribution system and have an electricity grid permit granted by the Finnish Energy Authority and whose grid has a nominal voltage of less than 110 kV. A closed distribution system operator is subject to the same rules concerning the responsibility to deliver data as a distribution system operator.

Electricity supplier company Electricity supplier companies are companies registered in Finland as electricity suppliers and which have an agreement with a balance responsible party or serve as a balance responsible party.

Transmission system operator

The transmission system operator with system responsibility (Fingrid Oyj)

Regional grid operator

Regional grid operators, also known as high voltage distribution grid operators, are companies that practise electricity grid operations in high voltage distribution grids and have an electricity grid permit granted by the Finnish Energy Authority.

Industrial grid

The operator of an internal electricity system of a property or a similar group of properties which does not act as a balance responsible party. The grid operator has agreed with the imbalance settlement unit eSett and the DSO to whose network the internal network of a property or a group of properties is connected, that the operator’s grid area is named as a separate Metering Grid Area.


15.4.2020


27 March 2020 8 (43)

Fingrid Datahub Oy


reg.



Production grid Over 1 MVA production plants will be handled as production grids

Service provider

Service providers that need to use the data in Datahub within the scope of authorisations can deliver authorisation information en masse in conjunction with data migration.

A market party can also be a service provider that delivers source data to the Titta service on behalf of another party.

2.1.1 Master data delivery responsibility and rules concerning data selection

The market parties deliver to Datahub the data for which they have maintenance

responsibility. Data for which maintenance responsibility lies with another market party

can also be stored in the parties’ information systems. With regard to accounting point

data, for example, each distribution system operator is responsible for maintaining the

accounting point data in its own region, but accounting point data can also be contained

in the electricity suppliers’ systems. In this case, only the distribution system operator

delivers the data to Datahub during the data migration work. The table below shows

which market party delivers the data, and the period of time covered by historical data

that is delivered in data migration.

TABLE 2 DATA DELIVERY RESPONSIBILITY AND RULES CONCERNING DATA SELECTION

Migration file Data owner (delivers data to Datahub)

Rules Data to be delivered

Customer information

Distribution system operator (grid agreement)

Electricity supplier (sales agreement)

If the information that different market parties have about the same customer differ, only the customer information of the supplier with the most recent (valid latest) contract with the customer will be delivered to the Datahub.

Customers with valid agreements. Customers with agreements that have ended within the past six weeks. Customers with future agreements (See Agreement information)

Accounting point data

Distribution system operator

If a DSO’s system contains accounting points outside its metering grid area, the DSO only supplies the accounting points for its own area. Only main metering points are transferred.

Accounting points in use are imported. (Under construction, connected or disconnected)

Accounting points removed from use are imported if an agreement specified for transfer is associated with them (see Agreement information).


15.4.2020


27 March 2020 9 (43)

Fingrid Datahub Oy


reg.





Agreement information.

Distribution system operator (grid agreement)

Electricity supplier (sales agreement)

-

Valid agreements and agreements that have ended within the past six weeks and confirmed agreements that will begin within the next 90 days.

Party information

All

Market parties deliver their own information.

Regional, industrial and production grid operators as well as the transmission system operator deliver their own party information, if the metering grid area has at least one exchange point with a DSO or closed DSO where the party is a metering responsible for this exchange point.

Only valid information.

Product information

Distribution system operator (grid product)

Electricity supplier (sales product)

- Valid products and prices connected to transferable agreements.

Authorisation information

Service provider (third party)

Authorisations are based on third-party rights to process data for a specified customer and accounting point.

The service provider specifies the valid authorisation information needed for the migration file.

Metering grid area data

All grid operators

The DSO and the operator of a closed distribution grid deliver the MGA of their respective network

The operator of a regional grid, production grid, industrial grid and transmission system deliver the MGA data of their respective network, if the conditions for delivering the data are met on the basis of exchange points.

Only valid data


15.4.2020


27 March 2020 10 (43)

Fingrid Datahub Oy


reg.





Exchange point data

All grid operators

Only the metering responsible party delivers data for an exchange point between two distribution grids.

The operators of a regional grid, industrial grid, production grid and main grid system deliver the data of exchange points that fall under the scope of Datahub and are associated with distribution grids or closed distribution grids, and the grid operator is responsible for metering in such an exchange point.

Only valid data

Production unit data


- Only valid data

Additional accounting point addresses


Additional addresses can be reported if necessary for those accounting points that are imported in the Accounting point data.

See Accounting point data

Accounting point supplier data


The valid supplier data is reported for accounting points that are in Accounting point data

Valid agreements and agreements that have ended in the past six weeks. Confirmed agreements that will begin within the next 90 days.

2.1.2 Metering data delivery responsibility and rules concerning data selection

Delivery responsibility for metering data:

The DSO and the operator of a closed distribution network delivers metering data

for accounting points, production units and exchange points in its own metering

grid areas.

Only the metering responsible party delivers metering data for an exchange point

between two distribution grids.


15.4.2020


27 March 2020 11 (43)

Fingrid Datahub Oy


reg.



A regional grid operator delivers the metering data of an exchange point that lies

between a regional grid and a distribution network, when the operator is the party

responsible for metering the exchange point.

An industrial grid operator delivers the metering data of an exchange point that

lies between an industrial grid and a distribution network, when the operator is the

party responsible for metering the exchange point.

A production grid operator delivers the metering data of an exchange point that

lies between a production grid and a distribution network, when the operator is the

party responsible for metering the exchange point.

The transmission system operator delivers the metering data of an exchange point

that lies between the transmission grid and a distribution grid, when the operator

is the party responsible for metering the exchange point.

The figure 1 below presents the rules concerning the transfer of metering data history.

The red vertical line describes the time prior to which data is transferred to Datahub.

During the data migration project, data is uploaded several times, so a red vertical line

may also depict the moment when data is extracted from the system for data checking.

FIGURE 1. RULES RELATED TO THE TRANSFER OF METERING DATA HISTORY DATA


15.4.2020


27 March 2020 12 (43)

Fingrid Datahub Oy


reg.



The items presented in figure 1 refer to the following cases:

1. The accounting point grid agreement has been valid for a shorter period than six

years, so metering data is imported from the whole agreement period.

2. The accounting point grid agreement has been valid throughout the six-year period.

Metering data is transferred for the entire six-year period. Metering data for a longer

period is not imported to Datahub.

3. If an accounting point grid agreement has expired no more than six weeks before the

data was extracted from the source system, the metering data from these accounting

points is also taken. Metering data is collected for the validity period of the agreement

after it ends, but only for a maximum of six years from the time of extraction.

4. Only metering data within the scope of hourly time series metering is taken into

consideration in metering data. If the metering method has changed during the

agreement validity period, only the hourly time series metered data is included in the

metering data.

5. If the consumption of the accounting point is measured by some other means than an

hourly time series, the use of these accounting points must be reconciled by means of

an equalising calculation by Datahub Go-Live or sooner, and the practice of using an

hourly time series or hourly time series modelling must be adopted for these

accounting points.

Metering data for production units and exchange points is imported for a six-year period

or for the operating period of the metering point if the metering point has been in use for

less than six years.

The data migration process for metering data will be clarified as planning of

commissioning progresses.

2.2 Responsibilities of Fingrid Datahub Oy

Fingrid Datahub Oy’s responsibilities and obligations for the entire Datahub project and

system are specified in the Electricity Market Act and decrees.

Fingrid Datahub Oy is responsible for the progress and schedule of the data migration

project. Monitoring of progress will be carried out by monitoring the amounts of data

delivered to the Titta system by market parties and the quality of source material and

comparing these observations to the data migration stages presented in chapter 3. As

part of monitoring, surveys can be sent to the market parties, which can be used to

assess the market party situation in more detail.


15.4.2020


27 March 2020 13 (43)

Fingrid Datahub Oy


reg.



Fingrid Datahub Oy is responsible for coordinating, scheduling and communicating about

the simultaneous deliveries of source data for the entire industry.

Fingrid Datahub Oy is responsible for the quality, development and maintenance of the

Titta service.

Fingrid Datahub Oy is responsible for communication and instructions related to the data

migration project. The instructions are centralised in the Titta service and the Ediel.fi

portal. Communication will be carried out as needed using different channels.

Fingrid Datahub Oy is actively monitoring the industry’s preparations for changes brought

by Datahub as part of the data migration project. Monitoring is carried out using a range

of surveys targeting the industry. Monitoring the preparation of the industry is vitally

important in order to carry out the planning and commissioning of the whole Datahub

project in an efficient way.

2.3 Responsibilities of the data migration service provider

The data migration service provider (Solteq Oyj) is responsible for the technical

development and maintenance of the Titta service. The service provider is also

responsible for technical user support provided to the market parties.

2.4 Responsibilities of the Datahub system supplier

The Datahub system supplier (CGI) is responsible for compliance of the Datahub system

with the Titta service and for uploading data to the Datahub system and producing the

data migration reports.

2.5 RACI matrix

The tasks and responsibilities of data migration work are described in the accompanying

RACI matrix (Responsibility assignment matrix). The roles are divided as follows in the

matrix:

R = responsible

A = accountable

C = consulted

I = informed


15.4.2020


27 March 2020 14 (43)

Fingrid Datahub Oy


reg.



TABLE 3 DATA MIGRATION TASKS AND RESPONSIBILITIES

Task Market party

Data migration service supplier

(Solteq Oyj)

The Datahub system supplier

(CGI)

Fingrid

Datahub Oy

Communications

Communications and contact directed at data migration project market parties

C/I C C/I A/R

Communications directed at data migration project system suppliers

C/I C A/R

Communications and reporting directed at the data migration project pilot group

C/I R C/I A/R

Communications for other external stakeholders in the data migration project

C C A/R

Training and technical support

Instructions for using the Titta service

C/I R A

Training for market parties and system suppliers

C/I C/I A/R

Data migration project technical support

I R A

Titta service

The commissioning of source data checking rules based on the data standard

C/I R

A

Maintenance and management of the Titta service

R A

Establishment of master data in the Titta service

R A

Maintenance of user information in the Titta service

A/R C I

Source data checking and correction

Compiling migration file examples C I A/R

Extraction tools for production of migration files

A/R C


15.4.2020


27 March 2020 15 (43)

Fingrid Datahub Oy


reg.



Task Market party

Data migration service supplier

(Solteq Oyj)

The Datahub system supplier

(CGI)

Fingrid

Datahub Oy

The production of migration files based on specifications

A/R

Uploading of migration files to the Titta service

A/R

The checking of uploaded data R A

Production of a company-specific check and correction report

I R A

Overall data migration project reporting

R I A

Data correction and enrichment in the market parties’ own systems

A/R C C

Necessary changes to market parties’ own data systems

A/R

Datahub system upload tool A/R

Data migration from the Titta system to Datahub

R A/R

3 Data migration project

The Datahub data migration will be carried out as a sub-project of the Datahub system

project. The aim of the data migration project is to harmonise the data supplied from

source systems, ensure data quality, and initial load of the Datahub system. The Datahub

data standard, in other words the specified data model, and external data sources will be

utilised when harmonising data and ensuring quality.

The functionality of the data migration process and performance of the Titta service have

been verified in a separate data migration pilot project. The data migration processes

were tested in the pilot project using proper source data from pilot companies and the

instructions were improved on the basis of the experiences gained. The pilot project was

completed in March 2018.

3.1 General description and schedule of the data migration project

In the data migration project, the market parties extract the data to be transferred to

Datahub from their own systems and deliver it to Titta in a format that complies with the

migration file specification. The data quality must meet the quality objectives specified in

the check point for each data migration stage. The criteria for data quality and delivery


15.4.2020


27 March 2020 16 (43)

Fingrid Datahub Oy


reg.



are tightened from one stage to another, in order to reach the final target. The quality

requirements are described in chapter 3.2 and the delivery requirements in 3.3.

In order to facilitate achievement of the targets in each stage, Fingrid Datahub Oy has

compiled role-specific task lists that are presented in appendices 7–10. Actions

concerning preparation and actual commissioning are specified in the commissioning

plan.

The Datahub data migration plan is divided into five stages, and the preliminary

schedules and targets specified for the first three stages are presented in figure 2. If

desired, a market party can also progress faster than the schedule presented in figure 2,

taking into account the fact that certain actions that apply to the entire industry must be

performed simultaneously in a centralised manner throughout the industry. These actions

include consistency checks, mass data checks against external registers, and

deployment of GS1 IDs. The schedules can be updated if necessary when the law takes

effect.

FIGURE 2 DATA MIGRATION PROJECT PHASING AND SCHEDULE

3.2 Requirements for source data quality

The data supplied by different parties is checked in the Titta service against rules

specified in advance. Quality targets for the source data have been separately set for

each project checkpoint.

In the data migration project, the parties must ensure that data is corrected in the source

system and that it acceptably passes the check.

The data in the source data is divided into four quality requirement levels according to

table 4.


15.4.2020


27 March 2020 17 (43)

Fingrid Datahub Oy


reg.



TABLE 4 SOURCE DATA QUALITY REQUIREMENT LEVELS

Level Description Explanation

1 Essential for Datahub

Data necessary in terms of the data standard’s data model is given in a format that complies with the data standard.

Incomplete data causes rejection of the entire record.

2 Essential for electricity market operations

Data necessary in terms of the Datahub business rules, such as conditionally mandatory data that is not mandatory from the data model perspective.

Data in the wrong format is rejected.

3 Useful data, but not essential for market and Datahub operations

Useful data from the perspective of electricity market functionality.

Data in the wrong format is rejected.

4 Possible error Data deviates from the information in the external data source but may not necessarily be wrong.

The quality criteria for the checkpoint in each data migration stage are presented in table

5. The checkpoints are consistent with the data migration stages, and progress to the

next stage only takes place when all market parties associated with Datahub have

achieved the quality criteria.

The quality criteria presented in table 5 apply to large changing materials, which are:

Customer information

Agreement information.

Product information

Accounting point data

Accounting point supplier information

Metering data (starting from checkpoint 2)

Additional accounting point addresses

Authorisation information


15.4.2020


27 March 2020 18 (43)

Fingrid Datahub Oy


reg.



The quality of the following migration files must be 100% at all levels starting from

checkpoint T1:

Party information


Exchange point data

Production units, depending on the number of units


15.4.2020


27 March 2020 19 (43)

Fingrid Datahub Oy


reg.



TABLE 5 PARTY-SPECIFIC QUALITY CRITERIA FOR SOURCE DATA BY CHECKPOINT

Level Description T1

5/2019 T2

12/2019 T3

12/2020 T4

5/2021 T5

2/2022

Party, metering grid area and exchange point as well as production units, depending on the number of units

100% 100% 100% 100% 100%

1 Essential for Datahub

95.00% 98.00% 99.50% 99.90% 99.98%

2 Essential for electricity market operations

80.00% 95.00% 98.00% 99.00% 99.50%

3

Useful data, but not essential for market and Datahub operations

No requirement

No requirement

No requirement

No requirement

No requirement

4 Possible error No requirement

No requirement

No requirement

No requirement

No requirement

Applied validation checks for data quality

File Check Integrity Check

Consistency Check

Consistency Check

Consistency Check

3.3 Requirements for the duration of migration file production and delivery

Migration file production means all the necessary actions related to extracting the source

data from the source system and writing in a format that complies with the migration file

instructions.

In terms of master data, migration file delivery means that all the migration files have

been transferred to Titta and the user has pressed the “Lataa kaikki” button. In terms of

metering data, migration file delivery means that all the migration files have been

transferred to Titta’s SFTP server.

The duration of source data production is measured from the beginning, but the related

requirements are only added to the acceptance criteria as the project progresses. The

requirements are market party specific.


15.4.2020


27 March 2020 20 (43)

Fingrid Datahub Oy


reg.



TABLE 6 REQUIREMENTS FOR PRODUCTION OF SOURCE DATA AND DURATION OF

DELIVERY

Material T1

5/2019

T2

12/2019

T3

12/2020

T4

5/2021

T5

2/2022

Master data migration file production and delivery to Titta.

Measured 4 hrs 4 hrs 4 hrs 4 hrs

Metering data migration file production and delivery to Titta.

No requirement

Measured 120 hrs 120 hrs

120 hrs

3.4 Targets of the project stages and acceptance criteria for checkpoints

Table 7 lists the key targets for each data migration stage and the acceptance criteria for

the related checkpoints.

TABLE 7 TARGETS AND ACCEPTANCE CRITERIA FOR STAGES 1–3 OF THE DATA

MIGRATION PROJECT

Targets Acceptance criteria for checkpoints

Checkpoint T0

The Titta service is ready to start the data migration project

Instructions concerning data migration are sufficient and up-to-date

Data security and data protection for the Titta service has been verified

The legislation concerning Datahub has taken effect

Stage 1, 01/2019 - 05/2019

All market parties have signed the data migration service agreement.

The market parties have implemented the master data extraction tools as a system function.

The market parties deliver up-to-date master data in its entirety from their source systems.

The market parties have routines for maintaining data quality.

The market parties have a plan for collecting missing personal identification numbers.

Checkpoint T1

All the fields of the master data files uploaded by the market parties to the Titta service are in a format that complies with the data standard and the migration file specification.

The scope and quality of the source data meets the criteria set for the stage in chapters 3.2 and 3.3.

Each market party has measured the duration of all master data extraction.

All distribution system operators have provided Fingrid with a complete list of exchange points and adjacent grids.

Stage 2 6/2019 - 12/2019 Checkpoint T2


15.4.2020


27 March 2020 21 (43)

Fingrid Datahub Oy


reg.




All distribution system operators have implemented the metering data extraction tools.

All market parties have delivered data for the coordinated mass check of customer information and addresses.

The mass check of customer information and addresses has been performed and the continuous checking service is in use.

The market parties have performed the party-specific integrity checks on all the source data.

o The parties perform a sufficient number of integrity checks until the data quality is at an acceptable level in accordance with the description in chapter 3.2.

The market parties have ordered new party IDs (GLN), metering grid area IDs (EIC) and metering point IDs (GSRN).

o Only distribution system operators order EIC and GSRN IDs

The data has been checked against external data sources maintained by third parties.

All the parties have performed integrity checks.

The scope and quality of the source data meets the criteria set for the stage in chapters 3.2 and 3.3.

Master data extraction performance meets the criteria set for the stage.

Stage 3 01/2020 - 12/2020

The market parties have deployed the new IDs in migration files.

The GSRN ID service has been taken into use. The service maintains the GSRN IDs until commissioning of Datahub.

A consistency check covering the entire industry has been performed.

o A sufficient number of consistency checks are performed until data quality and performance are at an acceptable level in accordance with the description in chapters 3.2 and 3.3.

Migration source data that covers the entire industry has been uploaded to the Datahub system and the data migration process has been verified based on reports produced by Datahub.

The data migration process is ready to move to the Datahub commissioning preparation stage.

Checkpoint T3

Metering point IDs have been changed to GSRN IDs.

Metering grid area IDs have been changed to EIC IDs.

The parties have begun using GLN party IDs.

Consistency checks covering the entire industry have been performed.

The quality of the source data meets the criteria set for the stage.

The master data and metering data extraction performance meets the criteria set for the stage in chapters 3.2 and 3.3.

Performance of Datahub uploading and data migration verification meets the requirements for Datahub commissioning.


15.4.2020


27 March 2020 22 (43)

Fingrid Datahub Oy


reg.



The preliminary targets and acceptance criteria for Datahub commissioning preparations

and the actual commissioning are described in table 8. The targets and acceptance

criteria will be clarified as the commissioning plan progresses.

TABLE 8 TARGETS AND ACCEPTANCE CRITERIA FOR COMMISSIONING PREPARATIONS AND

FOR COMMISSIONING WITH REGARD TO DATA MIGRATION


Stage 4 01/2021 - 05/2021

Preparations for commissioning

The quality of data and data migration process performance is at a sufficient level for commissioning of Datahub in accordance with the description in chapters 3.2 and 3.3.

The commissioning exercises have been completed, and possible observations made during the exercises have been corrected and the corrections verified.

The data migration process is ready to move to the Datahub commissioning stage.

Checkpoint T4

The source data imported via data migration has been verified in the Datahub business processes.

The source data meets the criteria set for the stage in chapters 3.2 and 3.3.

Other criteria are specified in a separate commissioning plan.

Stage 5 06/2021 – 02/2022

Commissioning

The quality of data is at a sufficient level for commissioning of Datahub in accordance with the description in chapters 3.2 and 3.3.

Checkpoint T5

The source data meets the criteria set for the stage in chapters 3.2 and 3.3.

Other criteria are specified in a separate commissioning plan.

3.5 Detailed schedule of Stage 3

In Stage 3 of data migration, the quality check of the source data will be based on

consistency checks. The consistency checks compare the data submitted by the parties

to Titta. Due to the nature of the consistency check, low quality in the data submitted by

one party produces errors for other market parties. Therefore, each market party must

submit their entire source data (basic and metering data) for consistency checks in a

comprehensive and timely manner. In addition, the market parties will be required to meet

specific quality requirements in an integrity check performed for each party before the

consistency check. The requirements and milestones for the quality of the integrity check

are set out in Table 9. Based on the results of integrity checks, it is easier to demonstrate

likely sources of error. The market parties shall ensure in advance that the quality of their

integrity check that is a part of the consistency check is sufficient. This shall take place on

the market parties’ own schedule by running integrity checks to the source data and


15.4.2020


27 March 2020 23 (43)

Fingrid Datahub Oy


reg.



amending the data as appropriate between the consistency checks. Figure 3 presents a

timeline of the milestones of stage 3 of data migration.

FIGURE 3. SCHEDULE OF STAGE 3 OF DATA MIGRATION

Table 9 describes the schedule of data migration stage 3 in more detail, including

milestones and their requirements, and whether failures to meet the acceptance criteria

are reported to the supervising authority. Stage 3 of data migration will concentrate on the

deployment of new IDs, the quality of data and the simultaneous submission of data

throughout the entire industry. The emphasis is on the new IDs and data quality. The

rationale of the consistency checks to be carried out in stage 3 is that all market

participants extract basic data from the same market situation. The time window for

extracting data is stated separately for each milestone to enable reliable comparison of

data between the parties. The market parties shall agree with each other on procedures

for correcting errors noticed in the consistency check.

The schedule of stage 3 is designed to have a delay of one week between the extraction

window and the consistency checks. The scheduled delay is intended as a buffer against

any challenges the market participants might meet in the supplying of information to Titta.

A primary requirement for consistency checks is that all parties are able to extract the

data from the same market situation and deliver the source data to Titta before the

consistency check begins. The performance requirement (basic data 4 hrs / metering

data 120 hrs) is a secondary quality requirement. However, in the later stages,

shortcomings in performance are also reported to the supervising authority, but any

challenges met in performance do not constitute a reason for not submitting the source

data, as long as the source data can be delivered before the beginning of the consistency

check.

In order to verify the performance of the metering data extraction, the extraction of

metering data shall be started within the same extraction window as for the basic data

(Titta enables the comparison of the closing time of the extraction window to the

submission time of the metering data).


15.4.2020


27 March 2020 24 (43)

Fingrid Datahub Oy


reg.



When the integrity check has been approved, a market party shall not deliver any more

data to Titta, nor run a new integrity check before the consistency check. This rule is in

place to ensure that the consistency check is performed on the correct and desired data.

TABLE 9 MILESTONES AND REQUIREMENTS FOR DATA MIGRATION STAGE 3

Check or event Acceptance criterion

1. Consistency check 21 January 2020 at 10:00 am The market parties submit all master data to Titta and will perform an integrity check for the master data by themselves before the consistency check begins. Old IDs shall be used in the checks.

The master data must correspond to the market situation on 14 January 2020 from midnight to noon (Extraction window 12 hrs, extraction time 4 hrs).

Preliminary quality requirement: The results of the integrity check must meet the following quality requirements: Quality Category 1: ≥ 98.00% Quality Category 2: ≥ 95.00%

If necessary, the requirements of future consistency checks might be specified further or changed based on the results of this consistency check. The results are not reported to the authority

Submission of GLN IDs to Titta 31 January 2020 All market parties must submit their GLN ID to Titta by the end of January 2020. The submission is made manually from the user interface. The required functionality will be implemented in Titta by the end of 2019. Titta will also contain a view for converting new and old party IDs.

The transmission of GLN IDs to Titta will not impose any requirements on the market party’s own systems. It is sufficient that the ID exists and has been delivered to Titta, even if it cannot be stored in the market party’s own systems at the moment.

The result is not reported to the authority

Check against external registers 4 February 2020 at noon The data to be included in the check must be submitted to Titta by noon on 4 February 2020. The

The result is not reported to the authority


15.4.2020


27 March 2020 25 (43)

Fingrid Datahub Oy


reg.



check will inspect the master data in Titta that have passed the file check without category 1 errors.

The results of the inspection check shall be reported separately for each party after the check is complete. The completion depends on the controller of the external register. After the check, the basic data are checked in real-time (on-line).

2. Consistency check 17 March 2020 at 10:00 am The market parties submit all master data to Titta and will perform an integrity check for the master data by themselves before the consistency check begins. Old IDs shall be used in the checks.

The master data must correspond to the market situation on 10 March 2020 from midnight to noon (Extraction window 12 hrs, extraction time 4 hrs)


Acceptance criteria for the party-specific consistency check:

Quality Category 1: ≥ 95.00% Quality Category 2: ≥ 90.00%

The results are not reported to the authority

The deployment of new IDs 20 April 2020 – 30 April 2020

Starting from noon on 20 April 2020, Titta will only accept transfer files with new IDs. The new GLN IDs must also be used in transfer file names.

By noon on 23 April 2020, the distribution system operators shall deliver the transfer files of supplier data for market parties, accounting points and exchange points to Titta with the new IDs (own GLN, supplier GLN and GSRN IDs). The accounting point data must indicate both the new and old identifiers. Titta will report to the suppliers the GSRN IDs of the accounting points based on the supplier data of the accounting points reported by the grid. Each party can download their report from Titta in an xlsx format. All parties can download from Titta a conversion table for converting between new metering area IDs (EIC IDs) to the old metering area IDs.

By noon on 30 April 2020, all market parties must submit to Titta all master data with new IDs for a


With the deployment of the new IDs, consistency checks are not run because the data collection is not coordinated.

The grids ensure individually that the metering data (SAF files) also contain the GSRN IDs.


15.4.2020


27 March 2020 26 (43)

Fingrid Datahub Oy


reg.



consistency check and run an integrity check on the master data. The data extraction window is not set at this point. The extraction can be started as soon as the new IDs are available in the transfer files.

On 30 April 2020 at noon, a consistency check shall start to verify that all market parties have deployed the IDs successfully for basic data.

3. Consistency check 29 May 2020 at 10:00 am The market parties submit all master data to Titta and will perform an integrity check for the master data by themselves before the consistency check begins.

The master data must correspond to the market situation on 22 May 2020 from 6:00 am to noon (Extraction window 6 hrs, extraction time 4 hrs)

Preliminary quality requirement: New IDs (GS1, GLN, GSRN and EIC) have been deployed in the data migration and the data submitted to Titta does not contain any old IDs The results of the integrity check must meet the following quality requirements: Quality Category 1: ≥ 99.00% Quality Category 2: ≥ 97.00%



Deviations and deficiencies in pre-qualification requirements or approval criteria shall be reported to the authority separately for each party

Submission and inspection of metering data and master data 11 June 2020 – 16 June 2020

DSOs submit all master data to Titta and perform an integrity check on the data by themselves.

The master data must correspond to the market situation on 11 June 2020 from 6:00 am to noon (Extraction window 6 hrs, extraction time 4 hrs)

The grid operator shall start the extraction of metering data at the same time as the extraction of master data and submit the metering data to Titta within 120 hours. The grid operator performs a check for the metering data in Titta.

The metering data and master data must be submitted to Titta by 16 June 2020.

The extraction and delivery of metering data is subject to a performance requirement of 120 h

Acceptance criteria for the party-specific metering data:


The grid operator shall correct any errors and deviations in the metering data in relation to


15.4.2020


27 March 2020 27 (43)

Fingrid Datahub Oy


reg.



The requirements also apply to grid owners that have exchange points to distribution grids and are parties responsible for metering at the exchange points

The event does not apply to electricity suppliers

the submitted accounting point data.


4. Consistency check 22 September 2020 at 10:00 am The market parties submit all master data to Titta and will perform an integrity check for the master data by themselves before the consistency check begins

The master data must correspond to the market situation on 15 September 2020 from 8:00 am to noon (Extraction window 4 hrs, extraction time 4 hrs)

In addition, the grid operator shall start the extraction of metering data at the same time as the extraction of basic data and submit the metering data to Titta within 120 hours. The grid operator performs a check for the metering data in Titta.




Quality Category 1: ≥ 99.00% Quality Category 2: ≥ 97.00% Acceptance criteria for the party-specific metering data:



The grid operator should also correct any errors found in the check of metering data. These errors are not reported to the authority.

Performance requirements: Extraction of basic data ≤ 4 hrs

Extraction of metering data ≤ 120 hrs Performance requirements will not be reported to the authority at this stage

5. Consistency check 25 November 2020 at 10:00 am Checkpoint T3 The market parties submit all master data to Titta and will perform an integrity check for the master data by themselves before the consistency check begins

The master data must correspond to the market situation on 20 October 2020 from 8:00 am to noon (Extraction window 4 hrs, extraction time 4 hrs)

In addition, the grid operator shall start the extraction of metering data at the same time as the extraction of basic data and submit the metering data to Titta within

Acceptance criteria for the party-specific consistency check: Quality Category 1: ≥ 99.50% Quality Category 2: ≥ 98.00% Acceptance criteria for the party-specific metering data:



15.4.2020


27 March 2020 28 (43)

Fingrid Datahub Oy


reg.



120 hours. The grid operator performs a check for the metering data in Titta.







3.6 Detailed preliminary timetable for stages 4 and 5

The schedule for data migration stages 4 and 5 follows the testing and commissioning

schedule of datahub. According to the testing and go-live plans, several datahub

migration verifications and trial production runs of datahub will be performed before

Datahub Go-Live. For these purposes, the industry is required to submit its own source

data to Titta for data migration in order to ensure that datahub is always loaded with new

source data. Table 10 shows a preliminary schedule and requirements for data migration

stages 4 and 5. The market parties must provide all master data to Titta for all checks in a

comprehensive and up-to-date format. In addition, DSOs shall provide complete metering

data for all checks. The requirements concerning the submission of master data and

metering data for data migration stages 4 and 5 also apply to all other grid operators that

have exchange points to distribution networks and are themselves responsible for

metering at these exchange points.

Please note! In stages 4 and 5, it is likely that the metering data will be submitted

phased, with the majority of the metering data being submitted in advance and only a

small part of the metering data on the due date. This approach will be decided by June

2020 at the latest.

The schedule, requirements and objectives for stages 4 and 5 will be specified further in

2020.


15.4.2020


27 March 2020 29 (43)

Fingrid Datahub Oy


reg.



TABLE 10 MILESTONES AND REQUIREMENTS FOR DATA MIGRATION STAGES 4–5

Check or event Acceptance criterion

Stage 4 1. Verification of the migration Consistency check 26 January 2021 The market parties submit all master data to Titta and will perform an integrity check for the data by themselves before the consistency check begins

The market situation from which the data shall be extracted shall be determined later (Extraction window 4 hrs, extraction time 4 hrs) The grid operator shall schedule the extraction of metering data so that it completes at the same time the extraction of master data completes, or earlier. The grid operator must submit the metering data to Titta within 120 hours from the start of the extraction. The grid operator performs a check for the metering data in Titta.




Quality Category 1: ≥ 100.00% Quality Category 2: ≥ 99.90% Deviations and deficiencies in pre-qualification requirements or approval criteria shall be reported to the authority separately for each party




Stage 4 2. Verification of the migration (Checkpoint T4) Consistency check 30 March 2021 The market parties submit all master data to Titta and will perform a data integrity check by themselves before the consistency check begins




15.4.2020


27 March 2020 30 (43)

Fingrid Datahub Oy


reg.



The market situation from which the data shall be extracted shall be determined later (Extraction window 4 hrs, extraction time 4 hrs)

The grid operator shall schedule the extraction of metering data so that it completes at the same time the extraction of master data completes, or earlier. The grid operator must submit the metering data to Titta within 120 hours of the start of the extraction. The grid operator performs a check for the metering data in Titta.







Stage 5 1. Trial production run Consistency check 7 June 2021 The market parties submit all master data to Titta and will perform an integrity check for the data by themselves before the consistency check begins

The market situation from which the data shall be extracted shall be determined later (Extraction window 4 hrs, extraction time 4 hrs) The grid operator shall schedule the extraction of metering data so that it completes at the same time the extraction of master data completes, or earlier. The grid operator must submit the metering data to Titta within 120 hours of the start of the extraction. The grid operator performs a check for the metering data in Titta. This round does not provide all historical metering data to the datahub unless there is a special need for it. The time interval to be provided will be specified as part of the planning of the production trial run.

Acceptance criteria for the party-specific consistency check: Quality Category 1: ≥ 99.95% Quality Category 2: ≥ 99.25% Extraction of basic data ≤ 4 hrs Extraction of metering data ≤ 120 hrs Acceptance criteria for the party-specific metering data:


The grid operator should also correct any errors found in the check of metering data. These


15.4.2020


27 March 2020 31 (43)

Fingrid Datahub Oy


reg.





errors are not reported to the authority.

Stage 5 2. Trial production run (Inspection point T5) Consistency check 7 September 2021 The market parties submit all master data to Titta and will perform a data integrity check by themselves before the consistency check begins


The grid operator shall schedule the extraction of metering data so that it completes at the same time the extraction of basic data completes, or earlier. The grid operator must submit the metering data to Titta within 120 hours of the start of the extraction. The grid owner performs a check for the metering data in Titta. This round does not provide all historical metering data to the datahub unless there is a special need for it. The time interval to be provided will be specified as part of the planning of the production trial run.







15.4.2020


27 March 2020 32 (43)

Fingrid Datahub Oy


reg.



Final rehearsal of Datahub Go-Live Consistency check 30 November 2021 The market parties submit all master data to Titta and will perform a data integrity check by themselves before the consistency check begins


The grid operator shall schedule the extraction of metering data so that it completes at the same time the extraction of master data completes, or earlier. The grid operator must submit the metering data to Titta within 120 hours of the start of the extraction. The grid operator performs a check for the metering data in Titta.






4 Instructions for data migration work

This chapter presents an upper-level description of the tasks to be performed in data

migration. The chapter presents the principles according to which the market parties

deliver data to the Titta service and describes how the service processes the data. More

detailed descriptions of the issues in the chapter are available in the appendices of this

document.

4.1 Titta service

The data migration service – “Titta” – is the most important tool in data migration work.

The Titta service is separate from Datahub, so that the market parties can independently

check the data that they have transferred to datahub during the data migration project.


15.4.2020


27 March 2020 33 (43)

Fingrid Datahub Oy


reg.



The parties can repeat the upload-check-upload process until the quality of the data is at

the required level.

Among others, Titta offers the following functions for market parties:

Delivery of master data via an interface

Delivery of metering data via an SFTP connection

Checking of master data against the data standard

Integrity check of metering data structural information

Checking of customer information and addresses against external data sources

Reports about the quality of the delivered source data

Observation reports for the checks

Market party users will be provided with user support directly from the Titta service. More

detailed instructions regarding use of the Titta service and the technical description are

available in the User instructions for Titta and the Data migration service description

appendix to the service agreement.

4.2 Description of data migration process

Figure 4 presents a description of the data migration process. Data migration work is by

nature a repetitive process, and the number of repetitions depends on matters such as

source data quality, source system compliance with the data standard, the quality of

source data extraction tools, and the market party’s ability to maintain quality during the

data migration.


15.4.2020


27 March 2020 34 (43)

Fingrid Datahub Oy


reg.



FIGURE 4 DATA MIGRATION PROCESS

4.3 Market party tasks during the data migration process

In each stage of the data migration project, the market parties repeat the process stages

until the data quality meets the acceptance criteria for the checkpoint.

Production of data migration files Each market party produces the data

specified by the data standard from its own data systems for the migration files.

The structure and data content of the migration files are described in greater detail

in the document, Siirtotiedosto-ohjeistus (Appendix 1).

Importing migration files to Titta. The market parties upload the master data to

the Titta service via an interface. Metering data is transferred to Titta via a

separate SFTP connection.

Data checking. Titta checks that the data delivered by the market parties

complies with the data standard. A detailed description of the checks is presented

in the document, Data checking in Titta (Appendix 3).

Processing of observations. Titta produces party-specific check reports by

source data. The check reports are in xlsx format and they are saved in the file

archive. The parties can retrieve their own reports from the Titta service interface.

Data correction and enrichment. Based on the check reports, the market parties

correct the data, enrich data in the source systems or correct the data extraction

tools.


15.4.2020


27 March 2020 35 (43)

Fingrid Datahub Oy


reg.



Tasks to be performed by Fingrid Datahub Oy. Fingrid Datahub Oy is responsible for

transferring the data to Datahub and verifying the data migration.

Data transfer to datahub. Data that has passed the check is transferred to the

Titta service publishing area, from where it is available to the Datahub system.

Data uploading to datahub. The Datahub system retrieves data from the Titta

service and uploads it to its own database.

Data migration verification. Verification of data migration is performed on the

basis of the data migration reports produced by Datahub by comparing the reports

to the data transferred to Datahub. The market parties can utilise the Datahub

reports to ensure that the data uploaded to Datahub corresponds to the data in

the source system.

The following chapters describe the work stages on a more detailed level.

4.3.1 Production of data migration files

In order to ensure reliability and performance, the production of migration files must

primarily be implemented as a semi-automated system function and the use of manual

phases is not recommended. The migration files must be produced from up-to-date

material that corresponds to the market situation.

The importance of reliability and performance is emphasised when running consistency

checks, in which case each market party must deliver up-to-date data to Titta in a very

short time so that it is possible to ensure that every party’s data is current. Minimisation of

run time is also of utmost importance in conjunction with Datahub commissioning.

However, migration files that contain only a small amount of data, are nearly unchanged

by nature, and can presumably be made completely error-free with little work can be

created manually. These migration files are:

Party information


Exchange point data

Production units, depending on the number of units

4.3.2 Import and checking of master data

The market party delivers the source data as migration files to the Titta service for

checking. The source data should be extracted from the production system or from a


15.4.2020


27 March 2020 36 (43)

Fingrid Datahub Oy


reg.



system containing up-to-date data equivalent to production data. In each data upload, the

party delivers a migration file containing all the rows of master data, not only the changed

rows. However, this does not mean that all the migration files have to be delivered in

every upload. For example, the party can deliver only customer information several times

as part of the data quality improvement work.

Delivery of the material takes place as self-service, in which case each market party

handles their own schedules and ensures that all the material is delivered to Titta for

checking.

If necessary, source material is also delivered in a coordinated and simultaneous manner

by the entire industry. The market party’s obligation is to ensure that the source material

is delivered to Titta at the requested time.

The following checks are performed on the source data in Titta:

File check: The format of the migration file delivered by the party complies with

the data standard.

Integrity check: The source data delivered by the party is internally complete

from the perspective of the data standard and Datahub business processes.

Consistency check: The source data delivered by the market party is consistent

with the material supplied by other parties.

The market parties monitor the quality of the uploaded data by means of reports

produced by the Titta service, which can be downloaded from the user interface. The

check report lists all the errors found in the data, including error type and category, data

content of the entire incorrect row, and a possible correction recommendation.

The following sections provide an upper-level description of the data migration checks. A

detailed description is provided in Appendix 3.

4.3.2.1 File check

The file check involves checking an individual migration file to ensure that its format

complies with the specification in Appendix 1. In addition, every data field is checked

against the Datahub data standard.

Quality requirement levels are specified for the material imported in the data migration,

and the manner in which errors in different quality requirement data affect importing of the

data is specified in the Titta service. For example, errors in quality requirement level 1

data cause rejection of the entire row. More detailed specifications related to the check

processes are provided in Appendix 3.


15.4.2020


27 March 2020 37 (43)

Fingrid Datahub Oy


reg.



4.3.2.2 Integrity check

After a market party has delivered all the mandatory migration files for its role, an integrity

check can be performed on the data. This involves checking the reference integrities

between the migration files. For example, a customer specified in the agreement must be

found with a given ID from customer information imported to Titta by the market party.

Integrity check also takes into account that a value of a field in one migration file may

depend on a value of another field in another migration file. The comparison only targets

data uploaded by the party itself.

The market party launches the integrity check when the migration files containing the

reference integrities have been delivered to the Titta service.

4.3.2.3 Source data consistency check

Fingrid Datahub Oy initiates a Datahub-level consistency check after the parties have run

integrity checks and the quality measured in the file and integrity checks is at a level that

makes it appropriate to run consistency checks.

The following checks are done as part of a consistency check:

Reference integrity checks between parties (for example, the accounting point ID

in an electricity supplier’s agreement must be found in the distribution system

operator’s material).

Unambiguous periods of validity (for example, the same accounting point may not

have agreements simultaneously valid between two different suppliers).

Functional integrity (for example, an accounting point has a sales and a grid

agreement, metering time series are available for the validity of the agreements).

Duplicate checks (for example, the same accounting point may not have several

records).

Only material that complies with the data standard data model is published in Datahub.

Rows in which quality requirement level 1 errors have been observed are not transferred

to Datahub. Final approval of the source data is done in this stage.

A consistency check is not performed on a party-specific basis, so the progress of its

process is not visible in the party’s user interface.

4.3.3 Processing of observations produced in the Titta service

The task of the market parties is to check the error reports produced by the Titta service

and, if necessary, correct the material in their own source system as required. The errors


15.4.2020


27 March 2020 38 (43)

Fingrid Datahub Oy


reg.



in the reports may also be due to an error in the extraction tool, in other words, the error

may not necessarily be in the source system data.

Performance of file and integrity checks and correction of the material based on the error

reports produced by these checks takes place as self-service. Each market party ensures

that all of its material meets the quality requirements described in chapter 3.2 in each

stage of data migration.

Each market party is responsible for the correctness of its own material.

4.3.4 Import and checking of metering data

The distribution system operators, transmission system operator and regional grid

operators deliver metering data via a separate SFTP server. The Titta service user

launches the metering data check after the metering data and master data have been

imported to Titta.

Among others, Titta carries out the following checks:

The format of the migration file structural data complies with the specification.

The metering point ID in the metering data can be found in the master data.

The active metering points reported in the master data can be found in the

metering data.

The time series for energy leaving the metering grid area and energy entering the

metering grid area have been imported for the exchange points (from the metering

responsible party’s perspective).

4.3.5 The transfer of data to the Datahub system

Once a market party’s data has passed the consistency check and meets the quality

requirements specified for data acceptance (see chapter 3.3), the data is transferred to a

separate publishing area. The Datahub system uploads the data from the publishing area

to the Datahub system database.

4.3.6 Data migration check

After the datahub data upload, the datahub system creates so-called data migration

reports, which are returned to Titta. The data migration reports contain all master data

uploaded to Datahub on a market party-specific basis.


15.4.2020


27 March 2020 39 (43)

Fingrid Datahub Oy


reg.



The Titta service publishes the data migration reports for the market parties so that they

can compare the data in the data migration reports with the data in the source systems,

and thus verify that the data in Datahub and the source systems is consistent.

Titta also checks that the data uploaded to Datahub corresponds to the data transferred

to the Datahub system. Based on the check results, the Titta service produces reports

about possible deviations for the administrator users.

With regard to metering data, Datahub delivers the following reports by distribution

system operator:

A list of accounting points, production units and exchange points that are missing

all or some metering data during the period of validity of the grid agreement.

Metering time series that Datahub was not able to link to a master data structure.

Metering grid area-specific loss series.

Supplier-specific total series by metering grid area.

That is, Datahub does not return the uploaded metering data to Titta.

4.4 Data check against external data sources

In the mass data check against an external register carried out in stage 2 and 3 of the

data migration project, each market party must ensure that all basic data have been

delivered to the Titta service prior to the check so that the basic data is included in the

check. After the mass checks, the system will begin using real-time on-line checking in

conjunction with file checks.

The checks performed and the registers utilised in the checks are described in more

detail in the document, Tietojen tarkastus Titassa (Appendix 3).

4.5 The deployment of new IDs

GSRN accounting point IDs and GLN party IDs will be deployed in stage 3 of the data

migration project, and every market party must obtain these IDs itself. The instructions

are available in the EDIEL.fi portal in the document "Use of GS1 IDs in the electricity

retail market" (https://www.ediel.fi/Datahub/business-processes/Datahub-ohjeet).

In addition, several coordinated checks and other actions that apply to all market parties

will be carried out in the data migration project. Participation will be obligatory.

https://www.ediel.fi/sites/default/files/GS1-tunnuksien%20k%C3%A4ytt%C3%B6%20s%C3%A4hk%C3%B6n%20v%C3%A4hitt%C3%A4ismarkkinoilla_0.pdf

https://www.ediel.fi/sites/default/files/GS1-tunnuksien%20k%C3%A4ytt%C3%B6%20s%C3%A4hk%C3%B6n%20v%C3%A4hitt%C3%A4ismarkkinoilla_0.pdf


15.4.2020


27 March 2020 40 (43)

Fingrid Datahub Oy


reg.



5 Data security for the data migration service

5.1 General information about data security for server rooms

Data security controls are observed in the server room service provider’s server rooms.

The server rooms meet the requirements of Vahti classification and all the service

provider’s services are Katakri-audited.

The confidentiality, integrity and availability of data have been taken into account when

implementing data security. The server room service provider has a data security policy

approved by the management and a valid ISO20001 certificate that sets requirements for

data security policy, including a deviation management process and risk management.

The data security practices of the server room service provider are based on the

accompanying certification requirements and on ITIL practices:

KATAKRI II auditing was acceptably completed in 2016.

The ISO20000 IT service management system was certified in 2017.

The ISO27000 information security management system was certified in 2017.

5.2 Data ownership

Ownership of the data delivered to the data migration service by the market parties

remains with the market parties. The obligation to secrecy is specified in the service

agreement between Fingrid Datahub Oy and the market parties and in the agreements

between Fingrid Datahub Oy and the subcontractors.

Data will not be transferred outside the borders of Finland during any stage of Fingrid

Datahub Oy’s processing.

5.3 Maintenance of data security

Datahub’s data migration service applications were data security-audited prior to the start

of the data migration pilot stage and again after the commissioning project.

The following general principles apply to data security auditing:

All medium-serious and serious observations must be corrected in order to allow approval of the service’s data security and so that the service can be taken into production use.

All new external and inter-application interfaces must be data security-audited before production use.


15.4.2020


27 March 2020 41 (43)

Fingrid Datahub Oy


reg.



If necessary, the service will be audited again if significant changes or a large number of

functional changes are made to the service.

6 Data protection for the data migration service

In terms of data protection, Fingrid Datahub Oy acts as a personal data processor within

the scope of the data migration service. In other words, Fingrid Datahub Oy processes

personal data on behalf of the market parties. When the data is transferred to the

Datahub system, a personal data register is formed from the data, and Fingrid Datahub

Oy is the registrar.

Data protection for the data migration service is described in Appendix 3.


15.4.2020


27 March 2020 42 (43)

Fingrid Datahub Oy


reg.



7 Definitions

Terms and abbreviations

Description

Datahub A centralised information exchange solution for the electricity retail market.

Data standard A specification that describes the Datahub business data and dependencies between the data.

Integrity check

A check on the source data of one market party carried out by Titta service. The system ensures that the source data supplied by the market party is complete with regard to the data model in the data standard. The integrity check is a different concept than the consistency check, which examines the consistency of the source data at the market level.

Fingrid Datahub Oy A subsidiary established for the operational activities of Datahub and wholly owned by Fingrid.

Business process A group of tasks linked to each other, which are done in order to achieve a set target, for example, the switching of a customer's supplier.

Source data Data delivered to the Titta service by the market parties.

Master data Master data are all the data delivered to Titta in migration files.

Migration file A file used in data migration that is used to export data from the source system to Datahub in the specified format.

File check Checking performed on an individual file of the data migration service. This includes a syntax check, logical checks of data content and duplicate tests.

Market party

A company that has signed a data migration service agreement with Fingrid Datahub Oy. In conjunction with Datahub data migration, Market party is also used to refer to companies that use the Titta service but do not have a retail electricity market party ID.

Data migration service

A system service in which a check of data coming from market party source systems is carried out, and about which possible deviations are reported back to the market parties. The data migration service gives summary data to Fingrid about the progress of the migration process and what stage each market party is at. The name of the data migration service is “Titta”.

Data migration work group

A group of 11 pilot companies (pilot group) selected by Finnish Energy that, together with Fingrid and the data migration plan partner, participates in specification of the documentation and data migration process associated with data migration.

Data model A data model is an abstract model, which organises data fields and specifies how they are connected to each other.

Consistency check A check performed on the entire source data by the Titta service. The system checks that the source data delivered by different market


15.4.2020


27 March 2020 43 (43)

Fingrid Datahub Oy


reg.



parties is unambiguous and non-contradictory. The consistency check is a different concept that the "integrity check", which is performed on source data delivered by a single market party.

8 Appendices

Appendix 1: Migration file instructions.

https://www.ediel.fi/Datahub/tietokonversio/siirtotiedosto-ohjeistus

Appendix 2: Data check in Titta: Source data check in Titta service

https://www.ediel.fi/datahub/tietokonversio/siirtotiedosto-ohjeistus

Appendix 3: Data protection for the data migration service

https://www.ediel.fi/Datahub/tietokonversio/tietokonversiopalvelun-tietosuoja

Appendix 4: User instructions for Titta

https://konversio.Datahub.fi/Content/Tietokonversiopalvelun_kayttoohje.pdf

Appendix 5: Description of the data migration service

https://www.ediel.fi/sites/default/files/Liite%201%20Tietokonversiopalvelun%20kuvaus.pd

f



https://www.ediel.fi/datahub/tietokonversio/tietokonversiopalvelun-tietosuoja

https://konversio.datahub.fi/Content/Tietokonversiopalvelun_kayttoohje.pdf

https://www.ediel.fi/sites/default/files/Liite%201%20Tietokonversiopalvelun%20kuvaus.pdf

https://www.ediel.fi/sites/default/files/Liite%201%20Tietokonversiopalvelun%20kuvaus.pdf

Data Migration Plan 2.1 – UNOFFICIAL · Data Migration Plan 2.1 – UNOFFICIAL 15.4.2020 Finnish...

Documents

Transcript of Data Migration Plan 2.1 – UNOFFICIAL · Data Migration Plan 2.1 – UNOFFICIAL 15.4.2020 Finnish...