Transition Guidelines: Managing legacy data and information · technical/management requirements...

Transition Guidelines:

Managing legacy data and information

November 2013

v.1.0

NSW Transition Guidelines: Managing legacy data and information v.1.0

of 21

Document Control

Document history

Date Version No. Description Author

October 2013

0.1 Draft Department of Finance and Services

November 2013

1.0 Final Department of Finance and Services

Approvals

Name Date

ICT Leadership Group

Document custodian

Name Signature

Department of Finance & Services, Strategic Policy


of 21

CONTENTS 1. PURPOSE ......................................................................................................................................... 4

2. INTRODUCTION .............................................................................................................................. 4

3. SCOPE ............................................................................................................................................. 5

4. REQUIREMENTS FOR TRANSITION ................................................................................................. 6

5. OPTIONS FOR TRANSITION ........................................................................................................... 15

6. IDENTIFYING MATERIAL FOR DIGITISATION ................................................................................. 17

7. RELATED GUIDANCE ..................................................................................................................... 18

8. CONTACT ...................................................................................................................................... 18

APPENDIX A – GLOSSARY ...................................................................................................................... 19

APPENDIX B – RESOURCES FOR MANAGING TRANSITION ................................................................... 20

APPENDIX C – INFORMATION MANAGEMENT PRINCIPLES ................................................................. 21


of 21

1. Legacy data and information is data and information stored in superseded formats and systems.

The purpose of these guidelines is to assist agencies to manage or transition legacy data and

information in ways that will minimise business risks.

The guidelines provide direction to address risks associated with:

migrating data to new systems, including transitioning from paper to digital records;

transitioning to shared services, Cloud computing or other third-party arrangements;

maintaining legacy data and/or legacy systems;

disposing of legacy data and/or legacy systems.

These guidelines will assist agencies in making decisions to ensure data and information remains

reliable and fit-for-purpose, to ensure core business information is sustained and time expired

information is accountably destroyed, to enable better service delivery and better value investment

in the NSW public sector. The guidelines will be implemented by agencies in various ways according

to their specific business needs and operating environment.

This document does not include procedures for managing or conducting a transition project.

Appendix B indicates resources which address specific transition scenarios and provide more

detailed procedural guidance for migration or conversion.

The guidelines can help agencies implement the Information Management Framework and best

practices for managing information across the NSW public sector.

2.

2.1 Information Management Framework

A key initiative of the NSW Government ICT Strategy is the development of an Information

Management Framework to support the way government designs systems, and administers and

uses data and information.

The Framework is a set of standards, policies, guidelines and procedures which are implemented

either manually or, where possible, automated through technology. This Framework will enable

data and information to be managed in a secure, structured and consistent manner.

It will ensure that data and information can be appropriately shared or re-used – immediately and

in the longer term – by agencies, individual public sector staff, the community or industry for better

services, improved performance management and a more productive public sector.

http://www.finance.nsw.gov.au/ict/


of 21

2.2 About transition

As they move to new ways of working or ICT sourcing arrangements, agencies will often have to

determine how best to manage or transition data and information stored in superseded formats

and systems. This could require the transition from one hardware or software platform to another;

or the proactive digitisation of incoming paper records for current business; or retrospective

digitisation of older paper records.

Questions about legacy data management can also arise during periods of administrative change.

Where responsibilities are transferred, agencies may relinquish or inherit relevant data and

information in a range of formats and systems.

In transition scenarios, some legacy data and information may be needed to support ongoing

business operations, while some may be accountably destroyed.

It is important to assess the risks and benefits associated with either:

Maintaining data and information in superseded formats or systems; or

Converting or migrating data and information to new formats or systems.

Requirements for managing legacy data and information will vary according to the drivers for

transition; the business case and objectives for transition; the use and value of the data or

information being transitioned.

Agencies need to ensure appropriate controls continue to be applied to data and information,

during and after a transition.

These guidelines identify the issues that need to be considered to support and manage business

information assets through transition scenarios.

3.

These guidelines apply to the management and maintenance of data and information assets within

all NSW Government Departments, Statutory Bodies and Shared Service Providers and related

entities of departments/divisions.

The guidelines are recommended for adoption by NSW:

State-owned corporations (SOCs);

Public trading entities (PTEs);

Entities fully funded by the NSW Government; and

Local Government Authorities.


of 21

4. Requirements for managing legacy data and information will vary according to the drivers for

transition; the business case and objectives for transition; the use and value of the data or

information being transitioned. Understanding these requirements will enable agencies to identify

the best option or approach to transition.

This section describes the issues and critical factors agencies should consider to determine

pathways for managing data and information with ongoing business value. Effective transition

strategies can maximise the process efficiencies and cost savings made possible in the digital

environment, whilst also enabling the routine and authorised destruction of time-expired

information.

4.1 Drivers for transition

Various circumstances may compel the need to transition and each scenario will highlight different

areas of risk to be addressed as a priority.

DRIVER FOR TRANSITION CRITICAL FACTORS

Automating a business process

(eg scanning paper documents to create a seamless

digital work process)

Identify documents for scanning and capture into

the new system.

Determine which controls and functionality are

required to support it in the new system.

Understand the “lifetime” of legacy data and

information, and its relationship to newly-created

data or information to determine needs and

limitations for back-capture.

Management arrangements for:

source paper documents, once scanned

legacy paper documents which are not

scanned,

- including search, accessibility, security, retention

and destruction or transfer to State Records.

Retiring or decommissioning a system

(eg due to upgrade or new application)

Identify data for migration to the new system.

Determine which controls and functionality are

required to support it in the new system.

Management arrangements for legacy data which is

not migrated – including search, accessibility,

security, retention and destruction or transfer to

State Records.


of 21

DRIVER FOR TRANSITION CRITICAL FACTORS

Implementing a completely new system

(eg when a new service is commenced

Dependencies between this new source of data,

particularly any requirements to integrate or merge

datasets for analysis.

Potential for re-purposing; value of new data for

informing decision making in other operational

areas.

Continuity or preservation of data/information

required long-term

(eg changing the format, storage medium, or

transferring to Digital State Archives)

Identify data or information that can be transferred

to the Digital State Archives maintained by State

Records.

Management arrangements for legacy data which is

not transferred – including search, accessibility,

security, retention and destruction.

Particular requirements for future format

conversions or system migrations that may be

required to support long-term preservation.

Administrative change

(eg where functions move from one agency to

another; or a function ceases to be carried out by

NSW Government)

Identify which data or information should be

transferred with the function, and which is required

to meet ongoing business or legal requirements.

Clarifying custodianship responsibilities of the

relinquishing agency and the receiving agency.

Reconciling legal or business needs for data with

the realities of different ICT environments.

Moving to a new provider or service delivery model

(eg shared services, cloud services, or other contract

arrangements)

Clarify the responsibilities of each party (agency,

provider) for applying information controls.

Identify which data should be transitioned into (and

subsequently, out of) service arrangements.

Performance monitoring and reporting.


of 21

4.2 Business considerations

A number of considerations in the broader business environment can potentially influence

decisions around whether to transition legacy data or information; and if so, which legacy data or

information should be transitioned and how.

These factors will be assessed in combination with the specific data or information management

requirements, to determine the overall value proposition associated with each transition option.

The business case for the transition should be clearly established, to ensure the transition meets

the objectives and delivers anticipated benefits.

KEY ISSUES FOR TRANSITION CRITICAL FACTORS

Have the business case and/or anticipated benefits of

transition been clearly established and agreed by all

key stakeholders?

Confirm that transition is required and/or can

deliver the anticipated benefits.

Identify any alternatives to transition which may be

suitable for consideration (eg staff training; process

re-design; policy enforcement; data disposal).

Will service delivery be improved by transition?

What will be the impact on service delivery during

transition?

Requirements for business process or

organisational change.

Customer service and communications.

Is there sufficient capability/expertise to:

identify business needs and

technical/management requirements for the

legacy data and information?

carry out the transition process?

maintain legacy data or information systems

for the duration of their “lifetime”?

Availability of key staff or cost of external

contractors for business analysis, information

management and technical tasks.

Is there sufficient understanding of how the legacy

data or system was used?

What are the implications of not transitioning?

Availability of system documentation and/or staff

with applied knowledge of the original purpose of

the data or system; and its current use within the

organisation.

Does the resourcing and funding model cover the

entire transition process; and/or the maintenance of

legacy data or information for its entire “lifetime”?

Risk of incomplete transition or legacy data and

information becoming unsupported

Has a plan for communications or change

management been prepared?

Have staff training needs been considered?

Impact on costs, timelines, business processes

Who will determine the feasibility of the transition? Input required from management, operational

staff, information management and ICT staff.


of 21

4.3 Data and information management considerations

Agencies need to ensure appropriate controls continue to be applied to data and information,

during and after a transition.

It must be possible to find, use and understand data and information; it must be stored and

managed in a secure, efficient and accountable manner – whether it is migrated, converted,

maintained in a superseded format or system, or until it is accountably destroyed.

Summary checklist

Steps to manage the transition process:

Define, assign and resource accountable roles for managing legacy data or information,

including senior responsibility.

Define appropriate information governance frameworks that will direct the management of

data and information in the new custodian/service arrangement, system or format.

Inform, train and support users of legacy data or information.

Monitor and review ongoing management of legacy data or information against business

objectives.

Steps to manage legacy data or information:

Review legacy data or information to understand how it is created, organised and used.

Understand also accountabilities, opportunities, compliance obligations, cost, value and

risks associated with the data or information.

Determine whether any or all legacy data or information needs to be transitioned to new

custodian, system or format.

Define pathways for identified legacy data or information to be transitioned to new

custodian, system or format, managed by appropriate governance frameworks and tagged

with relevant metadata that enables its ongoing use and accessibility.

Establish systems or processes that support effective management and appropriate internal

and/or external use of legacy data or information that is not transitioned.

Establish a system of ongoing checks to ensure that legacy data or information remains

accessible and usable for as long as it required.

Establish systems or processes which enable legacy data or information that is not

transitioned to be routinely and accountably destroyed when it is no longer required, in

accordance with the State Records Act.


of 21

Detailed questions

The following table provides more detailed questions to help clarify the relative costs and risks

associated with either maintaining or transitioning legacy data and information.

The key issues are organised below according to which business outcomes and principles of the

Information Management Framework are being supported.

Refer to Appendix C for more detail around the IM Principles.

IM PRINCIPLE: GOVERN Data and information are governed according to their value, as assets for which we are accountable


Who has responsibility for the business process? Is

this being transitioned?

Who is the data/information owner, custodian?

Can all existing governance requirements be met by

the new custodian/service, system or format? What

alternate strategies can be put in place if they are

not?

Decisions to retain some legacy data and information

may incur ongoing licensing or storage costs.

Clarify roles, responsibilities and accountability.

Assign critical tasks for data and information

management (SLA, MOU, etc).

Ensure transparency and effective decision-making.

Define and document transition pathways and the

information governance decisions made.

Establish the governance framework for effectively

managing data and information in the new format,

system or custodian/service environment.

Cut off periods between legacy and new business

environments must be clear and understood. If they

are not, duplicate transactions and the lack of a

single source of business truth can cause risks.


of 21

IM PRINCIPLE: COLLECT

Data and information are captured to enable services and operations; and collected in line with standards

so they are fit-for-purpose


What is the quality of the data and metadata to be

transitioned?

In which different business, system and network

environments is the legacy data stored?

How was it collected, and for what purpose?

Is data collection ongoing?

(eg the business process is current)

Does the transition require data to be collected in a

different format?

Does the transition propose a new use for an existing

source of personal information?

Identify possible privacy issues

Evaluate the risk/benefit of managing data in

multiple formats or converting and consolidating

Estimate growth and storage requirements


of 21

IM PRINCIPLE: ORGANISE Data and information are organised for easy retrieval and efficient management


How is the information organised?

What relationships exist between data and

information components or aggregations; or

between the legacy system and other systems?

What metadata is present in the legacy system?

How does it map to the target system or dataset?

(can legacy systems be cross-referenced?)

Has it been / can it be standardised?

What metadata is necessary to access, use, manage

and understand legacy data and information? How

can this be maintained and/or carried forward?

Metadata can be critical for authenticating and

managing legacy data and information? How can

this form of metadata which may not be connected

directly to the data but which may be kept in

security or audit logs be maintained as required?

Can all data and information be tagged with

relevant retention and destruction requirements

during transition?

Is some form of data management on the legacy

data required before it can be transitioned? For

example, legacy email accounts should be

appropriately managed and not transitioned as a

whole to a cloud service environment.

Retain metadata especially where it provide context

and meaning to data:

Data can be used and understood

Helps inform decisions made under other

principles.

Technical capacities of systems may limit the legacy

data and information that can be transitioned.

Limited information management frameworks in

some business environments may make identification

of legacy data and information to be transitioned or

managed challenging.

Building these requirements into new custodian,

system and format management environments will

streamline management, reduce ongoing costs and

facilitate future transitions.

Large demands can be placed on staff organising

information for transition and staff in these roles

need to be adequately resourced and supported.

Resourcing of the retrospective organisation of data

and information can be a significant cost but

expending money on this can yield significant cost,

management and accountability dividends in the new

custodian, system or format environment.


of 21

IM PRINCIPLE: SECURE Data and information are secured to protect their integrity and with respect for their sensitivity


Is any of the data/information personal, private,

privileged, confidential, classified or otherwise

sensitive?

What protective markings or access controls have

been applied? (Note: controls may have been applied

via roles/positions, workflows, or data and

information classifications/containers.)

Can these controls be applied in the new format,

system or custodian/service environment?

Can the security be down-graded?

If not now, when and how?

Evaluate the functionality required and any cost

implications associated with maintaining controls.

Implement appropriate security measures to ensure

the data and privacy are not compromised.

Consider rights and permission management for

legacy data and information.

Does the target system (through functionality or the

presence of other data) allow de-identified data to

be re-identified?

IM PRINCIPLE: USE Data and information are used to support evidence-based policy, targeted service delivery, and re-used to derive maximum return on investment


Who needs to / could use the legacy data?

(within the agency, across business units)

How will the legacy data be used? Is it still in use for a

specified business purpose; what is the frequency of

retrieval?

How will it be accessed?

Do on-going use considerations identify that different

transition strategies should be applied to different

types of legacy data? For example, it may be

necessary for action officers to reference the last 2

years of customer enquiries and so these records

should be digitised, but there is not a sufficient use

case to justify the digitisation of the last 10 years of

these records so the remaining paper records will not

be digitised.

What is the impact of transition on the business

process and users? Will transition improve the

accessibility of data and information?


implications associated with maintaining usability

and providing access.

Multiple parties may require access to legacy data,

across a range of business environments. Can this

be enabled?


of 21

IM PRINCIPLE: SHARE Data and information are shared to streamline service delivery and reporting; to enable value adding; and support transparency


If appropriate, can the legacy data or information be

shared or published? (outside the agency)

Free? Or at what cost?

What licensing options are applicable?

Does the transition propose sharing, linking or

matching personal information between different

organisations?

Identify possible privacy issues .

Consider rights and permission management.

Evaluate the risk/benefit of managing data in

multiple formats or converting and consolidating.

IM PRINCIPLE: MAINTAIN Data and information are maintained for as long as they have value then disposed systematically and accountably


Is the legacy data or information considered vital for

ongoing operations?

Is the legacy data or information stored in a format

that is proprietary or standard? Is it still supported?

Is the legacy data or information in a stable

condition, or is it unstable/fragile?

Does the legacy data or information need to be

kept? Can it be destroyed or transferred according

to an applicable disposal authority?

If not now, when and how?

Maintain metadata mappings from former to new

(target) systems

Ensure adequate system and configuration

documentation

Does full testing of the new custodian, system or

format demonstrate a successful validation? Have

the results been verified? Is the transition

successful, with all anticipated information,

governance and functionality?

Post validation, can source data be maintained for a

business appropriate validation period then

destroyed?

Will transition help preserve the data or information

for a longer period of time?

Is the format unique? Is a recognised/standard

migration path or conversion process available?

Will transition expose the data to an unacceptable

risk of loss or damage?

Is any management or reporting functionality lost or

gained in the transition?


implications required to maintain the data for the full

duration of its retention period.

Implement appropriate mechanisms to implement

approved disposal.

Develop appropriate backup strategy and disaster

management plan for legacy data and information.

Opportunity to implement stable, long-term formats,

to limit format and system dependencies.


of 21

5.

There are several options or models which agencies may consider for managing legacy data and

information. The most suitable approach will depend on the unique combination of requirements

identified according to the risks and benefits assessed in the preceding sections.

Each approach involves a combination of:

Maintaining data and information in superseded formats or systems; and/or

Converting or migrating data and information to new formats or systems.

Custodian agencies must ensure the capability, functionality and resources required to apply

appropriate controls to data and information, during and after a transition.

It is important to ensure the new environment incorporates robust information management

controls and data disposal mechanisms.

The selected approach will enable the strategic application of governance frameworks and the

maintenance of core business information in both new and legacy environments.

OPTION CRITICAL FACTORS

1. Transition all legacy data or information (with appropriate

metadata) to new custodian, system or format.

After transition is completed, reviewed and success is

confirmed, destroy any source data or information

according to an applicable disposal authority.

Ongoing access to legacy data

or information is necessary for

process continuity.

Standard migration /

conversion paths available

2. Transition some legacy data or information required by

ongoing business processes (with appropriate metadata) to

new custodian, system or format.

Establish the governance and management requirements

needed by the new custodian/service, system or format to

support the legacy data or information.

Retain remaining legacy data or information in the

superseded format or system, until it is eligible for

destruction.

Determine which legacy data

or information is required for

ongoing business needs.

Identify the governance

requirements/responsibilities

to be transitioned.

Determine whether retained

legacy data or information will

be maintained in-house or

stored and managed by a third

party.


of 21

OPTION CRITICAL FACTORS

3. Transfer some legacy data or information (with appropriate

metadata) to the Digital State Archive (State Records).

Transition remaining legacy data or information (with

appropriate metadata) to new custodian, system or format.

Data or information identified

as State archives in an

applicable disposal authority.

Agree on an appropriate

migration plan with State

Records for the archived data

or information.

4. Transfer some legacy data or information (with appropriate

metadata) to the Digital State Archive (State Records).

Retain remaining legacy data or information (with

appropriate metadata) in the superseded format or system,

until it is eligible for destruction.

Data or information identified

as State archives in an

applicable disposal authority.





party.

5. Maintain all legacy data or information in the superseded

format or system, until it is eligible for destruction.

Understand the retention

period for the data or

information. Specify when and

how it will be disposed.

Resources are available to

maintain superseded format

or system and ensure data or

information remains

accessible for the retention

period.





party.


of 21

6.

The requirements and options for transition, outlined in these guidelines, apply to data and

information in any format – including digital, paper or microforms.

Most agencies have large stores of paper files and may be considering whether or not there is a

business case for digitising (or scanning) these documents.

The following checklist provides high-level questions to determine whether digitisation could be

beneficial and/or feasible.

Further resources are listed in Appendix B. They offer more detailed procedural guidance for

transitioning from paper records to digital and managing digitisation projects.

YES

High activity – if documents are frequently accessed and requested by multiple users. Digitisation will significantly enhance access or increase use by a specific user group (staff,

customers, general public). High volume – if savings in storage and consolidation outweigh the costs of digitisation. Documents at risk of loss or inaccessibility due to deterioration, before retention period

expires – cost of digitisation outweighs alternative preservation options. Specific workflow or process efficiencies can be demonstrated. Identified business objectives or service delivery targets will be met or enabled by

digitisation.

NO

× Retention and disposal requirements are unknown.

× Retention period has expired and documents are being maintained "just in case" (ie there is no defined legal or business requirement).

× Low activity – documents are rarely accessed and remaining retention period is short (eg less than 3 years).

× Inadequate systems – agency does not have skilled staff, appropriate software or policy framework for managing digital records (ensuring security, integrity and accessibility).

× To create a “backup” of documents that have been legally disposed; or have been transferred to a new custodian.


of 21

7.

This document is consistent with:

guidance provided by the NSW State Records Authority for records management;

guidance provided by the NSW Information and Privacy Commission for information access

and privacy; and

requirements for information security outlined under Premier’s Memorandum M2012-15

Digital Information Security Policy.

8.

For assistance with any aspect of this document, please contact:

Principal Policy Officer, Information Strategic Policy Department of Finance & Services Level 15, McKell Building 2-24 Rawson Place SYDNEY NSW 2000 (02) 9372 8291


of 21

–

TERM DEFINITION

Custodian The agency with possession, control of and/or responsibility for the legacy data or information. The custodian is primarily responsible for:

the development, management, care and maintenance of a specified dataset or information asset;

ensuring that all legal, regulatory and policy requirements are met in relation to the management of the specified dataset or information asset; and

determining the conditions for appropriate use, sharing and distribution of the specified dataset or information asset.

Legacy data/information Data or information retained in a superseded format or system.

Legacy system Database or information system which has been superseded or decommissioned for current business processes.

Transition A change in the responsibilities for controlling or managing a particular set of data or information; and/or

A conversion from one format to another, for a particular set of data or information; and/or

A migration from one system or storage location to another, for a particular set of data or information.


of 21

–

The following resources provide more detailed guidance or procedures relating to specific scenarios for

transitioning or managing legacy data and information.

(IPC) = Information and Privacy Commission (SRA) = State Records Authority

Identifying Privacy Impacts

Identifying privacy issues – checklist (IPC)

Providing Access

GIPA Act Compliance checklist for agencies (IPC) – 3.5 Searching for information; 3.12 Providing access

Transferring State Archives

Digital Archives Migration Methodology (SRA)

Transition to Shared Services

Recordkeeping In Brief 44 - Using shared services for records management (SRA)

Migration to New Systems

General Retention and Disposal Authority GA33: Source records that have been migrated (SRA)

Guideline 22: Managing digital records – 4. Effectively manage the migration of your digital records (SRA)

Transition to Cloud Services

Recordkeeping In Brief 62 - FAQs about cloud computing (SRA)

Recordkeeping In Brief 54 Storage of State records with service providers outside of NSW (SRA)

General Retention and Disposal Authority GA35: Transferring records out of NSW for storage with and

maintenance by service providers based outside of the State (SRA)

Transitioning Paper Records to Digital

General Retention and Disposal Authority GA36: Imaged records (SRA)

Guideline 25: Managing digitisation programs and projects – 12. Managing original paper records (SRA)

Guideline 25: Managing digitisation programs and projects – 13. Disposal of original paper records after

digitisation (SRA)

Transfer of System Ownership/Responsibility

Recordkeeping In Brief 23 - Managing records in administrative change (SRA)


of 21

–

NSW Government data and information are managed in a way that enables better service delivery and

better value investment. They are…

Governed

as assets of strategic, operational and administrative value to NSW Government agencies;

in a manner that is transparent and accountable to NSW citizens and organisations.

Collected

to document or facilitate delivery of services and the functions of NSW Government agencies;

with regard for the privacy of NSW citizens and the confidentiality of NSW organisations;

once, according to standards that support relevance, accuracy and consistency so they are reliable, fit for purpose and can be appropriately re-used by NSW Government agencies to improve service delivery or management reporting.

Organised

described and linked to related data or information, so they are easy for NSW Government agencies to search, retrieve, use and compare;

identified and integrated into systems that allow NSW Government agencies to routinely track and manage them according to their value.

Secured

against unauthorised access, alteration, loss or deletion, to ensure their integrity and ongoing value to NSW Government agencies;

using controlled and auditable processes that demonstrate to NSW citizens and organisation the protection of sensitive data and information.

Used

to support planning, decision-making, resource allocation, reporting, communications and transactions by NSW Government agencies;

processed and analysed by NSW Government agencies to develop evidence-based policy and deliver targeted services to NSW citizens and organisations.

and re-used, so NSW Government agencies derive maximum benefit from their investment in these assets.

Shared

with regard for the privacy of NSW citizens and the confidentiality of NSW organisations;

with other NSW Government agencies to reduce duplication of effort, streamline service delivery and provide a consolidated view of customer needs or public sector performance;

published and made available for discovery by NSW citizens and organisations, providing opportunities to communicate, consult and collaborate or to engage in value-added processing, analysis and development.

Maintained

using cost-effective, risk-based measures that facilitate business continuity for NSW Government agencies;

to ensure their availability and reliability, for as long as they support service delivery and accountability by NSW Government agencies;

and then systematically destroyed when their use and value has ceased, to minimise the costs and risks to NSW Government agencies of over-retention; or systematically archived to protect the enduring rights and interests of NSW citizens and organisations.

Transition Guidelines: Managing legacy data and information · technical/management requirements...

Documents

Transcript of Transition Guidelines: Managing legacy data and information · technical/management requirements...