1

The Data Quality Assessment Framework

OECD Meeting of National Accounts Experts

October 2001

2

Purpose of this Presentation

To describe: The IMF’s Data Quality

Assessment Framework (DQAF), and

Experience to date with the DQAF for Reports on Observance of Standard and Codes (ROSCs) and beyond.

3

Plan for Presentation Origins of DQAF DQAF Approach

Framework: what is it? Process: how was it developed? Draft framework: an overview The DQAF suite of assessment tools The work ahead

Links to SDDS/GDDS Working with the DQAF

4

Origins of Recent Work SDDS and GDDS: broadening the scope

of data standards to strengthen the link with data quality

Provision of data by members to the IMF: a need to be clearer about what is called for

ROSC’s: a need for an even-handed approach to assessing data quality

5

Increased Interest in Data Quality

More widely, interest in quality follows from explicit use of statistics in policy formulation and goal setting:

Inflation targeting (spotlight on CPI) Stability Pact in the context of EMU

(spotlight on debt/deficit ratios to GDP) UN Conferences on Least Developed

Countries (inclusion and graduation from list is based on specified economic indicators)

6

The IMF’s Approach

7

The IMF’s Approach Data Quality Reference Site at the

IMF’s Dissemination Standards Bulletin Board http://dsbb.imf.org/dqrsindex.htm

The Site provides an introduction to the topic of data quality and includes a selection of reference materials and articles on data quality issues.

8

The DQAF: What is its Purpose?

Its potential uses To guide data users—to complement the

SDDS and GDDS To guide IMF staff

in assessing data for IMF surveillance and operations,

in preparing ROSCs, and in designing Technical Assistance

To guide country efforts (self-assessment)

9

The DQAF: Requirements Given these differing potential uses,

the framework should be: Comprehensive Balanced between experts’ rigor and

generalists’ bird’s-eye view Applicable across various stages of statistical

development Applicable to the major macroeconomic datasets Designed to give transparent results Arrived at by drawing on national statisticians’

best practices

10

The DQAF : What Is It?

Generic

Dataset-Specific

NA

etc

.

BO

P GFS

etc

. etc

.

11

How the DQAF Was Developed

We engaged a national statistical office to help develop the generic framework

In parallel, IMF staff worked on frameworks for several datasets National accounts was reviewed in June 2000 National accounts (revised) and four other

specific frameworks were circulated informally in the international statistical community for comment in August-September 2000

12

How the DQAF Was Developed

Drafts were discussed in topical or regional meetings, e.g. East Asian Heads of NSOs ECB Working Group on Money and

Banking Statistics IMF BOP Statistics Committee GFS Expert Group meeting

13

How the DQAF Was Developed

IMF staff tested the frameworks in the field

A paper for the Statistical Quality Seminar in December 2000 presented: Revised generic framework Revised BOP dataset-specific framework Alternatives for a preview (“lite”) tool Sample summary presentations of results

To access the paper: http://dsbb.imf.org/dqrsindex.htm

14

DQAF: an Overview Uses a cascading structure

• Five dimensions of quality- and for each dimension,

• Elements that can be used in assessing quality- and for each element,

• Indicators that are more concrete and detailed- and for each indicator,

• Focal issues that are tailored to the dataset• - and for each focal issue

• Key points

15

DQAF: an Overview

The five dimensions of the IMF’s

Data Quality Assessment Framework

1. Integrity

2. Methodological soundness

3. Accuracy and reliability

4. Serviceability

5. Accessibility

16

DQAF: an Overview Also, some elements/indicators are

grouped as “prerequisites of quality” Pointers that are relevant to more

than one of the five dimensions Generally refer to the umbrella

agency Example: quality awareness

17

Prerequisites for Quality

Legal and institutional framework Roles and responsibilities of statistical

agencies Data sharing and coordination between

data producing agencies Access to administrative and other data

for statistical purposes Nature of reporting Resources Quality awareness

18

Elements of Integrity

Professionalism Transparency Ethical standards

19

Elements of Methodological Soundness

Concepts and definitions Scope Classifications Basis for recording: accounting

rules and valuation principles

20

Elements of Accuracy

Source data Statistical techniques: compilation

procedures and statistical methods and adjustment

Assessment and validation

21

Elements of Serviceability

Relevance of the national accounts program

Timeliness and periodicity Consistency Revision policy and practice

22

Elements of Accessibility

Data accessibility Metadata accessibility:

documentation Assistance to users: service and

support

23

Indicators of Consistency Temporal consistency Internal consistency Intersectoral consistency

24

Focal Issues for Internal Consistency

Internal consistency of the annual accounts

Internal consistency between quarterly and annual estimates

25

Key Points Internal Consistency of the National

Accounts

Discrepancies between approaches shown?

Size of discrepancies? Differences between growth rates? Supply and use framework applied? Do total supply and use match? Does net lending/borrowing match

between sectors?

26

General Reactions “Welcome initiative” “Fills important gap” “Is careful and thoughtful” “Provides basis for coherent and

practical way forward in a complex field”

27

General Reactions Some other points

Is the framework really operational for small countries?

Can it be used without giving a “black mark” for points that are irrelevant to a country?

Is the framework able to identify “poor” statistics prepared within a developed statistical system?

28

General Reactions Some other points (cont’d)

Expand the range of datasets covered Coordination with other organizations

working on data quality is important Continue working in a consultative

manner

29

The DQAF Suite of Tools DQAF “Lite”

Background: interest in a version that might serve as a diagnostic preview or for a non-statistician’s assessment

IMF is field testing a “Lite” made up of 13 indicators.

30

Summary presentation of results Background: Interest in a presentation

of results for, e.g., policy advisors IMF is testing a summary presentation

For each dataset, a one-page table At the two-digit level (21 elements)

On a 4-point scale, from “practice observed” to “practice not observed”

With an “n.a.” column With a “comments” column

The DQAF Suite of Tools

31

Data Quality Assessment Framework

Summary for [dataset]

 Note: O = Practice Observed; LO = Practice Largely Observed: MNO = Practice Materially Nonobserved;

NO = Practice Nonobserved; NA= Not Applicable Comment: only if different from O.

32

Dataset

(6-digit)

“Lite”Generic

(3-digit) Summary of

Results

DatasetSpecific

(5-digit)

NA

etc

.

BO

P GFS

etc

. etc

.

The DQAF Suite of Tools

33

Work ahead

Test the suite in a wider range of country situations especially with non-statisticians

Refine and revise the suite Complete supporting materials

A Glossary Supporting Notes for specific datasets A Methodology (a how-to-do-it guide)

Develop frameworks for other datasets

34

Links to the SDDS/GDDS

Summary: The DQAF complements the SDDS/GDDS

All of the elements of the SDDS/GDDS are also found within the DQAF

35

Links to SDDS/GDDS The purpose and scope of the

SDDS/GDDS and DQAF differ: In SDDS/GDDS, as dissemination standards,

quality is a dimension. That dimension takes an indirect approach to

dealing with, e.g., accuracy--it calls for dissemination of relevant information.

In DQAF, as an assessment tool, quality is the umbrella concept.

That concept covers collection, processing, and dissemination of data.

36

Links to SDDS/GDDS The DQAF definition of “quality”

has been brought into line with the emerging consensus that quality is a multidimensional concept. Some aspects relate to the product Some aspects relate to the institution

37

Links to SDDS/GDDS DQAF is “more active” in dealing with,

e.g., conformity with international guidelines, accuracy, and reliability. SDDS and to a lesser degree GDDS left

users on their own to make judgments DQAF guides users in making such

judgments by providing two structured dimensions:

Methodological soundness Accuracy and reliability

38

Working with the DQAF The earlier list of potential uses of

the DQAF included “To guide IMF staff “ Largely this refers to staff of the IMF

Statistics Department Interrelated uses:

in assessing data for IMF’s use in surveillance and operations,

in preparing ROSCs, and in designing technical assistance

39

Working with the DQAF We are now using the DQAF in the

field In capacity building advisory missions In ROSCs

40

Working with the DQAF What do we see from the experiences?

Advantages Provides more structure to technical assistance Promotes consistency across staff/experts Potentially provides input for useful database Places data standards in the center of work on the

international financial architecture

Challenges Puts premium on consistency Calls for explicit judgments

41