Enhancing Data Quality of Distributive Trade Statistics

Workshop for African countries on the Implementation of International Recommendations for Distributive Trade Statistics

27-30 May 2008, Addis Ababa, Ethiopia

UNITED NATIONS STATISTICS DIVISION Trade Statistics Branch

Distributive Trade Statistics Section

Outline of the presentation

Quality measurement of Distributive Trade Statistics (DTS)

Quality indicators versus direct quality measures

Metadata on DTS Recommendations

Quality measurement of DTS Goal of quality measurement

To provide the user with sufficient information to judge whether or not the data are of adequate quality for their intended use

“Fitness for use” of the data

The users must be able to: Verify that the conceptual framework and definitions that

would satisfy their particular data needs are the same as, or sufficiently close to those employed in collecting and processing the data

Asses the degree to which the accuracy of the data is consistent with their intended use or interpretation

Quality management - all the measures that NSO takes to assure quality of statistical information

Data quality assessment frameworks

QAFs – integrate various dimensions (aspects) of quality, their definitions and quality measurement

Overall aim of QAFs Standardize and systematize statistical quality

measurement and reporting across countries Allow an assessment of national practices to be

made against internationally accepted statistical approaches for quality measurement

Use of QAFs Guide countries ’ efforts for strengthening their

statistical systems by providing a self-assessment tool and for identifying areas of improvement

Technical assistance purposes Reviews of particular statistical domains performed

by international organization Assessment by other groups of data users

0. Prerequisites of quality 0.1 Legal and institutional environment 0.2 Resources 0.3 Relevance 0.4 Other quality measurement 1. Assurance of integrity 1.1 Professionalism 1.2 Transparency 1.3 Ethical standards 2. Methodological soundness 2.1 Concept and definitions 2.2 Scope 2.3Classification/Sectorization 2.4 Basis for recording 3. Accuracy and reliability 3.1 Source data 3.2 Assessment of source data 3.3 Statistical techniques 3.4 Assessment and validation of intermediate data and statistical outputs 3.5 Revision studies 4. Serviceability 4.1 Periodicity and timeliness 4.2 Consistency 4.3 Revision policy and practice 5. Accessibility 5.1 Data accessibility 5.2 Meta data accessibility 5.3 Assistance to users

Institutional and

organizational arrangements

Core statistical process

Statistical products

Relevance

Comparability across

countries

Accuracy

Timeliness and

punctuality

Coherence

Accessibility and clarity

Relationship between IMF DQAF, Eurostat Quality Definition and OECD Quality Measurement Framework

IMF DQAF (incl. elements) Eurostat OECD

Relevance

Credibility

Accuracy

Interpretability

Timeliness

Coherence

Accessibility

Dimensions of quality (1) Prerequisites of quality

All institutional and organizational conditions that have an impact on the quality of DTS data

Elements – legal basis; adequacy of data sharing and coordination; assurance of confidentiality; adequacy of human, financial, and technical resources; quality awareness

Relevance Degree to which DTS data meet the real needs of users Measuring relevance requires identification of user

groups and their needs Credibility

Confidence that users place in the data based on the image of the statistical agency that produces the data

Trust in objectivity of the data Data are perceived to be produced professionally in

accordance with appropriate statistical standards Policies and practices are transparent

Dimensions of quality (2) Accuracy

Degree to which the data correctly estimate or describe the characteristics they are designed to measure

Defined in terms of errors in statistical estimates Systematic errors Random errors

Timeliness Delay between the end of the reference period to which

the data pertain and the date on which the data are released

Closely related to the existence of a publication schedule

Involved in a trade-off against accuracy Accessibility

Ease with which data can be obtained from the statistical office

Suitability of the form or the media of dissemination through which the information can be accessed

Dimensions of quality (3) Methodological soundness

Application of international standards, guidelines and good practices in production of DTS

Elements - adequacy of the definitions and concepts, target population of units, variables and terminology underlying the data; information describing the limitations of the data

Closely related to the interpretability of data Interpretability reflects the ease with which the user

may understand and properly use/analyze the data Coherence

Degree to which the data are logically connected and mutually consistent

Coherence within datasets Coherence across datasets Coherence over time Coherence across countries

Quality indicators versus direct quality measures

Quality measures Items that measure directly a particular aspect of

quality - time lag from the reference date to the release date

Most of them are difficult or costly to calculate in practice

Quality indicators Summarize quantitative information to provide

evidence about the quality or standard of data Do not measure quality directly but provide enough

information for the assessment of quality - response rate is a proxy quality indicator for measurement of non-response bias

Quality measures and quality indicators can either supplement or act as substitutes for the desired quality measurement

Quality Indicators

Criteria for defining quality indicators Cover part or all of the dimensions of

quality Methodology for their compilation is well

established Indicators are easy to interpret

Types of quality indicators Key indicators – coefficient of variations

(accuracy), time lag (timeliness) Supportive indicators – average size of

revisions (accuracy) Indicators for further analysis – user

satisfaction survey (relevance)

Key quality indicators for DTSQuality dimension Quality measures/ indicators

Relevance R1. Identification of gaps between key user interests and compiled distributive trade statistics in terms of concepts, coverage and detail R2. Conducted users’ satisfaction surveys

Accuracy A1. Sampling errors - Coefficient of variation A2. Non-sampling errors - Unit response rate - Item response rate A3. Quantity response rate (% of total sales reported) A4. Number and average size of revisions of distributive trade data

Timeliness T1. Time lag between the end of the reference period and the date of the first release (or the release of final results) of distributive trade data

Methodological soundness MS1. Number and rates of differences in concepts and measurement procedures used in the collection/compilation of distributive trade statistics from the relevant international statistical standards

Coherence CO1. Comparison and joint use of related distributive trade data from different sources

Accessibility AC1. Number and types of means used for dissemination of distributive trade statistics AC2. Distributive trade statistics datasets made available by mode of dissemination as a percentage of total DTS datasets produced

Content of statistical data

Microdata - data on the characteristics of units of the population

Macrodata - derived from the microdata by grouping or aggregation

Metadata - “data about data”, describes the microdata, macrodata or other metadata

Statistical metadata Fundamental purposes of metadata

Describe or document statistical data Facilitate sharing, querying, and

understanding of statistical data over the lifetime of the data

Help users understand, interpret and analyze the data

Help the producers of statistics to enhance the production and the dissemination of the data

A bi-directional relationship between metadata and quality

Metadata describe the quality of statistics Metadata are a quality component

Provide a mechanism for comparing national practices in the compilation of DTS

Metadata on DTS Levels of metadata

Structural metadata – integral part of DTS data tables

Reference metadata - provide details on the content and quality of data, may accompany the tables or may be presented separately

Components of DTS metadata Data coverage, periodicity, and timeliness Access by the public Integrity of disseminated data Data quality Summary methodology Dissemination formats

Recommendations (1) Quality dimensions are overlapping and interrelated

and form a complex relationship. NSOs can decide to:

Implement directly one of the existing QAFs Develop national QAFs that fit best their countries

practices and circumstances

Not all quality dimensions should be addressed for all data

Countries are encouraged to select those quality measures/indicators that together provide an assessment of the overall strengths, limitations and appropriate uses of a given dataset

Quality review of DTS should be undertaken every 4 to 5 years or more frequently if significant methodological changes or changes in the data sources occur

Recommendations (2)

Countries are encouraged to:

Accord a high priority to development of metadata

Consider their dissemination an integral part of dissemination of DTS

Adopt a coherent system and a structured approach to metadata across all areas of economic statistics, focusing on improving their quantity and coverage

Identify user needs of metadata and arrange users into groups so a layered approach to metadata presentation can be applied

Issue regularly, quality reports as part of their metadata

Thank You