Quality Frameworks:Implementation and Impact

Notes by Michael Colledge

Terms of Reference

Some organisations are using quality frameworks others are not. Does it really make a difference?

What is important, the existence of a quality framework, support of top management, quality of the organisation?

Content

What is a quality (assurance) framework?

Who are stakeholders in a quality framework?

What are costs and benefits?

Which organisations have a quality framework?

What is impact on quality of a quality framework?

To what extent can quality frameworks converge?

What is a quality (assurance) framework?

Umbrella for quality practices

Refers to survey programme rather than single survey

Covers range of survey operations rather than single subject

Involves template that can be used for quality assessment

Scope of a quality framework?

TransformationInput Output

Scope of a quality framework?

Product (output data, services)

Process (collection transformation, dissemination)

Input data

Organisation

Difference in scope limits convergence

Dimensions of Data Quality

Generic set: Relevance, accuracy, timeliness, accessibility,

interpretability, comparability, coherence

Could be standardised

Could be validated by feedback from users

Check against ISO 20252

The other dimension: cost/performance

Quality

Effectiveness

Product

“Doing the right thing”

Quality indicators

Performance

Efficiency

Process

“Doing the thing right”

Performance indicators

Quality Related Mechanisms and Topics

Mechanisms Review (self, peer, external) Reporting (indicators, reports) Labelling, declarations Sharing best practices

Topics Metadata management Programme monitoring /evaluation, audit TQM Management

Types of Quality Framework by Owner and Applier

NSO owned and applied Statistics Canada quality framework

IO owned and applied OECD quality framework

IO owned NSO applied IMF DQAF

Types of Quality Framework by Owner and Applier

NOS owned IO applied What value does IO add? Is every data item actually used? What efforts have been made to harmonise

across IOs? How does IO quality assure its outputs?

Questions would be answered by IO owned and applied framework

Stakeholders in Quality Framework

Designers/owners

Organisational units that apply framework

Users of data collected via framework template

Costs and Benefits

Benefits Increased awareness of quality concepts,

components, best practices Systematic quality assessment Identification of quality problems Comparisons over time, across organisational units Indications of additional resource and training needs

Costs - staff time

Which Organisations Have Quality Frameworks?

National

International/Supranational

Impacts of Quality Frameworks?

Statistics Canada Quality Guidelines

Statistics Canada Quality Framework

ESS Self-Assessment Checklist (DESAP)

OECD Quality Framework

IMF Data Quality Assessment Framework (DQAF)

Australian Statistical Clearing House

Impacts of Quality Frameworks: Conclusions

Quality frameworks serve range of useful purposes

Does not have to be developed from scratch Can adapt existing ones (best practice)

Development process itself has benefits

Convergence of Quality Frameworks

Benefits Common terminology Better understanding of concepts Promoting current best practice Reducing reporting burden

Limitations Differing objectives, scope, users For example, process orientation v product orientation