European Conference on Quality in Survey Statistics
-
Upload
mikayla-ward -
Category
Documents
-
view
41 -
download
3
description
Transcript of European Conference on Quality in Survey Statistics
![Page 1: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/1.jpg)
European Conference on Quality in Survey Statistics
![Page 2: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/2.jpg)
Quality in Official Statistics:Some Recent and Not so Recent Developments
Lars Lyberg
Statistics SwedenQ2006
![Page 3: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/3.jpg)
Why We Have a Q Conference
• One of the LEG recommendations
• The ESS mission, where it is stated that ESS shall provide the EU and indeed the world, with high quality information, available to everyone, on various areas and levels for decision-making, research and debate.
• The ESS vision with keywords such as world leader, scientific principles, continuous improvement, harmonization, and basis for democracy and progress.
• Westat, Inc.
![Page 4: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/4.jpg)
Contents of Q so Far (# ms)
• Evaluation of data quality (92)
• Sampling and estimation (65)
• Nonresponse (44)
• Questionnaire development and testing (22)
• Confidentiality (17)
• Burden (5)
• Knowledge economy (5)
• Quality management of systems and organizations (63)
• Frameworks (11)
• Reporting (52)
• Process control (36)
• Auditing and self-assessment (11)
• Customers (36)
• Standards (11)
• Harmonization (12)
![Page 5: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/5.jpg)
Somewhat Neglected Topics
• Respondents
• Costs
• Trade-offs
• Standardization
• Fitness for use
• User perception of quality
• Trust
• Audits and self-assessment
• The nitty-gritty of QM
![Page 6: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/6.jpg)
Issue No.10The Concept of Quality
• Statistical Process Control (30’s and 40’s)
• Small errors indicate usefulness (Kendall, Jessen, Palmer, Deming, Stephan, Hansen, Hurwitz, Tepping, Mahalanobis)
• Decomposition of MSE around 1960
• Data quality (Kish, Zarkovich 1965)
• Quality frameworks 70’s
• CASM movement 80’s
• Quality and users
• The UN Fundamental Principles
![Page 7: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/7.jpg)
Components of Quality
![Page 8: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/8.jpg)
Defining Quality
• Fitness for use or fitness for purpose
• Framework components
• Getting the job done, on time, within budget, so that it meets the specified requirements
![Page 9: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/9.jpg)
Quality Assurance and Quality Control
• QA makes sure that the processes are capable of delivering a good product
• QC makes sure that the product is actually good
![Page 10: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/10.jpg)
Controlling Quality
Scores,
Strong and weak points,
Are we measuring up?
Excellence models, CoP,
Reviews, Audits, Self-assessments
NSI, owner,
society
Organization
Variation via control charts, other paradata analysis
Process variables, SPC, CBM, SOP , checklists
Survey designerProcess
Framework dimensions, error est., MSE
Product specsUserProduct
Measures Indicators
Control instrument
Main stake-holders
Quality Level
![Page 11: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/11.jpg)
Issue No. 9Quality Measurement and Quality Reporting
• Objective: To ensure that users have access to measures or indicators of quality, presented in ways that meet their particular needs
• The typical framework: relevance, accuracy, timeliness and punctuality, accessibility and clarity, comparability, and coherence
![Page 12: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/12.jpg)
Examples of Reports
• Dataset-specific quality assessments for different kinds of economic statistics (IMF)
• Process data handbook (LEG/UK)
• Quality guidelines (Stats Canada, Stats Finland)
• Questions and Answers (OMB)
• National or organizational frameworks
• Quality profiles
• Guidelines for quality reporting (Stats Can, ONS,, Stats Sweden, FCSM)
![Page 13: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/13.jpg)
Concerns
• The user has not been consulted
• How should dimensions be measured?
• How do we handle information gaps?
• Some quality indicators are dubious
• Dimensions are in conflict
• What happened to total survey error or total quality? Särndal and Platek (2001)
• Do we need global harmonization?
![Page 14: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/14.jpg)
Issue No. 8Deming’s 13 points
• The 13 factors that affect the usefulness of a survey
• To point out the need for directing effort toward all of them in the planning process with a view to usefulness and funds available
• To point out the futility of concentrating on only one or two of them
• To point out the need for theories of bias and variability that correlate accumulated experience
![Page 15: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/15.jpg)
The 13 Points
1. Variability in Response
2. Differences between Different Kinds and Degrees of Canvass
3. Bias and Variation Arising from the Interviewer
4. Bias of the Auspices
5. Imperfections in the Design of the Questionnaire and Tabulation Plans
![Page 16: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/16.jpg)
13 Points Continued
6. Changes that Take Place in the Universe before Tabulations Are Available
7. Bias Arising from Nonresponse
8. Bias Arising from Late Reports
9. Bias Arising from an Unrepresentative Selection of Data for the Survey or of the Period Covered
10.Bias Arising from an Unrepresentative Selection of Respondents
![Page 17: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/17.jpg)
13 Points Continued
11.Sampling Errors and Biases
12.Processing Errors
13.Errors in Interpretation
![Page 18: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/18.jpg)
Issue No. 7The Race for the No.1 Spot
• Started with The Economist’s ranking
• There is an element of positioning in some of the visions presented by statistical organizations
![Page 19: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/19.jpg)
But:
• There is no justification for competition
• There is no framework, jury or reward
• Statistical organizations have the same problems and tasks and need to collaborate
• Statistical organizations should capitalize on their strengths and develop excellence centre networks and share knowledge
![Page 20: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/20.jpg)
Global Coordination
• Kotz (2005): The statistical community is witnessing an astonishing lack of coordination between many hundreds of statistical offices and agencies scattered throughout the world….
• ..without an overall planning, some of the efforts of civil servants and researchers are largely wasted….
• …well-planned international measures are urgent….
• …new basic global definitions of basic concepts need to be developed…
![Page 21: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/21.jpg)
Issue No. 6Quality Management
• TQM, Business reengineering, Balanced scorecard, business excellence models, Six Sigma
• Tools and core values
• Aversion to QM acronyms
• The management principles cannot be used uniformly across countries and companies
• Operations vs research culture
• Culture eats strategy for breakfast
• We are left with a set of very useful tools and work principles
![Page 22: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/22.jpg)
Examples
• The process view– Key process variables, paradata, control charts
• Spirit of continuous improvement
• Extensive user involvement
• Adoption of the PDCA cycle
• The importance of leadership– Organizing work, inspiration, focussing on important issues,
going for root cause, benchmarking, developing staff competence, evaluating approaches used, promoting good examples, empowerment, communication
![Page 23: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/23.jpg)
From Good to GreatJim Collins
• What’s so special with businesses that have
• been very successful for at least 15 years?
• Level 5 leadership
• First who, then what
• Confront the brutal facts, yet never lose faith
• The hedgehog concept
• Culture of discipline
![Page 24: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/24.jpg)
Issue No. 5Competence
• Staff competence– Excellent programs within the U.S. Federal System (JPSM,
USDA)– Stats Canada, INSEE, ONS, ABS– Excellent university programs
• User competence
![Page 25: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/25.jpg)
Competence Issues
• Existing programs heavy on methodology– Sampling and estimation– Software
• Specialization
• Not much on broader aspects of quality
• Many NSIs talk about the need to skill up
• Any examples of vigorous attempts vis-a-vis the user?
![Page 26: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/26.jpg)
Issue No. 4 Comparative Studies
Comparative studies are increasingly
• important:
• Short term economic indicators
• Literacy surveys
• Social surveys (EU-SILC, ESS)
• Education surveys
![Page 27: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/27.jpg)
Examples of challenges
• Existing systems for input and output harmonization are not sufficient
• Developing a questionnaire that works in all countries and languages
– Concepts, questions, translation, interpretation
• Extensive quality control and supervision
• Varying methodological and financial resources
• Increased distance between user and producer
![Page 28: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/28.jpg)
Issue No. 3The Process View
• Traditional large-scale evaluations are expensive and results come too late
• Small-scale evaluations must be conducted to get estimates of error components (gold standard, latent class analysis, responsive designs, multi-level modelling)
• Long-term improvements are achieved via improved processes controlled by paradata
![Page 29: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/29.jpg)
Generic Control Chart
Upper control limit (UCL)
Lower control limit (LCL)
The central limit (CL)
Time
Pro
cess
cha
ract
eris
tics
![Page 30: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/30.jpg)
Understanding Variation (I)
Common cause variation
• Common causes are the process inputs and conditions that contribute to the regular, everyday variation in a process
• Every process has common cause variation
• Example: Percentage of correctly scanned data, affected by people’s handwriting, operation of the scanner…
![Page 31: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/31.jpg)
Understanding Variation (II)
Special cause variation
• Special causes are factors that are not always present in a process but appear because of particular circumstances
• The effect can be large
• Special cause variation is not present all the time
• Example: Using paper with a colour unsuitable for scanning
![Page 32: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/32.jpg)
Action
• Eliminate special cause variation
• Decrease common cause variation if necessary
• Do not treat common cause as special cause
![Page 33: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/33.jpg)
Standards
• Purposes– To control processes, variability and costs– To improve comparability– To define a minimum level of performance
• Examples– Classification– CBMs and checklists– Standard Operating Procedures– ISO
![Page 34: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/34.jpg)
Problems with Standards
• They must be adhered to
• They must be maintained and updated
• In stovepipe systems it’s easy to find excuses to deviate
• Standard, policy, guideline, best practice, recommended practice……?
![Page 35: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/35.jpg)
Issue No. 2The UserIn place:
• The principle of openness (OMB 1978)
• Responsibility to inform users (many agencies in the 70’s)
• Dissemination procedures
• Customer satisfaction and image surveys
• Councils and service level agreements
Problems:
• How should quality information be communicated?
• How do we distinguish between different kinds of users?
• How do users and producers use quality information and metadata?
• How do producers and users collaborate on fitness for use? (ABS)
![Page 36: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/36.jpg)
Issue No. 1Image Is Everything
![Page 37: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/37.jpg)
A. Eliminate special cause variation
B. Decrease common cause variation if necessary
![Page 38: European Conference on Quality in Survey Statistics](https://reader035.fdocuments.us/reader035/viewer/2022081506/56812c00550346895d907068/html5/thumbnails/38.jpg)
European Conference on Quality in Survey Statistics