Mixed Modes and Measurement Error

31
Mixed Modes and Measurement Error Gerry Nicolaas

description

Mixed Modes and Measurement Error. Gerry Nicolaas. Mixed Modes and Measurement Error. Funded under the ESRC Survey Design and Measurement Initiative 3-year contract starting 1 Oct 2007 Collaboration between academics and data collection organisation. Research Team. - PowerPoint PPT Presentation

Transcript of Mixed Modes and Measurement Error

Page 1: Mixed Modes and Measurement Error

Mixed Modes and Measurement Error

Gerry Nicolaas

Page 2: Mixed Modes and Measurement Error

Mixed Modes and Measurement Error• Funded under the ESRC Survey Design and

Measurement Initiative• 3-year contract starting 1 Oct 2007 • Collaboration between academics and data

collection organisation

Page 3: Mixed Modes and Measurement Error

Research Team• National Centre for Social Research (NatCen)

Gerry NicolaasSteven Hope

• Institute for Social & Economic Research (ISER)Peter LynnAnnette JäckleAlita NandiNayantara Dutt

• Freelance Survey Methods ConsultantPam Campanelli

Page 4: Mixed Modes and Measurement Error

Background

Page 5: Mixed Modes and Measurement Error

• Interviewer-administeredface-to-face interview (PAPI, CAPI)telephone interview (PAPI, CATI)

• Self-administeredpostal questionnaireself-completion questionnaires with interviewer

present (PAPI, CASI, A-CASI)IVR and TDE in telephone surveysweb/email

Modes of data collection:

Page 6: Mixed Modes and Measurement Error

Choice of mode = trade-off

Quality

Time Cost

Coverage error

Non-responseerror

Measurement error

Recording error

Samplingerror

Page 7: Mixed Modes and Measurement Error

Differences across modes• coverage error

different members of the population have a zero chance of inclusion depending on the mode

• non-response errordifferential non-response bias

• measurement errorrespondents give different answers in different

modes

Page 8: Mixed Modes and Measurement Error

Increasing use of mixed modes:• To maximise response rates (unit & item)

offer choice of mode to respondentsuse alternative mode among non-respondents

• To reduce measurement errorself-completion modes for sensitive items within

a f2f or tel interview• To reduce costs

sequentially with cheapest mode firstdifferent waves of a longitudinal study

Page 9: Mixed Modes and Measurement Error

Mixing modes:• Different modes to collect different data items from

the same respondentse.g. CAPI with CASI module data comparability not affecteddata quality may be improved

• Different modes to collect the same data from different respondents e.g. tel follow-up among postal non-responderse.g. panel survey with wave 1=CAPI & wave 2=CATIpotential for mode effects

Page 10: Mixed Modes and Measurement Error

For an overview of using mixed modes in surveys:

• Edith D. de Leeuw (2005) “To Mix or Not to Mix Data Collection Modes in Surveys”, Journal of Official Statistics, 21(2), pp. 233–255

• One of Edith’s conclusions: Hardly any theoretical or empirical knowledge is

available on how to design optimal questionnaires for mixed-mode data collection.

Page 11: Mixed Modes and Measurement Error

Study Design

Page 12: Mixed Modes and Measurement Error

Main Objective

• Practical advice on how to improve portability of questions across modesWhich mode combinations are likely to produce

comparable responses?Which types of questions are more susceptible

to mode effects?

Page 13: Mixed Modes and Measurement Error

Research Design• A literature review & framework of mixed modes

develop a theoretical framework identify gaps in evidence base and formulate

hypotheses to address gaps• Quantitative data analysis

test hypotheses using existing datasets and new experimental data

• Cognitive interviewingexplore how respondents process questions in

different modes

Page 14: Mixed Modes and Measurement Error

The Quantitative Data• Existing datasets, e.g.

1999 Welsh Assembly Election Study2005 Social Capital SurveyEuropean Social Survey mode experiments2006 Health Survey for England London boost

• New experimental datafollow-up surveys to BHPS & NatCen Omnibusfocus on f2f, tel and web comparisons

Page 15: Mixed Modes and Measurement Error

NatCen Omnibus Survey• Two rounds of face-to-face data collection

Jul/Aug 2008 and Sep/Oct 2008• Follow-up surveys after 6 months

Omnibus respondents who agreed to follow-up and web access

Random allocation = 400 f2f, 400 tel, 400 web• Cognitive interviews after 6 months

Purposively selected sample of 36 respondents from follow-up surveys

Page 16: Mixed Modes and Measurement Error

British Household Panel Study• BHPS Wave 18 (all f2f)

Sep 2008 – Dec 2008• Follow-up surveys after 6 months

BHPS respondents who agreed to follow-up and have web access

Random allocation = 400 tel, 400 webNo separate f2f data collection at this stage

• BHPS Wave 19 (all f2f)Sep 2009 – Dec 2009

Page 17: Mixed Modes and Measurement Error

20 BHPSquestions

20 BHPSquestions

20 BHPSquestions

Another 40questions

20 BHPSquestions

20 BHPSquestions

20 BHPSquestions

20 BHPSquestions

20 BHPSquestions

Another 40questions

Another 40questions

Another 40questions

Another 40questions

Other modules

Other BHPS

questions

Other BHPS questions

NatCen Omnibus

BHPS W18

BHPS W19 (12 months after W18)

F2F interview after 6 months

Tel interview after 6 months

Web q’naire after 6 months

Tel interview after 6 months

Web q’naire after 6 months

Cognitive interviews after 6 months

Page 18: Mixed Modes and Measurement Error

Key Features of Design• Repeated measures

enables estimation of mode effects in measures of change

• Random allocation to modes • Compare ‘seasoned’ panel members with ‘fresh’

survey sample members• Cost-efficient design to collect very rich

experimental data

Page 19: Mixed Modes and Measurement Error

Limitations• Restricted to respondents with web access

primarily relying on randomisation within the sample as our basis for inference

relatively broad basis for extrapolation to general population compared to other mixed mode studies

• BHPS f2f follow-up is 12 months later rather than 6 monthsovercome by comparing data from Omnibus and its f2f

follow-up after 6 months with data from BHPS and its f2f follow-up after 12 months

Page 20: Mixed Modes and Measurement Error

Literature Review

Page 21: Mixed Modes and Measurement Error

Aims of literature review

• A review of the evidence of differences in measurement due to mode for:Different types of questionsDifferent combinations of modes

• Identify gaps (mode pairs and question types) in the evidence base

• Formulate hypotheses to address these gaps

Page 22: Mixed Modes and Measurement Error

The literature review• Initially over 700 papers identified• Currently screening papers for relevance and

summarising relevant papers• Criteria for inclusion:

comparison of 2 or more modesmodes of survey data collectionmeasurement error

Page 23: Mixed Modes and Measurement Error

Classify existing evidence:• The question

question type (e.g. attitude, behaviour, other factual) question format (e.g closed/open, scale, # categories) task difficulty sensitivity of question

• The mode comparison interviewer presence (face-to-face, telephone, none) delivery of question (visual, aural) response list (visual, aural) recording of responses (oral, written)

Continued on next slide

Page 24: Mixed Modes and Measurement Error

Classify existing evidence:• The results

Hypotheses tested Indicators and statistical methods Results

Page 25: Mixed Modes and Measurement Error

Synthesis of literature• Causes of differential measurement error

e.g. interviewer presence, cognitive task• Nature of differential measurement error

e.g. social desirability bias, survey satisficing• Magnitude of differential measurement error

Page 26: Mixed Modes and Measurement Error

Causes of Mode Effects on Measurement (Roberts, Jäckle & Lynn; 2006)

Comprehension Retrieval Judgement Response

Sufficient Effort?

Willingness to disclose?

Task difficulty R motivation R ability* *

Stimulus: Cognitive task

Interviewer presence:Pace, non-verbal communication, multitasking

Interviewer presence: Anonymity vs. rapport

Privacy/legitimacy

Shortcuttin

g

Social desirability

bias

Page 27: Mixed Modes and Measurement Error

Initial observations from lit review:• Many experiments are not theory-driven

Focus is on descriptive comparisons of response distributions across modes

Lack of generalisable inferences about causal mechanisms

• Many papers provide insufficient information about the questions and modes being testedquestion type & format, sensitivity of question, task

difficulty interviewer presence, delivery of question and

response options, recording of responses

Page 28: Mixed Modes and Measurement Error

Next steps

Page 29: Mixed Modes and Measurement Error

Next steps:

• Complete the literature review and develop theoretical framework

• Identify gaps in the evidence base• Design extra 40 questions for mixed modes

experiment

Page 30: Mixed Modes and Measurement Error

RC-33 Conference in Naples, 1-5 Sept 2008• Research team is chairing a session on Mixed

Modes and Measurement Error• Steven Hope presenting a paper on preliminary

results of literature review

Page 31: Mixed Modes and Measurement Error