What is meant by mode effect on measurement?

A research study to identify causes of mode effects

Gerry Nicolaas

Background• Increasing use of mixed modes

Falling response ratesRising costs of data collection

• Risk of reduced data comparabilityCoverage errorNon-response errorMeasurement error

• Need for practical advice to inform decisions about when to mix modes and how

Mixed Modes and Measurement Error• Funded under the ESRC Survey Design and

Measurement Initiative• 3-year contract starting 1 Oct 2007 • Collaboration between NatCen, ISER and

independent survey methods consultant

Research Team• National Centre for Social Research (NatCen)

Gerry NicolaasSteven HopeDavid Hussey

• Institute for Social & Economic Research (ISER)Peter LynnAnnette JäckleAlita NandiNayantara Dutt

• Independent Survey Methods ConsultantPam Campanelli

Main Objective

• Practical advice on how to improve portability of questions across modesWhich mode combinations are likely to produce

comparable responses?Which types of questions are more susceptible

to mode effects?

Research Design• A literature review & framework of mixed modes

develop a conceptual framework identify gaps in evidence base and formulate

hypotheses to address gaps• Quantitative data analysis

test hypotheses using existing datasets and new experimental data

• Cognitive interviewingexplore how respondents process questions in

different modes

Mode is defined by:

• Interviewer presence (face-to-face, phone, none)• Oral and/or visual transmission of information

Question deliveryDelivery of response optionsRecording of response

Face-to-Face Interview

ACASIWritten (CAI)Visual (CAI)Aural (CAI)

ACASIWritten (CAI)Aural (CAI)Aural (CAI)

SAQ in-int’wWritten (paper)

Visual (paper)Visual (paper)

CASIWritten (CAI)Visual (CAI)Visual (CAI)

FTF (card)OralVisualAural

FTF (no card)OralAuralAural

DescriptionResponseResponse Options

Question

Telephone interview

IVROral (CAI)Aural (rec)Aural (rec)

Phone with showcards

OralVisualAural

TDEWritten (CAI)AuralAural

TelephoneOralAuralAural

DescriptionResponseResponse Options

Question

Self-completion

A-WebWritten (CAI)Aur & Vis (CAI)

Aur & Vis (CAI)

Web / emailWritten (CAI)Visual (CAI)Visual (CAI)

SAQ (e.g. mail)

Written (paper)

Visual (paper)Visual (paper)

DescriptionResponseResponse Options

Question

Causality

• How, when and why to mix/choose modes• Need to better understand causal mechanisms• Development and testing of behavioural theory

Comprehension Retrieval Judgement Response

Depth of cognitive processing: Sufficient Effort?

Social norms: Willingness to disclose?

Cognitive demands

Aural/visualInterviewer presence

Interviewer presence Privacy/legitimacyI-R interaction

Context information: Influence on processing?

Time pressureR distractionR motivation

Add. explanationsI characteristics

Sequential/Simultaneous

Qs

Control Qaire

Social desirability

bias

Satisficing

Hypotheses• Short versus long response lists

Effects of interviewer presence on satisficing • Agree-Disagree scales

Is acquiescence caused by satisficing, cognitive ability, social desirability?• Ranking versus Rating

Effects of interviewer presence • Fully-labelled versus End-labelled scales

Effects of visual stimulus and interviewer presence• Showcards versus No Showcards (face-to-face interview)

Effects of visual stimulus• “Branched” versus “Non-branched” questions

Effects of item design (no mode differences expected)• “Yes/No” versus “Code all that apply”

Effects of item design (no mode differences expected)

Question selection

Questions were designed to vary by:• Task difficulty• Sensitivity• Question type

SatisfactionOther attitudinalBehaviouralOther factual

Mixed Modes experiment

• Follow-up surveys to NatCen Omnibus (& BHPS)Face-to-face, telephone and web comparisonsExperimental design with random allocation

• LimitationRestricted to respondents with web access

Acquiescence (1)• Hypothesis

Acquiescence is a form of satisficingMore satisficing in web than f2f & tel

(no interviewer to motivate, explain, probe, etc)• 12 Agree/Disagree questions

5-point agree/disagree scaleUse of opposite statements

Acquiescence (2)• Initial results from the experiment

More acquiescence in f2f & tel compared to web• Results from the cognitive interviews

Only 2 out of 23 cases of agreeing to opposite statements due to acquiescence

Justifiable explanations given for other 21 cases

Acquiescence (3) • Example of justifiable agreement with opposite

statementsN36: Compared to other neighbourhoods, this

neighbourhood has more properties that are in a poor statement of repair.

N38: Compared to other neighbourhoods, this neighbourhood has more properties that are well kept.

Respondent: In this village, … it’s like half and half…”

Acquiescence (4)

• Interim conclusionsUse of opposite statements to detect acquiescence

bias brought into questionWhy higher rate of acquiescence in f2f & tel

compared to web?

Other forms of satisficing (1)• Hypotheses:

More satisficing in web than f2f & tel (no interviewer to motivate, explain, probe, etc)

More satisficing in tel than f2f (lack of physical presence of interviewer, lack of non-verbal communication, distractions, etc)

• Indicators of satisficingPrimacy effects in visual modes, recency effects

in aural modes, middle category effects, item non-response

Other forms of satisficing (2)• Initial results from the experiment

Primacy & recency effects: inconsistent patterns Item non-response: no mode differences Middle category effects: web respondents more

likely to select middle categories than f2f & tel resps (also for agree/disagree scales – see acquiescence results)

• Results from cognitive interviewsResults suggest more satisficing in web & tel than

in f2f interview

Mode effect or question design effect?

• Example:“Code all that apply” frequently used in f2f

interviews and self-completion surveysAlternative format for tel interviews tends to be a

series of Y/N questions • Hypothesis

No mode effecti.e. No differences if series of Y/N questions used

across all modes

Mode effect or question design effect?

• Initial results from the experiment F2F & tel resps more likely than web resps to say

“Yes” in series of Y/N questions Similar mode effect not found for f2f and web

respondents in “code all that apply” format (nb this format not used in tel mode)

• Results from cognitive interviews Questions raised about validity of “Yes” answers

in Y/N series that may differ by mode

• Continue with analysis of experimental data• Papers

Causes of mode effects on survey measurementThe role of the interviewer in producing mode effectsThe role of visual/aural stimuli in producing mode effectsThe role of question format in producing ‘mode’ effectsUsing cognitive interviews to explore mode effects

• Training courseQuestionnaire design for mixed mode surveys

What next?

For more details on this project, contact:

Gerry.Nicolaas@natcen.ac.uk