-Being Successful Meant Speaking Well, And Speaking Well Meant Speaking Standard English-. Aproxima
What is meant by mode effect on measurement? A research study to identify causes of mode effects
description
Transcript of What is meant by mode effect on measurement? A research study to identify causes of mode effects
![Page 1: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/1.jpg)
What is meant by mode effect on measurement?
A research study to identify causes of mode effects
Gerry Nicolaas
![Page 2: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/2.jpg)
Background• Increasing use of mixed modes
Falling response ratesRising costs of data collection
• Risk of reduced data comparabilityCoverage errorNon-response errorMeasurement error
• Need for practical advice to inform decisions about when to mix modes and how
![Page 3: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/3.jpg)
Mixed Modes and Measurement Error• Funded under the ESRC Survey Design and
Measurement Initiative• 3-year contract starting 1 Oct 2007 • Collaboration between NatCen, ISER and
independent survey methods consultant
![Page 4: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/4.jpg)
Research Team• National Centre for Social Research (NatCen)
Gerry NicolaasSteven HopeDavid Hussey
• Institute for Social & Economic Research (ISER)Peter LynnAnnette JäckleAlita NandiNayantara Dutt
• Independent Survey Methods ConsultantPam Campanelli
![Page 5: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/5.jpg)
Main Objective
• Practical advice on how to improve portability of questions across modesWhich mode combinations are likely to produce
comparable responses?Which types of questions are more susceptible
to mode effects?
![Page 6: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/6.jpg)
Research Design• A literature review & framework of mixed modes
develop a conceptual framework identify gaps in evidence base and formulate
hypotheses to address gaps• Quantitative data analysis
test hypotheses using existing datasets and new experimental data
• Cognitive interviewingexplore how respondents process questions in
different modes
![Page 7: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/7.jpg)
Mode is defined by:
• Interviewer presence (face-to-face, phone, none)• Oral and/or visual transmission of information
Question deliveryDelivery of response optionsRecording of response
![Page 8: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/8.jpg)
Face-to-Face Interview
ACASIWritten (CAI)Visual (CAI)Aural (CAI)
ACASIWritten (CAI)Aural (CAI)Aural (CAI)
SAQ in-int’wWritten (paper)
Visual (paper)Visual (paper)
CASIWritten (CAI)Visual (CAI)Visual (CAI)
FTF (card)OralVisualAural
FTF (no card)OralAuralAural
DescriptionResponseResponse Options
Question
![Page 9: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/9.jpg)
Telephone interview
IVROral (CAI)Aural (rec)Aural (rec)
Phone with showcards
OralVisualAural
TDEWritten (CAI)AuralAural
TelephoneOralAuralAural
DescriptionResponseResponse Options
Question
![Page 10: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/10.jpg)
Self-completion
A-WebWritten (CAI)Aur & Vis (CAI)
Aur & Vis (CAI)
Web / emailWritten (CAI)Visual (CAI)Visual (CAI)
SAQ (e.g. mail)
Written (paper)
Visual (paper)Visual (paper)
DescriptionResponseResponse Options
Question
![Page 11: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/11.jpg)
Causality
• How, when and why to mix/choose modes• Need to better understand causal mechanisms• Development and testing of behavioural theory
![Page 12: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/12.jpg)
Comprehension Retrieval Judgement Response
Depth of cognitive processing: Sufficient Effort?
Social norms: Willingness to disclose?
Cognitive demands
Aural/visualInterviewer presence
Interviewer presence Privacy/legitimacyI-R interaction
Context information: Influence on processing?
Time pressureR distractionR motivation
Add. explanationsI characteristics
Sequential/Simultaneous
Qs
Control Qaire
Social desirability
bias
Satisficing
![Page 13: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/13.jpg)
Hypotheses• Short versus long response lists
Effects of interviewer presence on satisficing • Agree-Disagree scales
Is acquiescence caused by satisficing, cognitive ability, social desirability?• Ranking versus Rating
Effects of interviewer presence • Fully-labelled versus End-labelled scales
Effects of visual stimulus and interviewer presence• Showcards versus No Showcards (face-to-face interview)
Effects of visual stimulus• “Branched” versus “Non-branched” questions
Effects of item design (no mode differences expected)• “Yes/No” versus “Code all that apply”
Effects of item design (no mode differences expected)
![Page 14: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/14.jpg)
Question selection
Questions were designed to vary by:• Task difficulty• Sensitivity• Question type
SatisfactionOther attitudinalBehaviouralOther factual
![Page 15: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/15.jpg)
Mixed Modes experiment
• Follow-up surveys to NatCen Omnibus (& BHPS)Face-to-face, telephone and web comparisonsExperimental design with random allocation
• LimitationRestricted to respondents with web access
![Page 16: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/16.jpg)
Acquiescence (1)• Hypothesis
Acquiescence is a form of satisficingMore satisficing in web than f2f & tel
(no interviewer to motivate, explain, probe, etc)• 12 Agree/Disagree questions
5-point agree/disagree scaleUse of opposite statements
![Page 17: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/17.jpg)
Acquiescence (2)• Initial results from the experiment
More acquiescence in f2f & tel compared to web• Results from the cognitive interviews
Only 2 out of 23 cases of agreeing to opposite statements due to acquiescence
Justifiable explanations given for other 21 cases
![Page 18: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/18.jpg)
Acquiescence (3) • Example of justifiable agreement with opposite
statementsN36: Compared to other neighbourhoods, this
neighbourhood has more properties that are in a poor statement of repair.
N38: Compared to other neighbourhoods, this neighbourhood has more properties that are well kept.
Respondent: In this village, … it’s like half and half…”
![Page 19: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/19.jpg)
Acquiescence (4)
• Interim conclusionsUse of opposite statements to detect acquiescence
bias brought into questionWhy higher rate of acquiescence in f2f & tel
compared to web?
![Page 20: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/20.jpg)
Other forms of satisficing (1)• Hypotheses:
More satisficing in web than f2f & tel (no interviewer to motivate, explain, probe, etc)
More satisficing in tel than f2f (lack of physical presence of interviewer, lack of non-verbal communication, distractions, etc)
• Indicators of satisficingPrimacy effects in visual modes, recency effects
in aural modes, middle category effects, item non-response
![Page 21: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/21.jpg)
Other forms of satisficing (2)• Initial results from the experiment
Primacy & recency effects: inconsistent patterns Item non-response: no mode differences Middle category effects: web respondents more
likely to select middle categories than f2f & tel resps (also for agree/disagree scales – see acquiescence results)
• Results from cognitive interviewsResults suggest more satisficing in web & tel than
in f2f interview
![Page 22: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/22.jpg)
Mode effect or question design effect?
• Example:“Code all that apply” frequently used in f2f
interviews and self-completion surveysAlternative format for tel interviews tends to be a
series of Y/N questions • Hypothesis
No mode effecti.e. No differences if series of Y/N questions used
across all modes
![Page 23: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/23.jpg)
Mode effect or question design effect?
• Initial results from the experiment F2F & tel resps more likely than web resps to say
“Yes” in series of Y/N questions Similar mode effect not found for f2f and web
respondents in “code all that apply” format (nb this format not used in tel mode)
• Results from cognitive interviews Questions raised about validity of “Yes” answers
in Y/N series that may differ by mode
![Page 24: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/24.jpg)
• Continue with analysis of experimental data• Papers
Causes of mode effects on survey measurementThe role of the interviewer in producing mode effectsThe role of visual/aural stimuli in producing mode effectsThe role of question format in producing ‘mode’ effectsUsing cognitive interviews to explore mode effects
• Training courseQuestionnaire design for mixed mode surveys
What next?
![Page 26: What is meant by mode effect on measurement? A research study to identify causes of mode effects](https://reader036.fdocuments.us/reader036/viewer/2022062520/568159e3550346895dc72cc7/html5/thumbnails/26.jpg)