Interviewers, nonresponse bias and measurement error

25
NCRM is funded by the Economic and Social Research Council NCRM is funded by the Economic and Social Research Council 1 Interviewers, nonresponse bias and measurement error Patrick Sturgis University of Southampton Research Methods Festival, Oxford, 2-5 July 2012

description

Interviewers, nonresponse bias and measurement error. Patrick Sturgis University of Southampton. Research Methods Festival, Oxford, 2-5 July 2012. Co-authors. Ian Brunton -Smith (University of Surrey) Joel Williams (TNS-BMRB ). Background and Motivation. Common Causes of Survey Error. - PowerPoint PPT Presentation

Transcript of Interviewers, nonresponse bias and measurement error

Page 1: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council1

Interviewers, nonresponse bias and measurement

errorPatrick Sturgis

University of Southampton

Research Methods Festival, Oxford, 2-5 July 2012

Page 2: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Co-authors

• Ian Brunton-Smith (University of Surrey)

• Joel Williams (TNS-BMRB)

Page 3: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Background and Motivation

Page 4: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Common Causes of Survey Error

• Recent attention has focused on common causes of nonresponse and measurement error (cf. 2010 special issue of POQ on Total Survey Error)

• Agencies often target field resources at persuading reluctant respondents to meet response rate targets

• These respondents are less motivated and potentially less able to complete questionnaire accurately

4

Page 5: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Error trade-offs• So potential reduction in nonresponse bias my

be offset by increase in measurement error

• Growing evidence that this does happen in practice (Kreuter et al 2010; Sakshaug et al 2010)

• This work has focused on respondents so far

• What about interviewers?

5

Page 6: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Interviewers as common cause?• Interviewers can cause nonresponse bias and

measurement error• Success in obtaining contact and cooperation related to

interviewer characteristics:– Tailoring– Maintaining an interaction– Personality– Attitudes and beliefs

• Some interviewers don’t get interviews, where ‘better’ interviewers would

• If these ‘lost’ respondents are different on survey variables, result is biased population estimates

6

Page 7: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Interviewer variance

• Interviewers cause measurement error through the way they administer the questionnaire

• At the individual level, these can be considered biases (response different to true value)

• But across respondents and interviewers the result is larger variances

Page 8: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Interviewer Variance • E.g. interviewer always reads same question

incorrectly

• Across interviews, this creates within-interviewer correlation – same as geographical clustering

• Interviewer contribution to variance of estimator denoted ρInt

8

Page 9: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

How is ρInt related to nonresponse?

• Anecdotal evidence that more successful interviewers on the doorstep are less diligent at sticking to questionnaire wording and instructions

• Alternatively, some interviewers are good at what they do, others are not so good

• Either way, we should anticipate a correlation between response rate and ρInt?

Page 10: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Conceptual model 1

• Implications for total survey error• MSEint = bias2 + varianceint

Agreeable-ness

Response rate

Deviation from questionnaire wording and instructions

+

ρInterviewer

Tailoring

++

Nonresponse bias

VarianceInt+

-+

-

10

Page 11: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Conceptual model 2

• Implications for total survey error• MSEint = bias2 + varianceint

Conscientiou-sness

Response rate

Deviation from questionnaire wording and instructions

+

ρInterviewer

Tailoring

++

Nonresponse bias

VarianceInt

+

-+

+

11

Page 12: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Analytical approach

• Fit cross-classified multilevel models to face to face interview data

• Partition ρ into area and interviewer components

• Examine variation in ρInt across distribution of measures of interviewer success in obtaining contact and cooperation seperately

12

Page 13: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Measuring interviewer success

• Average response rate problematic as indicator of interviewer success on the doorstep

• Our measure of interviewer success– Calculate ‘expected’ response propensity for all original issue cases

based on geodemographic characteristics and paradata– Take mean of ratio of expected to observed rate across all cases for

each interviewer– Do this separately for contact and cooperation– Group into ‘success quantiles’

13

Page 14: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Modelling strategy

• Adjusts estimates for clustering of interviews within sample points• Models also include individual, interviewer and area controls to account for

non-random allocation of respondents to interviewers

Cross-classified multilevel models with a complex interviewer error structure

Allows simultaneous estimation of separate ρInt for each interviewer success quintile

Page 15: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Data and analysis

• British Crime Survey (2005/06)– 43,465 respondents , 472 interviewers, 3,782 areas

• 36 items asked of all respondents which were non-factual and included probes and/or show-cards

• Cross-classified multilevel model with complex error term at interviewer level

• Controls– Individual - Gender, age, ethnicity, education– Interviewer - Gender, age, ethnicity, experience level (months worked)– Area - Socio-economic disadvantage, urbanisation, ethnic diversity, housing

structure, age profile, population turnover

15

Page 16: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council16

Table 1. Coefficient estimates for 'Overall rating of health' (higher score = less healthy)

Estimate (S.E)

FIXED EFFECTS

Constant 2.08 0.01 **Respondent level Male 0.04 0.01 **

Age 0.27 0.005 **Nonwhite 0.00 0.02Education (contrast: No qualifications ): GCSE -0.20 0.01 **

A level -0.26 0.01 **Degree -0.36 0.01 **Non-traditional/ foreign qualification -0.16 0.02 **

Neighbourhood level Socio-economic disadvantage 0.10 0.005 **Urbanicity 0.03 0.01Population mobility 0.02 0.01 **Age profile 0.01 0.005 *Housing structure 0.02 0.01 **Ethnic diversity 0.05 0.05

Interviewer level Male 0.04 0.02 *Age -0.01 0.01Nonwhite -0.06 0.05Experience (months working) 0.00 0.01

RANDOM EFFECTS Contact Cooperation

Top success quintile 0.014 0.0150.020 0.0200.017 0.0180.023 0.017

Bottom success quintile 0.033 0.035Area 0.003 0.003Individual 0.690 0.690** P<(.01)* P<(.05)

Results I: Example Model

Page 17: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Results II: Overall estimates

• For contact measure bottom quintile has largest variance on 25/36 items (15 at p<0.05)

• For cooperation measure bottom quintile has largest variance on 20/36 items (13 at p<0.05)

• For contact, bottom group had 74% higher variance across all items compared to top group

• For cooperation, bottom group had 34% higher variance across all items compared to top group

17

Page 18: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Mean Interviewer variance components across 36 items by contact and cooperation success quintiles

Least successful

quintile

Most successful

quintile Contact 0.059 0.048 0.048 0.041 0.034 Cooperation 0.059 0.041 0.039 0.046 0.044

Page 19: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

ITEMS WITH A DOWNWARD TREND ASSOCIATION BETWEEN

CONTACT SUCCESS AND INTERVIEWER VARIANCE

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

1 2 3 4 5

contact success quintile groups

inte

rvie

we

r v

ari

an

ce

co

mp

on

en

t

Page 20: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

ITEMS WITH A U-SHAPED ASSOCIATION BETWEEN COOPERATION SUCCESS AND INTERVIEWER VARIANCE

0

0.02

0.04

0.06

0.08

0.1

0.12

1 2 3 4 5

cooperation success quintile groups

inte

rvie

we

r v

ari

an

ce

co

mp

on

en

t

Page 21: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

• Are these differences really due to interviewers deviating from the script?

• Could also arise due to differential nonresponse bias

• We find no differences across quintile groups on a range of background variables

• If due to nonresponse bias, should observe uniform

gradients over different question types that vary the degree of interviewer involvement

An alternative explanation?

21

Page 22: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Results III: by question type

22

Least successful

quintile

Most successful

quintile Contact No interviewer effort items (n=7) 0.007 0.007 0.005 0.007 0.005 interviewer effort items (n=36) 0.059 0.048 0.048 0.041 0.034 Cooperation No interviewer effort items (n=7) 0.005 0.007 0.005 0.006 0.008 interviewer effort items (n=36) 0.059 0.041 0.039 0.047 0.045

Page 23: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Discussion

• Historical measure of contact and cooperation success negatively correlated with interviewer variance

• Different pattern for contact and cooperation – contact broadly linear, cooperation evidence of u-shaped distribution

• Pattern of findings across question types suggest effect is due to interviewer behaviour in questionnaire administration

• Relatively small group of ‘poor’ interviewers make disproportionate contribution to total survey error

• Suggests response rate may be used as indicator of potential problems with interviewer behaviour/ training

23

Page 24: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Some extensions

Page 25: Interviewers, nonresponse bias and measurement error

NCRM is funded by the Economic and Social Research CouncilNCRM is funded by the Economic and Social Research Council

Conceptual model

Agreeableness

Response rate

Deviation from questionnaire wording and instructions

+

ρInterviewer

Tailoring

++

Nonresponse bias

VarianceInt+

-+

-