Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014...

1Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs

Nonresponse Bias and Measurement Error in Web surveys

Presented at TSE 2015Baltimore, MD, USASeptember 21st, 2015

Web survey response rates are generally lower than response rates of other self-

administered surveys such as mail surveys (Couper, 2001).

Web survey researchers have established various strategies to reduce nonresponse

rates: appealing invitation e-mails, e-mail reminders, incentives etc. (Fan & Yan, 2010).

Substantial nonresponse has the potential to induce nonresponse bias.

However, incorporating nonrespondents into Web surveys does not necessarily reduce

nonresponse bias and might cause additional measurement error.

Less motivated respondents might provide less accurate responses.

Background


Research Questions


Do nonrespondents have an impact on data quality?

Do initial nonrespondents induce nonresponse bias?

Do survey break-offs induce nonresponse bias?

Should survey break-offs be incorporated into Web surveys?

Does convincing survey break-offs to participate in Web surveys cause additional measurement

error?

Data BasisStudies of University Applicants


Study 1 Study 2 Study 3 Study 4

Target population: University applicants

Field phase:July/August

2012July/August

2013July/August

2014July/August

2015

Completes*: n=5,977 n=7,395 n=5,996 n=4,034

AAPOR RR6: 32% 40% 35% 24%

No. of Questions: 43 41 47 44

No. of Pages: 38 35 39 36

* Respondents who answered 50 percent or more of all applicable survey questions

3240 34

23

6473 71 67

1814

14

11

3627 29 33

51 46 5266

0

10

20

30

40

50

60

70

80

90

100

Study 1(2012)

Study 2(2013)

Study 3(2014)

Study 4(2015)

Study 1(2012)

Study 2(2013)

Study 3(2014)

Study 4(2015)

in p

erce

nt

(%)

Completes* Questionnaire Breakoff Initial Nonresponse

Gross Sample Net Sample

Response & Nonresponse RatesStudy 1-4


* Respondents were excluded who completed the questionnaire but answered less than 50 percent of all applicable survey questions. Respondents

who answered 50 percent or more of all applicable survey questions but did not complete the questionnaire were counted as survey break-offs.

Device used to open the surveyStudy 1-4


84%

75%

69%

63%

16%

25%

31%

37%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

2012 2013 2014 2015

Desktop Mobile

Survey Breakoff RatesStudy 1-4


36%

27%29%

33%34%

24%26%

30%

45%

37% 36%38%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

2012 2013 2014 2015

percen

tag

eo

fth

en

et

sam

ple

Breakoff (total) Breakoff (desktop) Breakoff (mobile)

64 3 2

0

5

10

15

20

25

30

35

40

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35% o

fth

en

et

sam

ple

last page

Study 2 (2013)

%

cum %

75

2

8

0

5

10

15

20

25

30

35

40

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

% o

fth

en

et

sam

ple

last page

Study 1 (2012)

%

cum %

Breakoff by Page NumberStudy 1 & 2


open-e

nded

sin

gle

response

sin

gle

response

sin

gle

response

welc

om

epage

list-

sty

le o

pen

-ended

grid

welc

om

epage

Breakoff by Page NumberStudy 3 & 4


84

25

2

0

5

10

15

20

25

30

35

40

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36% o

fth

en

et

sam

ple

last page

Study 4 (2015)

%

cum %

64

6

0

5

10

15

20

25

30

35

40

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39% o

fth

en

et

sam

ple

last page

Study 3 (2014)

%

cum %

welc

om

epage

we

lco

me

page

sin

gle

response

sin

gle

response

grid

multip

le r

esponse

grid g

rid

Standarad ErrorStudy 1-4: grid question


Item 1

Item 2

Item3

Item 4

Item 5

Item 6

Item 7

Item 8

Item 9

Item 10

Item11

2012 (15/38)

SEMEAN (c) .006 .016 .010 .008 ,016 .013 .015 .012

SEMEAN (c+b) .006 .015 .010 .008 .015 .012 .014 .012

2013 (16/35)

SEMEAN (c) .010 .011 .011 .013 .011 .010 .009 .010

SEMEAN (c+b) .009 .011 .010 .013 .010 .009 .009 .010

2014 (12/39)

SEMEAN (c) .011 .013 .012 .015 .011 .011 .010 .012

SEMEAN (c+b) .010 .012 .011 .014 .011 .010 .010 .011

2015 (6/36)

SEMEAN (c) .027 .017 .030 .042 .040 .017 .029 .014 .039 .029 .024

SEMEAN (c+b) .025 .016 .028 .039 .037 .016 .027 .013 .036 .027 .023

Relative Nonresponse Bias


(Groves, 2006)

Relative Nonresponse BiasSociodemographic Variables


-0,045

-0,011

0,005

0,060

-0,046

-0,011

0,006

0,064

-0,015

-0,0030,000

0,015

Gender(male)

Area of Studies(HSS)

Semester(1st)

Region(ZIP with 6)

2012

-0,033

-0,016

0,0130,017

-0,018-0,012

0,015 0,017

-0,002-0,011

0,001 0,004

Gender(male)


Semester(1st)

Region(ZIP with 6)

2013

-0,031 -0,032

0,010

0,033

-0,032

-0,045

0,011

0,034

-0,008 -0,014

0,001 0,008

Gender(male)


Semester(1st)

Region(ZIP with 6)

2014

-0,002

-0,054

0,0050,014

-0,001

-0,053

0,006 0,014

-0,004-0,020

0,0000,006

Gender(male)


Semester(1st)

Region(ZIP with 6)

2015

Nonresponse Bias


(Groves, 2006)

Nonresponse BiasSociodemographic Variables


-0,07

-0,02

0,01

0,09

-0,07

-0,02

0,01

0,10

-0,04

-0,010,00

0,04

Gender(male)


Semester(1st)

Region(ZIP with 6)

2012

-0,05

-0,03

0,02 0,03

-0,03

-0,02

0,03 0,03

-0,01

-0,04

0,00 0,02

Gender(male)


Semester(1st)

Region(ZIP with 6)

2013

-0,05 -0,05

0,02

0,05

-0,03

-0,07

0,02

0,06

-0,05

-0,05

0,000,03

Gender(male)


Semester(1st)

Region(ZIP with 6)

2014

0,00

-0,07

0,010,02

-0,01

-0,07

0,01 0,02

0,00

-0,06

0,000,02

Gender(male)


Semester(1st)

Region(ZIP with 6)

2015

SummaryResults: NRB - Sociodemographic Variables


Bias estimates for all four variables were at a very low level.

The initial nonresponse bias and the survey breakoff bias add to each other.

In all studies the relative nonresponse bias for all variables is caused by the initial

nonresponse bias to a greater extent and this is not due to the high percentage of initial

nonrespondents.

Therefore, it seems to be more important to reduce initial nonresponse.

However, it is easier and more cost-effective to convince survey breakoffs to participate

in Web surveys by optimizing the questionnaire design.

Does incorporating survey breakoffs cause additional measurement error?

Measurement ErrorIndicators

Degree of differentiation in grid questions(Kaminska et al., 2010; Barge & Gehlbach, 2011)

Number of item missing in grid questions(Barge & Gehlbach, 2011)

Response time of grid questions(Barge & Gehlbach, 2011)

Number of characters of answers to narrative open-ended questions(Smyth et al. 2009)


Measurement Error(Degree of differentiation)


0,570,55 0,54

0,37

0,57 0,57 0,56

0,45

2012 2013 2014 2015

Breakoff

Completes

Significance n.s. p<.001 p<.01 p<.05

Question type grid grid grid grid

Question position 15/38 16/35 12/39 6/36

Measurement Error(# of item missing)


0,06

0,04

0,06

0,19

0,00 0,00 0,00

0,14

2012 2013 2014 2015

Breakoff

Completes

Significance p<.001 p<.001 p<.001 n.s.



Measurement Error(Response time in seconds)


4341

50

66

4649

51

68

2012 2013 2014 2015

Breakoff

Completes

Significance p<.01 p<.001 n.s. n.s.



Measurement Error(# of characters)


128

64 6557

145

107

81

69

2012 2013 2014 2015

Breakoff

Completes

Significance p<.01 p<.001 p<.05 p<.05

Question type open-ended open-ended open-ended open-ended


Measurement Error(Degree of differentiation)



Beta Sig Beta Sig. Beta Sig. Beta Sig.

Semester(semester 2+)

Region(area code ≠ 6)

Area of studies(engigneering/science)

Gender(female)

Device(mobile)

-.006

.004

-.054

.054

.007

.775

.359

.000

.000

.198

.024

-.004

-.014

.019

.028

.106

.220

.000

.000

.000

.005

-.002

-.022

.025

.021

.776

.667

.000

.000

.000

--

.057

.049

-.039

.032

--

.062

.092

.178

.271

Breakoff(complete)

.007 .347 -.018 .003 -.019 .003 -.080 .017

Constant

R²

.566

.068

.000 .526

.019

.000 .540

.022

.000 .399

.031

.000



Note. Reference categories in brackets. Linear regression model.

Measurement Error(# of item missing)







Gender(female)

Device(mobile)

.00

-.00

-.00

-.01

-.00

.786

.711

.164

.096

.659

.00

.00

-.00

.00

.00

.717

.271

.563

.888

.924

.01

-.00

-.00

.00

-.00

.594

.598

.444

.821

.754

.16

.03

-.03

-.02

-.01

.574

.132

.123

.297

.586

Breakoff(complete)

.06 .000 .04 .000 .06 .000 .05 .049

Constant

R²

.00

.021

.831 -.00

.032

.690 -.00

.043

.758 -.00

.001

.996




Measurement Error(Response Time)







Gender(female)

Device(mobile)

-.04

.66

-3.96

1.18

-10.99

.992

.405

.000

.144

.000

2.42

1.71

-6.21

.00

-.04

.638

.109

.000

.999

.975

-5.79

1.91

-4.57

1.37

-2.90

.198

.031

.000

.126

.003

12.24

7.91

-3.60

-2.71

-8.83

.660

.000

.015

.072

.000

Breakoff(complete)

-3.88 .007 -8.73 .000 -.58 .690 -2.35 .233

Constant

R²

57.46

.019

.000 49.40

.006

.000 58.76

.007

.000 60.11

.016

.031




Measurement Error(# of characters)







Gender(female)

Device(mobile)

22.4

3.2

3.5

-12.0

5.3

.098

.224

.185

.000

.147

-14.3

-11.0

7.6

-23.6

20.6

.349

.000

.009

.000

.000

-14.8

2.3

-4.2

9.8

13.8

.341

.503

.217

.005

.000

--

.8

-3.6

.9

12.2

--

.790

.211

.759

.000

Breakoff(complete)

-25.0 .000 -39.6 .000 -14.0 .030 -9.5 .072

Constant

R²

121.2

.015

.000 124.4

.050

.000 80.4

.019

.000 61.9

.018

.000

Question type open-ended open-ended open-ended open-ended



SummaryResults: Measurement Error


Breakoffs differentiate significantly less than completes.

Breakoffs show significant higher item missing rates than completes.

Breakoffs take significantly less time to answer questions (2 of 4).

Breakoffs report shorter answers with a significantly lower number of characters than

completes.

Controlling the differences of the sample composition due to nonresponse these results

stay significant.

Therefore, differences can be interpreted as measurement error.

Incorporating breakoffs will increase the measurement error.

Conclusion


Convincing survey break-offs to participate in Web surveys would increase response

rates, reduce the standard error of the mean and nonresponse bias to a small extent.

However, incorporating break-offs causes additional measurement error.

Further analyses need to examine if the quality of estimates becomes better with our

without including break-offs. Mean Squared Error

Convincing initial nonrespondents to participate in Web surveys would reduce

nonresponse bias to a greater extent but further research is necessary to analyze the

data quality of initial nonrespondents.

Darmstadt University of TechnologyDepartment 02Institute of SociologyResearch MethodsDipl.-Soz. Anke Metzler

Erst-Ludwig-Str. 3/ Im Carree

64283 Darmstadt

Germany

Phone: +49 6151/16-70972Fax: +49 6151/16-72070

[email protected]

Thank you.

intrest of

subject

contribute to

social changes

personal

developmenttalent help others

scientific

interest

contact with

people

problem

solving

Mean(c) 1,18 2,72 1,62 1,41 2,65 1,96 2,43 1,95

Mean(b) 1,22 2,79 1,69 1,47 2,71 2,01 2,43 2,02

Estimate(b) 1,18 2,80 1,64 1,45 2,58 1,89 2,32 1,85

Nonresponse BiasSubstantial Answers - Study 1 (2012)


NRB -0,04 -0,07 -0,07 -0,06 -0,06 -0,05 0 -0,07

Meas Bias 0 -0,08 -0,02 -0,04 0,07 0,07 0,11 0,10

grid question position 15/38:



broad basic

knowledge

foreign

language

organisation

talent

computer

literacywriting skills speaking skills

working

independent

ability to

cooperate

Mean(c) 1,84 2,37 1,96 2,82 1,96 1,76 1,67 1,83

Mean(b) 1,81 2,31 1,98 2,75 1,96 1,71 1,67 1,78

Estimate(b) 1,84 2,24 1,80 2,47 1,90 1,61 1,52 1,62

NRB 0,03 0,06 -0,02 0,07 0 0,05 0 0,05

Meas Bias 0 0,13 0,16 0,35 0,06 0,15 0,15 0,21




broad basic

knowledge

foreign

language

organisation

talent

computer

literacywriting skills speaking skills

working

independent

ability to

cooperate

Mean(c) 1,83 2,27 1,84 2,76 1,84 1,71 1,56 1,84

Mean(b) 1,85 2,21 1,98 2,81 1,89 1,73 1,60 1,86

Estimate(b) 1,92 2,24 2,02 2,75 1,97 1,88 1,62 1,98

NRB -0,02 0,06 -0,14 -0,05 -0,05 -0,02 -0,04 -0,02

Meas Bias -0,09 0,03 -0,18 0,01 -0,13 -0,17 -0,06 -0,14


cb

informationof profs

informationof students

studentcounselling

(phone/email)

studentcounselling(personal)

studentcounselling(faculty)

flyer etc. newspaper homepage TUDayinformation

events of theTUD

online self-assessment

Mean(c) 2,47 2,18 2,40 2,55 2,50 2,34 2,75 2,10 2,59 2,46 2,40

Mean(b) 2,40 2,29 2,54 2,40 2,43 2,39 2,57 2,15 2,56 2,38 2,28

Estimate(b) 2,43 2,40 2,61 2,45 2,43 2,43 2,61 2,12 2,67 2,49 2,27

Nonresponse BiasSubstantial Answers- Study 4 (2015)


NRB 0,07 -0,11 -0,14 0,15 0,07 -0,05 0,18 -0,05 0,03 0,08 0,12

Meas Bias 0,04 -0,22 -0,21 0,10 0,07 -0,09 0,14 -0,02 -0,08 -0,03 0,13


Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014...

Documents

Transcript of Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014...