Response Rates Impact Data Quality, But not How you Might Think
-
Upload
stephanie-eckman -
Category
Data & Analytics
-
view
136 -
download
1
Transcript of Response Rates Impact Data Quality, But not How you Might Think
![Page 1: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/1.jpg)
www.rti.orgRTI International is a registered trademark and a trade name of Research Triangle Institute.
Response Rates Impact Data Quality, but Not How you Might Think
Based on 2 papers:
Eckman, S and Koch, A. “The Relationship between Response Rates,
Sampling Method and Data Quality: Evidence from the European Social
Survey” Under Review
Eckman, S, Himelein, K and Dever, J. “Innovative Sample Designs Using
GIS Technology" forthcoming in Advances in Comparative Survey
Methods: Multicultural, Multinational and Multiregional Context.
Stephanie Eckman, RTI Fellow
![Page 2: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/2.jpg)
Motivation
� Relationship between RR & Data Quality
� High response rates signal data are good quality
� Response rates uncorrelated with data quality
– High RR survey no more accurate than low (Keeter et al, 2000)
– Merkle & Edelman (2002)
– Groves & Peytcheva (2008)
2
![Page 3: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/3.jpg)
RR NR bias (Merkle Edelman)
3 Merkle & Edelman 2002
![Page 4: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/4.jpg)
RRs do not Correlate with Nonresponse Bias
4 Groves & Peytcheva 2008
![Page 5: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/5.jpg)
Motivation
� Relationship between RR & Data Quality
� High response rates signal data are good quality
� Response rates uncorrelated with data quality
– High RR survey not more accurate than low (Keeter et al, 2000)
– Merkle & Edelman (2002)
– Groves & Peytcheva (2008)
� But maybe high response rates are a sign that data are crap?
5
![Page 6: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/6.jpg)
Data Quality
� Total Survey Error Framework
– Undercoverage
– Nonresponse
– Measurement error
– Editing error
– Processing error
– etc.
� Misrepresentation error
– Undercoverage + Nonresponse
� Tradeoff between undercoverage & NR
– Eckman & Kreuter 2017
6
Image: http://makeagif.com/dkjuuc
![Page 7: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/7.jpg)
European Social Survey
� 7 waves
� 30+ countries
� Central Committee sets standards
– Core questionnaire
– Minimum effective sample size
– Paradata collection
– Documentation
– Face to face attempts
– RR standard 70%
� Our data: 136 country-rounds in first 6 waves
7
![Page 8: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/8.jpg)
Sampling Methods in Analysis
8
SamplingMethod Includes
Field Staff Involvement in Selecting
nHousehold Person
Individual Register
None None70
HouseholdRegister
Household Register
Address Register
None
Interviewer
None
Interviewer
41
HouseholdWalk
Listing
Random Walk
Lister
Lister
Interviewer
Interviewer
25
![Page 9: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/9.jpg)
RRs by Sample Type
9
![Page 10: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/10.jpg)
2 Measures of Data Quality
� External measure:
– How different is ESS from Labor Force Survey?
– On 6 categorical variables: age, gender, HH size, marital status, etc.
– Index of dissimilarity measures how different 2 surveys are
– Average over 6 variables
– Assumes LFS is higher quality
� Internal measure:
– 50% of all respondents from gender heterogeneous couples should be
women
– ��,� > 1.96 indicates significant deviation from 50%
10
��,� =%female�,�−50
50 ∗ 50/�
��,�,� = 0.5 ∗�|��,�,���� − ��,�,� !� |�
![Page 11: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/11.jpg)
2 DVs, 2 IVs
� Dependent variables: misrepresentation error
– External measure
– Internal measure
� Independent variables
– RR
– Sampling method
� Joint effect of RR and sampling method on data quality
11
![Page 12: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/12.jpg)
2 Measures vs RR, by Sample Type
12
![Page 13: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/13.jpg)
Regression Models
13Estimated Regression Coefficients
![Page 14: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/14.jpg)
Implications
� High RRs might signal that you have problems with your data
– When interviewers select samples
– Interviewers seem to manipulate selection process to keep RRs high
� Note that ESS does better random walk than other surveys
– Listing should be done by someone other than interviewer
� Other problems with random walk
– Walker effects
– No probabilities of selection
14
![Page 15: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/15.jpg)
Possible Solutions
� What are some alternatives to random walk?
– Satellite Photos
– Reverse Geocoding
– Qibla Method
– Geosampling
– Listing with Drones
15
![Page 16: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/16.jpg)
GIS Resources
� Turn by turn directions on phone
� Satellite images
– Daytime images
– Small-sat revolution
– Nighttime lights
� Other remote sensing data
� How can we exploit these resources for sampling?
– And avoid random walks problems
16
![Page 17: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/17.jpg)
Satellite Photo Mogadishu
17
![Page 18: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/18.jpg)
Reverse Geocoding
� Geocoding: address→ coordinate
� Reverse geocoding: coordinate → address
– Select random points in segment
– Identify closest address
– Many online tools
– Used in Italy ISSP 2009, 2011
18
![Page 19: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/19.jpg)
Example of Reverse Geocoding
19
![Page 20: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/20.jpg)
Example of Reverse Geocoding
20
![Page 21: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/21.jpg)
Example of Reverse Geocoding
21
![Page 22: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/22.jpg)
Qibla Method
� Qibla is Arabic for “in the direction of Mecca”
� Given random starting coordinate
– Interviewer walks in the direction of Mecca
– Selects first HH encountered
22
![Page 23: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/23.jpg)
Example of Qibla Method
23
![Page 24: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/24.jpg)
Example of Qibla Method
24
![Page 25: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/25.jpg)
Geosampling
� Select first stage units
– Administrative units
– Or 1km squares
� Select second stage units
– Smaller squares
� Visit and interview all households in smaller unit
25
![Page 26: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/26.jpg)
Geosampling: First & Second Stage
26
![Page 27: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/27.jpg)
� Eliminates separate listing step
� Still vulnerable to interviewer manipulation
� Possible QC by interviewer GPS tracks? (Himelein et al, 2014)
Geosampling: Second & Third Stage
27
![Page 28: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/28.jpg)
Use of UAVs for Listing
� RTI has tested listing from drone images
– Galapagos & Guatemala
28
Amer et al 2016
![Page 29: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/29.jpg)
Listing with Drones
29
Amer et al 2016
![Page 30: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/30.jpg)
Listing with Drones
30
Amer et al 2016
![Page 31: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/31.jpg)
Listing with Drones
� Still testing use of drones
� Legal issues
� Use local staff to code from images
31
![Page 32: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/32.jpg)
Conclusions
� Ideal method:
– Removes influence of interviewer
– Results in equal probability sample of HUs
– With known probabilities
� No alternative is perfect
– High involvement of interviewers
– High data requirements
� Drones may prove useful
32
![Page 33: Response Rates Impact Data Quality, But not How you Might Think](https://reader031.fdocuments.us/reader031/viewer/2022030318/5a65b4ce7f8b9a38648b4b77/html5/thumbnails/33.jpg)
33
Stephanie Eckman, PhD
Fellow, Survey Research Division
RTI International
stepheckman.com