What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · •...
Transcript of What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · •...
![Page 1: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/1.jpg)
1
What is Survey Weighting?
Chris Skinner
University of Southampton
![Page 2: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/2.jpg)
2
Outline
1. Introduction
2. (Unresolved) Issues
3. Further reading etc.
![Page 3: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/3.jpg)
3
Sampling
![Page 4: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/4.jpg)
4
Representation
4 out of 8 1 out of 10
10
![Page 5: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/5.jpg)
5
Weights
8/4 = 2 10/1 = 10
![Page 6: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/6.jpg)
6
Data
Sample unit Weight Survey variable
1 2 1 2 2 3 3 2 0 4 2 0 5 10 4
![Page 7: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/7.jpg)
7
Estimation of Mean of Survey Variable
Respondent i Weight iw Survey
variable iy i iw y
1 2 1 2 2 2 3 6 3 2 0 0 4 2 0 0 5 10 4 40
Total 18 8 48
Unweighted mean = 8/5 1.60= Weighted mean = 48/18 2.67=
![Page 8: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/8.jpg)
8
Weighted Estimation as Replication Weighting equivalent to
(i) replicating each respondent iw times (provided iw is
integer)
(ii) using unweighted estimation with replicated dataset
respondent i represents iw population units
![Page 9: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/9.jpg)
9
Why Weight?
Want to estimate statistical characteristics of population.
If don’t weight will estimate characteristics of sample, which
may not be representative of population.
Difference between two → bias of unweighted estimator.
![Page 10: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/10.jpg)
10
Example: Workplace Employee Relations Survey 1997 (Purdon, 2004)
Population Sample Sampling fraction (1
in ..)
Weight
10-24 197358 362 545 545
25-49 76087 603 126 126
50-99 36004 566 64 64
100-199 18701 562 33 33
200-499 9832 626 16 16
500+ 3249 473 7 7
![Page 11: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/11.jpg)
11
Example of Bias in WERS
Unweighted employee-level estimates will overrepresent
characteristics of employees in larger workplaces
(when population of all employees is of interest).
Weighting needed to avoid bias.
![Page 12: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/12.jpg)
12
Circumstances when Weighting may be
useful
• unequal probability sampling
• non-probability sampling
• nonresponse (usually unit)
• other selection e.g. undercoverage
![Page 13: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/13.jpg)
13
Construction vs. Use of weights
Construction of weights and incorporation into data files
by survey agency
• may use detailed information, technical expertise and
evaluations not available to standard data user.
Use of weights by researchers / survey data users
![Page 14: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/14.jpg)
14
Construction of weights
![Page 15: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/15.jpg)
15
Weights
8/4 = 2 10/1 = 10
![Page 16: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/16.jpg)
16
Alternative Weights
5/3 =1.67
2 10/1 = 10 3/1=3
![Page 17: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/17.jpg)
17
Approaches to Constructing Weights
Inverse probability weights (Hansen and Hurvitz, 1943; Horvitz and Thompson, 1952)
Weight = 1/ Probability that unit is selected
For probability sampling, assume probabilities of selection
‘measurable’
For nonprobability sampling or nonresponse, need to model
probability of selection e.g. using logistic regression.
Sometimes called propensity weights
![Page 18: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/18.jpg)
18
Auxiliary Information
Weights use information on auxiliary variables = weighting
variables
in population, e.g. national age-sex population estimates
or (in case of non-response) on sample, e.g. characteristics of
area in which respondent or nonrespondent lives.
Techniques include post-stratification, (generalized)
regression estimation, raking, calibration estimation.
![Page 19: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/19.jpg)
19
Use of Weights
Different kinds of weights:
• sampling weights to compensate for unequal sampling
probabilities
• nonreponse weights to compensate for nonresponse
- may have alternative weights corresponding to different
responding subsamples, e.g. in surveys combining
interview & diary
• final/combined weights (e.g. combining sampling and
nonresponse weights
![Page 20: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/20.jpg)
20
• scaled (standardised) weights, which sum to 1 or 100
across sample (may be easier to interpret, but not suitable
for estimating totals)
• weights for different units of analysis, e.g. household vs.
individual weights
• weights for different survey variables, but usually try to
avoid this, weights are multipurpose unlike e.g. imputation
![Page 21: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/21.jpg)
21
Dependence on assumptions for bias
adjustment
Probability sampling – sampling weights can correct for bias
without further assumptions
Non-probability sampling and nonresponse – weights only
correct for bias under strong assumption that selection is
unrelated to survey variable, conditional on auxiliary
weighting variables
![Page 22: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/22.jpg)
22
Weighting and Variance
Weighting affects variance as well as bias. Can inflate
variance or can reduce it. Variance inflation can offset
advantage of bias-correction.
Unless, effect is allowed for, standard errors, confidence
intervals, significant tests etc. produced by standard statistical
software can be misleading
Other features of complex sampling (stratification and
multistage sampling) also affect variance.
![Page 23: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/23.jpg)
23
Using weighted data
Want software which allows for weighting, stratification and
multistage sampling in analysis.
Usually require data file which includes fields for:
weight, stratum and primary sampling unit
Software includes:
STATA (version 10+) - svy procedures
SAS (SAS/STAT)
SPSS (version 12+) – Complex sample module
R (Lumley, 2010)
![Page 24: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/24.jpg)
24
Issues: construction of weights
• which weighting variables to use in particular applications
e.g. use of past vote in election polling (Curtice, 2010)
• how many weighting variables to use (and with what
detail)?
• whether & how to incorporate social science theory of
survey participation in nonresponse weights (Groves and
Couper, 1995)
![Page 25: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/25.jpg)
25
Issues: construction of weights (new methods) • weighting for multilevel modelling (Pfeffermann,
Skinner…, 1998)
• links with biostatistical and econometric literatures on
inverse probability and double robust weighting (Kang and
Schafer, 2005)
![Page 26: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/26.jpg)
26
Issues: alternative approaches
• whether (and when) to weight (Fienberg, 2008)
• regression models to control for selection variables as
alternative (Gelman, 2007)
• ‘efficient’ statistical modelling approaches (Bayesian,
likelihood) (Chambers and Skinner, 2003)
• multiple imputation as alternative, even for unit
nonresponse
![Page 27: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/27.jpg)
27
Future Short Course
Introduction to Survey Sampling and Estimation
Pedro Silva
4th-6th October 2010
Southampton
Courses in Applied Social Surveys
http://www.s3ri.soton.ac.uk/cass/
![Page 28: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/28.jpg)
28
Further Reading Paper (last updated April 2010) and Presentations at ‘Weighting the Social Surveys’ meeting, ESDS Government,
March 2004,
http://www.ccsr.ac.uk/esds/events/past.shtml#weighting
Practical Exemplars on the Analysis of Surveys (PEAS)
http://www.restore.ac.uk/PEAS/
(website last updated May 2009)
Heeringa, S.G., West, B.T. and Berglund, P.A. (2010) Applied Survey Data Analysis. Boca Raton: CRC Press.
(Section 2.7 on Weighting in the Analysis of Survey Data; uses STATA in book but has other software on
http://www.isr.umich.edu/src/smp/asda/ ).
Lumley, T. (2010) Complex Surveys: A Guide to Analysis Using R. Hoboken: Wiley.
Bethlehem, J. (2009) Applied Survey Methods: A Statistical Perspective. Hoboken: Wiley (Chapter 10: Weighting
adjustment).
Kalton, G. and Flores-Cervantes, I. (2003) Weighting methods. Journal of Official Statistics, 19, 81-97.
Biemer, P. and Christ, S. (2008) Constructing the survey weights. Chapter 16 in P.Levy and S.Lemeshow Sampling
of Populations: Methods and Applications. Hoboken: Wiley.
![Page 29: What is Survey Weighting? - NCRMeprints.ncrm.ac.uk/1358/1/Weighting Festival 2010.pdf · • weights for different units of analysis, e.g. household vs. individual weights • weights](https://reader034.fdocuments.us/reader034/viewer/2022050217/5f6358e111ab006ed126072b/html5/thumbnails/29.jpg)
29
Additional References
Chambers, R. and Skinner, C. eds. (2003) Analysis of Surveys. Chichester: Wiley.
Curtice, J. (2010) Is it safe to past vote weight? Presented at seminar on “Taking the electoral pulse: three approaches to the methodology of
election polling”, January 2010, http://eprints.ncrm.ac.uk/892/
Fienberg, S. (2008) To weight or not to weight? A Bayesian Model-based update. Paper presented in Southampton
http://www.s3ri.soton.ac.uk/conferences/ssbs08/papers/14%20Stephen%20Fienberg.pdf
Gelman, A. (2007) Struggles with survey weighting and regression modelling. Statistical Science, 22, 153-164
Groves, R. and Couper, M. (1995) Theoretical motivation for post-survey nonresponse adjustment in household surveys. Journal of Official
Statistics, 11, 93- 106.
Hansen, M. and Hurwitz, W. (1943) On the theory of sampling from finite populations. Annals of Mathematical Statistics, 14, 333-362.
Horvitz, D. and Thompson, D. (1952) A generalization of sampling without replacement from a finite universe. Journal of American Statistical
Association, 47, 663-685.
Kang, J. and Schafer, J. (2007) Demystifying double robustness: a comparison of alternative strategies for estimating a population mean from
incomplete data source: Statistical Science, 22, 523-539.
Pfeffermann, D., Skinner, C.J., Holmes, D.J., Goldstein, H. and Rasbash, J. (1998) Weighting for unequal selection probabilities in multilevel
models (with discussion). Journal of the Royal Statistical Society, Series B, 60, 23-56.
Purdon, S. (2004) Weighting NatCen surveys. Presentation at http://www.ccsr.ac.uk/esds/events/2004-03-12/slides.shtml