How survey design affects analysis

Susan Purdon

Head of

Survey Methods Unit

National Centre for Social Research

Two general principles

• Weighting affects both survey estimates and standard errors

• Complex sample design (clustering, stratification) affects standard errors but not survey estimates (as long as survey isn’t weighted).

Why weights are used

Two main reasons:• Because sample units are selected with non-equal

probabilities of selection• To reduce non-response bias

Weighting for non-equal probabilities

Non-equal probabilities of selection can be used:

• For good statistical reasons: stratification with disproportionate allocation (e.g over-sampling by region)

• For convenience: because sampling frame is of larger units than survey units, and need to sub-sample within units (e.g selection of one person per household).

• Weights are calculated as the inverse of the probability of selection.

Example 1: WERS98

Population Sample Samplingfraction (1

in ..)

Weight

10-24 197358 362 545 545

25-49 76087 603 126 126

50-99 36004 566 64 64

100-199 18701 562 33 33

200-499 9832 626 16 16

500+ 3249 473 7 7

Effect of selecting one adult per household

H’hldsize

H’hlds (per 100) Adults Adults selected Weight

1 38 38 38 1

2 51 102 51 2

3 9 27 9 3

4+ 2 8 2 4

Total 100 175 100

Weighting for non-response

• Many surveys use weights to reduce non-response bias.

• Up-weight low responding groups; down-weight high responding groups.

Weighting for non-response

Age Sample Population(pro-rata on

fullresponsesample)

Response rate Weight

16-24 510 920 55.4 1.8

25-34 650 1080 60.2 1.7

35-44 340 500 68.0 1.5

1500 2500

Effect of weights on estimates

• Weighting changes almost all survey estimates (means, percentages, odds ratios, correlation coefficients, regression coefficients etc.)

Effect of weights on estimates

Unweighted estimate=

(25%x38 + 17%x102+15%x27+9%x8)

/(38+102+27+8)=20%

Weighted estimate=

(25%x38x1 + 17%x102x2+15%x27x3+9%x8x4)

/(38x1+102x2+27x3+8x4)=18%

H’hldsize

Adults Weight % in poorhealth

1 38 1 25

2 102 2 17

3 27 3 15

4+ 8 4 9

Total 175 100

Effect of weighting on standard errors

• Standard errors for weighted and unweighted estimates are not the same.

• Weighting because of non-equal probabilities of selection tends to increase standard errors.

• Weighting for non-response sometimes increases/sometimes decreases ses. Impact tends to be smaller.

Impact of other design features - clustering

• Most face-to-face surveys are clustered• Clustering doesn’t change estimates, but it does

increase standard errors• Degree of increase depends on cluster size and

cluster homogeneity• To account for clustering need to identify the

primary sampling unit (psu) on dataset.

Impact of other design features - proportionate stratification

• Most surveys use proportionate stratification (either overall or within regions)

• Does not affect estimates. Tends to reduce standard errors. Degree depends on choice of stratifiers.

In summary:

• To get unbiased estimates need to use survey weights.

• To get correct standard errors need to take into account survey design, in particular weighting, clustering and stratification.