1

Nonresponse and Measurement Error in Employment Research

Gerrit Müller (IAB)

joint with: Frauke Kreuter (JPSM – U Maryland), Mark Trappmann (IAB)

2

Do survey respondents recruited with extra effort, provide answers of lower quality?

Are cooperators more motivated to provide accurate data? Or, are late respondents hampered by recall deficits?

How does extra effort affect total bias?

Research Questions

3

Survey Data

Panel Study “Labor Market and Social Security” (PASS)

Dual frame survey (benefit recipients / residential population) Wave1: 12,000 HH

20,000 P RR1: 30.5% (within HH: 85%)

Mixed mode survey (sequential CATI -> CAPI)

4

Record Linkage I

Individual survey data linked with individual administrative data (80% of all Rs agreed; 72% successfully matched)

Administrative records on: employment, earnings, unemployment, labor market programs

,

Contact data on HH-level only

recordsRy ,surveyRy ,

5

Record Linkage II

Administrative data linked with paradata for the gross sample of recipients (from unemployment register)

Contact data on HH-level only Indicator for Respondents / Nonrespondents on HH-level only ,recordsNRRy , recordsRy ,

6

Hypotheses about Measurement Error(response process model, Tourangeau ’84)

Unemployment benefit (UBII) July 2006 Nov 2006 at time of interview

Income in month prior to interview Occupation Educational degree

Relationship between ME and response propensity (number of contact attempts)

7

Contact Quintiles and Follow-Up Efforts

# of contacts: _min _maxQ1 (high contactability) 1 2Q2 3 4Q3 5 7Q4 8 14Q5 (low contactability) ≥15

Transfer CATI to CAPI CATI NR follow-up of “soft refusals”

8

Measurement Error (in percent) by Contact Quintiles and Follow-up Efforts

UB IIJuly 2006

UB IIInterview Date

Q1 (high contactability) 12 11Q2 13 12Q3 14 13Q4 16 17Q5 20 15To CAPI 18 14

CATI NR follow-up 21 14

9

Measurement Error by Contact Quintiles and Follow-up Efforts

Income(abs. dev.)

Years of Edu.(% mismatch)

Q1 (high contactability) 311 25Q2 343 28Q3 370 27Q4 335 26Q5 380 24To CAPI 451 25

CATI NR follow-up 407 27

10

11

5. NR-ME Bias Decomposition

)()(

)()(

,,,,

,,

surveyRrecordsRrecordsRrecordsRNR

surveyRrecordsRNR

yyyy

yyybiastotal

for Recipient sample only HH-level variables only (!)

UBII in Jul06 not feasible for bias decomposition UBII in Nov UBII at date of interview

12

Figure 1. Cumulative mean over (quintiles of) no. of contact attempts; UB II recipience in Nov 06

0,7

0,75

0,8

0,85

0,9

0,95

1

1 (High) 2 3 4 5 (Low) NR f'up to CAPI

Contactability

UB

II r

ec

ipie

nc

e in

No

v 0

6

Records

Target

Survey reports

13

Figure 2. Cumulative mean over (quintiles of) no. of contact attempts; UB II recipience at date of interview

0,7

0,75

0,8

0,85

0,9

0,95

1

1 (High) 2 3 4 5 (Low) NR f'up to CAPI

Contactability

UB

II r

ecip

ien

ce

at

da

te (

mo

nth

) o

f in

terv

iew

/ la

st

co

nta

ct

Records

Target

Survey reports

14

“Pick your brains”

ME-Model for UBII in July06 (handout)Puzzle: high ME for the young? HH-interview by target head?

Administrative data not always „Gold Standard“ (error-free)Assumption: ME in register data unrelated to ME in survey reports and response propensity

15

“Pick your brains”

Decomposition findings statistic-specific Extend analyses to P-level variables (e.g. employment,

income) Problem: unknown on individual level

How to go ahead?

recordsRy ,