E969 Peter Lynn

Reasons for Unit Non-Response • Failure of the data collector to locate/identify the

sample unit; • Failure to make contact with the sample unit; • Refusal of the sample unit to participate; • Inability of the sample unit to participate (e.g. ill

health, absence, etc); • Inability of the data collector and sample unit to

communicate (e.g. language barriers); • Accidental loss of the data/ questionnaire. Unit non-response is defined relative to the eligible sample. In other words, if the sampling frame contains ineligible units, these do not contribute towards response/ non-response. Unit non-response is often divided into three components: non-contact, inability to respond, lack of co-operation (refusal). Response rates can usefully be divided into these (or other) components.

E969 Peter Lynn

Item non-response A sample unit participates but data for some survey items are not available for analysis. Reasons could include:

• Refusal to provide an answer • Inability to provide an answer • Other failure to answer (e.g. by accident) • Provided answer being of inadequate quality (e.g.

incomplete, implausible, failing an edit/consistency check, etc.)

Item non-response can be caused by:

• the action of the sample member (e.g. refusal to answer);

• the action of an interviewer (e.g. failure to ask a question that should have been asked, or failure to record the answer adequately);

• the survey design (e.g. poor routeing instruction). In practice, these factors interact.

E969 Peter Lynn

Response Pattern We can define the survey response pattern by a matrix R = [rjk], where rjk = 1 if item j is observed for unit k, and rjk = 0 otherwise. Possible response patterns include: Unit (k) y1k y2k y3k … yqk

1 1 1 1 … 1 full response

2 1 0 1 … 1 item non-response

3 1 0 0 … 1 item non-response

4 0 0 0 … 0 unit non-response Note that, due to item non-response, the set of units available for analysis depends on the item or items required.

E969 Peter Lynn

Non-Response Error The effect of non-response on a survey estimate yr can be defined:

)( nrrnr yynnryy −+=

where yr = statistic for the r responding units, yn = statistic for all n sample units, ynr = statistic for the nr non-responding units. (Groves, 1989, p.133).

)( nrr yynnr

− is the estimate (the realisation in one implementation of the survey) of total non-response error.

E969 Peter Lynn

Components of Non-Response Error The non-response error has two elements:

• the non-response rate;

• the difference between responding and non-responding units in terms of y.

Both of these elements are important. Note that the second element will result from both non-response bias and variance. In practice, studies suggest that bias is usually the main component.

E969 Peter Lynn

Components of Non-Response Error ctd.

Non-response errors can be divided into two components: • Errors due to unit non-response • Errors due to item non-response Each of these can in turn be decomposed into sub-sources, for example: Unit non-response due to: • Non-contact • Refusal to respond • Inability to respond Item non-response due to: • Routing (instrument) error • Routing (interviewer) error • Refusal to respond • Inability to respond Etc.

E969 Peter Lynn

Non-Response in Different Types of Surveys

Survey structure:

• Cross-sectional

• Longitudinal

• Rotating Panel

• Flow sample Respondent type:

• Business

• Individual

• Household

• Individual within household

E969 Peter Lynn

Cross Sectional Surveys

Components of non-response:

• Refusals

• Non-contact

• Inability to respond Constraints on non-response:

• Limited field time

• Limited budget

• Survey population

• Survey task (burden)

E969 Peter Lynn

Longitudinal Surveys

Components of NR same as for cross-sectional surveys. But structure of NR different:

• Complete NR

• Wave NR (gaps)

• Attrition

NR patterns depend partly on survey policy and partly on field efforts (especially "tracing").

E969 Peter Lynn

Patterns of response to a 4-wave panel survey

Policy 1: Issue all eligible cases at every wave. 16 possible patterns:

Wave 1 2 3 41 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

E969 Peter Lynn

Policy 2: Issue only wave 1 responding cases at each subsequent wave. 9 possible patterns:

Wave 1 2 3 41 2 3 4 5 6 7 8 9

Policy 3: At each wave, issue only cases responding to previous wave. 5 possible patterns:

Wave 1 2 3 41 2 3 4 5

E969 Peter Lynn

Example: England and Wales Youth Cohort Study

Panel with 3 waves; Implements policy 1 (issue all eligible cases at every wave). 8 possible response patterns:

Wave: 1 2 3 No. of cases

% of cases

1 8,396 44.9 2 2,555 13.7 3 2,660 14.2 4 900 4.8 5 386 2.1 6 575 3.1 7 352 1.9 8 2,855 15.3

(From Lynn, Purdon, Hedges and McAleese, 1994)

E969 Peter Lynn

Panel Attrition Specific to longitudinal studies. Can result from non-traceable or refusal cases Impacts can be quite different from cross sectional surveys Differences should be studied (using data from earlier waves). Design stage: strategy for dealing with attrition (both non-response and “population exits”) Response rates as products of wave response rates

E969 Peter Lynn

Panel Attrition ctd.

Non-response due to mobility (e.g. sample unit has moved and cannot be traced): • not unique to longitudinal surveys • but can be especially problematic • particularly if long time gap between waves Experience of the first wave may alter the response behaviour on subsequent waves: • Almost unique to longitudinal studies • Interview experience therefore very important

E969 Peter Lynn

Survey Types and Data Collection Modes

• Interviewer surveys − Face-to-face surveys − Telephone surveys

• Self completion − Mail surveys − Web surveys − Diary keeping

• Mixed mode strategies − Supervised self-completion − Mail questionnaire with interviewer follow-up − Interview and diary keeping period − Interview and self-completion − Different modes for different waves in panel surveys

E969 Peter Lynn

Survey Types and Data Collection Modes ctd.

Face-to-face surveys typically yield higher response rates than telephone surveys.

Response for mail surveys usually much lower than other modes of data collection.

Differences are mainly due to level of respondent motivation possible in different modes.

In mixed mode strategies: motivation of respondents vs. time and costs.

Interview surveys: role of the interviewer is crucial.

Telephone surveys: scope for interviewer influence is more limited.

Mail surveys: only communication is via document design and content (q’re, advance or covering letter, reminders).

E969 Peter Lynn

Measuring and Presenting Response Rates

Response rates measure the proportion of eligible sample units that successfully provided data, i.e. the reduction due to non-response of cases available for analysis. Response rates are more likely to be reported than almost any other survey process quality indicator. Response rates are often mistakenly used as a measure of quality of survey statistics. Remember, response rate is just one component of non-response error.

E969 Peter Lynn

Refusals Factors influencing household survey participation (Groves & Couper, 1998):

1. Societal-level factors

2. Attributes of the survey design

3. Characteristics of the sample person

4. Attributes of interviewer

5. Respondent-interviewer interaction

Social environment

Household Interviewer

Survey design

Decision to co-operateor refuse

Household-interviewerinteraction

E969 Peter Lynn

1. Societal-level factors

• The degree of social responsibility felt by sample persons

• Legitimacy of societal institutions

• The degree of social cohesion (Goyder, 1987).

• Interviewer respondent interaction:

→ Particular persuasion strategies (on the part of the interviewer)

→ Decision making strategies (on the part of the respondent)

(Groves, Cialdini and Couper, 1992).

2. Attributes of the survey design

The mode of the initial contact affects

• The number of channels of communication between interviewer

and respondent (Groves, 1978).

• The selection of persuasion strategies to employ and the

effectiveness of alternative strategies (Groves, Cialdini and

Couper, 1992).

E969 Peter Lynn

Other attributes:

• Perceived burden (e.g. length of the interview being requested,

sensitivity of questions)

• Respondents’ level of interest and knowledge in the survey (topic,

description)

• Incentives

3. Characteristics of the sample person

• Socio-economic characters

• Psychological characteristics

• Past experiences (Goyder, 1987).

These factors tend to produce a set of pre-dispositions that affect

the decision. They also affect the initial approach of the

interviewer to the sampling unit.

(Groves, Cialdini and Couper, 1992).

E969 Peter Lynn

4. Attributes of interviewer

Observable socio-demographic characteristics of the interviewer may

affect the ‘script’ evoked in the respondent’s mind at the first contact

with the interviewer. (Groves, Cialdini and Couper, 1992).

No research has yet found strong links between stable interviewer-

personality characteristics and success in gaining co-operation.

Reasons for this might be interviewers being relatively homogeneous

group, tailoring, social skills, and other adoptive behaviours (Groves

and Couper, 1998). Morton-Williams (1993) argues that social skills

can be taught and offers an outline of such training.

Those with greater interviewing experience tend to achieve higher

rates of co-operation than those with less experience. It is still

unclear, whether this is a selection effect (less successful interviewers

terminate their employment earlier) or a training effect (due to the

benefits of coping over time with diverse situations in recruiting

respondents) or both (Groves and Couper, 1998).

E969 Peter Lynn

There is some evidence that interviewers who, prior to the survey, are

confident about their ability to elicit co-operation tend to achieve

higher co-operation rates (Groves and Couper, 1998).

5. Respondent-interviewer interaction

Persuasion strategies employed by the interviewer are determined not

only by the interviewer’s ability, expectations, and so on but also by

features of the survey design and by characteristics of the immediate

environment and broader society.

Similarly, the responses that the sample person makes to the request

are affected by a variety of factors, both internal and external to the

respondent, and both intrinsic and extrinsic to the survey request.

(Groves, Cialdini and Couper 1992).

E969 Peter Lynn

Non-Contact A conceptual model for contacting sample households (Groves & Couper 1998):

Contact likelihood is a function of three factors: 1) Physical impediments that prevent visiting

interviewers from alerting the household to their presence

2) When household members are at home 3) When and how many times the interviewer visits the

household.

Social environmental attributes

Physicalimpediments

Socio-demographicattributes

Accesible at-home patterns

Likelihood of contact

Interviewer attributes

X Timingof calls

Number of calls

X

E969 Peter Lynn

Reasons for Non-Contacts Association between when household members are at home and when interviewer calls:

• Life styles that lead to reduced time at home (patterns of work, related patterns of shopping and entertainment)

• Rural/urban differences • Local crime rate (perceptions) • Ethnic/cultural differences • One-person households • households with no children <5 or adults >70 Hidden refusals: • Fear of crime/strangers/salesmen • Too busy • Answer machine screening

E969 Peter Lynn

Example: The Survey Participation Process in the British Crime Survey

It can be useful to identify the logical stages of the survey participation process. This was done for BCS (next page). It can be seen that there are many stages at which either a non-contact or a refusal could occur (see diagram on next page, from Laiho and Lynn, 2000). This illustrates the heterogeneity of the survey non-response phenomenon. Survey design and implementation features should be used to address each of these possible non-response outcomes.

E969 Peter Lynn

Survey participation process in BCS (Laiho & Lynn 2000)

Advance letter to thesampled address

Multiple DU

1) List in systematic order by flat number

2) random selection of the dwelling unit

Ineligible

Contact attempt

- Insufficent address- Not traced- Not yet built- Vacant/ derelicted / demolished- Empty- Business/industrial only- Other Listing of all adult (16+)

members of the DU

Random selection of therespondent

Information about numberof persons 16+ refused

No contact with responsible/any adult in selected DU(after 5+ attempts)

No contact with selectedperson after 5+ attempts

Completed interview

Contact attempt - respondent

Personal refusal byrespondent

Proxy refusal on behalf ofselected respondent

Broken appointment,no recontact

Refusal

Respondent contacted

Away/in hospital duringsurvey period

Inadequate English

Senile/incapacitated

Ill at home

Other reasonfor non-response

Other

Office refusal

Dwelling Unit (DU)

Single DU

E969 Peter Lynn

Studying Non-Response Designs for Determining Characteristics of Non-respondents: • Special studies of non-respondents

• Using information on the sampling frame

• Asking others about non-respondents or having interviewer provide information about them (example on next page)

• Comparison of respondent characteristics by call number

• Comparison of respondent characteristics to census or other external information

• Studying persons who drop out of a panel survey after an initial interview

E969 Peter Lynn

ALL RESIDENTIAL ADDRESSES (CONTACTS AND NON-CONTACTS – INCLUDING VACANTS) 17. Does the address have an

entryphone? Yes 1 No 2

18. Which of the following are visible at the sampled address?

CODE ALL THAT APPLY

Burglar alarm 1 Security gate over front door 2 Bars/grilles on any windows 3 Other security device(s) 4 Estate/block security lodge/guards 5 None of these 0

INTERVIEWER ASSESSMENTS:

19. Are the houses/flats in this immediate area in a good or bad physical state?

Mainly good 1 Mainly fair 2 Mainly bad 3 Mainly very bad 4

20. Is the sampled house/flat in a better or worse condition outside than the others in this area?

Better 1 Worse 2 About the same 3 Does not apply 4

21. Do you know or think that the occupants are probably…

… white 1 … black 2 … Asian 3 … Other: __________________ 4 Don’t know 5

22a. SAMPLED DWELLING IS: Whole house:– detached 1 Semi-detached 2 IF NO DWELLING mid-terrace 3 SELECTED, CODE end terrace 4 FOR ADDRESS Maisonette 5 Flat:– purpose-built 6 converted 7 Rooms, bedsitter 8 Unable to code 9

IF FLAT ETC (5-7 AT a.) ANSWER b-e. OTHERS - END

b. CODE TYPE OF FLAT ETC: Self-contained 1 Not self-contained 2 Don’t know 8

c. BUILDING HAS: Fewer than 5 floors 1 5 floors or more 2 Unable to code 3

d. FLOOR LEVEL OF MAIN ACCOMMODATION:

Basement/semi-basement 1 Ground floor/street level 2 First floor 3 2nd/3rd floor 4 4th – 9th floor 5 10th floor or higher 6

e. BUILDING HAS: Common entrance: lockable 1 Common entrance: not lockable 2 No common entrance 3

E969 Peter Lynn

Example: Using Interviewer Observation Data

Data were collected by interviewers, using the form on the previous page, on the 1996 British Crime Survey. The following table uses the answers to question 22a. Response Selected Responding rate sample sample % % House: detached 82.6% 19.5 21.0 semi-detached 79.6% 30.2 31.3 end terrace 79.2% 7.3 7.6 mid-terrace 77.7% 20.4 20.6 Maisonette 74.9% 1.7 1.7 Flat: converted 72.3% 2.9 2.7 purpose-built 70.3% 11.7 10.7 Rooms/bedsit 75.6% 0.3 0.3 Unable to code 51.2% 6.0 4.0 Base 13,117 10,059 Source: Lynn P (1996) Weighting for non-response, in Survey and Statistical Computing 1996, Chesham: Association for Statistical Computing

E969 Peter Lynn

Example: Using Sample Frame Data

These data are from the Scottish School Leavers Survey, a postal survey for which the sampling frame is a list of pupils and their school exam results. Highest Response Selected Responding Qualification rate sample sample % % 5+ Higher grades 91.1% 18.0 21.4 3-4 Higher grades 85.1% 13.0 14.5 1-2 Higher grades 81.7% 15.0 16.1 5+ Standard grades 1-3 76.4% 8.1 8.1 3-4 Standard grades 1-3 74.1% 9.1 8.8 1-2 Standard grades 1-3 69.1% 14.5 13.1 Standard grades 4-7 only 62.6% 14.4 11.8 No qualifications 59.6% 7.8 6.1 Base 4,542 3,469 Source: Lynn P (1996) Weighting for non-response, in Survey and Statistical Computing 1996, Chesham: Association for Statistical Computing

E969 Peter Lynn

Example: Using Geographical Data

The following data are from the Italian Multipurpose Survey carried out by ISTAT. The figures presented are regression coefficients from a model to predict propensity to refuse. It can be seen that refusals are most likely in metropolitan areas and least like in small (rural) municipalities. Parameter s.e.

Metropolitan area 2.341 (0.147)

Met. area ring 1.287 (0.168)

Large municipality 0.915 (0.114)

Small municipality 0

Source: Baldazzi et al (2002) Interviewer’s effect on refusal risk in the Italian Multipurpose Survey: a multilevel approach, paper presented at the annual conference of the Italian Statistical Society, May 2002.

E969 Peter Lynn

Example: Using Number of Calls

These data are from the 1987 British General Election Survey. This kind of analysis can be done for any survey, even if there is no useful information from the sampling frame or from interviewer observation (e.g. postal or telephone survey).

Interviewed

at 1st call Interviewed at 2nd call

Interviewed at 3rd call

Interviewed after 4+ calls

Non-respondents

Social Class % % % % % Non-manual 41.3 44.3 49.7 50.4 Unknown Manual 46.4 44.5 39.5 36.9 Unknown Self-employed 6.2 6.3 6.1 7.9 Unknown Unclassifiable 6.0 4.9 4.7 4.8 Unknown Base 726 1,024 722 1,354 1,637

Source: Lynn P (1996) Weighting for non-response, in Survey and Statistical Computing 1996, Chesham: Association for Statistical Computing

E969 Peter Lynn

Example: Comparing Non-Contacts, Refusals and “Easy-to-get” Households

Data are from 1996 Health Survey for England. Lynn et al (2002) analysed calls record data and classified respondents as easy or difficult to contact, and willing or reluctant. It can be seen, for example, that persons who were difficult to contact were much younger than average and much more likely to be employed; but reluctant persons (potential refusals) were slightly less likely than others to be employed. Estimate

Difficult to contact (6+

calls) A

Reluctant (temporary

refusal) B

Hard-to-get

A+B

Easy-to-get households

C

All responding households

A+B+C

Male (%) 46.7 40.5 45.7 45.5 45.5

Age (mean) 39.4 46.5 40.7 47.9 46.7

Owner-occupier (%) 66.8 74.1 68.1 72.8 72.0

Employed (ILO

definition) (%)

66.6 47.7 63.3 50.9 53.0

White (%) 92.0 90.9 91.8 94.1 93.7

Source: Lynn P, Clarke P, Martin J and Sturgis P (2002) The effects of extended interviewer efforts on nonresponse bias, in Survey Nonresponse (ed.s R M Groves, D A Dillman, J L Eltinge and R J A Little), New York: Wiley.

E969 Peter Lynn

Common Non-Response Patterns

1. Many (most) household surveys find

response rates lower in urban areas. 2. Many self-completion surveys find response

rates higher amongst those with more education.

3. Interview surveys often find higher contact rates amongst many-person households than amongst 1- or 2- person households.

4. Many household surveys find contact rates higher amongst persons aged 65+, but co-operation rates lower.

5. In most (but not all) European countries, response rates are usually higher for women than men.

6. On most business surveys, response rates are higher amongst larger businesses.

But… These relationships may not be true for all countries, cultures and surveys.