QUALITY ISSUES IN TELEPHONE SURVEYS: COVERAGE, NON-RESPONSE and QUOTA SAMPLING

Austral. J . Statist. 38(1), 1996, 15-34

QUALITY ISSUES IN TELEPHONE SURVEYS: COVERAGE, NON-RESPONSE AND QUOTA SAMPLING

DAVID STEEL', JOE VELLA' AND PETER HARRINGTON2

University of Wollongong and lllawarra Regional Information Service

Summary

The quality of a telephone survey is affected by several factors: telephone coverage, non-response, the methods used to select households and persons, and the quality of responses obtained from respondents. Data are provided which show that a large proportion of Australian households have telephone connections. However, telephone coverage is not uniform and some subgroups of the population have much lower connection rates. This paper reviews evidence of the effect of non-response and the effectiveness of repeated call backs, and reports the results of a new study. The use of quota sampling to select respondents from randomly selected households is also examined. The results suggest that telephone surveys under-represent older persons and the unemployed, and over-represent middleaged persons. It is shown that while call backs C&I increase the response rate, the effect on the composition of the sample and resulting estimates is minimal. The main effects are due to refusals and variation in coverage rates.

Key words: Non-sampling errors; response rates; non-response bias; quota sampling; telephone surveys; telephone coverage.

1. Introduction

Telephone surveys are now widely used in social and market research. Com- pared with the traditional data collection method of face-to-face interviewing, telephone surveys can provide information more quickly and at lower cost and allow easier control and supervision of interviewing. Telephone surveys also remove the need to cluster the sample geographically. The quality of estimates produced from a telephone survey is determined by a number of factors. This paper considers three of the major factors: the coverage of the target population, the method of selecting persons from selected households, and non-response.

The other main determinant of data quality is how the use of the telephone affects the quality of the responses given by respondents. This mode effect has

Received August 1994; revised May 1995; accepted October 1995.

2522. anlawarra Regional Information Service, 22 Porter St, North Wollongong, NSW 2500. Admowiedgments. The authors thank an anonymous referee for some useful comments, and Mary Yannis for her assistance in providing unpublished data from the August 1991 telephone connections survey.

Dept of Applied Statistics, University of Wollongong, Northfields Ave, Wollongong, NSW

16 DAVID STEEL, JOE VELLA AND PETER HARRINGTON

been studied extensively (e.g. Groves & Kahn, 1979; Sykes & Collins, 1987). In an analysis of 25 studies comparing telephone and face to face surveys De Leeuw & Van der Zouwen (1988) showed that in most cases no statistically significant differences were found. Combining the information from the separate studies they concluded that any differences are small and have reduced over time as researchers gain more experience with telephone surveys. Where responses could be checked with external sources no mode effect was found. Telephone surveys were found to have slightly lower item non-response, more socially desirable answers and give less information in response to open-ended or checklist questions, but these effects were small.

The proportion of households in Australia which have a telephone connected has grown steadily over the last 20 years and is now so high that in general there are few concerns about bias due to the non-coverage of households without a telephone. Section 2 considers details of the telephone connection levels in Australia. Random selection of household telephone numbers is relatively straightforward, but selection of people from selected households can be made in a variety of ways and these are considered in Section 3. All surveys have a degree of non-response which is partly determined by the follow-up procedures used. An advantage of telephone surveys is that the data can be collected over one or two days and results produced quickly. However, the shorter the interviewing period the higher the level of non-contact that can be expected, unless considerable extra resources are used in the survey. Non-response issues are discussed in Section 4.

There is little publicly available information concerning quality issues associated with telephone surveys in Australia. In Sections 5 and 6 we report the design and results of a study conducted in the Wollongong area. It examined the effects of non-response bias and quota sampling of people in selected households.

2. Telephone Coverage in Australia

The Australian Bureau of Statistics (ABS) has conducted several surveys that provide information on household telephone coverage. The most recent figures available show only 5.6% of private households in Australia did not have a telephone connected in August 1991 (ABS, 1991). Earlier surveys conducted by the ABS show a non-connection.rate of 8.7% in March 1986, 14.7% in March 1983 and 38% in 1974 (Steel & Bod, 1988).

Telephone non-coverage varies by State with 3% of households in the Aus- tralian Capital Territory not being connected, followed closely by Victoria with a 3.4% non-connection rate. The Northern Territory had the highest non- connection rate of 16.3%.

The continual reduction in the proportion of households without a telephone connected has lessened concern that telephone surveys conducted in Australia would be biased because they could not represent people in such households. However, an analysis of data from the March 1986 ABS survey showed that while the overall percentage of households without a telephone connection was

QUALITY ISSUES IN TELEPHONE SURVEYS 17

TABLE 1 Percentage of Australian households without a

telephone connection, by State/Territory, August 1991 State/Territory percent

ACT 3.0 Victoria 3.4 South Australia 5.2 Western Australia 5.7 New South Wales 6.1 Tasmania 6.9

Northern Territory 16.3 Australia 5.6

Queensland 7.7

(Source: ABS 1991)

8.7%, accounting for 7.2% of persons in private households, there were subgroups in the population with a considerably higher non-connection rate (Steel & Bod, 1988). These included, for example people living alone, people in a low-rise flat or rented accommodation, people in the age bracket 15-24, and people whose principal source of income was government benefits. The resulting relative bias due to non-coveiage was significant for some variables; for example -12.2% for unemployment rate, -8.9% for the proportion of households with low income, -4.7% for the health insurance rate, but +1.1% for the labour force participa- tion rate and 0.6% for female. Steel & Bod concluded that surveys in which people with low income, young people, or people in rented accommodation are important, should not be made by telephone. However, in many surveys under- representation of these groups may not be a problem.

While there has been a further increase in the household telephone connection rate since March 1986, unpublished data still show some groups with lower coverage. Table 2 gives details of telephone coverage rates for selected subgroups in the population from the August 1991 survey (ABS, 1994). These figures show that 4.3% of the population aged 15 or more, living in private households, were without a telephone connected, though this figure was higher for some subgroups. Non-coverage was significantly higher for the unemployed, single parent households with dependent children, households comprising one adult, and divorced and separated people. Non-coverage decreased with size of household. While the differences are not large, the non-coverage of males, people aged less than 30, those living in non-metropolitan axeas and Plant Operators, Labourers etc. are higher than the overall population. There is little difference according to birthplace, with those born outside Australia having slightly higher coverage, although it is lower for those born in New Zealand, Oceania and south-east Asia. If the analysis is taken further, groups with much higher non-connection rates can be found, e.g. divorced and separated males (13%), males aged 15-24 living alone (32%) (ABS, 1992). Recent surveys have not collected data on income,


TABLE 2 Telephone non-coverage for persons over 15 (August 1991, selected subgroups)

subgroup non-coverage rate % malek females aged 15-29 aged 30-64 aged 64 and over employed unemployed not in labour force

4.8 3.9

6.5 3.3 3.7

2.8 13.6 5.4

born in Australia 4.6 born outside Australia 3.6

3.4 born Europe and former USSR (excluding UK, Ireland) 2.8 born New Zealand and Oceania 6.9 born south-east Asia 5.1

one-adult household 10.2

born United Kingdom and Ireland

marriedlde facto divorced/separated widowedlnever married couple-only household couple-with-dependent-children household single-parent-with-dependent-children household metropolitan extra-metropoli tan plant operators, labourers etc. managers, adminstrators

2.9 9.4 6.1

3.1 2.8

10.4

2.8 6.9

6.1 1.0

total population over 15 4.3

(Source: ABS 1994)

although cost is given as the main reason for not having a telephone connected. Analysis of the 1986 Household Expenditure Survey and the March 1986 survey showed an increasing coverage rate as income increased. These patterns are similar to those in the USA where telephone coverage is lower for households which are low income, rural, in rented accommodation, comprise one adult or a lot of adults, whose head is young, never married or divorced, of low education, male or black (Thornberry & Massey, 1988; Smith, 1987). -

Australia’s proportion of household telephone connections is comparable with that of most developed countries. In an international comparison of telephone coverage, Trewin & Lee (1988) showed that Australia (91%) ranked 8th, behind the USA (94%), and it had significantly higher coverage rate than the United Kingdom (71%). The highest proportion reported was 98% for Sweden. The Australian figures refer to households in private dwellings. Approximately


4% of the population lives in non-private dwellings such as caravan parks, hospi- tals, hotels, motels, boarding houses etc. and is usually excluded from household telephone surveys.

The preceding figures refer to a l l telephone connections. However, many telephone surveys in Australia are based on publicly available lists of telephone subscribers, and use either printed telephone directories or the ‘electronic White Pages’ as the basic sampling frame. For example Smith et d. (1992) describes a health survey using numbers randomly selected from the Melbourne White Pages. This means that households with unlisted numbers are not covered by such a telephone survey. In Australia about 13% of private telephone numbers are unlisted, and there is no information available on the characteristics of the people in such households. Collins & Sykes (1987) quotes information indicating that in the USA 20% of private telephone numbers are unlisted, and gives an estimate of 12% for Britain. However, the rate is higher in some cities in the USA, and London has the highest rate (25%) in Britain. Wyllie et d. (1994) estimates that 10% of households in New Zealand have unlisted numbers. In Britain, New Zealand and the USA the percentage of unlisted connections is higher for households with a young head and lower income or social grade. These groups also have relatively lower connection rates so the two effects reinforce each other.

Unlisted nuinbers have a chance of selection if random digit dialling (RDD) is used as the selection procedure. This method is used by some survey organisations in Australia. Using RDD creates some difficulties in obtaining a response from people who have unlisted numbers. Moreover, a high proportion of numbers selected are non-working or non-household numbers, increasing costs. In the USA the two stage Waksburg-Mitofsky method (Waksburg, 1978) is used to increase the chances of dialling working household numbers. Casady & Lepkowski (1993) propose increasing the efficiency of RDD-based selection by using information on listed numbers to stratify the population of all possible numbers into one stratum with a high proportion of working household numbers and another stratum with a low proportion. These methods use features of the structure of the telephone numbers in the USA which do not apply in Australia. Collins & Sykes (1987) notes that RDD methods do not appear to be widely used in Britain.

3. Method of Respondent Selection

Most telephone surveys start with an equal probability sample of numbers, at least within geographically defined strata. The August 1991 survey (ABS, 1994) found 2.4% of households with telephones had more than one telephone connection. Thus in all but a very small proportion of cases a household can be uniquely associated with one telephone listing, so that listed households have the same chance of selection. To achieve an equal probability sample of people, each person needs to be associated with one household and all persons in the selected household who are in the scope of the survey must be selected. However, it is


common to adopt a selection procedure in which only one person is selected from each selected household.

Selecting only one person per household reduces the reporting load on the selected households. Statistically there are advantages and disadvantages in this practice. If all persons were included in the sample then there would be a clustering effect introduced due to intra-household correlation which would increase sampling variance for some variables. However, if only one person is selected there is no clustering effect, but the probability of a person being selected is inversely related to the number of persons in the household, assuming equal probability selection is used within households. Generally, giving people unequal chances of selection tends to increase sampling errors. Sample weights should be applied to account for the different probabilities of selection. In practice this is not always done, although weighting to independent official age-by-sex population estimates is often carried out.

Several methods have been developed for selecting a single person from a household selected in a telephone survey. Biases may result if the sample is selected from only the people who are available at the time of the call. The selection procedure shodd be simple and objective enough to reduce biases introduced by the interviewer or person initially contacted in the household, without appearing invasive or threatening, and produce high cooperation rates. A simple method is the Last Birthday procedure, in which the person selected is the one in the household who had the most recent birthday. Kish (1965) and Troldahl & Carter (1964) (see also Bryant, 1975) attempt to ensure a balance of the selected persons’ age and sex. Another method is the Male/Femde Alternation procedure, which tries to ensure a balanced representation of males and females in the sample. Oldendick et d. (1988) compared the Kish and Last Birthday methods and found very few differences in terms of response rate, demographic composition of the sample or substantive results.

While it is desirable to have some control over the selection procedure, the more control specified the more difficult it can become to obtain the required selection, increasing the number of calls that may be necessary to obtain the required sample size, and affecting time and cost. An alternative selection procedure adopted by some private survey organisations is to use quota sampling. In this approach quotas are set, usually according to sex and broad age groups, and interviewers are instructed to fill their quotas as best they can for the randomly selected household. Since the households have been selected using probability sampling this method is a type of probability sampling with quotas (Sudman, 1966). It is also used commonly in field household surveys. Discussion with some major survey organisations reveals that all the methods described here are in use in Australia.

4. Non-Response

All surveys have a degree of non-response which is usually high initially,


but by following up initial non-respondents the final level of non-response can be reduced considerably. An effective follow-up phase involves extra cost and time, but an advantage of telephone surveys over face-to-face household surveys is the relative cheapness of follow-up. To produce results quickly, only limited follow-up is undertaken in some telephone surveys.

Non-response can be broken into three categories: a) non-contact: persons associated with selected telephone numbers but with

whom an interviewer never makes contact; b) refusal: once contacted the selected person refuses to participate in the

survey; and c) incompetent interviewees: people who are unable to provide the survey

information because of illness, physical or mental disability or language dif-

Non-respondent units are selected in the sample but not measured, whereas non-covered units have no chance of selection because their household has no telephone or has an unlisted number.

Response rates depend on many factors such as the topic of the survey, the length of the interview, the organisation conducting the survey, the initial contact procedures used, the number of call attempts and length of the interview period. Non-response is 'higher for voluntary non-government surveys than compulsory government surveys. Advance letters may increase the response rate by between 5% and 13% (Traugott et d., 1987), but increase costs and can only be used for samples selected from telephone directories. Response rates appear to be higher in face-to-face surveys, although to make a fair comparison we need to allow for the extra costs of follow-up in such surveys. Wiseman 8z McDonald (1979) (reported by Groves, 1989 p.155) shows that the median non-contact rate of 182 telephone surveys conducted by 32 survey firms was 39.1%, and the median refusal rate was 28% of contacts (i.e. 17.1% of sample). Response rates of four telephone surveys in the UK varied between 46% and 65%, compared with 60% to 73% for equivalent face-to-face surveys (Collins et d., 1988). In the telephone surveys, the refusal rate varied between 19% and 38%, the non-contact rate was between 6% and 11% and other categories of response varied from 1% to 16%. For the survey with highest response rate, the refusal rate was 19%, non-contact rate 7% and other response categories were 9%. In comparing telephone and face- to-face surveys Groves & Lyberg (1988, p.203) note that, for telephone surveys, most non-response is due to refusal. De Leeuw & Van der Zouwen (1988), in a study of 25 investigations comparing telephone and face-to-face surveys, reported average response rates of 69% and 75% respectively.

The non-contact component of non-response can be reduced by conducting the survey on those days of the week and at those times of the day when many people are at home, but not at socially unacceptable times when higher refusal rates are likely.

In a large study using RDD in the USA, Kulka & Weeks (1988) found that

ficulty.


the percentage of first calls answered was highest (73%) on weekday evenings (5.00 pm to 10.00 pm), compared with 60% for weekday mornings and 59% for weekday afternoons. Contact rates were higher on Monday, Tuesday and Wednesday evenings than on Thursday and Friday evenings. Sunday evening had a contact rate of 71%, while Sunday afternoon had 62%. Saturday morning and afternoon had contact rates of 68% and 63% respectively. Kulka & Weeks showed that after an unsuccessful first call, contact rates were improved by more appropriate scheduling of the subsequent attempts. Contact rates and costs are also affected by the number of times the telephone is allowed to ring before an unanswered call is terminated. Smead & Wilcox (1980) observed that the majority of telephone calls axe answered within the first five rings. However, for a telephone survey which involves interviewing elderly adults Groves & Ly- berg (1988) suggest that more rings be allowed, to compensate for the higher proportion of people with physical disabilities.

Good survey practice is to use repeated call backs to increase the response rate, though cost and time pressures mean that a limit has to be set on the number of follow-up calls. Drew et al. (1988) examined the number of call attempts needed to maximise the response rate from a study based on the Cana- dian Labour Force Survey. For a sample selected from telephone directories, one call produced a response rate of 54%. That increased to 87% for five attempts, but it took up to nine calls to achieve a response rate of 91% and 17 c d s to reach a response rate of 93%. Sebold (1988) increased the response rate from 74% to 80% by extending the survey period from two to four weeks. Massey et al. (1981) increased the response rate from 74% to 80% by lengthening the survey period by 30 days. They also found a higher household non-response rate (30%) for unanswered numbers recycled during the follow-up period than the non-response rate (22%) for households contacted in the initial data collection period.

The response rate is affected by the method of obtaining respondents from selected households. Cannell et al. (1987) experienced a response rate of 75% when using a randomly selected respondent and 81% when using a knowledgeable adult to provide the information. Sebold (1988), in a survey ip which all household members aged 12 or more were interviewed, found that only 9% of interviews were contacted at the 1st call, 32% by the 3rd call, 48% by the 5th call, 71% by the 10th call and 91% by the 25th c d .

The response rate is also affected by the policy on following-up initial refusals. Many survey organisations report that 25% to 45% of sample cases initially refusing later provide an interview. The success of following-up refusals depends on the strength of the refusal (Groves & Lyberg, 1988 p.209). Alexan- der et al. (1986) reports conversion rates of 30% to 45%. However, Collins et d. (1988) only obtained a 13% response rate for refusals that were recycled for interview.

A proportion of numbers that remain unanswered after a reasonable number


of call attempts may be in unoccupied private households. Groves & Kahn (1979) suggests that only 5% of numbers dialled over 12 times are working household numbers. Common survey practice in Australia is to limit the number of call attempts to between three and five calls.

The bias due to non-response is determined by the response rate and the differences in the characteristics of respondents and ncjn-respondents. Groves (1989 pp.201-208) reviews evidence (not all of it consistent) from USA surveys with a range of approaches and content. Generally, refusal rates seem higher for older and less educated people and those living in smaller households. Groves & Lyberg (1988) review evidence and show that elderly persons have higher refusal rates, and that higher non-response is experienced among the lower educated. A similar, but weaker, tendency is found for age in face-to-face surveys. Cannell et al. (1987) shows response rates of 57% among people 65 years or older in a health survey, compared with overall response rates of 75%. Response rates were lower for males, young people (17-24), older persons (65+), the lower educated, widowed, single and those in the ‘other’ category of economic activity, and non- white. These figures are for a random respondent rule: smaller effects were found when a knowledgeable adult rule was used. Using a random respondent rule, Brown & Bishop (1982) found 7.5% of people over the age of 60 years refused an inteiview, whereas the age brackets of 18-39 and 40-59 registered refusal rates of 1.8% and 3.0% respectively. The effects of age and education on response rates are possibly confounded because elderly people also tend to have a lower educational level. Further analysis of response rates by age and educational level is required to isolate their separate effects.

The characteristics of people contacted in the follow-up phase can be compared with those reached in the initial survey period. This does not give direct evidence about the final non-response, but sheds light on the effect of follow- up. The results depend on what constitutes the initial period. In a study using the National Crime Survey, Sebold (1988) assessed the respondents contacted in an initial survey period of four weeks, compared with those contacted in a follow-up period of a further two weeks. Generally little difference was found, although the average household size of those people who were interviewed in the initial period was 2.6 persons, compared with 2.0 in the follow-up period. The follow-up phase had proportionally more ‘never married’ persons and fewer married persons. Massey et d. (1981) compared the initial respondents with those contacted in the follow-up period and found the two groups to be similar except for a larger proportion of people over the age of 65 years among the follow-up respondents.

Face-to-face surveys also experience non-response. Collins et d. (1988) compared the compositions of a telephone survey and a corresponding face-to-face survey. To remove coverage effects the comparison was done for households with telephones. Overall few differences were found and they concluded there was nothing to distinguish nm-respondents in telephone and face-to-face surveys.


The main effect noted was that telephone surveys appeared better for contact- ing working couples.

5. Study Design

There is little information available concerning telephone surveys in Aus- tralia. To investigate the effectiveness of follow-up and the effects of non- response, a telephone survey of the Wollongong area was conducted in conjunc- tion with the Illawarra Regional Information Service (IRIS). The survey followed the usual procedures used by IRIS, but with more extensive follow-up so we could examine the characteristics of people who would not usually be included in an IRIS survey. IRIS uses a quota sampling method to select a single person from each selected household, so this study also examined the effect of the quota sampling method.

IRIS usually sets quotas based on the most recently available population estimates for the Wollongong area according to sex and the age groups 15-24, 25-39, 40-54 and 55+. To achieve a total sample of, say, 200 respondents, a sample of 500 private household telephone numbers is selected and each of four interviewers is asked to fill a quota of 50 respondents. Thus the quotas are set at the interviewer level.

Usually an interviewer calls a household and determines if there are any persons who belong to any quota which has not yet been filled. If there is exactly one such person then he or she is selected; if more than one such person is available the interviewer usually selects first the person in the most difficult to fill quota. If no people are available for the unfilled quotas, the household is not used and the interviewer rings the next selected telephone number. Interviewing usually stops when the quotas are all filled. In IRIS experience, quotas can be filled in two or three nights of interviewing, provided enough interviewers are used.

For this study, a sample of 500 private household telephone numbers was selected at random from the current Wollongong telephone directory (042 area code). The survey was conducted over an eight-day period from Monday 17 May 1993 to Tuesday 26 May 1993. Interviewers were instructed to obtain the allocated quotas between 5.30 pm and 8.30 pm in the initial caUing period of the first three evenings. Follow-up to complete the interviewing was conducted on the following Saturday and Sunday mornings between 9.00 am and 12 noon, and on Tuesday evening. Interviewers .were instructed to attempt each of the selected telephone numbers at least once in the initial calling period. A particular telephone number was tried at most two times in the same time shift and up to ten calls were made within the time frame of the survey. In the event, 485 of the 500 telephone numbers initially selected were attempted at least once in the initial call period, and analysis was confined to these numbers since we wished to examine the characteristics of households which were attempted but not available in this period.

QUALITY ISSUES IN TELEPHONE SURVEYS

TABLE 3

25

Response status

status first call $4 initial call period % follow-up period % final % interview obtained 54 66 57 73 refusal 14 17 14 18 language problem 2 3 4 3 sick 3 3 0 3 non-contact 28 12 25 3

The survey collected demographic information on age, sex, employment status and country of birth for all the people in the selected household. Persons aged 15 years or more were asked their opinions about issues associated with Australia becoming a republic. It was not considered feasible to interview all persons concerning their opinions. Instead the interviewer called a household and, based on the possible outcomes, chose one of the following options to obtain the sample for the opinion questions:

select the person who answers the telephone if they are within an unfilled quota cell; interview the person who answers the telephone, even if he or she is not within an unfilled quota, and also select and interview another person in the household who is within an unfilled quota; interview the person who answers the telephone, even if no-one who lives at the household comes from an unfilled quota; interview the person who answers the telephone, even if all the quotas have been completely filled.

This design enables comparison of the demographic characteristics of the following samples: i) the household sample, i.e. all persons in the selected households; ii) the quota sample; iii) the sample consisting of the person who initially answers the call. The quota and first person samples can be compared in terms of the opinion questions. The effect of different follow-up strategies can also be examined. For example the sample obtained in the initial calling period (IC) and the sample obtained on the first call attempt can be compared with the final sample.

6. Results

6.1. Response Rates Of the total 485 telephone numbers used in the survey, 472 were working

private household numbers; the remaining 13 (2.7%) were either business, fax, disconnected numbers or households occupied by visitors to the area.

Table 3 summarises the response status for households at the first call, in the initial call period, those contacted in the follow-up (FU) period and the final


TABLE 4 Cumulative contact and response rate by cdl attempt

attempt no. contact rate % response rate % 1 72.8 54.2 2 87.2 65.3 3 91.5 68.9 4 93.0 69.7 5 94.8 71.2 6 95.9 72.0 7 96.3 72.2 8 96.9 72.9 9 96.9 72.9

10 97.1 73.1

response profile. Table 4 shows the number of attempts required before a call was answered.

Most telephone numbers were answered at the first attempt. The contact rate of 72.8% obtained at the first attempt increased to 91.5% at the third attempt. This contact rate gives an estimate of the probability of establishing contact with a selected household on an early weekday evening and is very close to that reported by Kulka & Weeks (1988). The average number of attempts, before a call was answered or the limit of 10 attempts was reached, was 1.76. However, 2.9% of the households sampled could not be contacted after ten attempts were made to call them. The final non-response rate of 27% was composed mainly of refusals and households where language problems were experienced or members were sick. The final response rate of 73.1%, assumes that the 2.9% of household not contacted were in the scope of the survey. Little improvement in response rate was obtained by going past five calls, but the response rate increased from 54.7% to 71.2% in going from one call to five and so, in terms of response rate, it is definitely worth going past the first call. The contact, response and refusal rates are very similar to those experienced in the US and Britain as reviewed in Section 4. The 56 households contacted in the follow-up period included 11 households contacted in the initial call period who had indicated that it was an inconvenient time to be surveyed. Of these, six gave an interview. This shows that it is worth following up such cases. The refusal rate in the follow-up period was no higher than in the initial call period.

There were 345 responding households containing a total of 967 people, 768 of whom were 15 years of age or more. Ninety one percent of the total quota sample was obtained on the first call and 99% by the third call. The fineness of the quota cells influences the difficulty of filling the specified quotas.

6.2. Biases in the Household Sample Taking into account the response rate, the loss of coverage due to unlisted

and non-connected households and the omission of non-private dwellings the survey effectively covers 58% of the population. Comparison of the demographic


variables for the sample of all persons in responding households with the 1991 Census of Population and Housing for the Wollongong Statistical District enables an assessment of the net effect of non-coverage and non-response biases. Using a strict probability sampling scheme to select respondents from households would produce this sample in expectation. In examining these comparisons it must recognised that there may also be some effect due to the 17 months’ time difference between the census and the survey and differences in the data quality of the collection methods. However, these comparisons should be capable of indicating major differences. Major changes in the demographic composition of the area are unlikely to occur in this period. However, some changes can occur in the composition of the labour force in such a period. The results of this survey were also compared with those of the labour force survey for May 1993 (Table 5).

The distribution of the variables in the telephone survey were compared with the census and the labour force survey using chi-squared tests. The differences between the survey and the census were statistically significant, except in the case of gender. As expected, the household sample under-represents smaller households due to a combination of lower connection rates and the smaller chance that at least one person would be at home at the time of the call. There is also some under-representation of larger households. Children and the older age groups are under-represented and persons aged 40 to 54 are over-represented. Many surveys are confined to the population over 14 and so are not affected by under-representation of children. Table 2 shows this group is also under- represented in households with a telephone connection. Table 5 indicates a small over-representation of the Australian-born and under-representation of the British-born. The under-representation of British-born may be due to the topic of republicanism leading to a higher refusal rate for this group. The differences found we larger than can be explained by the differential telephone coverage of these groups in the population.

Biases were found for employment status: unemployed and part-time work- ers were significantly under-represented. When the full-time and part-time employment categories are combined the difference becomes smaller. It is possible that the different collection methods lead to differences in respondents’ under- standing of what constitutes full-time work. These differences are consistent in direction with the telephone connections data, but are larger than can be explained by this factor alone. The main effect is that the telephone survey has under-represented the unemployed and, therefore, the labour force, and over- represented those not in the labour force. The under-representation of the older age group suggests that the over-representation of those not in the labour force occurs for those of working age.

Many of the differences found, while statistically significant, may not be important for many surveys, depending on the purposes of those surveys and the accuracy required. (The chi-squared tests could be affected by the clustering of some characteristics within households. However, the results stay statistically


variable full sample % population census % relative bias % 14.5 18.2 -20.3 household

TABLE 5 Comparison of full sample and population

census, Labour Force Survey May '93

size

gender

age

country of birth

age

employment status

1

2 3 4 5 6+

male female 0-14 15-24 25-39 40-54' 55 and over Australia Britain other English-speaking European other

34.2 20.3 20.9 8.1 2 .o

all persons 50.8 49.2

18.8 15.5 25.3 21.1 19.3

75.6 6.7 2.7 6.2 8.8

31.0 17.6 19.4 9.6 4.2

50.0 50.0

22.4 15.5 23.3 18.1 20.7

74.1 9.0 1.3 6.7 8.8

persons 15 years and over 15-24 19.1 20.0 25-39 31.1 30.1 40-54 26.0 23.2 55 and over 23.8 26.6

employed full-time 45.7 38.1 employed part-time 7.6 13.7 total employed 53.4 51.0

total labour force 56.7 60.0 not in labour force 43.3 38.6 unemployment rate 5.8 13.7

unemployed 3.3 8.2

+10.3 +15.3

+7.7 -15.6 -50.4

'+1.6 -1.6

-16.1 -0.1 +8.3

+16.6 -6.4

+1.9 -25.2 +99.5

-7.3 -0.5

-4.4 +3.4

+11.4 -10.5

LFS Survey 43.6 11.5 55.1

8.6 63.7 36.3 13.4

significant even when reasonable allowance is made for the potential effect of this clustering.)

6.3. Non-Response Effects I

The household response rate at the first call was 54% and in the initial calling period it was 66%. The contact rates were 73% at the first c d , and 88% during the initial calling period. The improvements in contact and response rate in continuing the survey after the first call or the initial calling period add to the cost of the survey and the time to produce the results. An analysis was


TABLE 6 Household sample, calling-period effects for country of birth,

all persons

country of birth IC period % FU period % final sample

Australia 75.5 77.1 75.6 Britain 7.2 1.2 6.7 other English-speaking 2.9 0.0 2.7 European 6.4 3.6 6.2 other 7.9 18.1 8.8

TABLE 7 Comparison of first call and final household sample,

all persons

employment status

country of birth

variable first call sample % final sample % employed full-time 37.1 37.1 employed part-time unemployed school student tertiary student home duties pensioner other Australia Britain other English speaking European other

6.5 2.9

10.0 3.2

10.4 16.6 13.3

75.1 7.0 2.9 6.0 8.9

6.2 2.7

10.4 3.6

10.6 16.0 13.2

75.6 6.7 2.7 6.2 8.8

conducted to see if the additional respondents surveyed by going past the first call attempt or the initial calling period had any significant effect on the survey results.

No statistically significant differences were found between the persons in households surveyed in the initial calling period and the follow-up period in terms of age, gender or employment status. However, there was a small but statistically significant difference for country of birth, with the proportion born in non-English-speaking or non-European countries being higher in the follow-. up period. This may be due to additional follow-up being required because of language problems. While the difference is statistically significant the actual difference is not large because the follow-up period accounts for only 8.6% of responding households (Table 6).

Twenty-five percent of the responding household sample was obtained after the first call, but, again, no statistically significant differences were found between the age, gender or employment status of households surveyed at the first calling attempt and those surveyed in later calls. In fact the similarity of the


TABLE 8 Comparison of samples, demographic variables, persons 15 years and older

variable first person quota household population sample % sample % sample % census (LFS) %

age

gender

employmen t status

country of birth

15-24 25-39 40-55 55 and over male female employed full-time employed part-time total employed unemployed total labour force not in labour force unemployment rate Australia Britain other English-speaking European .

15.7 33.0 23.5 27.8

45.8 54.2

39.7 8.4

48.1 3.2

51.3 48.7

6.2

71.9 10.7 2.6 4.6

18.6 32.7 24.1 24.6

50.8 49.2

44.7 9.1

53.8 3.0

56.8 43.2

5.3

66.8 12.6 3.0 5.5

19.1 31.1 26.0 23.8

51.7 48.3

45.7 7.6

53.4 3.3

56.7 43.3

5.8

71.2 8.3 3.2 *

7.4

20.0 30.1 23.3 26.6

50.0 50.0

35.6 (43.6) 12.8 (11.5) 51.0 (55.1) 8.1 (8.6)

59.1 (63.7) 39.4 (36.3) 13.7 (13.4)

74.1 9.0 1.3 6.7

other 10.1 12.1 9.9 8.8

samples obtained in the first calls and the find calls is remarkable (Table 7). The results are similar if the analysis is confined to those aged 15 years and over. The results suggest that any biases in telephone surveys may be mainly due to non-coverage and refusal rather than non-contact.

6.4. Effect of Person Selection Method For the demographic variables, the effect of taking the first person to answer

the call as the selected person, or of using the quota sampling method, can be compared with the sample consisting of all persons in the households and the population census. Good survey practice does not normally allow selection of the first person, but in some cases, pressure of time available to obtain a specified sample size, or poorly controlled quota sampling, may lead to the selection method being close to this method. Comparison with the household sample iso- lates the effects due to the method of respondent selection. Since we selected persons 15 years and over in the quota or first person sample we confine the comparisons to this group.

The opinion questions were only put to the selected persons and can only be compared for the first person and quota samples (Tables 8 and 9). There is some overlap in the samples, with the first person sample comprising 45% of the adults in the household sample and the quota sample comprising 53% of the first person sample.

I

QUALITY ISSUES I N TELEPHONE SURVEYS 31

TABLE 9 Comparison of samples, opinion variables, persons 15 years and over

variable first person sample % quota sample % Awareness of Yes 81.7 85.9 republican issues no 18.3 14.1

support Y e s 55.4 for repu bEc no 29.0

don’t know 15.7

support for Yes 40.3 current Aag no 52.5

don’t know 7.3

support for Y e s 76.2

British Commonwealth don’t know 14.5 staying in no 9.3

56.3 31.2 12.6

43.7 51.3 , 5.0

77.4 9.6

13.1

As expected the quota sample was representative in terms of age and sex, whereas the first person sample under-represented 15-24 year olds and males. For employment status the quota sample corresponds very well to the household sample. The first person sample surveyed fewer employed persons than did the quota or household samples and over-represented those not in the labour force. The quota sample under-represented the Australian-born and over-represented British-born persons, but if these categories are combined the difference is small. The results suggest that quota sampling is superior to the first person sample and gives a sample similar to the household sample.

For the opinion questions, similar results were obtained, although there was a tendency for the first person sample to have a lower awareness level and more ‘don’t knows’.

We examined the data for any effects of going past the initial call or first call, for both the quota and first person samples. We found no statistically significant differences for the quota sample, although it must be remembered that all the quota sample was obtained in the initial call period and 91% of the quota sample was obtained in the first c d . For the first person sample, few statistically significant differences were found between the final sample and the samples obtained in the initial call period or at the first call. The exceptions were. that a higher proportion of females, people who were unaware of the republican debate or didn’t know whether to change the flag, were found in the follow-up period. There was higher support for changing the flag in the first call sample.

I

7. Discussion

Nearly all the available literature on telephone surveys deals with experience in the USA, Britain or Europe. The study reported here is limited in its geographic scope and in ithe variables considered, but it provides some useful


evidence concerning telephone surveys in Australia. The connection, contact, response and refusal rates are consistent with overseas experience. Analysis of differences between this survey and the Population Census and the Labour Force Survey suggests that telephone surveys may under-represent the young, old and unemployed, and over-represent those not in the labour force and the middle-aged. This does not suggest telephone surveys are inferior to face-to- face surveys, since those can experience similar problems. Moreover, the level of bias may not be important for many surveys. The consistency with overseas experience suggests results of those studies can be used as a guide to Australian responses.

The analysis suggests that there is little or no effect of calling back a large number of times; in fact taking only those contacted at the first call attempt has produced the same results as the final sample obtained after ten call attempts. While we think it would be dangerous to conclude there is no need to call back, the results suggest the current practice of three to five calls is satisfactory.

Biases are mainly due not to non-contact but to refusals and non-coverage. The biases are larger than can be explained by the proportion of non-connections, but since we have no information on the characteristics of households with unlisted numbers we can not attribute them entirely to refusals. We believe that the people who refuse,to participate in telephone surveys are different from the general population, but this may not apply to those who are more difficult to contact.

Information is needed about the characteristics of households with unlisted telephone numbers so researchers can evaluate the improvement in coverage that could be achieved by using RDD. The quality of telephone surveys should be improved by finding ways of reducing the level of refusals.

References

AUSTRALIAN BUREAU OF STATISTICS (1991). Household Telephone Connections, Australia,

- (1992). Social Indicators, Australia, August 1992. Catalogue No.4101.0. - (1994). Unpublished Tables, Household Telephone Connections Survey, Australia, August

1991. '

ALEXANDER, C., SEBOLD, J. & PFAFF, P. (1986). Some results with an experiment with telephone sampling for the US Nationai Crime Survey. In Proc. of the Section on Survey Research Methods, pp.351-356. Washington: Amer. Statist. Assoc.

BROWN, P.R. & BISHOP, G.F. (1982). Who Refuses and Resists in Telephone Surveys? Some New Evidence. University of Cincinnati.

BRYANT, B. (1975). Respondent selection in a time of changing household composition. J. Marketing Res. 12, 129-135.

CANNELL, C.F., GROVES, R.M., MAGILAVY, L., MATHIOWETZ, N. &z MILLER, P.V. (1987). An Experimental Comparison of Telephone and Personal Health Surveys. Health Centre for Health Statistics, Technical Series 2, No 106.

Methodology 19, 103-113.

August 1991. Catalogue No.4110.0.

CASADY, R.J. & LEPKOWSKI, J.M. (1993). Stratified telephone survey designs. Survey


COLLINS, M. & SYKES, W. (1987). The problem of non-coverage and unlisted numbers in

-, -, WILSON, P. k BLACKSHAW, N. (1988). Nonresponse: the UK experience. In R.M.

DE LEEUW, E.D. k VAN DER ZOUWEN, J. (1988). Data quality in telephone and face to

DREW, J.D., CHOUDRY, G.H. k HUNTER, L.A. (1988). Non-response issues in government

GROVES, R.M. (1989). Survey Errors and Survey Costs. New York: John Wiley. - k KAHN, R.L. (1979). Surveys by Telephone: A National Comparison With Personal

- & LYBERG, L. (1988). An overview of non-response issues in telephone surveys. In R.M.

-, BIEMER, P.B., LYBERG, L.E., MASSEY, J.T., NICHOLS, W.L. k WAKSBERG, J. (Eds)

KISH, L. (1965). Survey Sampling. New York, John Wiley. KULKA, R.A. & WEEKS, M.F. (1988). Toward the development of optimal calling protocols

for telephone surveys: a conditional probabilities approach. J. Official Statist. 4, 319-332. MASSEY, J.T., BARKER, P.R. k HSIUNG, S. (1981). An investigation of response in tele-

phone surveys. In Proc. Section on Survey Research Methods, 426-431. American Statistical Association.

SORENSON, S.B. & TUCHFARBER, A.F. (1988). A comparison of the Kish and Last Birthday method of respondent selection in telephone surveys. J. Official Statist. 4, 307-318.

SEBOLD, J. (1988). Survey period length, unanswered numbers, and non-response in telephone surveys. In R.M. Groves et al. (1988), 247-256.

SMEAD, R.J. k WILCOX, J. (1980). Ring policy in telephone surveys. Public Opinion Quar- terly 44, 115-116.

SMITH, A.M.A, DENNERSTEIN, L., MORSE, C.A., HOPPER, J.C. k GREEN, A. (1992). Costs and benefits of the use of commercial market research approaches in large scale surveys. Med. J. Austral. 157, 504.

SMITH, T.W. (1987). Phone home? An analysis of household telephone ownership. In Internat. Conf. Telephone Survey Methodology, Charlotte, November 1987.

STEEL, D. & BOAL, P (1988). Accessibility by telephone in Australia: implications for telephone surveys. J. Official Statist. 4, 285-297.

SUDMAN, S. (1966). Probability sampling with quotas. J. Amer. Statist. Assoc. 61, 749-771. SYKES, W. k COLLINS, M. (1987). Comparing telephone and face-to-face interviewing in the

THORNBERRY, 0.". & MASSEY, J.T. (1988). Trends in US telephone coverage across time

TRAUGOTT, M.W, GROVES, R.M. & LEPKOWSI, J. (1987). Using dual frame designs to

TREWIN, D. k LEE, G. (1988). International comparisons of telephone coverage.

TROLDAHL, V.C. & CARTER, R.E. Jr. (1964). Random selection of respondents within house-

WAKSBURG, J. (1978). Sampling methods for random digit dialling. J. Amer. Statist. Assoc.

WISEMAN, F. k McDONALD, P. (1979). Non contact and refusal rates in consumer telephone

telephone surveys in Britain. J. Roy. Statist. SOC. Ser. A 50, 241-253.

Groves et al. (1988), pp.213-232.

face surveys: a comparative meta analysis. In R.M. Groves et al. (1988), pp.283-300.

telephone surveys. In R.M. Groves et al. (1988), pp.233-246.

Interviews. New York: Academic Press.

Groves et al. (1988), pp.191-222.

(1988). Telephone Survey Methodology. New York: John Wiley.

OLDENDICK, R.W., BISHOP, G.F.,

UK. Survey Methodology 13, 15-28.

and subgroups. In R.M. Groves et al. (1988), pp.25-50.

reduce non-response in telephone surveys. Public Opinion Quarterly 51, 522-534.

In R.M. Groves et aJ. (1988), pp.9-24.

holds in phone surveys. J. Marketing Res. 1, 71-76.

73, 40-46.

surveys, J. Marketing Res. XVI, 478-484.


WYLLIE, A., BLACK, S., ZHANG, J. F. & CASSWELL, S. (1994). Sample frame bias in telephone-based health research in New Zedand. New Zealand Statist. 29, 40-53.

QUALITY ISSUES IN TELEPHONE SURVEYS: COVERAGE, NON-RESPONSE and QUOTA SAMPLING

Documents

Transcript of QUALITY ISSUES IN TELEPHONE SURVEYS: COVERAGE, NON-RESPONSE and QUOTA SAMPLING