Pre-testing the ESS-questionnaire using interaction analysis€¦ · Pre-testing the...

Pre-testing the ESS-questionnaire using

interaction analysis

Yfke Ongena

Department of Social Research Methodology

Faculty of Social Sciences, Vrije Universiteit Amsterdam

De Boelelaan 1081c, 1081 HV Amsterdam, The Netherlands

E-mail: [email protected]

2

Introduction

The goal of survey interviews is to obtain answers that reflect the actual attitudes and

behaviours of respondents. However, the scores may be distorted by the questionnaire design,

respondent or interviewer characteristics. By means of studying the verbal interaction in the

interview some of these distortions may be detected.

Especially in case of evaluating the quality of questions, interaction analysis is relevant.

Interaction analysis, as it examines verbal behaviours provides information about the

interviewer’s presentation of the question. When interviewers often change the wording of

certain questions, this might be an indication of problems with these questions, for example

because they are too long or too complicated. Interaction analysis is also useful to detect

question order-problems. In some cases, due to a certain question order the information

required by the question may already have been provided before a target question was posed.

Therefore interviewers may tend to skip questions (and fill the scores in themselves without

asking the question).

The formulation of the respondent’s answer may be even more informative. When

respondents give inadequate answers, this may indicate for example a problem in the design

of the question or it’s response alternatives. Furthermore, interaction analysis also yields

information in the form of verbal expressions of the respondent when processing the

question, as a reaction to the question. For example, when the respondent asks for

clarification, the question might be ambiguous, and when respondents provide verbal

considerations before or even instead of a proper answer, the question might be requiring too

much cognitive effort to answer.

Interaction analysis as a research method involves several steps. First the audio data is

transcribed and simultaneously divided into Question answer sequences (Q-A sequences),

that is all utterances of the interviewer and respondent that belong to one question. Next each

Q-A sequence is divided into meaningful utterances of interviewer and respondent, and

finally each utterance is coded, preserving the sequence of utterances.

For transcribing, coding and analysing the sequences we used the sequence viewer program

(Dijkstra, 2002). Figure 1 shows one so-called Q-A sequence in a sequence viewer-file. In the

large field the transcriptions can be seen. To the left of this field, the codes appear before

each utterance. We used a coding system consisting of a number of variables, each describing

a particular aspect of the utterance. When all utterances in all Q-A sequences are coded it is

3

possible to analyse the coded sequences in order to identify causes and effects of problematic

behaviours. Problematic behaviours can be produced by either the respondent or the

interviewer.

Figure 1

Screenshot of the Sequence Viewer program

4

Respondent problematic behaviours

We define respondent problematic behaviours as utterances by the respondent that require

action by the interviewer, in order to explain the respondent’s task. By far the most frequently

occurring respondent problematic behaviour is a mismatch answer. This is an answer that is

not directly scorable but at best resembles the answer alternatives. As is illustrated by the

following example ‘most days’ is considered as a mismatch answer, as the original question

required an exact number between zero and seven.

Fragment 1

QA sequence containing a mismatch answer

I: How many days a week do you watch

television on average?

R: Well, most days

Interviewer problematic behaviours

Interviewer problematic behaviours concern utterances by the interviewer that are expected to

affect the validity of eventual response negatively. Suggestive probing, i.e. the interviewer

suggests an alternative that is not warranted by previous utterances of the respondent, is a

frequently occurring interviewer problematic behaviour. Apart from suggestive probing, a

frequently occurring behaviour is ‘choosing’. This means that the interviewer enters a

response alternative although the respondent did not give an adequate answer or did not even

answer at all. Because the interviewer does not mention what is entered, the respondent does

not have an opportunity to correct the response chosen by the interviewer. This makes

choosing a worse problem than suggestive probing, which in principle offers possibilities for

repair.

We can use interaction analysis of these problematic behaviours to evaluate survey questions.

When a lot of problematic sequences take place with respect to a particular question, this is

an indication of certain problematic aspects of the question. In this report the questions of the

pilot study of the European Social Survey were evaluated by using interaction analysis.

5

The Dutch ESS pilot study

The interviews of the Dutch pilot study of the European Social Survey concern face-to-face

interviews that were conducted in the spring of 2002. The questionnaire consisted of 268

CAPI questions, and an additional CASI questionnaire. The CAPI interviews were taped and

digitised and coded by three different coders. Reliability of the coding, based upon 1100

sequences coded twice by two of the coders, yielded a Kappa of 0.70. The dataset eventually

coded and used for analysis was checked for unlikely and rare codes, therefore the reliability

of this dataset is probably higher.

It appeared that only 23 interviews with a good recording quality were available for

coding. Due to time constraints, not all 268 questions of all interviews were transcribed.

Eight interviews were transcribed completely, for the other 15 interviews at least the first 100

questions of the interview were coded.

In table 1 the numbers of sequences that were coded and that were not coded for

various reasons are depicted. It appears that the number of incorrectly skipped questions is

quite high, considering the fact that the survey was administered on a computer. The high

percentage is partly explained by the behaviour of one interviewer in a single interview. This

interviewer approached his instructions to be an interviewer in his own peculiar way. He did

not read all of the questions, but instead let the respondent complete the questionnaire

himself, only assisting when the respondent faced difficulties. Therefore, from this interview

only for the first few sequences, and for some problematic ones verbal utterances were

available. It is not clear if this interviewer dealt with all interviews in this way, but the fact

that only this one interview from this interviewer was recorded did strongly suggest that.

Table 1

Number and percentage of QA sequences coded in the ESS-data

Number of sequences: percentage:

Asked and coded 3684 59.77%

Correctly skipped 353 5.73%

Incorrectly skipped 250 4.06%

Part of interview not on tape 87 1.41%

Not coded 1790 29.04

6164 100.00

6

We divided the QA sequences into 4 types of sequences. The first type was a paradigmatic

sequence. This is a sequence that evolves as intended by the researcher: the interviewer poses

the question adequately, the respondent immediately gives an adequate answer, and the

interviewer may before posing the next question, acknowledge the answer by saying ‘yes’,

‘mhm’ or repeating the answer. Non-problematic sequences develop along the same pattern,

but some non-problematic deviations may occur, for example the respondent may ask for

repetition of the question, a long silence may occur etc. In the remaining two types of

sequences, a problematic deviation occurs. The difference between these two types, however,

is that in the first, this problematic deviation is solved. For example, after giving a mismatch

answer, the respondent finally gives an adequate answer, because the interviewer probed non-

directively. In the last type of sequence this problematic deviation is never solved, for

example, the respondent never gives an adequate answer. Sequences that contain suggestive

probing were always categorized as problematic sequences, because it is practically

impossible to solve the problem of suggestive probing. Once the interviewer has suggested a

particular response, this response has become more easy to choose for the respondent, and the

respondent will think of this response as more appropriate, even if the interviewer will give

suggestions of all other responses afterwards.

We analysed the occurrences of the 4 types of QA sequences in the ESS interviews. As is

depicted in table 2, more than half of the sequences were paradigmatic, and over 78% of the

sequences was either paradigmatic or non-problematic. We were curious whether certain

question characteristics would have caused higher percentages of problematic sequences.

7

Table 2

Types of sequences for questions with and without show cards

percentage of sequences

for questions without

show cards

percentage of sequences

for questions with show

cards

Paradigmatic QA sequence 59.76 50.76

Non-problematic deviation in QA

sequence

18.67 27.90

Solved problematic deviation in

QA sequence

5.95 6.56

Non-Solved problematic deviation

in QA sequence

15.62 14.78

Total 1312 2362

Cramer’s V = 0.11, p <0.01

Different question types in the ESS data.

The 268 ESS Survey Questions can be divided into several question types. Some important

discriminators are:

-The type of question proper (open or closed question, for closed questions: choice question,

assertion, or yes-no question)

-Whether a show card was used

-Whether response alternatives were listed or implicit, and whether the interviewer was

instructed to read them (within the question or after the question, reading all or a subset of

questions)

-Whether the question was preceded by an introduction or definition (either or not including

examples of response alternatives). This latter distinction was not very important, it merely

indicates that the question is the first of a series of similar questions.

Three questions (A2, H2 and H4) concerned questions that were hardly ever asked (i.e. the

gender of the respondent and country of birth. It appeared that 60% of the questions in the

ESS are questions with use of a show card. From earlier studies it appeared that in QA-

sequences concerning questions with show cards mismatch answers occur significantly less

8

frequent than in QA-sequences concerning question without the use of show cards. We were

curious whether this was also the case for the ESS.

As can be seen from the first row of table 3, there appears to be only a small, hardly

significant difference with respect to the percentage of problematic sequences for questions

that use show cards and questions that do not use show cards.

However, show cards may have been used for apparently the most problematic questions. For

all assertion questions for example, show cards were used. When we ignore the assertions,

and only look at the closed questions (second row of table 3) we can still see a positive effect

from show cards; closed choice questions with show cards yield only 12,15% problematic

sequences whereas questions without show cards yield 24.44% problematic sequences.

Show cards may also have negative effects. A negative aspect of the use of show cards, is

that, although the response task is easier for the respondent, the use of show cards may be

tiring. Interviewers continually have to check whether the right show card is presented.

Furthermore, when questions were just yes-no questions the respondents had nevertheless

gotten used to the use of show cards and waited for their instruction to pick a show card or

the interviewer read out the possible answer alternatives.

Table 3

Percentage of problematic sequences for all and closed choice questions only,

with and without the use of show cards

Percentage of problematic sequences:

No show card

Show card

All questions 15.61% 14.82%

Cramer’s V = 0.10, p< 0.01

Closed choice questions

24.44%

12.15%

Cramer’s V = 0.16, p <0.01

Show cards may stimulate respondents to be more focused on the task, and therefore yield

less problematic behaviours, but more requests for clarification. In an earlier study on

different datasets we also found that more requests for clarification occurred in QA sequences

9

concerning questions with show cards (Dijkstra & Ongena, 2002). However, it is also

possible that respondents ask for clarification about which show card to use, although the

question is clear to them. This is caused by the fact that, as in many surveys, respondents are

asked to pick out the appropriate show card after the interviewer’s request. Reading show

cards may also distract the respondents from accurately listening to the question.

However, overall we find that show cards make the respondent’s task far more easier.

The respondent has to choose one of the alternatives that are presented to him or her visually.

It is likely that for different questions without show cards there might be differences with

respect to the percentage of problematic sequences.

Problematic sequences: fitting questions and answer alternatives

In table 4 the percentage of problematic sequences for questions without show cards is

depicted. In this case closed choice questions and yes-no questions are compared for the

answer alternatives that accompanied the question.

Table 4

Percentage of problematic sequences for questions without show cards

with different types of answer alternatives

Choice Question Yes-No Question

Implicit alternatives

28% (of 277 sequences)


Listed alternatives

(not read)



Alternatives read within Q


-

Alternatives read after Q



Cramer’s V

0.11

(p<0.01)

0.33

(p<0.01)

There appears to be a significant difference for the alternatives used with respect to the

percentage of problematic sequences. Especially for yes-no questions with implicit

10

alternatives, the percentage of problematic sequences was low. The implicit alternatives were

just yes and no, and the question formulation perfectly fitted these implicit alternatives. For

yes-no questions with answer alternatives after the question the percentage of problematic

sequences was far higher. These questions are therefore labelled as improper yes-no

questions, because not only ‘yes’ and ‘no’ are prescribed alternatives, but also specifications

of ‘yes’, as you can see in example 1.

Example 1

ESS question F8:

Are you hampered in your daily activities in any way by any longstanding illness, or

disability, infirmity or mental health problem?

1 Yes, a lot

2 Yes, to some extent

3 No

8 (Don’t know)

Although the question is formulated as a yes-no question, the answer alternatives that are

listed included not only ‘yes’ and ‘no’, but also specifications of ‘yes’. These specifications

were not indicated by the question, and therefore the question relies on good interviewer

behaviour to probe for this specification in case the respondent answers the question with

‘yes’.

As you can see in fragment 1 interviewers do not probe, even if the respondent does not

indicate a clear yes. This interviewer scored a ‘no’. In the final questionnaire F8 is still

included in the same formulation.

Fragment 1

QA sequence concerning ESS question F8

I: Are you hampered in your daily activities in any way by any

longstanding illness, or disability, infirmity or mental health problem?

R: At this age I am

I: hihi

11

Example 2 shows another improper yes-no question. Here the interviewer is instructed to

probe (which appears to be a rather cryptic instruction “PROMPT IN RELATION TO

PRECODES” ). Similar improper yes-no questions are ESS questions J48 and L15.

Example 2

ESS question J47:

Do you have any friends who have come to live in Britain from another country?

PROMPT IN RELATION TO PRECODES

Yes, several 1

Yes, a few 2

No, none at all 3

(Don’t know) 8

Occurrence of specific problematic behaviours

After having noticed the differences for problematic sequences, we were curious which

specific behaviours were causing the problematic sequences. We analysed the percentage of

occurrence of several different respondent and interviewer problematic behaviours in the ESS

data. As you can see in table 6, there is a difference in the frequency of occurrence of the

individual respondent behaviours.

Table 6 Percentage of sequences during which different respondent

problematic behaviours occur

Type of respondent problematic behaviour Percentage of occurrence

Mismatch answer 12.63%

Request for clarification 3.95%

Invalid answer 1.20%

Don’t know answer 0.82%

Qualified answer 0.44%

12

Occurrence of mismatch answers and different question types

As indicated above, mismatch answers occur in more than 12% of the Q-A sequences. This

number is low compared to surveys in general. For example, in the five telephone and face-

to-face surveys analysed by Dijkstra and Ongena (2002) the percentage of mismatch answers

ranged between 12.1% and 31.4%. In face-to-face interviews the number of mismatch

answers can be lower because of the use of show cards. In this survey a show card was used

for 60% of the questions. Although we would expect a difference with respect to the

occurrence of mismatch answers between questions with and without show cards, no

significant difference could be found. Mismatch answer occur in 9.87% of the Q-A sequences

concerning questions with show cards (n=2218), and in 10.64% of the Q-A sequences

concerning questions without show cards. However, some differences in the occurrence of

mismatch answers exist within questions without show cards.

The highest occurrences of mismatch answers could be found for yes-no questions that we

earlier labelled as improper. Apart from those questions many mismatch answers appear to

occur in Q-A sequences concerning closed questions without show cards with listed

alternatives that are read after the question. This is not surprising, as it is advisable to read

the alternatives within the question, to avoid the respondent from interrupting the interviewer

before all answer alternatives are read (Houtkoop-Steenstra, 2000). No mismatch answers

occur in QA sequences concerning open questions, as in practice it is by definition

impossible to give a mismatch answer for questions without specified answer alternatives.

Hardly any mismatch answers occur in QA sequences concerning yes-no questions.

Fragment 5 illustrates the occurrence of a mismatch answer for a question with a show card

(which is depicted in example 3). After the interviewer’s very subtle ‘acceptance’ of the

respondent’s offering interpretation of the question, the respondent gives a mismatch answer.

An interesting aspect of this question is that response category 5 is NOT indicating a

frequency. In fact this response option is suggesting a frequency (i.e. being less often than

once a month, but being more often than ‘less often’. However, it might be the case that

special occasions are more frequent than once a month, therefore ‘less often’ is even more

difficult to interpret. The numbers that accompany response alternatives are also confusing.

Suppose the respondent replies with “two times a month’. The interviewer would be tempted

to type ‘2’, and may have done so. Furthermore it is rather strange and confusing that the

question instructs respondents not to take ‘special occasions such as weddings, baptisms,

13

funerals, festivals and circumcisions’ into consideration when counting their frequency of

attending religious services, but ‘only on special holy days’ are nevertheless a response

option.

Example 3

ESS question G6:

CARD G2 :

Apart from special occasions such as weddings, baptisms, funerals, festivals and

circumcisions, about how often do you attend religious services these days?

Every day 1

More than once a week 2

Once a week 3

At least once a month 4

Only on special holy days 5

Less often 6

Never 7

(Don’t know) 8

Fragment 5

I: Uh, Apart from special occasions such as wedding baptisms, funerals, festivals and

circumcisions, I: about how often do you attend religious services these days?

R: well, it depends whether you take Masonry as a religious convention

I: yes

R: in practice hardly ever

R: on a weekly base I guess

14

Occurrence of Interviewer problematic behaviours

As we saw on the previous examples, interviewers often try to repair mismatch answers with

suggestive probing. This is therefore a frequently occurring interviewer problematic

behaviour. However, as the frequency of mismatch answers was low as compared to surveys

in general, the frequency of suggestive probing is also not so high, as can be seen in table 7.

Table 7 Percentage of sequences during which different interviewer

problematic behaviours occur

Type of interviewer problematic behaviour Percentage of occurrence

Suggestive probing 8.38%

Insufficient probing (choosing) 6.75%

Invalid question 1.20%

Another frequently occurring interviewer problematic behaviour is insufficient probing, or

‘choosing’, which occurs in 6.75% of the 3675 sequences. This problem is often caused by

the fact that respondent gave mismatch answers. Interviewers often neglect to probe until

they get an adequate answer, i.e. they just infer an adequate answer from the mismatch

answer. However, in almost 5% of the 3675 sequences it could not be judged whether the

interviewer probed insufficiently, as in those cases the answer of the respondent was not

perfectly audible. It is possible that the answer was audible for the interviewer, or the

respondent gave an answer non-verbally, for example by literally pointing at the relevant

answer on a show card. However, it appeared that non-audible answers did not occur more

frequently for questions with show cards.

Example 4

ESS questions G1-G4

G1: Do you consider yourself as belonging to any particular religion or denomination?

G2: Which one?

G3: Have you ever considered yourself as belonging to any particular religion or

denomination?

G4: Which one?

15

Example 4 depicts questions G1 to G4. In the ESS questionnaire, Question G4 appeared to

yield 8 problematic sequences out of 9, mainly because of suggestive probing and choosing

by the interviewer. The problem is caused by a question order problem and the problem of

field coding, that is, asking a choice question without presenting the alternatives to the

respondent. This, as was earlier shown by Smit (1995), very often causes interviewers to

probe suggestively.

If the respondent answered question G1 with yes, they also answered question G2. In case of

a yes-answer for G1, Question G3 seems to be redundant, but interviewers are instructed to

ask the question in all cases. However, when question G4 is about to be asked the

interviewers realise the redundancy and start to probe suggestively, suggesting the alternative

that was chosen as answer for question G2. However, in this case, the question is not as

redundant as it may seem, because a religion or denomination that respondents considered

themselves to belong to may be different from the religion they currently consider themselves

to belong to.

An easy way to reduce the effects of redundancy and supposed redundancy is never ask

questions G3 and G4 to respondents who answered G1 with yes. In the final questionnaire

this routing-instruction was included. It appears that the survey is not intended to collect

information about the number of religions that respondents may have considered themselves

to belong to, and their shifts in religious course of life. If, however this information is

relevant, for respondents who replied with a confirming answer to question G1, question G3

could be reformulated as ‘Have you ever considered yourself as belonging to any OTHER

particular religion or denomination?’, and question G4 could be reformulated as ‘What other

religions and denominations?’ (allowing multiple answers).

Invalid answers and questions

As tables 6 and 7 showed, both invalid questions and invalid answers occur in 1.20% of the

cases. However, they do not necessarily occur in the same sequences. In 30% of the cases,

invalid answers and invalid questions occur in the same sequence, which seems logic,

because when a respondent answers the interviewer’s invalid question, this answer will be

invalid as well.

16

An invalid answer is an answer to a question that is misunderstood by the respondent, as far

as can be determined by the interaction. For example, the question with the highest amount of

invalid answers, concerned participation in elections. It appeared that respondents did not

answer the question ‘which party they voted for with respect to the last GENERAL election’,

but with respect to the last election for their local community. The context of the interview

played an important role here: the CAPI interviews were conducted just after the local

elections. Therefore respondents, who were not paying attention to the phrase “the last

General election’, were answering the question invalidly.

An invalid question posed by the interviewer significantly alters the meaning of the

question, in the sense that the response to the question, may have been different if the

question was posed as worded in the questionnaire. It appeared that most invalid questions

occurred with respect to the questions considering work or previous work.

In the ESS interviews, after the interviewer posed the question “In what year were you last in

a paid job?’, the interviewer was instructed to change the wording of the subsequent

questions on the respondent’s professional status (L9 to L21) into either questions phrased in

present tense (if respondents were currently in paid work) or past tense (when respondent

were not in paid work, but had a job in the past). However, this instruction seemed too

difficult to apply. As interviewers are thoroughly trained to read questions exactly as worded,

most interviewers often did not adapt the question wording, and just read the question as

worded on their screen, that is, reading the questions in both present AND past tense. By

doing so, they changed the question wording, and when respondents responded to this

question it was not clear whether their answers applied to their current jobs or jobs in the

past. Therefore, these questions would be improved when they would not rely on the

interviewers as much. If this reliance on the interviewer cannot be avoided, as is the case in

paper-and pencil surveys, at least focus more on instruction of the interviewers is necessary.

It is necessary for example to repeat the instruction to reword the question for each individual

question. In computer assisted interviews it must be possible to let the computer adjust the

question wording (instead of the interviewer). However, a problem still exists with respect to

respondents who are not in paid work, but have had several jobs in the past: which of those

jobs should they have in mind when answering the questions?

17

Request for clarification

The media-questions (ESS questions B1-B7) appeared to yield a high percentage of requests

for clarifications of the question (between about 5% and 22% of the sequences contain a

request for clarification). For example, it was not defined what actually meant ‘watching

television’ (i.e. this may range from watching with complete attention, to having a TV-set

on, while being present in the room, but doing something else). The definition of ‘listening to

the radio’ also yielded problems, although interviewers were fairly capable of explaining that

listening to tapes or CD’s did not include listening to the radio (which therefore might better

be rephrased as ‘listening to radio broadcasts). ‘Reading the newspaper’ also created

ambiguousness of the definition of news papers (for example do they include free tabloids

etc). Aside from that, using the definite article ‘the’, the question is also leading in a very

subtle way: it implies that all respondents read a newspaper and it is also the same newspaper

that all respondents read. In the final questionnaire the plural ‘newspapers’ was used.

Although this does not imply that all respondents preferably read the same newspaper, but the

suggestion that the respondent ought to read a newspaper is still present. The same can be

said for all media questions, as they are posed; how often do you do X, implying that all

respondents do X. However, especially for the reading newspaper-question the suggestion is

problematic, as reading a newspaper is more socially desirable than watching television or

listening to the radio.

The final media question that created confusion was B7, considering the use of the

internet. The phrase ‘whether at home or at work’ and for ‘personal use’ seemed too

complicating.

Fragment 6

I: Will you please take card B1?

I: Would you take a look on this card and answer the following question by means of this

card?

I: How often do you use the internet, e-mail or the world-wide web, whether at home or at

work, for your personal use?

R: work and home?

I: yes but for personal use

R: oh yes

R: well that is several times a week.

18

Questions about racial laws (J45 and J46 in the final questionnaire) also yielded a high

number of requests for clarification. It was difficult for respondents to make a distinction

between the two questions: “a law against racial or ethnic discrimination” (the part “in the

work place was often ignored) and “a law against promoting racial or ethnic hatred” was the

same for many respondents. The phrase ‘law against’ also creates confusion, as fragment 7

illustrates. In this QA sequence, the question is reversed, and the respondent answers the

opposite question.

Fragment 7

QA sequence concerning ESS question J46

I: How bad or good is a law against promoting racial or ethnic hatred for a country?

R: For promotion?

I: For promoting racial hatred

R: A law for promoting

R: So that’s very bad

I: The last

(Interviewer scores ‘0’ , very bad)

Furthermore the questions concerning involvement in organizations were problematic with

respect to the meaning. Respondents had to indicate for a certain type of organization, for

example a sports club, whether they were a member, if they had participated, donated money

and or had done voluntary work for the organization. The definition of participation created

some trouble, and especially when people were member of an organization they were

confused whether this automatically concerned participation as well, and whether donating

money included their compulsory subscription’s fees. Furthermore, because multiple

organizations were dealt with within one question, answering the question became

complicated. Imagine a respondent who is member of an animal rights association, donated

money to an environmental organization and did voluntary work for a peace organization.

How should this respondent deal with this information within one question? Furthermore, in

the pilot study it was not indicated what period of time respondents should take as a reference

in answering the question. In the final questionnaire the phrase “in the last 12 months” was

included in the question. Although this might mean that due to telescoping effects

respondents will include memberships of longer periods, this will most likely improve the

validity of the answers to this question.

19

Finally, it appeared that for some respondents ‘every day’ is not specific enough as example 8

illustrates. However, the problem illustrated in this fragment may also be caused by a

question order effect. The questions that precede this question (B1-B6), ask for media-use on

an hourly basis at the daily level (i.e. on an average weekday how much time do you spend..).

When question B7 is asked, respondents have to switch the period of reference, from answers

indicating hours and minutes in the preceding questions to an answer indicating a frequency

ranging between every day and less than once a month or never.

Example 8

I: If you want to take card B1 please

R: yes

I: Uh, would you please take a look at the card while answering?

I: How often do you use the internet, e-mail or the world-wide web, whether at home or at

work, for your personal use?

R: per day?

I: eh yes

I: no, it’s on the card

R: oh yes

R: uhm well let’s say every day

20

Conclusion and discussion

So far, we have presented the most important problematic behaviours that illustrated

problematic aspects of some ESS questions. Interaction analysis appeared to be a useful

instrument to evaluate the quality of survey questions. Of course interaction analysis is not

the only way to pre-test survey questions. The fact that a sequence proceeds as a paradigmatic

sequence is no guarantee that the answer obtained adequately reflects the actual attitude or

behaviour of the respondent. Especially in case of questions requiring cognitive effort, the

respondent may for example as a strategy of satisficing the interviewer quickly respond with

a scorable answer, but neglect to put enough cognitive effort into the answering process to

assure an adequate answer.

Interaction analysis of the ESS questions showed that some questions heavily rely on the

interviewer to correct problematic respondent behaviour. Especially improper yes-no

questions relied on interviewers probing, which they are very easily tended to do in a

suggestive way. Also question order may stimulate suggestive probing. Especially when

answers to certain questions make later questions seem redundant. The use of show cards is

advisable. However, a danger exists of overusing show cards; it can make the respondent’s

task easier, but the interviewer’s more difficult, as she has to keep track of the right show

cards.

When no show cards are used, questions are best formulated in such a way, that answer

alternatives are incorporated in the question, and not to be read after the question.

Furthermore, a lot of concepts in questions can create confusion or misunderstandings: it is

therefore important that questionnaires are accompanied with definitions and clear

descriptions that interviewers can easily refer to.

21

Appendix 1

Summary of problematic questions

ESS

question

Original question wording Problem Solution

B1 On an average weekday, how much

time, in total, do you generally spend

watching television?

Unclear concepts

in question

-Include definitions that the interviewer can refer to (not necessarily

to be read aloud in all cases, only when respondents ask for

clarification or give indications of uncertainty)

-By watching television we mean watching with attention, being

present in the same room as the televison-set etc.

-By an average weekday we mean Monday to Friday, during

daytime as well as evenings and night time. Do not include

weekdays during holidays or special occasions.



listening to the radio

Subtly leading

question because

of definite article

‘the’

Substitute the phrase ‘listening to the radio’ with ‘listening to

radio broadcasts’



reading the newspapers?

Subtly leading

question because

of definite article

‘the’

Substitute the phrase ‘reading the newspapers’ with ‘reading

newspapers’

22

ESS

question

number


B7 Using this card, how often do you use the

internet e-mail or the World Wide Web –

whether at home or at work – for your

personal use?

Never Less than once a month Once a month Several times a month Once a week Several times a week Every day

Complicating

question, and

response options

may not be specific

enough

-Use of ‘world wide web’ is probably redundant, lengthens the

question unnecessarily.

-‘whether at home or at work for your personal use’ is a

complicating phrase, try to shorten the question.

-Use in an introduction to this question to separate the question

from the previous questions, that asked for numbers of hours and

minutes, or ask for the same level of detail (i.e. how much time..) if

required.

F8 Are you hampered in your daily activities

in any way by any longstanding illness,

or disability, infirmity or mental health

problem?

1 Yes, a lot 2 Yes, to some extent 3 No

Improper yes-no

question

When phrased as a yes-no question, only include ‘yes’ and ‘no’ as

alternatives, or phrase the question as a closed choice question:

Are you, due to any longstanding illness, or disability, infirmity or

mental health problem, ‘a lot’, ‘to some extent’ or ‘not at all’

hampered in your daily activities?

OR:

In what way are you hampered in your daily activities by any

longstanding illness, or disability, infirmity or mental health problem.

Would you say ‘a lot’, ‘to some extent’ or ‘not at all’?

23

ESS

question


G3 Have you ever considered yourself as

belonging to any particular religion or

denomination?

Redundant

question when G1

is answered with

yes

-Skip in question routing

OR

-rephrase question:

Have you ever considered yourself as belonging to any OTHER

particular religion or denomination?

G6 Apart from special occasions such as

wedding baptisms, funerals, festivals and

circumcisions, about how often do you

attend religious services these days?

Every day

More than once a week

At least once a month

Only on special holy days

Less often

Never

Response options

create confusion.

Introducing phrase

is confusing and

likely to be ignored

All response options must indicate a frequency,

-remove ‘only on special holy days’ or reorder the alternatives,

ending with ‘only on special holy days’ -Rephrase ‘less often’ as

‘less than once a month’

-Rephrase into two separate questions, first asking whether

respondents attend religious services apart from special occasions:

In the last past months did you, apart from special occasions such

as weddings, baptisms, funerals, festivals and circumcisions,

attend religious services?

If yes: About how often did you attend religious services?

G7 About how often do you pray outside of

religious services?

(same alternatives as questionG6)

Response options

create confusion

All response options must indicate a frequency,

-remove ‘only on special holy days’ or reorder the alternatives,

ending with ‘only on special holy days’

24

ESS

question


J45 How bad or good are each of these

things for a country?

A law against racial or ethnic discrimination in the workplace 0 Very Bad 10 Very good

Complicating

question because

of negation

‘against’

Avoid using negations.

J46 A law against promoting racial or ethnic

hatred

Complicating

question because

of negation

‘against’

Avoid using negations, especially in this case the words ‘against’

and ‘promoting’ are confusing.

Consider:

A law against racial or ethnic hatred.

Try to make the question more distinctive, this may be possible to

place the questions more distinct from each other in the

questionnaire.

25

ESS

question


J47/J48 Do you have any friends who have

come to live in Britain from another

country?

1 Yes, several 2 Yes, a few 3 No, none at all Do you have any colleagues at work

who have come to live in Britain from

another country?

1 Yes, several 2 Yes, a few 3 No, none at all

Improper yes-no

question


alternatives, or phrase as a closed choice question:

Do you have several, a few or no friends at all who have come to

live in Britain from another country?

OR:

How many friends who have come to live in Britain from another

country do you have? Would you say that are several, a few or

none at all?

26

ESS

question


K1:A-N For each of the voluntary

organisations I will now mention,

please use this card to tell me whether

any of these things apply to you now

or in the last 12 months, and, if so,

which

-Member -Participated -Donated money -Voluntary work -None A: Firstly, a sports club or club for out-door activities, do any of the descriptions on the card apply to you?

Response alternatives

are not mutually

exclusive/exhaustive,

complicating, double-

barrelled question

-Include definitions (not necessarily to be read aloud in all cases,

only when respondents ask for clarification or give indications of

uncertainty) :

-to distinct ‘participate’ from the other response alternatives

(what is the difference between ‘being a member’ and

‘participating’?)

-to clarify ‘donated money’: must obligatory donations

(membership fees) be included, or do only voluntary

donations count?

-The question is double-barrelled for respondents who are a

member of multiple sports-clubs etc. It should be made possible to

answer the question separately for different sports clubs etc.

L9-L21 -In your main job are/were you..

-How many employees (if any) do/did you have? -Do/did you have a work contract of.. -how many people are /were etc.

Change in question

wording relies to

much on interviewer

-Repeat instruction to rephrase the question in present OR past

tense, in order to remind the interviewer

OR

-In case of CAPI interviews: let the computer automatically apply

the right question wording

27

ESS

question


L15 Are/were you allowed to organise your

own work

Nearly all of the time Quite a lot of the time Not much of the time Or, not at all

Improper yes-no

question


alternatives, or phrase as a closed choice question:

Are you ‘Nearly all of the time’, ‘Quite a lot of the time’, ‘Not much

of the time’ or, ‘not at all allowed’ to organise your own work?

To what extent are/were you allowed to organise your own work?

Would you say that is ‘nearly all of the time’, ’quite a lot of the time’,

‘not much of the time’ or, ‘not at all’.

28

References

Dijkstra, W. (2002). Transcribing, Coding, and Analysing Verbal Interactions in Survey Interviews. In D. W. Maynard, H. Houtkoop-Steenstra, N. C. Schaeffer & J. v. d. Zouwen (Eds.), Standardization and Tacit Knowledge: Interaction and Practice in the Survey Interview. (pp. 401-425). New York: Wiley.

Dijkstra, W., & Ongena, Y. P. (2002). Question-Answer Sequences in Survey-Interviews. Paper presented at the International Conference on Questionnaire Development, Evaluation and Testing Methods, Charleston, SC.

Houtkoop-Steenstra, H. (2000). Interaction in the standardized survey interview: the living questionnaire. Cambridge: Cambridge University Press.

Smit, J. H. (1995). Suggestieve vragen in survey-interviews. Vóórkomen, oorzaken en gevolgen. Academisch proefschrift, Vrije Universiteit, Amsterdam.

Pre-testing the ESS-questionnaire using interaction analysis€¦ · Pre-testing the...

Documents

Transcript of Pre-testing the ESS-questionnaire using interaction analysis€¦ · Pre-testing the...