Designing and Using a Behaviour code frame to assess multiple styles

of survey items

Alice McGee and Michelle Gray

Presentation outline

• Background to study• Aims of research• Methodology• Designing a behaviour code frame• Using the behaviour code frame• Analysing the data• Lessons learned

Background to study

• English Longitudinal Study of Ageing (ELSA)• Dependent Interviewing (DI)• Two types of data item

Feed Forward (DI)Non-Feed Forward (non DI)

• Little evaluation of the impact of DI on data quality conducted to date

Aims of research

• Research aims:To assess how DI affects data qualityTo explore how Rs react to feed-forward phrasesTo find whether this varies by nature and

sensitivity of topic• Methodological aim:

To explore the combination of CARI and Behaviour Coding as methodological tools

Methodology

• Computer Assisted Recorded Interviewing (CARI)Computer acts as a sophisticated tape recorderUnobtrusively records interaction

• Behaviour CodingCodes systematically applied to interviewer-

respondent behavioursUncover and assess problems with questions

• Two methods combined for this study

Designing the code frame

Principles for good design

• Code frame adapted from Cannell et al (1989)• Short and straightforward • Few, easy to apply codes• Discrete• Broad rather than specific

Behaviours coded

• Question asking behaviour for interviewers• Immediate response behaviour for respondents• Whether partner intervened (concurrent

interviews)• Final outcome of the entire exchange

Two behaviour code frames

• Two code frames designed:DI (feed-forward) itemsnon DI (non feed-forward) items

• First level exchange (initial utterance)• Code what occured before other person speaks

Code frame

Behaviours coded

• Interviewer/Interviewer feed-forward• Respondent/Respondent feed-forward• Whether partner intervened• Final outcome

• One code per behaviour

Interviewer codes

• Exact Wording/Slight Change 01• *Major change 02• *Omission 03• *Question became a statement 04• *Inaudible Interviewer/Other 05• Not applicable 99

*denotes where notes must be made

Interviewer feed-forward codes

• FF item read as worded/slight change 01• *FF statement became a question 02• *FF question became a statement 03• *Other major change 04• *Omission 05• *Inaudible Interviewer/Other 06• Not applicable 99

*denotes where notes must be made

Respondent codes

• Adequate Answer 01• *Inadequate Answer/Elaboration 02• *Clarification 03• Question Re-Read 04• Don't Know 05• Refusal 06• *Inaudible Respondent/Other 07• Not applicable 99

*denotes where notes must be made

Respondent feed-forward codes

• *Affirmed FF item - adequate 01• *Disputed FF item - adequate 02• *Inadequate Answer/Elaboration 03• *Clarification 04• Question Re-Read 05• Don't Know 06• Refusal 07• *Inaudible Respondent/Other 08• Not applicable 99

*denotes where notes must be made

Partner intervention codes

• *Yes 01 • No 02• Not applicable (no partner present) 99

*denotes where notes must be made

• Code used for where the respondents partner intervened and subsequently answered for the respondent

Final outcome codes

• Adequate Answer 01 • *Inadequate Answer 02 • Don't Know 03• Refusal 04• *Inaudible/Other 99

*denotes where notes must be made

• Coding whether the final answer meet the objective of the question

Technical details

CARI equipment

• Equipment testingExternal microphones

• CARI built into Blaise program• Recording switched on and off at relevant items• Sound files automatically generated and saved• Sound files removed from interviewer laptops

Macro run Data sticks (USB)

Behaviour coding system

• Conducted within Blaise• Coding program designed for this purpose

Weststat testnote software• Three windows displayed simultaneously

Blaise interviewing screenCoding entry screenSound file (.wav)

• Automatically routed through interview• Tags to skip to relevant data items

Using the code frame

Sound file

Blaise interviewing screen

Coding program

Data preparation and analysis

Organising the data

• Two types of dataBehaviour codes (quantitative)Coder notes on non-standard behaviours

(qualitative)• All data automatically stored in Excel tab

delimited file• One Excel file produced for each coder• Excel files amalgamated• Exported into SPSS

Data preparation

• More cleaning than expected• Two main problems:

Duplicate files (limitations of system used)Incorrect code frame used at interviewer and

respondent behaviours (DI and non DI items)

Analysing the data

• SPSS• Frequencies and crosstabulations• Coder notes provided additional context• Very small base sizes at some items due to

routing

Advantages and disadvantages of our approach and lessons learned

What worked

• CARIUnobtrusive in natureMinimal impact on interviewers and respondents

• Behaviour codingAble to run statistical analysesAble to draw conclusionsMethod of coding easier than paper (routing)

What didn’t worked

• CARIHigh number of inaudible or hard to hear cases

(1/3 of respondents)Purchased speakers to help

• Next time…Fully re-test microphonesProbe respondents reasons for not giving

consent to being recorded

What didn’t worked

• Behaviour codingLengthy and costly process

Coding (approximately 45 mins per interview)

Data cleaning

Over complex code frameCoding method found cumbersome, limited and

error proneCoder judgement not measured

Next time...

• One code frame only• Build in sufficient time for each stage• Clear rationale for behaviour coding• Inter-coder reliability test (Kappa score)• Adequate sample for uncommon questions• Create more sophisticated, less error prone

coding system

Discussion & Questions...