Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and...
-
Upload
sophie-willis -
Category
Documents
-
view
218 -
download
2
Transcript of Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and...
Designing and Using a Behaviour code frame to assess multiple styles
of survey items
Alice McGee and Michelle Gray
Presentation outline
• Background to study• Aims of research• Methodology• Designing a behaviour code frame• Using the behaviour code frame• Analysing the data• Lessons learned
Background to study
• English Longitudinal Study of Ageing (ELSA)• Dependent Interviewing (DI)• Two types of data item
Feed Forward (DI)Non-Feed Forward (non DI)
• Little evaluation of the impact of DI on data quality conducted to date
Aims of research
• Research aims:To assess how DI affects data qualityTo explore how Rs react to feed-forward phrasesTo find whether this varies by nature and
sensitivity of topic• Methodological aim:
To explore the combination of CARI and Behaviour Coding as methodological tools
Methodology
• Computer Assisted Recorded Interviewing (CARI)Computer acts as a sophisticated tape recorderUnobtrusively records interaction
• Behaviour CodingCodes systematically applied to interviewer-
respondent behavioursUncover and assess problems with questions
• Two methods combined for this study
Designing the code frame
Principles for good design
• Code frame adapted from Cannell et al (1989)• Short and straightforward • Few, easy to apply codes• Discrete• Broad rather than specific
Behaviours coded
• Question asking behaviour for interviewers• Immediate response behaviour for respondents• Whether partner intervened (concurrent
interviews)• Final outcome of the entire exchange
Two behaviour code frames
• Two code frames designed:DI (feed-forward) itemsnon DI (non feed-forward) items
• First level exchange (initial utterance)• Code what occured before other person speaks
Code frame
Behaviours coded
• Interviewer/Interviewer feed-forward• Respondent/Respondent feed-forward• Whether partner intervened• Final outcome
• One code per behaviour
Interviewer codes
• Exact Wording/Slight Change 01• *Major change 02• *Omission 03• *Question became a statement 04• *Inaudible Interviewer/Other 05• Not applicable 99
*denotes where notes must be made
Interviewer feed-forward codes
• FF item read as worded/slight change 01• *FF statement became a question 02• *FF question became a statement 03• *Other major change 04• *Omission 05• *Inaudible Interviewer/Other 06• Not applicable 99
*denotes where notes must be made
Respondent codes
• Adequate Answer 01• *Inadequate Answer/Elaboration 02• *Clarification 03• Question Re-Read 04• Don't Know 05• Refusal 06• *Inaudible Respondent/Other 07• Not applicable 99
*denotes where notes must be made
Respondent feed-forward codes
• *Affirmed FF item - adequate 01• *Disputed FF item - adequate 02• *Inadequate Answer/Elaboration 03• *Clarification 04• Question Re-Read 05• Don't Know 06• Refusal 07• *Inaudible Respondent/Other 08• Not applicable 99
*denotes where notes must be made
Partner intervention codes
• *Yes 01 • No 02• Not applicable (no partner present) 99
*denotes where notes must be made
• Code used for where the respondents partner intervened and subsequently answered for the respondent
Final outcome codes
• Adequate Answer 01 • *Inadequate Answer 02 • Don't Know 03• Refusal 04• *Inaudible/Other 99
*denotes where notes must be made
• Coding whether the final answer meet the objective of the question
Technical details
CARI equipment
• Equipment testingExternal microphones
• CARI built into Blaise program• Recording switched on and off at relevant items• Sound files automatically generated and saved• Sound files removed from interviewer laptops
Macro run Data sticks (USB)
Behaviour coding system
• Conducted within Blaise• Coding program designed for this purpose
Weststat testnote software• Three windows displayed simultaneously
Blaise interviewing screenCoding entry screenSound file (.wav)
• Automatically routed through interview• Tags to skip to relevant data items
Using the code frame
Sound file
Blaise interviewing screen
Coding program
Data preparation and analysis
Organising the data
• Two types of dataBehaviour codes (quantitative)Coder notes on non-standard behaviours
(qualitative)• All data automatically stored in Excel tab
delimited file• One Excel file produced for each coder• Excel files amalgamated• Exported into SPSS
Data preparation
• More cleaning than expected• Two main problems:
Duplicate files (limitations of system used)Incorrect code frame used at interviewer and
respondent behaviours (DI and non DI items)
Analysing the data
• SPSS• Frequencies and crosstabulations• Coder notes provided additional context• Very small base sizes at some items due to
routing
Advantages and disadvantages of our approach and lessons learned
What worked
• CARIUnobtrusive in natureMinimal impact on interviewers and respondents
• Behaviour codingAble to run statistical analysesAble to draw conclusionsMethod of coding easier than paper (routing)
What didn’t worked
• CARIHigh number of inaudible or hard to hear cases
(1/3 of respondents)Purchased speakers to help
• Next time…Fully re-test microphonesProbe respondents reasons for not giving
consent to being recorded
What didn’t worked
• Behaviour codingLengthy and costly process
Coding (approximately 45 mins per interview)
Data cleaning
Over complex code frameCoding method found cumbersome, limited and
error proneCoder judgement not measured
Next time...
• One code frame only• Build in sufficient time for each stage• Clear rationale for behaviour coding• Inter-coder reliability test (Kappa score)• Adequate sample for uncommon questions• Create more sophisticated, less error prone
coding system
Discussion & Questions...