How do we assess whether we are improving instrument design? Alice McGee.
-
Upload
jayson-reynolds -
Category
Documents
-
view
219 -
download
1
Transcript of How do we assess whether we are improving instrument design? Alice McGee.
![Page 1: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/1.jpg)
How do we assess whether we are improving instrument design?
Alice McGee
![Page 2: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/2.jpg)
How do we assess whether we are improving instrument design?
Using multiple methods to evaluate whether a re-designed travel record
was 'better' than the existing one
![Page 3: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/3.jpg)
Presentation outline
• Background to NTS and Travel Record• Rationale for re-design• Research stages
A - Evaluate the existing recordB - Record re-designC - Evaluate the new record
• Findings• Conclusions
![Page 4: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/4.jpg)
Background to study
• National Travel Survey (NTS)• Measures travel patterns and behaviour
Face to face interview - household levelSeven-day Travel Record - individual level
Adult and child versionsNature of information - very detailed
![Page 5: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/5.jpg)
Existing Travel Record
![Page 6: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/6.jpg)
Rationale
• Concerns surrounding data qualityHigh proportions of item missingAnecdotal evidence suggesting measurement
error• International evidence supporting alternative
designsAxhausen:
Column based approach
![Page 7: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/7.jpg)
Research stages
• A - Evaluate the existing record• B - Re-design the record• C - Evaluate the new record
to assess whether the new was ‘better’ than the original
![Page 8: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/8.jpg)
A - Evaluation methods
Quantitative
1. Secondary analysis of survey data
Qualitative
2. Workshops with interviewers
3. Cognitive interviews4. Respondent suggestions5. Data editor comments
![Page 9: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/9.jpg)
B - Re-designing the record
• Two underpinning elementsExisting literature: theoretical basis for how
people approach self-completion documentsEmpirical evidence from Stage A
• Existing literatureJenkins et al:
Readers and Skimmers typologyPrinciples of good questionnaire design
![Page 10: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/10.jpg)
B - Aims of the re-design
• Make the task easier and more straightforward for all types of respondent through:providing navigational guides to help
respondents find their way around the recordimprove the way information was organised and
displayed• NB: unable to address the more conceptual
problems
![Page 11: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/11.jpg)
B - Main changes to record
• Colour and layout more user-friendly• Instructions brought together• Visual guides (arrows) and clearer signposting• Pictorial images• Tick boxes• Column heading posed as questions
![Page 12: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/12.jpg)
Re-designed Travel Record
![Page 13: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/13.jpg)
C - Evaluation methods
Quantitative
1. Comparison of sample of data from existing Record and pilot data (100 respondents)
2. Respondent debriefing questionnaires
Qualitative
3. Cognitive interviews4. Pilot interviewer
comments5. Data editor comments
![Page 14: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/14.jpg)
Hypotheses
• All evidence sources reviewed and triangulated• Hypotheses tested at Stage C:
Cognitive respondents would experience narrower range of problems
Problems relating to information organisation and navigation would diminish among cognitive respondents
Systematic review of respondent error would show the new Record produced lower error rates
![Page 15: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/15.jpg)
Findings and conclusions
• All three hypotheses proved correct• Triangulation of quantitative and qualitative
evaluation methods showed new Record to be ‘better’ than the original
• Study demonstrates how a combination of evaluation methods can be brought together to substantively conclude the best approach
![Page 16: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/16.jpg)
Re-designed Travel Record
![Page 17: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/17.jpg)
Future work
• Evaluation of Event History Calendar approach (EHC)England Longitudinal Study of Ageing (ELSA)Retrospective interview following Wave 3Interactive calendar administered through CAPI
• Package of work involving multiple evaluation methods (quantitative and qualitative)
![Page 18: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/18.jpg)
EHC - Evaluation methods
Quantitative
1. Behaviour coding 2. Secondary analysis
of survey data3. Respondent
debriefing questionnaires (closed responses)
Qualitative4. Interviewer diary5. Respondent
debriefing questionnaires (open responses)
6. Quasi transcripts7. Behaviour coding
(supporting notes)
![Page 19: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/19.jpg)
References
• Axhausen, K (1995) “Travel Diaries: An annotated catalogue, 2nd edition”
• Jenkins, C.R., Ciochetto, S. and Davies, W. (1992) 'Results of cognitive research on the public school 1991-92 field test questionnaire for the schools and staffing survey'. Unpublished
• Jenkins, C.R. and Dillman, D.A. (1995) 'Towards a theory of self-administered questionnaire design' in Lyberg et al (1997) 'Survey Measurement and Process Quality' Wiley
• McGee, A, Gray, M and Collins, D (2006) “NTS Travel Record Review Stage 1” DfT, web only
• McGee, A, Gray, M, Andrews, F, Legard, R, Wood, N and Collins, D (2006) “NTS Travel Record Review Stage 2” DfT, web only
![Page 20: How do we assess whether we are improving instrument design? Alice McGee.](https://reader035.fdocuments.us/reader035/viewer/2022062517/56649f125503460f94c257e1/html5/thumbnails/20.jpg)