Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen...
Transcript of Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen...
![Page 1: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/1.jpg)
Cognitive Interviewing
Kristen Miller, NCHS
Question Evaluation Methods WorkshopNational Center for Health Statistics
October 21 – 23, 2009
![Page 2: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/2.jpg)
State of the Method
Primary evaluation method used for federal surveys
No established guidelines or standards No directive regarding the accessibility of
testing reports Difficult to discern how the method is
actually conducted and applied Potential variation within and across
agencies
![Page 3: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/3.jpg)
NCHS QDRL Perspective
Context– Challenged to address long-standing criticisms Lacking scientific merit Anecdotal Impressionistic
– International testing projects– Qualitative methodology and sociological
training
![Page 4: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/4.jpg)
![Page 5: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/5.jpg)
![Page 6: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/6.jpg)
![Page 7: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/7.jpg)
![Page 8: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/8.jpg)
Interpretive processes throughout the response process
Specifically, meanings that emerge through interaction of the respondent and the survey question
Meaning/interpretation tied to social context; questions perform differently across different groups of people– Skeptical of “question design rules”– Argue for design based on empirical study
![Page 9: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/9.jpg)
Purpose: Identify and document phenomena captured by the question– Valid (Intended phenomena)– Non-valid (Unintended phenomena)
Essence of interview: R’s interpretations of the question response process
Narrative “Story” behind respondents’ answer; why they answered the
question as they did. In ideal form, by definition, untainted by interviewer
Role of interviewer: To facilitate a detailed and coherent narrative– (Analysis begins in the interview)
![Page 10: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/10.jpg)
Analytic process:– Identify types of textual information, make judgments
about quality: Narrative, Out-of-context interpretations, Respondents’ opinions about the question, Interviewer-imposed interpretations (reflexivity)
– Identify interpretive patterns throughout the response process
Goal of analysis: Produce a synthesized interpretive framework of the captured phenomena– (Captured phenomena = phenomena captured by the
survey question and represented in the survey data)
![Page 11: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/11.jpg)
Sample: Complete when all patterns are identified; Implicit desire to increase sample size to understand multiple perspectives
![Page 12: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/12.jpg)
Advantages Interpretive quality Capability of
capturing complexity Findings are
grounded
*Characteristicsparticularly beneficialfor mixed-methoddesigns
Requirements Standards for
evidence Deliberate method of
analysis Transparent process
![Page 13: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/13.jpg)
Example
Washington Group/UNESCAP ProjectEvaluation of Disability Questions
![Page 14: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/14.jpg)
![Page 15: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/15.jpg)
![Page 16: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/16.jpg)
![Page 17: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/17.jpg)
![Page 18: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/18.jpg)
![Page 19: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/19.jpg)
![Page 20: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/20.jpg)
![Page 21: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/21.jpg)
![Page 22: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/22.jpg)
![Page 23: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/23.jpg)
![Page 24: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/24.jpg)
![Page 25: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/25.jpg)
![Page 26: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/26.jpg)
![Page 27: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/27.jpg)
Maldives and Sri Lanka Field Test: Percentage reporting anxiety and depression
0
20
40
60
80
Daily Weekly Monthly Fewtimes a
year
Never
MaldivesAnxiety
MaldivesDepression
Sri LankaAnxiety
Sri Lanka
![Page 28: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/28.jpg)
Field Test Data Maldives and Sri Lanka: Characteristics of reported anxiety
0
10
20
30
40
50
60
70
80
90
100
Work Chest hurts Positive Interfere Economic Normal Clinical
Reports of anxiety
Maldives, N=538
Sri Lanka, N=215
![Page 29: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/29.jpg)
Field Test Data: Characteristics of reported depression
0
10
20
30
40
50
60
70
80
90
100
Grief Stays in bed Interfere Economic Normal Clinical
Reports of depression
Maldives, N=421
Sri Lanka, N=125
![Page 30: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/30.jpg)
Advantages to Survey Research
Interpretive quality Capability of capturing complexity Findings are grounded
1.Provides evidence-based understanding of how the question functions:– To make informed decisions about question design
2.Provides basis for interpreting quantitative data:– Other question evaluation methods– Survey data
![Page 31: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/31.jpg)
Methodological Criteria
Standards for evidence– Interviews consist of different types of
information, different types not all equal in quality, can be contradictory
Deliberate method of analysis
* Study findings can vary depending on:1. the type of evidence used, and 2. the completeness of analysis
![Page 32: Cognitive Interviewing - Centers for Disease Control and ... · Cognitive Interviewing Kristen Miller, NCHS Question Evaluation Methods Workshop. National Center for Health Statistics.](https://reader035.fdocuments.us/reader035/viewer/2022081402/5f0e1b667e708231d43da553/html5/thumbnails/32.jpg)
Methodological Criteria
Transparent process– Cognitive testing reports accessible– Comprehensive reports: methods, detailed
evidence found in interviews, not opinion-based
– Audit trail to show analytic steps; Linking findings to actual interviews