Workshop: Analysis and Reporting of Cognitive Interviews Gordon Willis, Ph.D. Applied Research...

Workshop: Analysis and Reporting of

Cognitive Interviews

Gordon Willis, Ph.D.Applied Research Program

Division of Cancer Control and Population SciencesNational Cancer Institute

[email protected]

Gordon Willis DC-AAPOR 2014

mailto:[email protected]

Sequence: Sequence: Conduct interviewsConduct interviews

Make conclusions and recommendationsMake conclusions and recommendations

What happens in this interval?

-How are interview results (data) processed?

-How are these turned into conclusions about question functioning?

- Where do recommendations for question revision come from?

?

Gordon Willis DC-AAPOR 2014Gordon Willis DC-AAPOR 2014


Factors that influence the Factors that influence the nature of the analysisnature of the analysis

I. Purpose – based on model from DC-AAPOR 2013:I. Purpose – based on model from DC-AAPOR 2013:(1) (1) ReparativeReparative: To find problems and improve survey : To find problems and improve survey

questions questions (typical ‘finding/fixing’)(typical ‘finding/fixing’)(2) (2) DescriptiveDescriptive: To determine functioning – what : To determine functioning – what

question ‘captures’ question ‘captures’ (reverse engineering)(reverse engineering)

Pretesting involves (1); Quality assessment involves (2)Pretesting involves (1); Quality assessment involves (2)

My observation: Purpose is associated with variation in My observation: Purpose is associated with variation in (a) conduct of interviews and (b) their analysis ->(a) conduct of interviews and (b) their analysis ->

Factors that influence the nature of Factors that influence the nature of the analysisthe analysis

(1) (1) ReparativeReparative: : In the past 12 months, have you been In the past 12 months, have you been bothered by pain in the abdomen?bothered by pain in the abdomen?

a) Applied focus – What client wantsa) Applied focus – What client wantsb) Limited time/resources - small (tiny?) sample sizesb) Limited time/resources - small (tiny?) sample sizesd) Involves iterative testing of question variantsd) Involves iterative testing of question variantse) Conduct and Analysis are intertwined/cyclical e) Conduct and Analysis are intertwined/cyclical

(Round 1 is ‘analyzed’ prior to Round 2)(Round 1 is ‘analyzed’ prior to Round 2)

Factors that influence the nature of Factors that influence the nature of the analysisthe analysis

(2) (2) DescriptiveDescriptive: : Would you say your health in general is Would you say your health in general is excellent… poor? (Schechter, 1994)excellent… poor? (Schechter, 1994)

a) Attempts more complete evaluation of itema) Attempts more complete evaluation of itemb) So test larger numbers -> ‘saturation’b) So test larger numbers -> ‘saturation’c) Pure form involves c) Pure form involves no no iteration (general health item)iteration (general health item)d) Analysis occurs after data collectiond) Analysis occurs after data collection

Thoughts…?Thoughts…?



II. Complexity -> II. Complexity -> LevelsLevels of analysis involved of analysis involved(1)(1) SimpleSimple: Young Adult Cancer Survivors: Young Adult Cancer Survivors

(consider: age, cancer type, gender)(consider: age, cancer type, gender)(2)(2) ComplexComplex::

- Multi-cultural: White/Black/Hispanic/Asian… - Multi-cultural: White/Black/Hispanic/Asian… - Multi-entity: Farmers/poultry/cattle ranchers…- Multi-entity: Farmers/poultry/cattle ranchers…

Primary care/specialist types…Primary care/specialist types…

Including multiple groups impacts both conduct (recruitment, Including multiple groups impacts both conduct (recruitment, sample size) and levels of aggregation in analysis sample size) and levels of aggregation in analysis (“Complete” analysis)(“Complete” analysis)

Miller model (from previous Q-Bank Website)Miller model (from previous Q-Bank Website)




III. Interviewer/Analysis Experience and CompetenceIII. Interviewer/Analysis Experience and Competence(1)(1) Multi-lingual investigation -> Multi-lingual investigation -> (2)(2) Novice interviewers ->Novice interviewers ->(3)(3) Increased restrictions (tight leash) ->Increased restrictions (tight leash) ->(4)(4) Scripted probing, highly structured analysis – Scripted probing, highly structured analysis –

especially concerning data reduction and especially concerning data reduction and aggregationaggregation



Aggregation approaches:Aggregation approaches:

(1)(1) Successive Aggregation Successive Aggregation

(2)(2) Collaborative (Joint) AnalysisCollaborative (Joint) Analysis

Laboratory #1: 12 interviews

Interviewer #1 results -Interviewer #1 results -Aggregated comments Aggregated comments from 4 interviewsfrom 4 interviews



Combined resultsCombined resultsAggregated comments Aggregated comments from:from: 12 interviews12 interviews 3 interviewers3 interviewers

Combined report, by Combined report, by Lab #1 DirectorLab #1 Director

Laboratory #2: 12 interviewsCombined report, as for Lab #1

Laboratory #3: 12 interviewsCombined report, as for Lab #1

Fully aggregated (3-lab) report from:36 interviews9 interviewers

1111

Overall during the past four weeks, how Overall during the past four weeks, how much difficulty did you have in much difficulty did you have in

recognizing the face of someone four recognizing the face of someone four yards away?yards away?

IDID Open Open Verbatim Verbatim responseresponse

ActivityActivity Repeat?Repeat? Knew/Knew/

Used Used MeasureMeasure

Used Used Time Time Period?Period?

S1S1 ““not any”not any” Watching TVWatching TV NoNo YesYes N/AN/A

S2S2 ““A lot of A lot of difficulty”difficulty”

Seeing someone a Seeing someone a block awayblock away

NoNo NoNo YesYes

S3S3 ““I don’t I don’t know know what you what you mean”mean”

Recognizing a friend; Recognizing a friend; could be sight or a could be sight or a memory questionmemory question

YesYes NoNo NoNo

Based on Willis/Miller: SAPOR analysis short course



Processing and combining of the results is a major Processing and combining of the results is a major issue worthy of discussion/development issue worthy of discussion/development

Questions: Questions: Are there alternatives?Are there alternatives?Is there clearly a best approach?Is there clearly a best approach?How would we know?How would we know?What should we be doing to tell?What should we be doing to tell?



IV.IV. Knowledge of the research objectives of the Knowledge of the research objectives of the evaluated itemsevaluated items

- This is vital for Reparative Studies- This is vital for Reparative Studies- Objective is to match research interpretation with - Objective is to match research interpretation with that of survey respondentsthat of survey respondents- Objectives are study-specific- Objectives are study-specific

Investigator intent

Participant interpretations




- May not be as important for pure Descriptive Study- May not be as important for pure Descriptive Study- We capture all of the potential interpretations- We capture all of the potential interpretations- Researcher(s) can decide whether item matches - Researcher(s) can decide whether item matches objectivesobjectives- NOT study-specific – findings generalize to future - NOT study-specific – findings generalize to future studies studies




- Plays out in several ways for Reparative study:Plays out in several ways for Reparative study:- Need to conduct ‘cognitive interview’ of client - Need to conduct ‘cognitive interview’ of client - First round may - First round may notnot result in recommendations for result in recommendations for changes, but in questions about objectiveschanges, but in questions about objectives- Purpose of initial analysis may be to identify variation in - Purpose of initial analysis may be to identify variation in

interpretation, and later analyses are to fix interpretation, and later analyses are to fix problemsproblems- So, Descriptive round followed by Reparative ones?- So, Descriptive round followed by Reparative ones?- Especially when client is unclear about own - Especially when client is unclear about own objectives?objectives?


Nuts and Bolts of Analysis:Nuts and Bolts of Analysis:What do we mean?What do we mean?

- Data definition – segmenting – reduction – coding – Data definition – segmenting – reduction – coding – ‘crunching’ – cross-tabulating – interpreting – ‘crunching’ – cross-tabulating – interpreting – reportingreporting

- Definition: What are our data sources?Definition: What are our data sources?- Answers to tested itemsAnswers to tested items- Interviewer Notes (live or from recordings)Interviewer Notes (live or from recordings)- Participant Quotes (selected or full transcript) (?)Participant Quotes (selected or full transcript) (?)

- What do we do with these?What do we do with these?- Produce Produce Text SummariesText Summaries- Conduct Conduct CodingCoding


1. Text Summary 1. Text Summary In words, describe observations, problems, In words, describe observations, problems,

etc.etc.- The ‘usual’ model: No explicit coding is doneThe ‘usual’ model: No explicit coding is done- Similar to what is done by ethnographers: Similar to what is done by ethnographers:

- Field notes Field notes at the individual interview levelat the individual interview level

- BUT: What guides the writing of the notes?BUT: What guides the writing of the notes?- What do we choose to write about…? What do we choose to write about…?

- There’s not much guidance on ‘writing up notes’ – or There’s not much guidance on ‘writing up notes’ – or separating the wheat from the chaff as we undergo separating the wheat from the chaff as we undergo data reductiondata reduction

- Is largely a subjective process, guided by:Is largely a subjective process, guided by:- Cognitive testing objectivesCognitive testing objectives- A model of likely effects (e.g., Tourangeau)A model of likely effects (e.g., Tourangeau)

- If independent observers ‘wrote up’ the same results, If independent observers ‘wrote up’ the same results, would their observations match?would their observations match?


Yes, because "they don't pay attention" to her. She is thinking of people (her children especially) attending to what she advises them - not literally understanding the words she is saying.

Conclusion: “Understanding you” is interpreted not as speech understanding, but interpreting a message/direction and then complying with it


2. Cognitive Coding2. Cognitive CodingFocus on the subject’s cognitive processesFocus on the subject’s cognitive processes

- Apply codes, such as Tourangeau’sApply codes, such as Tourangeau’s- Comprehension/InterpretationComprehension/Interpretation- RetrievalRetrieval- Decision/JudgmentDecision/Judgment- ResponseResponse

- Bolton and Bronkhorst (1996): Automatic coding of Bolton and Bronkhorst (1996): Automatic coding of think-aloud protocol to search for terms such as think-aloud protocol to search for terms such as “don’t remember” into a “Forget” cognitive category“don’t remember” into a “Forget” cognitive category

- Otherwise, a human needs to code each result Otherwise, a human needs to code each result according to the applicable code(s)according to the applicable code(s)

- Limitations:Limitations:- Only covers pure ‘cognitive’ codesOnly covers pure ‘cognitive’ codes- ““Comprehension” tends to overwhelm other codesComprehension” tends to overwhelm other codes



- Apply codes, such as Tourangeau’sApply codes, such as Tourangeau’s- Comprehension/InterpretationComprehension/Interpretation- RetrievalRetrieval- Decision/JudgmentDecision/Judgment- ResponseResponse

- Bolton and Bronkhorst (1996): Automatic coding of Bolton and Bronkhorst (1996): Automatic coding of think-aloud protocol to search for terms such as think-aloud protocol to search for terms such as “don’t remember” into a “Forget” cognitive category“don’t remember” into a “Forget” cognitive category

- Otherwise, a human needs to code each result Otherwise, a human needs to code each result according to the applicable code(s)according to the applicable code(s)

- Limitations:Limitations:- Only covers pure ‘cognitive’ codesOnly covers pure ‘cognitive’ codes- ““Comprehension” tends to overwhelm other codesComprehension” tends to overwhelm other codes

Willis, et al., 1991 – Cognitive Coding Study

Communication Otherproblems problems

NCHS 70.5% 29.5%C. I.

NORC 58.1% 41.9%C. I.



- Coding may be done (1) at Participant level (P1…Coding may be done (1) at Participant level (P1…Px) level, or at (2) Question/Item level (aggregated Px) level, or at (2) Question/Item level (aggregated over Participants):over Participants):

- (1) “52% of Participants experienced a (1) “52% of Participants experienced a Comprehension problem with Item 1”Comprehension problem with Item 1”

- (2) “52% of items illustrated Comprehension (2) “52% of items illustrated Comprehension problems”problems”

- Not well distinguished in the literature – both Not well distinguished in the literature – both have been donehave been done- Seems that (1) is more useful in assessing Seems that (1) is more useful in assessing

individual item function, (2) if we want to express individual item function, (2) if we want to express the general types of problems found in survey the general types of problems found in survey questionsquestions


3. Question Feature Coding3. Question Feature CodingFocus on the ‘behavior’ of the items Focus on the ‘behavior’ of the items

- Changes the focus: Changes the focus: ““What cognitive problems do people have with the What cognitive problems do people have with the question?” ->question?” ->““What features of the question cause people to have What features of the question cause people to have problems?”problems?”

- Does this type of coding scheme make the results more Does this type of coding scheme make the results more actionable (than cognitive coding scheme)?actionable (than cognitive coding scheme)?

Limitation: Again, ‘generic’ codes result in loss of detailsLimitation: Again, ‘generic’ codes result in loss of details

Question Appraisal System (Willis & Lessler)

Gordon Willis DC-AAPOR 2014Gordon Willis DC-AAPOR 2014Source: Willis & Lessler, 1999: Question Appraisal System

CNEST model of cross-cultural problems (Fitzgerald, et CNEST model of cross-cultural problems (Fitzgerald, et al, 2011al, 2011))


LLLLLee (2012) Appraisal System for Cross-National Surveys

Lee (2012) Appraisal System for Cross-National Surveys



3. Question Feature Coding3. Question Feature Coding

- Models vary in complexity – some are hybrids between Models vary in complexity – some are hybrids between Cognitive and Question Feature modelsCognitive and Question Feature models

- More elaborate models capture more in terms of More elaborate models capture more in terms of underlying theory concerning what should be coded ->underlying theory concerning what should be coded ->

Willis (in press) modelWillis (in press) model



4. Bottom-up approach: Theme 4. Bottom-up approach: Theme CodingCoding

Create labels to describe the phenomena Create labels to describe the phenomena observed observed

- Rather than applying preset codes Rather than applying preset codes (top-down(top-down), we ), we build the codes from the data build the codes from the data (bottom-up)(bottom-up)

- Borne of qualitative research - Borne of qualitative research - Grounded TheoryGrounded Theory: : the analysis is grounded by the datathe analysis is grounded by the data

- We mine the qualitative data for consistent We mine the qualitative data for consistent themethemes s which give rise to codeswhich give rise to codes

- Example: General health question:Example: General health question:- Develop categories from cognitive interviewing Develop categories from cognitive interviewing

results: results: Physical health, Mental health, Spiritual Physical health, Mental health, Spiritual health…health…

So - We are NOT imposing our worldview on the dataSo - We are NOT imposing our worldview on the dataBut - This gives rise to a unique coding system for each But - This gives rise to a unique coding system for each

investigation – not much general knowledge gainedinvestigation – not much general knowledge gained




- By creating codes that are grounded in the data:By creating codes that are grounded in the data:- We engage in a type of We engage in a type of Reverse EngineeringReverse Engineering- That is, we determine what the question is That is, we determine what the question is

composed of – according to how it is interpretedcomposed of – according to how it is interpreted- Miller: We Miller: We determine “What the question determine “What the question

captures”captures”- Note that there is NOT an emphasis on ‘problems’ Note that there is NOT an emphasis on ‘problems’

((dysfunctiondysfunction), at least initially), at least initially- Rather, we focus on the totality of question Rather, we focus on the totality of question functionfunction- There is a strong focus on:There is a strong focus on:

- Interpretation/ComprehensionInterpretation/Comprehension- Saturating the resultsSaturating the results

Overall, during the past 4 weeks, how much Overall, during the past 4 weeks, how much difficulty did you have with thinking clearly difficulty did you have with thinking clearly

and solving daily problems?and solving daily problems?

Respondent 5

Respondent 2

Respondent 6 Respondent 3

Respondent 4

Respondent 1

Alzheimer’s disease

Busy

Long term, medical problem

Specific experience- organizing tenants

Remembering detailed list

Fiscal functioning

Source: Miller, K. (2008)Gordon Willis DC-AAPOR 2014Gordon Willis DC-AAPOR 2014



Respondent 5

Respondent 2


Respondent 4

Respondent 1


Busy




Fiscal functioning

Could label these as “Chronic limitation or health problem”




Respondent 5

Respondent 2


Respondent 4

Respondent 1


Busy




Fiscal functioning

Could label these as “Activities/ problems of daily life”




Respondent 5

Respondent 2


Respondent 4

Respondent 1


Busy




Fiscal functioning

If designers are interested in just these, question is too wide-open



5. Pattern Coding5. Pattern CodingDiscover dominant associations in responsesDiscover dominant associations in responses

- Also a ‘bottom up’, Grounded Theory ApproachAlso a ‘bottom up’, Grounded Theory Approach- We categorize the behavior of subjects, in particular We categorize the behavior of subjects, in particular

focusing on inconsistencies across items, and focusing on inconsistencies across items, and across responses to probesacross responses to probes

- Best illustrated through an example (K. Miller) Best illustrated through an example (K. Miller) - Conducted multicultural cognitive interviewing Conducted multicultural cognitive interviewing

for disability measurementfor disability measurement- One set focused on vision problems (already One set focused on vision problems (already

shown) ->shown) ->


Overall during the past four weeks, how much Overall during the past four weeks, how much difficulty did you have in recognizing the face of difficulty did you have in recognizing the face of

someone four yards away?someone four yards away?








NoNo NoNo YesYes



YesYes NoNo NoNo


Overall during the past four weeks, how much Overall during the past four weeks, how much difficulty did you have in recognizing the face of difficulty did you have in recognizing the face of

someone four yards away?someone four yards away?








NoNo NoNo YesYes



YesYes NoNo NoNo

This ‘chart’ approach forces This ‘chart’ approach forces systematic attention to what was systematic attention to what was obtained across interviews, in obtained across interviews, in parallel fashionparallel fashion


Which approach is best?Which approach is best?Unknown: May depend on what you are Unknown: May depend on what you are

doing…doing…1)1) Fix my questionnaire, now! Fix my questionnaire, now! Text summary Text summary (1)(1)

• We want to plug the leaks…We want to plug the leaks…

2) Get C.I. report in a journal that wants quantitative 2) Get C.I. report in a journal that wants quantitative data, statistics -> data, statistics -> Cognitive/Question coding Cognitive/Question coding (2, 3) (2, 3) • We want to plug them but also count them…We want to plug them but also count them…

3) Find out everything about how people think about X 3) Find out everything about how people think about X (X = general health, employment, sexual (X = general health, employment, sexual identification…) -> identification…) -> Theme/Pattern coding Theme/Pattern coding (4,5) (4,5) • We want to understand everything about leaks…We want to understand everything about leaks…


Which approach is best?Which approach is best?

Writing up cognitive testing results Writing up cognitive testing results

• There is no standard formatThere is no standard format• Cognitive testing publications have lots of holes! Cognitive testing publications have lots of holes!

• Boeije & Willis (2013): Boeije & Willis (2013): Introduce Introduce Cognitive Cognitive Interviewing Reporting FormatInterviewing Reporting Format (CIRF) (CIRF)……

• In Special Issue of (journal) MethodologyIn Special Issue of (journal) Methodology

• Ten major elements, with listing of info to be Ten major elements, with listing of info to be included under each:included under each:

• Some elements are normally omitted from C.I. reportsSome elements are normally omitted from C.I. reports

• Some elements are included but under-specifiedSome elements are included but under-specified


Cognitive Interviewing Reporting Cognitive Interviewing Reporting Format (CIRF)Format (CIRF)

CIRF Element

(1) [ ] Research Objectives

(2) [ ] Research Design

(3) [ ] Ethics

(4) [ ] Participant Selection

(5) [ ] Data Collection

(6) [ ] Data Analysis

(7) [ ] Findings

(8) [ ] Conclusions, Implications, and Discussion

(9) [ ] Strengths and Limitations of testing process

(10) [ ] Report Format

Boeije, H., & Willis, G. (2013). The Cognitive Interviewing Reporting Framework (CIRF): Towards the Harmonization of Cognitive Testing Reports. Methodology, 9(3): 87-95.

Evaluation of CIRF Checklist Approach Evaluation of CIRF Checklist Approach Willis & Boeije (2013). Willis & Boeije (2013). Methodology 2013; Vol. 9(3):123–128.Methodology 2013; Vol. 9(3):123–128.

• Based on 3 empirical tests of its use by European researchers:Based on 3 empirical tests of its use by European researchers:• Potentially useful as a guide to developing Potentially useful as a guide to developing testing plan testing plan

• Forces attention to ‘mundane’ issues otherwise omitted, but vital for Forces attention to ‘mundane’ issues otherwise omitted, but vital for evaluation and replicationevaluation and replication

• Facilitates Facilitates comparisoncomparison across reports/standardization across reports/standardization

• But:But:• Can result in excessive Can result in excessive lengthlength for journal articles for journal articles

• May May not fit not fit the project if followed too dogmaticallythe project if followed too dogmatically

• ClientClient may not care about all the details may not care about all the details


Q-Bank: Accessing Cognitive Test Findings

Historically, cognitive test findings have been relatively inaccessible.

Implications Knowledge is lost Resources are wasted Lack of transparency and accountability

Important need for a medium to make findings available


Q-Bank Website:

http://wwwn.cdc.gov/qbank/Home.aspx

Q-Bank e-mail:

[email protected]


Workshop: Analysis and Reporting of Cognitive Interviews Gordon Willis, Ph.D. Applied Research...

Documents

Transcript of Workshop: Analysis and Reporting of Cognitive Interviews Gordon Willis, Ph.D. Applied Research...