Using Cognitive Interviews to Improve Survey Instruments
description
Transcript of Using Cognitive Interviews to Improve Survey Instruments
![Page 1: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/1.jpg)
Using Cognitive Interviews to Improve Survey Instruments
Heather Haeger, Indiana UniversityAmber Lambert, Indiana University-
BloomingtonJillian Kinzie, Indiana University-
BloomingtonJames Gieser, Indiana University
Presented at the Association for Institutional Research Forum
June 2-6, 2012
![Page 2: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/2.jpg)
Agenda• Introduction
• Conceptual Framework
• Methods
• Findings
• Conclusion and Discussion
![Page 3: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/3.jpg)
Current Context• Pressure to assess educational
practices remains high• Crucial to ensure that instruments
accurately measure educational practices and experiences o Questionnaires must measure what they
intend; respondents must understand & correctly interpret items
• NSSE’s widespread use (1,500 institutions) makes it particularly important to rigorously subject items to cognitive research testing
![Page 4: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/4.jpg)
Context for NSSE’s Cognitive Interviews
• Cognitive Interviews (CI) part of NSSE survey design from outset• Focus in 2005 to test survey among historically under-represented students• Planned NSSE update for 2013 provided occasion for multiple rounds of CIs
![Page 5: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/5.jpg)
• Purpose of cognitive interviewsoIdentify and analyze sources of
response error
• FocusoCognitive processes
oAccess respondents
interpretation & meaning of items
Purpose
![Page 6: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/6.jpg)
• In brief, cognitive interviews are meant to identify…oWhether subjects understand the
question…o In a way consistent across
subjects…oAnd in the way intended by
researchers.
Purpose (cont.)
![Page 7: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/7.jpg)
• Four actions of the cognitive process:oComprehend the questionoRetrieve informationoMake a judgment about relevance and
accuracyoFormulate and provide a response
Background
![Page 8: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/8.jpg)
• “Think-aloud”o Explicit activity in which the subject
verbalizes, his/her thought processes as s/he answers survey questions. • Interviewer reads the question, then observes and
records as the subject responds. • Interviewer is mainly passive in process, aside from
providing encouragement to “tell me what you’re thinking” if s/he hesitates or pauses.
o Advantages: Freedom from bias imposed by frequent interviewer interjections; minimal interviewer training requirements; open-ended design.
o Disadvantages: Subject usually requires training in method, or may resist technique; possibility for subject to stray from the topic at hand; subject may bias his/her description of his/her decision processing.
Methods
![Page 9: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/9.jpg)
• Verbal probing: concurrent and retrospectiveo Concurrent: after interviewer asks a question and
subject answers, interviewer asks more specific questions designed to elicit further information about the response. Probes can be scripted or spontaneous.
o Retrospective: at the end of interview, subject is asked to verbalize their thoughts about questions answered earlier when taking the questionnaire.
o Advantages: Interviewer maintains control of the interview; relative ease of training the subject.
o Disadvantages: Artificiality – criticism that this technique is not reflective of a real survey interview, in which interviewer simply asks questions and respondent answers them; potential for bias through poor selection of probes.
Methods (cont.)
![Page 10: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/10.jpg)
Analysis3 stages and related sub-stages that respondents faced during cognitive interviews:
1) Understanding the survey question and response options
a) Comprehending the survey questionb) Comprehending the response options
2) Performing the primary survey tasksa) Retrieving informationb) Deduction; making conclusions about
informationc) Mental arithmetic computation
3) Formatting responses a) Mapping data yielded by primary task
processes to an explicit response optionb) Response option is not available/offered
![Page 11: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/11.jpg)
Coding within these stages can address any of the following problems:
oLanguage problemso Inclusion/exclusion problemsoTemporal problems oLogical problemsoComputational problems
Types of Problems
![Page 12: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/12.jpg)
Question: In your experience at your institution during the current school year, about how often have you had serious conversations with people who differ from you in the following ways? (Never, Sometimes, Often, Very Often)
Example 1
a) Political views, b) Economic and social background, c) Religious beliefs or philosophy of life, d) Race, ethnic background, or country of origin, e) Sexual orientation
![Page 13: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/13.jpg)
• Language Problemo Too wordy, too complexoDiversity of interpretation• Talked with people who are different• Talked with people who are different but only about that topic• Talked about that topic with anyone
• Logic problemoSerious conversations + people who are
different from you + in the following ways
Example 1 (cont.)
![Page 14: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/14.jpg)
Question: Indicate the quality of your interactions with the following people at your institution: a) Student Affairs Professional
Example 2
![Page 15: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/15.jpg)
• Language problemo Not knowing what a Student Affairs Professional waso Wording was changed to “Student services staff
(campus activities, housing, career services, etc.)”
• Inclusion/exclusion problemo Students were including everyone
• Added parenthetical to help narrow focuso This raises a question for researchers: are we ok with
students thinking broadly about this question (e.g. including dining hall staff and campus security staff)?
• Formatting problemo Response options ranged from Poor=1 to Excellent=7o Need to add an NA option
Example 2 (cont.)
![Page 16: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/16.jpg)
Question: During the current school year, in about how many of your courses did you do the following?
Example 3
Participated in a community-based project as part of a regular course (i.e., service-learning)
![Page 17: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/17.jpg)
• Language problemoThis problem was site specific
• Inclusion/Exclusion problemoStudents wondered what activities to include • What counts as a “community-based” activity, and what did we mean by “community” (campus only? Surrounding neighbor/town too? Elsewhere in US/overseas?) • Some students thought of volunteering on their own, not in connection to a particular course
Example 3 (cont.)
![Page 18: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/18.jpg)
Question: Which of the following have you done or do you plan to do before you graduate from your institution?
Example 4
Participate in a formal program where groups of students take two or more classes together (sometimes called a learning community)
![Page 19: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/19.jpg)
• Language problemo As with service learning, students’ understanding
of this question was also site specific
• Formatting problemo Location on survey made students less likely to
complete the item if they didn’t understand itoSolution: We restructured where the item
appeared in the instrument• We moved the item lower on the page (it
originally was the first question at the top of the page)• We moved the item lower within the question
itself (it originally was the first item in series of 6 sub-questions
Example 4 (cont.)
![Page 20: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/20.jpg)
Question: In a typical week this year, about how many total pages have you read for all of your courses?
Example 5
![Page 21: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/21.jpg)
• Computation problem o Difficult for students to compute number
of pages, especially if sources didn’t have page numbers (e.g. web pages, e-readers)
• Inclusion Exclusion problemo Many students didn’t count readings in
non-traditional book format (e.g. readings online, lab reports)
o Some students counted what they actually had read, but some said that they counted all the pages assigned in the course, even those that they had not actually read.
Example 5 (cont.)
![Page 22: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/22.jpg)
Applications to Your Campus
• How might these methods help triangulate NSSE results on your campus?
• What questions are you most concerned about in terms of what your students mean by their responses?
• Are there item terms that may have less face validity with your student populations?
• How might it help to know more about students’ interpretations of response option in terms of what to do with findings?
• How might you initiate this activity on your campus? Who might be interested in this work? Who should conduct the interviews?
![Page 23: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/23.jpg)
Conclusion • Cognitive interviews and focus
groups can provide students an opportunity to reflect on their behaviors in college• Institutions gain concrete
information about THEIR students experiences, perceptionso Can then be more explicit about opportunities
and academic services of which students can take advantage
![Page 24: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/24.jpg)
Contact Information
• Heather Haeger – email: [email protected]• Amber D. Lambert – email:
[email protected]• Jillian Kinzie– email:
[email protected]• James Gieser – email:
![Page 25: Using Cognitive Interviews to Improve Survey Instruments](https://reader035.fdocuments.us/reader035/viewer/2022062810/56815e94550346895dcd2363/html5/thumbnails/25.jpg)
Introducing Updated NSSE!
• Retains NSSE’s focus on diagnostic & actionable information
• New Engagement Indicatorso Academic challengeo Deep approaches to learningo Collaborative learningo Quantitative reasoningo Experiences with facultyo Campus environmento Interactions with diversity
• Modules• New & Updated Items• Comparisons to Prior-Year
Results• FSSE & BCSSE Updates
Register Now for NSSE 2013
(deadline Sept. 25)