Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1....
Transcript of Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1....
![Page 1: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/1.jpg)
Population Research Seminar Series
Session 3: Evaluating Survey Questions
Jack Fowler
Center for Survey Research
Survey and Statistical Methods Core
![Page 2: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/2.jpg)
There are 7 kinds of standards
for questions
1. Ask the right question
2. Cognitive standards
3. Usability standards
4. Interpersonal standards
5. Psychometric standards
6. Multi-mode standards
7. Multi-lingual standards
![Page 3: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/3.jpg)
Ways to evaluate survey
instruments and wording
1. Appraisal forms
2. Focus groups
3. Cognitive testing
4. Pretests with observation and/or debriefing (for self administration)+ paradata
5. Pretests with behavior coding (for interviews)
6. Split-ballot experiments
7. Analytic assessments of reliability and validity
![Page 4: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/4.jpg)
Appraisal forms
• Can flag issues that are known to affect
understanding or usability
• However, most of them rely on some
judgments (not clear, difficult to recall) that
require additional information
![Page 5: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/5.jpg)
Focus groups
• Recruit a few groups of 6-8 people to
come talk about the issues the survey will
cover
![Page 6: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/6.jpg)
Focus groups
• Content standards– What people’s relevant experiences and
situations are
– What people know and think about a topic
– What kinds of answers they give
• Cognitive standards– Vocabulary—what different candidate words
mean to people and what words they use
– What questions they can answer
![Page 7: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/7.jpg)
Strengths of focus groups
1. An efficient way to gather info about
several people at a time
2. Sometimes being in a group stimulates
ideas or raises issues that would not be
found one-on-one
![Page 8: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/8.jpg)
Weakness of focus groups
• Do not get to probe into the way
individuals understand and answer
questions
• May not hear all the issues because each
person only gets so much air time
• That’s what one-on-one cognitive
interviews do
![Page 9: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/9.jpg)
Cognitive testing—a little history
• 1984 an NSF conference brought together
survey researchers and cognitive scientists
• NCHS established a laboratory with NSF
funding in the latter 1980s
• Bureau of the Census started a lab a little
later
• Cognitive testing did not start to become
common until the early 1990s
![Page 10: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/10.jpg)
What that means
• Questions that were designed before the
mid-1990s were not usually evaluated for
whether or not people understood the
questions or their answers meant what the
researchers hoped they meant
![Page 11: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/11.jpg)
Answering questions
1. Comprehend what is being asked for
2. Having information relevant to the answer
3. Working with the information to put it in a
form needed to answer
4. Providing the answer
![Page 12: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/12.jpg)
Error happens
• When one of these four steps is handled
problematically, validity is at risk
• The goal of question design is to minimize
the risk of problems at each of the steps of
the question answering process
• The goal of cognitive testing is to evaluate
how well each of these steps is performed
when someone answers a question
![Page 13: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/13.jpg)
Cognitive testing
• Recruit volunteers
• Usually pay them
• Interviews can take 1 -1.5 hours
• Interviews are almost always tape
recorded and reviewed after interview
![Page 14: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/14.jpg)
Protocols
• There is a lot of diversity in the way
interviews are done:
• Think aloud
• Follow-up probes after first asking
respondent to answer the test question
• Probes can be highly structured or
interviewers can be given a lot of flexibility
![Page 15: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/15.jpg)
Potentially probes can focus on
each aspect of the process1. What the question is asking
2. What the respondent knows or thinks
3. Refinement needed to get material needed for answer
4. Turning relevant material into an answer
![Page 16: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/16.jpg)
What question means
• IN THE PAST YEAR, HOW MANY TIMES
HAVE YOU SEEN OR TALKED WITH A
DOCTOR ABOUT YOUR OWN HEALTH?
Could you say in your own words what the
question is asking?
Do you think the question includes times when
you talked with a doctor on the telephone?
Do you think the question includes times when
you exchanged e-mails with a doctor?
![Page 17: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/17.jpg)
What the respondent knows or thinks
• How many times do you think you have been to
a doctor’s office about your own health in the
last year?
• How do you remember that?
• Have you had your eyes checked in the last year
by a doctor? (Did you include that in your
answer?)
• Was there a time when you saw a nurse
practitioner, but not the doctor? (Did you count
that?)
![Page 18: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/18.jpg)
Refining info to get close to answer
• So, could you take me through the process
you went through to decide how many times
you saw or talked with a doctor in the last
year?
• Were there any doubtful cases that you
decided to leave out that you thought about
including?
• Any that you included that you considered
leaving out?
![Page 19: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/19.jpg)
Continued
• How confident are you that you have the
right answer?
• If you are in error, what would you guess
is the most likely kind of error you could
have made?
![Page 20: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/20.jpg)
Providing an answer
• Did you consider more than one possible
answer before you gave your answer?
• How did you decide which one to give?
• Is there any way in which the answer you
gave might not be a good reflection of
what the true answer is?
![Page 21: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/21.jpg)
Example
• Do you think we are doing too much, too
little or about the right amount to fight
terrorism?
![Page 22: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/22.jpg)
Example
• How would you rate the importance of the
following behaviors that might promote good
health—very important, somewhat important,
or not important at all?
• Doing moderate exercise for 20 minutes at
least 3 days a week
• Having at least 2 glasses of wine, cans of
beer or 1.5 oz drinks of alcohol every day
![Page 23: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/23.jpg)
What cognitive testing can evaluate
• Asking right question?
• Cognitive standards
• Maybe psychometric issues
• Multi-lingual standards
![Page 24: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/24.jpg)
Conclusion
• The results depend on the investigator
juxtaposing the information provided in the
cognitive interview and the idealized way
that a respondent should answer if the
answer is going to be a ―valid‖ measure of
the target construct
![Page 25: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/25.jpg)
Usability testing
• Observation
• Debriefing
• Analysis of errors of navigation
• Paradata (when computer based)
![Page 26: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/26.jpg)
Pretests with behavior coding
Pretests of interviews with some people like
the planned respondents have been
standard for years
Main input from interviewer debriefing
Not very reliable
Not very informative
![Page 27: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/27.jpg)
What should question-answer
process look like?
1. I reads question exactly as worded
2. R understands question as intended
3. R retrieves information needed to answer
4. R puts answer in form required and tells
interviewer
![Page 28: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/28.jpg)
Premises of behavior coding
• 1. Deviations from this ideal may reflect
problems that are threats to validity of data
• 2. The wording of a question often is the
direct cause of these problems
• 3. The presence of problems can often be
inferred from the behavior of interviewers
or respondents
![Page 29: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/29.jpg)
How to do behavior coding
• Can be done live using an observer
• Mainly done by recording interview and
then having trained coders listen to
recordings
• NOTE: Experience shows that almost
everyone who is willing to be interviewed
agrees to be tape recorded
![Page 30: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/30.jpg)
What to code
• Unit of observation is usually the question
• The core focus of most coding is to count
deviations from an ideal question and
answer process
![Page 31: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/31.jpg)
Question reading
• Read exactly as worded
• Minor changes
• Major changes
• Interrupted
![Page 32: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/32.jpg)
Question answering
• Adequate (codable) answer given
• Inadequate (uncodable) answer given
• Qualified answer (―I think‖ ―It might be..‖)
• Refusals and ―don’t knows‖
![Page 33: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/33.jpg)
Other aspects of R behavior
• Asks for clarification
• Asks for all or part of question to be
repeated
![Page 34: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/34.jpg)
Sample output for each question
• Often results are reported like this:
– % read exactly as worded
– % interrupted reading
– % asked for clarification
– % gave inadequate answer
![Page 35: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/35.jpg)
Let’s try some behavior coding
![Page 36: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/36.jpg)
What do results mean?
• Requests for clarification often mean there
is an unclear term or concept.
• When did you move to New York?
• Does that mean the city, the metropolitan
area or the state?
![Page 37: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/37.jpg)
What do results tell us?
( Some examples)
1.Interruptions often occur when there is
dangling material in the question, such as
a definition after the question.
• How many children do you have? Do not
include step children.
![Page 38: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/38.jpg)
What do results mean?
2. Inadequate answers often occur when it is
unclear how to answer the question.
• When did you move to New York?
• Unclear whether interviewer wants date,
years ago, or stage in life cycle—and how
precise it has to be.
![Page 39: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/39.jpg)
What do results mean?
3. Requests for clarification often mean
there is an unclear term or concept.
• When did you move to New York?
• Does that mean the city, the metropolitan
area or the state?
![Page 40: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/40.jpg)
Effects on Data
• One of the clearest findings is that the
higher the rate at which interviewers have
to probe to get an adequate answer, the
higher the interviewer-related error
![Page 41: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/41.jpg)
More effects on data
• There is evidence that qualified answers
and response latency are related to the
―accuracy‖ of responses
![Page 42: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/42.jpg)
Strengths of behavior coding as question
evaluation method
• Low cost—easily integrated into pretest
• Evaluation is of how questions work under
realistic data collection conditions
• Results are reliable
• Results are objective—i.e. not dependent
on an individual’s subjective assessment
• Results are quantitative
![Page 43: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/43.jpg)
Weaknesses of Behavior Coding
1. Sometimes hard to diagnose reason for
results and how to fix it
2. Can’t always tell if an observed problem
actually affects data
3. Some question problems (such as
comprehension) do not show up in
behavior coding
![Page 44: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/44.jpg)
Split-ballot experiments
• About how many months has it been since
you last saw or talked to a dentist? Include
all types of dentists, such as orthodontists,
oral surgeons, or all other dental specialists,
as well as dental hygienists.
• About how many months has it been since
you last went to a dentist office for any type
of dental care?
![Page 45: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/45.jpg)
Results
ORIGINAL ALTERNATIVE
6 Months or less 60% 57%
More than 6 months
but not more than 1
year
14% 18%
More than 1 year 26% 25%
TOTAL 100%
(n=77)
100%
(n=79)
![Page 46: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/46.jpg)
Split-ballot experiment
• In the last 12 months, how often did you
get an appointment for regular or routine
health care as soon as you wanted -
always, usually, sometimes, never?
(“Always” recoded to “Yes”)
• In the last 12 months, were you always
able to get an appointment as soon as you
wanted?
![Page 47: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/47.jpg)
Results
ORIGINAL ALTERNATIVE
Yes 47% 66%
No 53% 34%
Total 100%
(261)
100%
(299)
![Page 48: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/48.jpg)
Split-ballot experiments are needed to
find out if wording affects estimates
They also are key to evaluating whether or
not questions meet
Multi-mode standards
Multi-lingual standards
Validity must be established via theory or
consistency with other evidence
![Page 49: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/49.jpg)
Psychometric analysis
• Reliability
• Validity
![Page 50: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/50.jpg)
Reliability
• Extent to which same people who should
not have changed much give same
answer at two points in time
• Extent to which people for whom true
value is the same give the same answers
![Page 51: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/51.jpg)
Validity
• Extent to which answers measure what
they are supposed to measure
• Note: Answers can be reliable and yet not
be valid
![Page 52: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/52.jpg)
How to measure validity
• Correspondence between answers and
some other ―true‖ measure of what is to be
measured. Record-check study is good
example.
• Correspondence between answers and
other answers or measures that are
supposed to be measuring or related to
the same thing
![Page 53: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/53.jpg)
And what is bias?
• A systematic difference between survey
answers and the ―true‖ scores
• Example: On average, people report their
weight as lower than a scale reading
• Answers can be biased and valid: that is,
they can be systematically different by
correlated highly
![Page 54: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/54.jpg)
More on Bias
• And bias is only meaningful with respect to
factual questions
• Since it is not possible to obtain a
calibrated ―true score‖ for a subjective
state (happiness?), bias is meaningless
![Page 55: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/55.jpg)
What does it mean to have a
“validated measure”
• Validity = a correlation coefficient
• If it is a positive number, that means there
is evidence that the answers measure the
―true scores‖ to some extent……
![Page 56: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/56.jpg)
Qualifications..
• In the population in which the data were collected
• Under the conditions in which the data were collected
• To the extent that you can be convincing that whatever it is correlated with has something to do with the true value you are trying to measure
![Page 57: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/57.jpg)
Psychometric Testing
• In the end, evidence that you are
measuring what you want to measure is
the ultimate question standard.
• Often, it is hard to gather that evidence
• And it usually requires forethought and
special effort to have the data to assess
validity
![Page 58: Evaluating Survey Questions · 2011. 3. 23. · Ways to evaluate survey instruments and wording 1. Appraisal forms 2. Focus groups 3. Cognitive testing 4. Pretests with observation](https://reader035.fdocuments.us/reader035/viewer/2022071410/6104fecf5a13b979f2374a05/html5/thumbnails/58.jpg)
Thank you.