Improving Survey Questions and Responses
description
Transcript of Improving Survey Questions and Responses
1
Survey Questions and Responses:Art and Research in Survey Design
Amy A. Germuth, Ph.D.
Prepared for the Research Triangle Park Evaluation MeetingAugust 13, 2010
Copyright 2010Amy A. Germuth, Ph.D.
2
3
Why do we care?
• Bias – decreases the reliability with which we can speak about results
• Multiple types of bias in surveys – we are interested in reducing bias associated with Measurement Error
4
Survey Questions
Can we identify any rules for what makes a good survey question?
5
Let’s take a survey!
6
What makes a good survey question?
1. The question needs to be consistently understood.
2. The question needs to be one that can be consistently administered or communicated to respondents.
3. What constitutes a good answer needs to be communicated via the question.
4. All respondents should have access to the information needed to answer the question (unless the question is meant to measure knowledge).
5. The questions must be ones to which respondents are willing to provide the answer called for.
7
Survey Responses
What can we say about best practices regarding survey response options?
8
Questions- Survey
Response Scales
Qualitative Anchors ?
Mid-Points or not?
Quantitative Anchors?
Number of Response Options?
9
1. How often do you use a hardcover/paperback telephone directory? Never / Sometimes / Frequently N________________S________________________________________F__________________ N_____________________________________S____________________________________F ______________N_______________________S_______________________F______________
Never, Sometimes, Frequently
10
3. To what degree do you agree with the following statement: “The NC economy is stronger than that of the US in general”? Disagree / Somewhat Disagree / Neither Disagree or Agree/ Somewhat Agree / Agree SD_________________D_______________ND/A________________A_________________SA _____SD______D_____________________ND/A_______________________A_____SA_____
The Agreement Continuum – 5 Point Scale
11
4. Please rate your agreement to this statement: The weather this past week has been great. strongly disagree, disagree, slightly disagree, slightly agree, agree, strongly agree SD_______D_______SLD___________________________________SLA______A_______SA SD_____________D_____________SLD____________SLA____________A____________SA
The Agreement Continuum – 6 Point Scale
12
If quantitatively anchored, would potential respondents be less confused about where
response options lie on a scale?
13
What do Quantitative anchors provide us?
• May more accurately describe qualitative descriptions• Provide scaling to qualitative descriptions• May be used to indicate response options are equal
interval in nature• Improves reliability decreases bias
14
Here are responses from just 6 respondents. I color-coded each respondent’s numeric representation of the descriptors as follows:
None One A Couple A Few Several
Even with only six respondents you see a pattern emerging that is replicated in the larger sample size.
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
Withdrawing Cash Question
15
1 2 3 4 5 6 7 8 9
Horrible Very Poor Poor Slightly less than Average
Average Slightly better than average
Good Very Good Excellent
What Service?
Atrocious Poor Leave me alone so I can eat
Average Above average
5-star
Needs improvement in all areas
Some improvement needed
Slightly below average
Average Slightly above average
Good Outstanding Superb
Dismal Lousy Poor Lame Average Good Great Excellent Exceptional
Very poor Poor Average Above average
Excellent
Extremely bad
Very bad Bad Somewhat bad
Neutral Somewhat good
Good Very Good Excellent
And our scale would be: 1 = Horrible/Very poor, 2 = Poor, 3 = Average, 4 =
Good, 5 = Very good/Excellent
Restaurant Service Question
16
What are problems with Qualitative anchors?
• Often ambiguous – not viewed reliably by respondents = introduces bias
• Fail to convey equal intervals• If multiple response options (e.g., 7 or 9) hard to
differentiate
17
How Many Response Options?
It depends:1. Cognitive Abilities – provide fewer responses for
children2. Middle Option (Neutral) – will determine if number
of response options are even or odd3. From findings above – is 7 too many? 5 or 6 appear
to be optimal as should provide enough variance in responses and can be both qualitatively and quantitatively anchored.
18
Odd-Numbered Scales
1. Given our state budget issues, do you think that the state sales tax should be increased or decreased?
2. Should more or less time be provided for Physical Education to students in North Carolina's public schools?
versus3. How true is this statement for you: “I support a
woman’s right to choose”?
19
• Persons lean towards one or the other polar alternatives and thus legitimate to force a selection.
• Persons really do favor the middle position and if forced to choose one of the alternatives will contribute to random or systematic error.
Middle Alternatives – Two Views
20
• 10-20% of respondents will choose that option• The decline appears to effect the polar alternative
categories equally • Persons who have low "intensity" about a subject are
most likely to choose a middle alternative . • Lack of or lower education does not appear to affect
whether someone choose a middle response or not.• Cognition does affect choices which is why one should be
careful how many choices children are provided on surveys.
If A Middle Alternative is Offered
21
Resources
• DeVellis, R. F. (2003). Scale development: Theory and applications, 2nd Edition, Thousand Oaks, CA: Sage.
• Dillman, D. A. (1999). Mail and internet surveys: The tailored design method (2nd. Ed.). New York, NY: John Wiley & Sons.
• Fowler, F.J. Jr. (1995). Improving survey questions: Design and evaluation. Thousand Oaks, CA: Sage
• Schuman, H. & Presser, S. (1996). Questions and Answers in Attitude Surveys. Thousand Oaks, CA: Sage
22
Amy A. Germuth, Ph.D.President, EvalWorks LLC
150 Solterra wayDurham, NC 27705
(919) [email protected]
Website: www.EvalWorks.comBlog: www.EvalThoughts.com
Contact Information