Reducing Bias in Public Opinion Polls
-
Upload
aacarter -
Category
Technology
-
view
1.936 -
download
1
description
Transcript of Reducing Bias in Public Opinion Polls
![Page 1: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/1.jpg)
Reducing Bias in Public Opinion Polls
Autumn CarterProfessor Jon Krosnick
Summer Research College 2009
![Page 2: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/2.jpg)
Satisfy Your Constituents...
Or Suffer the Consequences.
![Page 3: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/3.jpg)
So, how do we effectively determine what Americans
are feeling?
![Page 4: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/4.jpg)
Public Opinion Polling
![Page 5: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/5.jpg)
But Are the Polls Accurate?
?=
![Page 6: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/6.jpg)
Critics Don't Think So...
“The dirty little secret of the polling industry is that, all too often, its findings are based on flawed methodology and dubious assumptions.”
-Arianna Huffington, Huffington Post Founder
![Page 7: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/7.jpg)
Sample Public Opinion Question
Form 1
Individuals are more to blame than social conditions for crime and lawlessness in this country.
Agree
Disagree
![Page 8: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/8.jpg)
Sample Public Opinion Question
Form 1
Individuals are more to blame than social conditions for crime and lawlessness in this country.
Agree: 59.6%
Disagree: 40.4%
![Page 9: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/9.jpg)
Sample Public Opinion Question
Form 1
Individuals are more to blame than social conditions for crime and lawlessness in this country.
Agree: 59.6%
Disagree: 40.4%
Form 2
Social conditions are more to blame than individuals for crime and lawlessness in this country.
Agree
Disagree
![Page 10: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/10.jpg)
Sample Public Opinion Question
Form 1
Individuals are more to blame than social conditions for crime and lawlessness in this country.
Agree: 59.6%
Disagree: 40.4%
Form 2
Social conditions are more to blame than individuals for crime and lawlessness in this country.
Agree: 56.8%
Disagree: 43.2%Acquiescence Effect: 16.4%
![Page 11: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/11.jpg)
What is Acquiescence?
“...a presumed tendency for respondents to agree with attitude statements presented to them.”
-Schuman and Presser (1981)
![Page 12: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/12.jpg)
Why do Respondents Acquiesce?
Norms of Conduct: Be polite and agreeable
Status Differential: Defer to higher status
Satisficing: Choosing the easiest response because it requires less thinking
![Page 13: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/13.jpg)
Can we reduce acquiescence by
administering the survey via the Internet instead of
by telephone?
![Page 14: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/14.jpg)
Would Internet Administration Reduce Acquiescence?
Norms of Conduct: Be polite and agreeable Do you have to be polite to a computer?
Status Differential: Defer to higher statusIs there a status difference without a human interviewer?
Satisficing: Choosing the easiest response because it requires less thinking
Are people less likely to satisfice on Internet surveys? (Chang & Krosnick, in press)
![Page 15: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/15.jpg)
The Present Research
Replicate Schuman & Presser (1981)
Mode effects: Telephone vs. the Internet
Sample effects: Representative vs. Non-Representative
![Page 16: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/16.jpg)
How Do You Survey a Representative Sample Via the Internet?
Knowledge Networks randomly dialed phone numbers and invited people to join a panel (of about 50,000 adults).
Provided Internet access to homes without it.
Periodically emails about 1,000 panelists to complete surveys and receive incentives.
![Page 17: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/17.jpg)
A Classic Acquiescence Experiment
Form 1
Individuals are more to blame than social conditions for crime and lawlessness in this country.
Agree
Disagree
Form 2
Social conditions are more to blame than individuals for crime and lawlessness in this country.
Agree
Disagree
![Page 18: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/18.jpg)
Replication in a National Survey
Telephone survey of a representative sample:
Internet survey of a representative sample:
* p < 0.05+ p <0.10
11.5%*
7.1%+
TelephoneXInternet Interaction: n.s.
![Page 19: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/19.jpg)
What about Internet surveys of people who are not representative of the
national population?
![Page 20: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/20.jpg)
Non-Probability Sample Internet Surveys
Volunteers click on banner ads and “opt-in” into the panel.
![Page 21: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/21.jpg)
You've all seen these...
![Page 22: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/22.jpg)
Non-Probability Sample Internet Surveys
Volunteers click on banner ads and “opt-in” to the panel.
Participants receive money or prize.
Perhaps they acquiescence less because they like taking surveys and have more practice.
In fact, previous research has found less acquiescence vs. a telephone survey (Chang and Krosnick, in
press)
![Page 23: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/23.jpg)
Acquiescence BiasT
elep
hone
Pro
babi
lity
Web
Non
-Pro
b W
eb 1
Non
-Pro
b W
eb 2
Non
-Pro
b W
eb 3
Non
-Pro
b W
eb 4
Non
-Pro
b W
eb 5
Non
-Pro
b W
eb 6
Non
-Pro
b W
eb 7
0.0
5.0
10.0
15.0
20.0
25.0
Acquiescence Bias
Survey
Acq
uies
cenc
e B
ias
(in p
erce
nt)
*
*
**
*
†*
† †
*
![Page 24: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/24.jpg)
If anything, non-probability samples increase acquiescence.
So we should be cautious when interpreting these
types of surveys.
![Page 25: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/25.jpg)
What if we just didn't ask questions that ask
respondents to agree or disagree with a
statement?
![Page 26: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/26.jpg)
Balanced Forced-Choice Questions
Form 1
Which do you think is more responsible for crime and lawlessness in this country: individuals or social conditions?
Individuals
Social Conditions
Form 2
Which do you think is more responsible for crime and lawlessness in this country: social conditions or individuals?
Social Conditions
Individuals
![Page 27: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/27.jpg)
Response Order EffectsT
elep
hone
Pro
babi
lity
Web
Non
-Pro
b W
eb 1
Non
-Pro
b W
eb 2
Non
-Pro
b W
eb 3
Non
-Pro
b W
eb 4
Non
-Pro
b W
eb 5
Non
-Pro
b W
eb 6
Non
-Pro
b W
eb 7
-6.0
-4.0
-2.0
0.0
2.0
4.0
6.0
8.0
Survey
Diff
eren
ce b
etw
een
For
ms
(in p
erce
nt)
ns nsnsnsnsnsnsnsns
![Page 28: Reducing Bias in Public Opinion Polls](https://reader035.fdocuments.us/reader035/viewer/2022070303/54b850014a7959814f8b4624/html5/thumbnails/28.jpg)
Conclusions
Acquiescence bias is real and can undermine public opinion polls and possibly the democratic process.
Internet administration did not significantly reduce acquiescence.
Non-probability sample surveys showed, if anything, more acquiescence.
BUT WAIT! You can simply ask the questions using the balanced forced choice format, eliminating order effects.