Democracy International - Afghanistan Public Opinion Poll December 2013
The Key Factor of Opinion Poll Quality• Declining trend of listed telephone directories 9 The Key...
Transcript of The Key Factor of Opinion Poll Quality• Declining trend of listed telephone directories 9 The Key...
The Key Factorof Opinion Poll Quality
Shaw Tao, Ph.D.Director, Data Services
Future of Public Opinion Poll• “I mean clearly something's wrong. I am
concerned about the impact on thecredibility of the profession.”--Bruce Cameron, 2012
14/05/2015The Key Factor of Opinion Poll Quality2
Jeff McArthur, Radio AM 640
Pollster Bruce Cameron, on CBC
• “Alongside Tim Hudak who announced he’dbe stepping down as leader…should be a lotof pollsters stepping aside as well…”--Jeff McArthur, 2014
• “A Dark Election Day for U.K.’s Pollsters”--Charles Forelle, May 8, 2015 Charles Forelle,Editor, The WSJ
• “If you really want to know a likely electionresult, ignore the polls and look at the bettingmarkets.”--John Fund, May 9, 2015
John Fund, columnist, National Review
Recent Polling Results 1
14/05/2015The Key Factor of Opinion Poll Quality3
British Columbia Election 2013
GRN LIB NDP Firm MethodError of
Top 3ElectionResult 8.1 44.1 39.7
5/12/2013 9.0 37.0 45.0 Ipsos Reid Online 13.3
5/12/2013 13.0 34.5 40.5 EKOS IVR 15.3
5/10/2013 9.0 36.0 45.0Angus Reid PublicOpinion Online 14.3
5/09/2013 14.0 31.0 45.0Justason MarketIntelligence Online 24.3
5/08/2013 8.0 41.0 43.0 Forum Research IVR 6.56
14/05/2015The Key Factor of Opinion Poll Quality4
Quebec Election 2014
CAQ LIB PQ Firm MethodError of
Top 3
Election Result 23.1 41.5 25.4
4/04/2014 25.0 39.0 27.0 Angus Reid Online 6.0
4/03/2014 23.0 44.0 24.0 Forum Research IVR 4.0
4/03/2014 23.0 38.0 29.0 Léger Marketing Online 7.2
4/03/2014 21.1 39.8 27.0 EKOS IVR 5.3
4/02/2014 19.0 37.0 28.0 Ipsos Reid Online 11.2
Recent Polling Results 2
14/05/2015The Key Factor of Opinion Poll Quality5
Ontario Election 2014
LIB NDP PC Firm MethodError of
Top 3
Election Result 38.7 23.7 31.2
6/11/2014 35.0 26.0 32.0 Abacus Data Online 6.8
6/11/2014 33.0 30.0 31.0 Ipsos Reid Online 12.2
6/11/2014 41.0 20.0 35.0 Forum Research IVR 9.8
6/10/2014 36.6 21.5 30.2 EKOS IVR 5.3
6/10/2014 36.0 26.0 32.0 Angus Reid Online 5.8
6/08/2014 35.0 24.0 36.0 Oraclepoll ResearchTelephone 8.8
Recent Polling Results 3
14/05/2015The Key Factor of Opinion Poll Quality6
Alberta Election 2015Date NDP PC WR Firm Top 3 Error
5/6/2015 40.6 27.8 24.2Election Results5/4/2015 45 23 23Forum Research 10.45/4/2015 42 23 27Insights West 9.05/3/2015 44.3 22.5 24EKOS 9.25/2/2015 42 21 24Forum Research 8.4
4/29/2015 44 21 26Mainstreet Tech 12.04/29/2015 37 24 26Ipsos Reid 9.24/29/2015 42.2 23.1 21.3EKOS 9.24/28/2015 39 20 27ThinkHQ 12.24/28/2015 38 30 24Leger Marketing 5.04/28/2015 38 24 21Return On Insight 9.64/27/2015 44 20 191ABVote (Google Surveys) 16.4
Recent Polling Results 4
Historical Poll Results
14/05/2015The Key Factor of Opinion Poll Quality7
Election # of Polls Top3 ErrorOntario 2014 10 9.5Ontario 2011 13 6.1Ontario 2007 6 4.0British Columbia 2013 11 15.0British Columbia 2009 3 8.3British Columbia 2005 4 7.4Quebec 2014 6 7.0Quebe 2012 7 7.6Quebe 2008 3 7.3Quebe 2007 4 11.5
Current vs. Past: Errors have DOUBLED
published studies
14/05/2015The Key Factor of Opinion Poll Quality8
• Models used for polling results (such as the“likely” model) ;
• Effectiveness of different polling methods(IVR/telephone vs. online) ;
• Low response rates from polls;• Low turnout of voters;• Voters changed their minds at the last minute.
What Went Wrong?
• Declining trend of listed telephone directories
14/05/2015The Key Factor of Opinion Poll Quality9
Year Households Records Coverage
1996 10,899,427 9,650,412 88.5%
2001 11,546,320 10,963,435 95.0%
2006 12,386,625 11,162,433 90.1%
2011 13,320,615 12,015,463 90.2%
2015 14,206,364 9,626,078 67.8%
Coverage Rate in Canada
Data Source Is The Key
14/05/2015The Key Factor of Opinion Poll Quality10
• Random Digit Dialing (RDD, landline or cellphones)• Address Based Sampling (ABS)• Online panels• Social Media
Remedies
14/05/2015The Key Factor of Opinion Poll Quality11
• Fast turnaround time• Low cost• Coverage of people missed in listed
telephone directories (young householdsand apartment renters)
• Low intrusions to people
Advantages of Online Panels
14/05/2015The Key Factor of Opinion Poll Quality12
• Extremely low coverage rate (below 5%): How toget the responses from other 95% population?
• Low response rate (4% or lower)• Acceptance rate: true response rate =• Distribution Bias
– Internet accessibility (seniors, rural areas)– “Professional respondents”
• Respondent Fatigue
Limitations of Online Panels
4% * x% = y%
14/05/2015The Key Factor of Opinion Poll Quality13
• Canadian Internet Use Survey, 2012 (conducted byStatistics Canada)– In 2012, 83% of Canadian households had access to the
Internet at home, compared with 79% in 2010– 98% households with incomes of $94,000+ had home
Internet access, compared with 58% of households inthe low incomes of $30,000 or less
– While 83% of Canadians aged 16+ used the Internet forpersonal use, the rate of those who are 65+ was only48%
Internet Accessibility
14/05/2015The Key Factor of Opinion Poll Quality14
• Disinclined to continue participating;• Lead to invalid responses (‘‘test wise’’ );• A panel contains 200,000 members:
– 38.4% in Ontario (76,800);– Based on 5% response rate, 20,000 samples required
for 1000 respondents;– The company conducted six polls between May 9 and
June 11 for Ontario Election 2014;– Many panel members could have received 2+ polling
requests in one month’s time.
Respondent FatigueAn overlooked issue
14/05/2015The Key Factor of Opinion Poll Quality15
Alf Landon, Republican Franklin Roosevelt, Democrat
Who Won 1936 PresidentElection?
14/05/2015The Key Factor of Opinion Poll Quality16
• Sample Size: 10 million prospective voters, 2.4million responded.
• Prediction: Landon would get 57% of the voteversus Roosevelt's 43%.
• Actual result: Roosevelt received 62% versus 38%for Landon, and won 46 out of 48 states.
1936 Liberty Digest Poll
14/05/2015The Key Factor of Opinion Poll Quality17
• Sample distribution bias– From telephone directories, club member lists, magazine
subscribers;– They are wealthier than average Americans;– They tended to vote Republican;– “Garbage in, Garbage out.”
• Non-response bias– 2.4 million out of 10 million responded (24%);– People who respond to surveys are different from people
who don't.
History repeats, lessons not learned
What Went Wrong?
14/05/2015The Key Factor of Opinion Poll Quality18
Error of Top 3Online IVR/Tel
Mean 11.2 7.9Median 11.2 6.6Standard Deviation 5.9 3.9Sample Variance 34.8 14.9Kurtosis 2.4 1.7Skewness 1.4 1.3Range 18.5 11.3Minimum 5.8 4.0Maximum 24.3 15.3Count 9 7
Online vs. IVR/Tel•Mean: 41.8% higher•STD: 51.5% higher•Range: 63.7% higher
Recent Poll Result Analysis
14/05/2015The Key Factor of Opinion Poll Quality19
• Selection Bias: If each member of the relevantpopulation does not have an equal chance ofending up in the sample.
• Any self-selected sample is basically worthless as asource of information about the population beyonditself.
• Low response-rate samples invite bias.
Statistics Principles
14/05/2015The Key Factor of Opinion Poll Quality20
• Mainly due to extreme low coverage, onlinepanels can only be considered asconvenience, instead of probability, datasources.
• They can only be used for pilot studies, notfor studies representing whole population.
• Need to educate clients: while timing andcosts are important, reliability is VITAL!
Conclusions
14/05/2015The Key Factor of Opinion Poll Quality21
Wrong!Election Polls
Non-Election PollsMarket Researches• • • • • • • • •
How to regain our credibility?
14/05/2015The Key Factor of Opinion Poll Quality22
For further discussion: Questions?
Thank you!Detailed analyses and references are available at:http://www.marketingresearch.org/article/key-cause-polling-quality-deterioration
14/05/2015The Key Factor of Opinion Poll Quality23
• Pressures from clients is the main driving factor– Timing (“Clients want results yesterday.”)– Costs
Why Online Method in Favor?