Ray poynter advances in quant - 2011
-
Upload
ray-poynter -
Category
Marketing
-
view
28 -
download
0
Transcript of Ray poynter advances in quant - 2011
A Presenta*on from The NewMR “Advances in Quan*ta*ve Research” Event
19 September, 2012
Event sponsored by Affinnova All copyright owned by The Future Place and the presenters of the material
For more informa=on about Affinnova visit www.affinnova.com For more informa=on about NewMR events visit newmr.org
Scales in online surveys – An updated set of best prac=ces Ray Poynter, Vision Cri0cal University
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Agree/Disagree scales in online surveys LeK, right, top, and middle thoughts
Ray Poynter Director Vision Cri=cal University
@ Vision Cri=cal
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Agree/Disagree Scales Sometimes called Likert Scales
– More strictly Likert Items
Typical format – Agree strongly – Agree – Neither agree nor disagree – Disagree – Disagree strongly
Many variants exist
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Issues
Acquiescence bias
Left versus right
Top versus bottom
Mid-points
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Acquiescence Bias Saris, Krosnick, and Shaeffer http://goo.gl/LvAwS
Too many people agree!
Traditional solution – Include each factor several times, with slightly
different wordings – Phrasing half positive, half negative
Not practical in modern market research – Ask positively and only once
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Item Order Response Bias Many sources (e.g Krosnick, Chan, Friedman etc)
Items that appear first are more likely to be accepted
A Agree Neither Agree nor Disagree Disagree B Disagree Neither Agree nor Disagree Agree
In A there will be more Agrees In B there will be more Disagrees
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Left to Right To what extent do you agree or disagree with the statement Diet Coke, Orange juice, Milk and Red wine are healthy drinks?
A
B
Cells approx. 500, Vision Cri=cal Springboard UK Omni, 2012
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Left to Right Diet Coke Healthy
5%
10%
23%
35%
27%
2%
11%
24%
33% 31%
Agree strongly Agree Neither Nor Disagree Disagree Strongly
Diet Coke A Diet Coke B +/-‐ 3% = sig dif
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Left Versus Right
% Agree Strongly Agree on Left Agree on Right Diet Coke 5% 2% Orange Juice 21% 16% Milk 38% 30% Red Wine 6% 4%
% Disagree Strongly Disagree on
Right Disagree on
Left Diet Coke 27% 31% Orange Juice 1% 1% Milk 1% 1% Red Wine 5% 9%
People (in this context, in the UK) are more likely to pick the le` side of the scale
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Real but not big!
Healthy? %
Agree Mean
(5 points) Milk A 89% 4.2 Milk B 83% 4.1 Orange Juice A 76% 3.9 Orange Juice B 72% 3.8 Red Wine A 36% 3.1 Red Wine B 33% 3.0 Diet Coke A 15% 2.3 Diet Coke B 12% 2.2
Agree = Agree + Strongly Agree
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Observations so far 1. Acquiescence bias leads to higher
agreement
2. Putting agree on left increase the bias
3. Putting agree on right reduces the bias
4. The effect is not very large
5. So, DON’T MIX AND MATCH
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Top to Bottom A B
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Top to Bottom Water Healthy
83%
16%
1% 0% 0%
66%
29%
2% 2% 2%
Agree strongly Agree Neither Nor Disagree Disagree Strongly
Water A Water B +/-‐ 3% = sig dif
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Top Versus Bottom
% Agree Strongly Agree at Top Agree at Bottom Water 83% 66% White Wine 3% 1% Apple Juice 19% 13% Coke 2% 1%
% Disagree Strongly Disagree at
Bottom Disagree at
Top Water 0% 2% White Wine 5% 11% Apple Juice 0% 2% Coke 39% 46%
People (in this context, in the UK) are more likely to pick the top items
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Don’t Mix and Match
Horizontal -‐ standard Ver*cal -‐ visual % Agree + Agree Strongly
Agree Le`
Agree Right
Agree Top
Agree Bofom
Water 95% 94% 98% 95% White Wine 19% 14% 26% 18% Apple Juice 66% 66% 77% 70% Coke 5% 6% 6% 5%
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Response Order Thoughts
There are order effects
Putting Agree on the left/top compounds acquiescence
AOTBE put Disagree left/top
Don’t mix and match
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Mid-points
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Adding a Don’t Know
DKs: Water 0%, Red Wine 2%, Milk 0%, Coffee 0%, Diet Coke 0%
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Moving the Neither Nor
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Moving the Neither Nor
0%
10%
20%
30%
40%
50%
60%
70%
80%
Disagree Strongly
Disagree Neither agree nor ..
Agree Agree Strongly
Water -‐ Mid Water -‐ End
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Moving the Neither Nor
0% 5%
10% 15% 20% 25% 30% 35% 40% 45%
Disagree Strongly
Disagree Neither agree nor ..
Agree Agree Strongly
Red Wine -‐ Mid Red Wine -‐ End
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Moving the Neither Nor
0%
10%
20%
30%
40%
50%
60%
Disagree Strongly
Disagree Neither agree nor ..
Agree Agree Strongly
Milk -‐ Mid Milk -‐ End
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Moving the Neither Nor
0%
10%
20%
30%
40%
50%
60%
Disagree Strongly
Disagree Neither agree nor ..
Agree Agree Strongly
Coffee -‐ Mid Coffee -‐ End
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Moving the Neither Nor
0% 5%
10% 15% 20% 25% 30% 35% 40% 45% 50%
Disagree Strongly
Disagree Neither agree nor ..
Agree Agree Strongly
Diet Coke -‐ Mid Diet Coke -‐ End
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Neither Nor
In this sort of scale – Where the ends are opposites – And where people who do not agree or
disagree are likely to be in the middle – Neither agree nor disagree works well as a
mid-point in a five point scale Even if it does not always make semantic sense!
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Concluding Thoughts 1. The way we ask questions shapes the
answers
2. We can’t remove bias – Recognise it – Hold it constant
3. Most standard formats work reasonably well
4. But don’t mix and match!
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Thank you Ray Poynter
Director Vision Cri=cal University @ Vision Cri=cal
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Q & A
Sue York NewMR
Ray Poynter Vision Critical University
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Ray Poynter Vision Cri*cal University @Vision Cri*cal www.visioncri*caluniversity.com ray.poynter@visioncri*cal.com @RayPoynter
www.linkedin.com/in/raypoynter
A Presenta*on from The NewMR “Advances in Quan*ta*ve Research” Event
19 September, 2012
Event sponsored by Affinnova All copyright owned by The Future Place and the presenters of the material
For more informa=on about Affinnova visit www.affinnova.com For more informa=on about NewMR events visit newmr.org
Scales in online surveys – An updated set of best prac=ces Ray Poynter, Vision Cri0cal University