Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?
-
Upload
aidan-neal -
Category
Documents
-
view
39 -
download
4
description
Transcript of Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?
![Page 1: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/1.jpg)
Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?
Klaus Gebel
Centre for Chronic Disease Prevention
James Cook University
![Page 2: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/2.jpg)
German tourists in SpainGerman tourists in Spain
![Page 3: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/3.jpg)
Political efficacy in China and MexicoPolitical efficacy in China and Mexico
![Page 4: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/4.jpg)
Health inequalities between Indigenous and Health inequalities between Indigenous and non-Indigenous Australiansnon-Indigenous Australians
![Page 5: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/5.jpg)
Anchoring vignettesAnchoring vignettesExample: Self-rated healthExample: Self-rated health
ExcellentExcellent Very goodVery good GoodGood FairFair PoorPoor
Most common measure of health Often negative association with actual health
SF-36: In general, would you say your health is:
![Page 6: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/6.jpg)
Anchoring vignettesDifferential Item Functioning (DIF)
SelfSelf11
Low Low
HighHigh
Jo1
Pat1
Chris1
SelfSelf22
Jo2
Pat2
Chris2
Low
High
Chris2
Jo2
Pat2
SelfSelf22
![Page 7: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/7.jpg)
Recoding depending on self-rating and rating of vignettes
Self & vignettes Recoding
x>V1>V2>V3 7x=V1>V2>V3 6V1>x>V2>V3 5V1>x=V2>V3 4V1>V2>x>V3 3V1>V2>x=V3 2V1>V2>V3>x 1x>V1>V2=V3 7x>V1=V2=V3 7
![Page 8: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/8.jpg)
Next steps
Conversion from 7-point scale back to 5-point scale
Paired t-test to compare DIF-adjusted vs unadjusted scores
Wald’s test to compare associations of DIF-adjusted vs unadjusted scores with continuous or binary outcome measures
![Page 9: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/9.jpg)
Design of vignettes
Ideally equally spaced through distribution of self-assessments
Avoid extremes Gender specific Ideally 2-3 vignettes Design vignettes in focus group Test them with sub sample
![Page 10: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/10.jpg)
Example: Safety from traffic and cycling / walking
Safety and accessibility are the two most important environmental factors affecting activity participation across the lifespan.
DiPietro 2012, Human Kinetics
“”
![Page 11: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/11.jpg)
Background
![Page 12: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/12.jpg)
Background
Safety from traffic
Speed of traffic
Volume of traffic
Separation from traffic
![Page 13: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/13.jpg)
Results
Very unsafe
A little unsafe
Neither safe nor unsafe
A little safe
Very safe
How safe do you think it is to ride a bicycle in your local area?
1 2 3 4 5
Predictive validity of perceived safety from traffic
![Page 14: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/14.jpg)
Methods
Baseline data collected 2013Cycling and walking measured with travel diary
Outcome measures
• Cycle to work/study y/n (Probit)
• Frequency of cycling (ordered Probit)
• Mins/week of cycling and walking (Poisson)
![Page 15: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/15.jpg)
Results
n=871 58% women 18-55 (37±11.1 years) 14% use bicycle as main way travel mode to
work/study 29% cycle at least 1-2 days/week
![Page 16: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/16.jpg)
Results
Perceived safety from traffic
![Page 17: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/17.jpg)
Results
Ratings of vignettes
0
100
200
300
400
500
600
Very unsafe A littleunsafe
Neithersafe norunsafe
A little safe Very safe0
50
100
150
200
250
300
350
Very unsafe A little unsafe Neither safenor unsafe
A little safe Very safe0
100
200
300
400
500
600
Very unsafe A little unsafe Neither safenor unsafe
A little safe Very safe
![Page 18: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/18.jpg)
Results
DIF adjusted perceived safety from traffic
![Page 19: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/19.jpg)
Results
Unadjusted Adjusted for DIF
Wald’s test p-value
Cycle to work/study y/n 0.63 0.79 0.03
Frequency of cycling 0.71 0.78 0.06
Mins/week of cycling 0.53 0.78 >0.01
Mins/week of walking 0.48 0.62 0.01
Predictive validity of perceived safety from traffic
![Page 20: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/20.jpg)
Conclusion
Significant scale perception bias Anchoring vignettes predictive validity Anchoring vignettes might be a powerful tool
to improve validity of Likert-scale items
![Page 21: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/21.jpg)
Opportunistic evaluation of new bicycle paths in Cairns
![Page 22: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/22.jpg)
Bike path evaluation in CairnsBike path evaluation in Cairns
![Page 23: Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?](https://reader035.fdocuments.us/reader035/viewer/2022062720/5681351f550346895d9c817b/html5/thumbnails/23.jpg)
Thank you for your attention!Thank you for your attention!
Questions?Questions?