Post on 23-Feb-2016
description
VARIABILIT
Y OF L
IKERT SCALES
EFFECTS ON DATA
COLLECTIO
N RESULTS
C L A S S 7 – 2 0 1 1 / 2 0 1 2
MI M
E D - F M
UP
• Questionnaires• Likert scale: what is it?• Different Likert scales• Study objective• Methods
- Study participants - Study design
• Results• Conclusions
Questionnaries
Introduction Objective Methods
Results Conclusion
Currently, people’s opinion about an issue or service is a valuable way for researchers to acquire information in diverse areas of knowledge.
• A questionnaire is an essential instrument that aims to collect this information based on the inquisition of the population in study;• It consists in a certain amount of questions covering a topic of interest to investigators;• Is useful in almost every kind of job.
Likert Scale and Its Variables - A Comparative Study
Likert Scale and Its Variables - A Comparative Study
Having in mind the need to measure and classify this information, one reaches the conclusion of the pressing necessity for scales, tools that permits measuring people’s attitudes and opinions[1].
Likert scale
1. Zou KH, C.M., Quinn SA, Beta-mapping and beta-regression for changes of ordinal-rating measurements on Likert scales: a comparison of the change scores among multiple treatment groups. Stat Med, 2010. 29(24): p. 2486-500.
Introduction Objective Methods
Results Conclusion
LIKERT SCALE
A scale that consists in asking people to respond to a series of
statements about a topic, in terms of the extent to which they
agree or disagree with them.
What is it?
Measure pain, quality of life
Applications
Likert Scale and Its Variables - A Comparative Study
2. Zou KH, C.M., Quinn SA, Beta-mapping and beta-regression for changes of ordinal-rating measurements on Likert scales: a comparison of the change scores among multiple treatment groups. Stat Med, 2010. 29(24): p. 2486-500. 3.Huber JF, H.J., Zumstein MD, Ruflin G, Lüscher M., Visual circle scale (VCS)-a patient-friendly scale to measure pain compared to VAS and Likert scale. Z Orthop Unfall, 2007. 145(6): p. 795-7.
[2]
[3]
Introduction Objective Methods
Results Conclusion
DIFFERENT LIKERT SCALES
Different scales produce
different resultsThen, different
Likert scales may produce different
results
Search: some articles were found confirming our
idea
Likert Scale and Its Variables - A Comparative Study
4 Dawes, J., Do data characteristics change according to the number of scale points used? An experiment using 5-point, 7-point and 10-point scales. International Journal of Market Research, 2008. 50(1): p. 61-77.
[4]
Introduction Objective Methods
Results Conclusion
INFLUENCE OF A CENTRAL ANSWER
Different Likert scales
produce different results?
Does a neutral answer influence
the results?
Does both "I don't want to answer"
and neutral options influence the
results?Apparently not
studied yet.
Apparently not studied yet.
Likert Scale and Its Variables - A Comparative Study
5. Kulas JT. , S.A., Middle category endorsement in odd-numbered Likert response scales: Associated item characteristics, cognitive demands, and preferred meanings. Journal of Research in Personality, 2009. 43: p. 489–49
Already studied
[5]
Does a "I don't want to answer" option
influence the results?
Introduction Objective Methods
Results Conclusion
OBJECTIVES
Discover if the addition of a neutral element, a “refusal to answer” option or both
combined has repercussions in the results obtained.
Evaluate the reproducibility of the scales used in the questionnaire.
Likert Scale and Its Variables - A Comparative Study
6. Hansson LM, G.M., Bergström R, Factors affecting reproducibility of dietary reports using food frequency questionnaires. Eur J Clin Nutr, 2000. 54(8).
[6]
Introduction Objective Methods
Results Conclusion
Likert Scale and Its Variables - A Comparative Study
Study Design:
One questionnaire with two versions
Questionnaire 1 Questionnaire 2Number of questions in
each questionnaire:
12Same questions in both
questionnaires
Methods
Introduction Objective Methods
Results Conclusion
Type of study
Our study is included in a Longitudinal Study type.BECAUSE :
-There is a follow-up period of the individuals in study, which can be longer or shorter;
- Data is gathered at least in two points in time which allows us to observe changes in the population.
Likert Scale and Its Variables - A Comparative Study
Introduction Objective Methods
Results Conclusion
Students from classes 6 and 7
Study Participants:
All students that belong to the first year of the Integrated Master’s Degree in
Medicine of Faculdade de Medicina da Universidade do Porto, distributed by
classes 1 to 22
Exce
pt
Likert Scale and Its Variables - A Comparative Study
Methods
Introduction Objective Methods
Results Conclusion
Likert Scale and Its Variables - A Comparative Study
Monday Tuesday Wednesday
Thursday Friday
8:30-11 h 1 4 6 9
14-16:30h 2 x 7 1016:30-
19h 3 5 8These ten blocks where randomly divided in two
groups with SPSS.
Questionnaire 1Questionnaire 2
Study Participants:
Methods
Introduction Objective Methods
Results Conclusion
About the questionnaires:
• Theme of the questionnaire: Evaluation systems and study methods
Theme with less opinion change
probability
Likert Scale and Its Variables - A Comparative Study
Why?
Methods
Introduction Objective Methods
Results Conclusion
• Half of the participants answer to version 1 and the other half to version 2
• One month later, the first half answer to version 2 and the other half to version 1
Why?
Likert Scale and Its Variables - A Comparative Study
To avoid certain biases.
About the questionnaires:
Methods
Introduction Objective Methods
Results Conclusion
About the questionnaires:
• The questionnaires ought to be confidential.
How?The participants wrote their
names on the questionnaires and we
ensured confidentiality .
Why?In order to receive from
the participants their truly opinion
Likert Scale and Its Variables - A Comparative Study
Methods
Introduction Objective Methods
Results Conclusion
About the questionnaires:
Likert Scale and Its Variables - A Comparative Study
Comparison scale:I totally
agree I agree I disagree I totally disagree
… plus a neutral answer:Neither agree nor disagree
… plus refusal to answer:I don’t want to answer
Methods
Introduction Objective Methods
Results Conclusion
About the questionnaires:
Likert Scale and Its Variables - A Comparative Study
Three types of comparison
Comparison scaleVs
Comparison scale plus neutral answer
Comparison scale:Likert scale with “I totally
agree”, “I agree”, “I disagree” and “I totally disagree”
Comparison scaleVs
Comparison scale plus refusal to answer option
Comparison scaleVs
Comparison scale plus neutral and refusal to
answer option
Methods
Introduction Objective Methods
Results Conclusion
About the questionaires:
Likert Scale and Its Variables - A Comparative Study
To evaluate the reproducibility
Comparison scaleVs
Comparison scale
Comparison scale plus refusal to answer
optionVs
Comparison scale plus refusal to answer
option
Comparison scale plus neutral and refusal to
answer optionVs
Comparison scale plus neutral and refusal to
answer optionComparison scale:
Likert scale with “I totally agree”, “I agree”, “I disagree”
and “I totally disagree” Methods
Introduction Objective Methods
Results Conclusion
Questionnaire 1I1. Comparison Scale2. Comparison Scale3. Comparison Scale + neutral answer4. Comparison Scale5. Comparison Scale + refusal to answer6. Comparison Scale + neutral answer + refusal to answer7. Comparison Scale + refusal answer
II1. Comparison Scale + refusal to answer2. Comparison Scale + neutral answer + refusal to answer3. Comparison Scale4. Comparison Scale + neutral answer + refusal to answer5. Comparison Scale
Questionnaire 2I1. Comparison Scale + neutral answer2. Comparison Scale3. Comparison Scale 4. Comparison Scale + neutral answer5. Comparison Scale 6. Comparison Scale + neutral answer + refusal to answer7. Comparison Scale
II1. Comparison Scale + refusal to answer2. Comparison Scale
3. Comparison Scale + refusal to answer4. Comparison Scale 5. Comparison Scale + neutral answer + refusal to answer
Results
Introduction
Objective
Methods
Results
Conclusion
Study Participants:
Likert Scale and Its Variables - A Comparative Study
250
234
158
REPRODUCIBILITY
Introduction
Objective
Methods
Results
Conclusion
Valid
(n)
Question
Scale
Proportion of
Agreement
[Confidence interval]
Kappa
[Confidence interval]
Weighted Kappa
[Confidence interval]
157 I.2O sistema de avaliação por
exames é justo.
Standard 0,76[0,68;0,84]
0,54[0,42;0,67]
0,60[0,49;0,71]
156 II.1O meu estudo
rende mais quando trabalho
em grupo.
Standard+NR 0,64[0,56;0,72]
0,47[0,36;0,58]
N.A.
157 I.6O número de
alunos por turma no 1º ano da
FMUP é excessivo.
Standard+NR+Neutral
0,47[0,39;0,55]
0,30[0,20;0,41]
N.A.
STANDARD SCALE VS. NEUTRAL OPTION SCALE
Introduction
Objective
Methods
Results
Conclusion
Valid n Question Answers’
Optionsn(%)
Standart Scale
n(%)Standard Scale
+ Neutral Option
Proportion of Agreement[Confidence
Interval]
155
I.4. É justo ser necessário ter frequência a uma disciplina (frequentar pelo menos 2/3) das aulas para ser admitido a exame teórico.
Discordo Totalmente 15 (9,7) 12 (7,7) 0.29 [0,12;0,52]Discordo 37 (23,9) 33 (21,3) 0.37 [0,24;0,52]
Não Concordo nem Discordo ------- 33 (21,3) -------
Concordo 79 (51,0) 57 (36,8) 0,37 [0,28;0,48]Concordo
Totalmente 24 (15,5) 20 (12,9) 0,26 [0,13;0,44]
156
I.1O sistema de avaliação contínua é vantajoso para os alunos.
Discordo Totalmente 3 (1,9) 2 (1,3) 0 [0;0,54]Discordo 25 (16,0) 30 (19,2) 0,41 [0,26; 0,58]
Não Concordo nem Discordo ------- 31 (19,9) -------
Concordo 104 (66,7) 73 (46,8) 0,57 [0,47;0,66]Concordo
Totalmente 24 (15,4) 20 (12,8) 0,38 [0,22;0,56]
155I.3 Penso que o grau de exigência da FMUP é superior ao de outras.
Discordo Totalmente 0 (0) 1 (0,6) 0 [0;0,94]Discordo 7 (4,5) 3 (1,9) 0,43 [0,12;0,80]
Não Concordo nem Discordo -------- 12 (7,7) -------
Concordo 46 (29,7) 57 (36,8) 0,43 [0,32;0,55]Concordo
Totalmente 102 (65,8) 82 (52,9) 0,64 [0,55;0,73]
STANDARD SCALE VS. REFUSAL TO ANSWER OPTION SCALE
Introduction
Objective
Methods
Results
Conclusion
Validn Question Answers’
Options
n(%)Standard
Scale
n(%)Standard Scale
+ Refusal to Answer Option
Proportion of Agreement [Confidence
Interval]
158
I.5 Os professores do 1º ano da FMUP estão sempre disponíveis para tirar dúvidas
Discordo Totalmente 1 (0,6) 0 (0,0) 0 [0;0]Discordo 28 (18,1) 30 (19,4) 0,35 [0,21;0,51]Concordo 113 (72,9) 87 (56,1) 0,54 [0,45;0,63]
Concordo Totalmente 13 (8,4) 29 (18,7) 0,14 [0,05;0,30]
Não respondo ------- 9 (5,8) -------
158
I.7 Quando há avaliação contínua, acho antipedagógico os alunos não irem tendo ao longo do semestre feedback do seu desempenho
Discordo Totalmente 0 (0,0) 3 (1,9) 0 [0;0]Discordo 12 (7,6) 11 (7,0) 0,15 [0,04;0,39]Concordo 90 (57,3) 70 (44,6) 0,47 [0,37;0,57]Concordo
Totalmente 55 (35,0) 64 (40,8) 0,49 [0,38;0,60]
Não respondo ------- 9 (5,7) -------
158
II.3 Tenho necessidade de recorrer a bebidas energéticas e/ou vitaminas para acompanhar o ritmo de estudo exigido pelo curso.
Discordo Totalmente 70 (44,3) 71 (44,9) 0,52 [0,41;0,62]Discordo 57 (37,1) 50 (31,6) 0,35 [0,25;0,47]Concordo 25 (15,8) 22 (13,9) 0,31 [0,17;0,48]
Concordo Totalmente 6 (3,8) 11 (7,0) 0,31 [0,10;0,61]
Não respondo ------- 4 (2,5) -------
STANDARD SCALE VS. BOTH OPTIONS COMBINED SCALE
Introduction
Objective
Methods
Results
Conclusion
Validn Question Answers’
OptionsN(%)
Standard Scale
N(%)Standard Scale + Both Combined Options Scale
Proportion of Agreement [Confidence
Interval]
158
II.2 O estudo é mais proveitoso quando trabalho num local
silencioso.
Discordo Totalmente
0 (0,0) 0 (0,0) 0,00 [0,00;0,00]
Discordo 8 (5,1) 4 (2,5) 0,09 [0,01;0,43]Não Concordo nem Discordo ------- 11 (7,0) -------
Concordo 61 (38,6) 45 (28,5) 0,45 [0,34;0,57]Concordo
Totalmente 89 (56,3) 98 (62,0) 0,67 [0,57;0,75]Não Respondo ------- 0 (0,0) -------
158
II.4 Frequentemente acho mais rentável
estudar pelos apontamentos de
colegas mais velhos do que pela bibliografia
recomendada.
Discordo Totalmente
0 (0,0) 0 (0,0) 0,00 [0,00;0,00]
Discordo 13 (8,2) 8 (5,1) 0,31[0,12;0,59]Não Concordo nem Discordo ------- 11 (7,0) -------
Concordo 81 (51,3) 56 (65,4) 0.44 [0,34;0,55]Concordo
Totalmente 64 (40,5) 73 (46,2) 0,57 [0,46;0,68]Não Respondo ------- 2 (1,3) -------
158
II.5 É difícil conciliar as actividades extra-
curriculares (por exemplo praticar um desporto, aprender
um instrumento musical, etc…) com o trabalho exigido na
FMUP
Discordo Totalmente
8 (5,1) 3 (1,9) 0,10 [0,01;0,46]
Discordo 29 (18,4) 26 (16,5) 0,25 [0,14;0,41]Não Concordo nem Discordo ------- 22 (13,9) -------
Concordo 84 (53,2) 58 (36,7) 0,37 [0,27;0,47]Concordo
Totalmente 37 (23,4) 47 (29,7) 0,33 [0,22;0,46]Não Respondo ------- 2 (1,3) -------
Conclusion
Introduction
Objective
Methods
Results
Conclusion
Freedom of choice
It depends…
VS Reproducibility
Likert Scale and Its Variables - A Comparative Study
F A C U L D A D E D E M E D I C I N A D A U N I V E R S I D A D E D O P O R T O 2 º S E M E S T R E M A R Ç O D E 2 0 1 2
Abel Trzan
Ana Fonseca
Ana Filipa Ferreira
Ana Isabel Maduro
Catarina Tinoco
Guilherme Amorim
Joana Antunes
João Dias
José Carneiro
Maria João Bastos
Maria João Costa
Pedro Barros
Pedro Santos
Likert Scale and Its Variables - A Comparative Study