VARIABILITY OF LIKERT SCALES EFFECTS ON DATA COLLECTION RESULTS CLASS 7 – 2011/2012 MIMED - FMUP.
Transcript of VARIABILITY OF LIKERT SCALES EFFECTS ON DATA COLLECTION RESULTS CLASS 7 – 2011/2012 MIMED - FMUP.
VARIA
BILITY
OF
LIKERT
SCALE
S
EFFECTS
ON D
ATA C
OLLECTI
ON RESULT
S
CL
AS
S 7
– 2
01
1/ 2
01
2
MI M
ED
- F
MU
P
• Questionnaires
• Likert scale: what is it?
• Different Likert scales
• Study objective
• Methods
- Study participants
- Study design
• Results
• Conclusions
Questionnaries
Introduction Objective Methods
Results Conclusion
Currently, people’s opinion about an issue or service is a valuable way for researchers to acquire information in diverse areas of knowledge.
• A questionnaire is an essential instrument that aims to collect this information based on the inquisition of the population in study;
• It consists in a certain amount of questions covering a topic of interest to investigators;
• Is useful in almost every kind of job.
Likert Scale and Its Variables - A Comparative Study
Likert Scale and Its Variables - A Comparative Study
Having in mind the need to measure and classify this information, one reaches the conclusion of the pressing necessity for scales, tools that permits measuring people’s attitudes and opinions[1].
Likert scale
1. Zou KH, C.M., Quinn SA, Beta-mapping and beta-regression for changes of ordinal-rating measurements on Likert scales: a comparison of the change scores among multiple treatment groups. Stat Med, 2010. 29(24): p. 2486-500.
Introduction Objective Methods
Results Conclusion
LIKERT SCALE
A scale that consists in asking people to respond to a series of
statements about a topic, in terms of the extent to which they
agree or disagree with them.
What is it?
Measure pain, quality of life
Applications
Likert Scale and Its Variables - A Comparative Study
2. Zou KH, C.M., Quinn SA, Beta-mapping and beta-regression for changes of ordinal-rating measurements on Likert scales: a comparison of the change scores among multiple treatment groups. Stat Med, 2010. 29(24): p. 2486-500. 3.Huber JF, H.J., Zumstein MD, Ruflin G, Lüscher M., Visual circle scale (VCS)-a patient-friendly scale to measure pain compared to VAS and Likert scale. Z Orthop Unfall, 2007. 145(6): p. 795-7.
[2]
[3]
Introduction Objective Methods
Results Conclusion
DIFFERENT LIKERT SCALES
Different scales produce
different resultsThen, different
Likert scales may produce different
results
Search: some articles were found confirming our
idea
Likert Scale and Its Variables - A Comparative Study
4 Dawes, J., Do data characteristics change according to the number of scale points used? An experiment using 5-point, 7-point and 10-point scales. International Journal of Market Research, 2008. 50(1): p. 61-77.
[4]
Introduction Objective Methods
Results Conclusion
INFLUENCE OF A CENTRAL ANSWER
Different Likert scales
produce different results?
Does a neutral answer influence
the results?
Does both "I don't want to answer"
and neutral options influence the
results?
Apparently not studied yet.
Apparently not studied yet.
Likert Scale and Its Variables - A Comparative Study
5. Kulas JT. , S.A., Middle category endorsement in odd-numbered Likert response scales: Associated item characteristics, cognitive demands, and preferred meanings. Journal of Research in Personality, 2009. 43: p. 489–49
Already studied
[5]
Does a "I don't want to answer" option
influence the results?
Introduction Objective Methods
Results Conclusion
OBJECTIVES
Discover if the addition of a neutral element, a “refusal to answer” option or both
combined has repercussions in the results obtained.
Evaluate the reproducibility of the scales used in the questionnaire.
Likert Scale and Its Variables - A Comparative Study
6. Hansson LM, G.M., Bergström R, Factors affecting reproducibility of dietary reports using food frequency questionnaires. Eur J Clin Nutr, 2000. 54(8).
[6]
Introduction Objective Methods
Results Conclusion
Likert Scale and Its Variables - A Comparative Study
Study Design:
One questionnaire with two versions
Questionnaire 1 Questionnaire 2Number of
questions in each
questionnaire: 12
Same questions in both questionnaires
Methods
Introduction Objective Methods
Results Conclusion
Type of study
Our study is included in a Longitudinal Study type.
BECAUSE :
-There is a follow-up period of the individuals in study, which can be longer or shorter;
- Data is gathered at least in two points in time which allows us to observe changes in the population.
Likert Scale and Its Variables - A Comparative Study
Introduction Objective Methods
Results Conclusion
Students from classes 6 and 7
Study Participants:
All students that belong to the first year of the Integrated Master’s Degree in
Medicine of Faculdade de Medicina da Universidade do Porto, distributed by
classes 1 to 22
Exc
ept
Likert Scale and Its Variables - A Comparative Study
Methods
Introduction Objective Methods
Results Conclusion
Likert Scale and Its Variables - A Comparative Study
Monday Tuesday Wednesday
Thursday Friday
8:30-11 h 1 4 6 9
14-16:30h 2 x 7 10
16:30-19h 3 5 8
These ten blocks where randomly divided in two
groups with SPSS.
Questionnaire 1
Questionnaire 2
Study Participants:
Methods
Introduction Objective Methods
Results Conclusion
About the questionnaires:
• Theme of the questionnaire: Evaluation systems and study methods
Theme with less opinion change
probability
Likert Scale and Its Variables - A Comparative Study
Why?
Methods
Introduction Objective Methods
Results Conclusion
• Half of the participants answer to version 1 and the other half to version 2
• One month later, the first half answer to version 2 and the other half to version 1
Why?
Likert Scale and Its Variables - A Comparative Study
To avoid certain biases.
About the questionnaires:
Methods
Introduction Objective Methods
Results Conclusion
About the questionnaires:
• The questionnaires ought to be confidential.
How?The participants wrote their
names on the questionnaires and we
ensured confidentiality .
Why?In order to receive from
the participants their truly opinion
Likert Scale and Its Variables - A Comparative Study
Methods
Introduction Objective Methods
Results Conclusion
About the questionnaires:
Likert Scale and Its Variables - A Comparative Study
Comparison scale:I totally
agree I agree I disagreeI totally disagree
… plus a neutral answer:Neither agree nor disagree
… plus refusal to answer:I don’t want to answer
Methods
Introduction Objective Methods
Results Conclusion
About the questionnaires:
Likert Scale and Its Variables - A Comparative Study
Three types of comparison
Comparison scaleVs
Comparison scale plus neutral answer
Comparison scale:Likert scale with “I totally
agree”, “I agree”, “I disagree” and “I totally disagree”
Comparison scaleVs
Comparison scale plus refusal to answer option
Comparison scaleVs
Comparison scale plus neutral and refusal to
answer option
Methods
Introduction Objective Methods
Results Conclusion
About the questionaires:
Likert Scale and Its Variables - A Comparative Study
To evaluate the reproducibility
Comparison scaleVs
Comparison scale
Comparison scale plus refusal to answer
optionVs
Comparison scale plus refusal to answer
option
Comparison scale plus neutral and refusal to
answer optionVs
Comparison scale plus neutral and refusal to
answer option
Comparison scale:Likert scale with “I totally
agree”, “I agree”, “I disagree” and “I totally disagree” Methods
Introduction Objective Methods
Results Conclusion
Questionnaire 1I1. Comparison Scale2. Comparison Scale3. Comparison Scale + neutral answer4. Comparison Scale5. Comparison Scale + refusal to answer6. Comparison Scale + neutral answer + refusal to answer7. Comparison Scale + refusal answer
II1. Comparison Scale + refusal to answer2. Comparison Scale + neutral answer + refusal to answer3. Comparison Scale4. Comparison Scale + neutral answer + refusal to answer5. Comparison Scale
Questionnaire 2I1. Comparison Scale + neutral answer2. Comparison Scale3. Comparison Scale 4. Comparison Scale + neutral answer5. Comparison Scale 6. Comparison Scale + neutral answer + refusal to answer7. Comparison Scale
II1. Comparison Scale + refusal to answer2. Comparison Scale
3. Comparison Scale + refusal to answer4. Comparison Scale 5. Comparison Scale + neutral answer + refusal to answer
Results
Introduction
Objective
Methods
Results
Conclusion
Study Participants:
Likert Scale and Its Variables - A Comparative Study
250
234
158
REPRODUCIBILITY
Introduction
Objective
Methods
Results
Conclusion
Valid
(n)
Question
Scale
Proportion of
Agreement
[Confidence interval]
Kappa
[Confidence interval]
Weighted Kappa
[Confidence interval]
157 I.2O sistema de avaliação por
exames é justo.
Standard 0,76[0,68;0,84]
0,54[0,42;0,67]
0,60[0,49;0,71]
156 II.1O meu estudo
rende mais quando trabalho
em grupo.
Standard+NR 0,64[0,56;0,72]
0,47[0,36;0,58]
N.A.
157 I.6O número de
alunos por turma no 1º ano da
FMUP é excessivo.
Standard+NR+Neutral
0,47[0,39;0,55]
0,30[0,20;0,41]
N.A.
STANDARD SCALE VS. NEUTRAL OPTION SCALE
Introduction
Objective
Methods
Results
Conclusion
Valid n
QuestionAnswers’ Options
n(%)Standart
Scale
n(%)Standard Scale
+ Neutral Option
Proportion of Agreement[Confidence
Interval]
155
I.4. É justo ser
necessário ter
frequência a uma
disciplina (frequentar
pelo menos 2/3) das
aulas para ser admitido
a exame teórico.
Discordo Totalmente
15 (9,7) 12 (7,7) 0.29 [0,12;0,52]
Discordo 37 (23,9) 33 (21,3) 0.37 [0,24;0,52]Não Concordo nem Discordo
------- 33 (21,3) -------
Concordo 79 (51,0) 57 (36,8) 0,37 [0,28;0,48]
Concordo Totalmente
24 (15,5) 20 (12,9) 0,26 [0,13;0,44]
156
I.1
O sistema de avaliação
contínua é vantajoso
para os alunos.
Discordo Totalmente
3 (1,9) 2 (1,3) 0 [0;0,54]
Discordo 25 (16,0) 30 (19,2) 0,41 [0,26; 0,58]Não Concordo nem Discordo
------- 31 (19,9) -------
Concordo 104 (66,7) 73 (46,8) 0,57 [0,47;0,66]Concordo
Totalmente24 (15,4)
20 (12,8) 0,38 [0,22;0,56]
155
I.3 Penso que o grau de
exigência da FMUP é
superior ao de outras.
Discordo Totalmente
0 (0) 1 (0,6) 0 [0;0,94]
Discordo 7 (4,5) 3 (1,9) 0,43 [0,12;0,80]Não Concordo nem Discordo
-------- 12 (7,7) -------
Concordo 46 (29,7) 57 (36,8) 0,43 [0,32;0,55]Concordo
Totalmente102 (65,8) 82 (52,9) 0,64 [0,55;0,73]
STANDARD SCALE VS. REFUSAL TO ANSWER OPTION SCALE
Introduction
Objective
Methods
Results
Conclusion
Validn
QuestionAnswers’ Options
n(%)Standard
Scale
n(%)Standard Scale
+ Refusal to Answer Option
Proportion of Agreement [Confidence
Interval]
158
I.5 Os professores do
1º ano da FMUP estão
sempre disponíveis
para tirar dúvidas
Discordo Totalmente
1 (0,6) 0 (0,0) 0 [0;0]
Discordo 28 (18,1) 30 (19,4) 0,35 [0,21;0,51]
Concordo 113 (72,9) 87 (56,1) 0,54 [0,45;0,63]
Concordo Totalmente
13 (8,4) 29 (18,7) 0,14 [0,05;0,30]
Não respondo ------- 9 (5,8) -------
158
I.7 Quando há
avaliação contínua,
acho antipedagógico
os alunos não irem
tendo ao longo do
semestre feedback do
seu desempenho
Discordo Totalmente
0 (0,0) 3 (1,9) 0 [0;0]
Discordo 12 (7,6) 11 (7,0) 0,15 [0,04;0,39]
Concordo 90 (57,3) 70 (44,6) 0,47 [0,37;0,57]Concordo
Totalmente55 (35,0) 64 (40,8) 0,49 [0,38;0,60]
Não respondo ------- 9 (5,7) -------
158
II.3 Tenho necessidade
de recorrer a bebidas
energéticas e/ou
vitaminas para
acompanhar o ritmo
de estudo exigido
pelo curso.
Discordo Totalmente
70 (44,3) 71 (44,9) 0,52 [0,41;0,62]
Discordo 57 (37,1) 50 (31,6) 0,35 [0,25;0,47]
Concordo 25 (15,8) 22 (13,9) 0,31 [0,17;0,48]
Concordo Totalmente
6 (3,8) 11 (7,0) 0,31 [0,10;0,61]
Não respondo ------- 4 (2,5)
-------
STANDARD SCALE VS. BOTH OPTIONS COMBINED SCALE
Introduction
Objective
Methods
Results
Conclusion
Validn
QuestionAnswers’ Options
N(%)Standard Scale
N(%)Standard Scale + Both Combined Options Scale
Proportion of Agreement [Confidence
Interval]
158
II.2 O estudo é mais
proveitoso quando
trabalho num local
silencioso.
Discordo Totalmente
0 (0,0) 0 (0,0) 0,00 [0,00;0,00]
Discordo 8 (5,1) 4 (2,5) 0,09 [0,01;0,43]Não Concordo nem Discordo
------- 11 (7,0) -------
Concordo 61 (38,6) 45 (28,5) 0,45 [0,34;0,57]Concordo
Totalmente89 (56,3) 98 (62,0) 0,67 [0,57;0,75]
Não Respondo ------- 0 (0,0) -------
158
II.4 Frequentemente acho mais rentável
estudar pelos apontamentos de
colegas mais velhos do que pela bibliografia
recomendada.
Discordo Totalmente
0 (0,0) 0 (0,0) 0,00 [0,00;0,00]
Discordo 13 (8,2) 8 (5,1) 0,31[0,12;0,59]Não Concordo nem Discordo
------- 11 (7,0) -------
Concordo 81 (51,3) 56 (65,4) 0.44 [0,34;0,55]Concordo
Totalmente64 (40,5) 73 (46,2) 0,57 [0,46;0,68]
Não Respondo ------- 2 (1,3) -------
158
II.5 É difícil conciliar as actividades extra-
curriculares (por exemplo praticar um desporto, aprender
um instrumento musical, etc…) com o trabalho exigido na
FMUP
Discordo Totalmente
8 (5,1) 3 (1,9)0,10 [0,01;0,46]
Discordo 29 (18,4) 26 (16,5) 0,25 [0,14;0,41]Não Concordo nem Discordo
------- 22 (13,9) -------
Concordo 84 (53,2) 58 (36,7) 0,37 [0,27;0,47]Concordo
Totalmente37 (23,4) 47 (29,7) 0,33 [0,22;0,46]
Não Respondo ------- 2 (1,3) -------
Conclusion
Introduction
Objective
Methods
Results
Conclusion
Freedom of choice
It depends…
VS Reproducibility
Likert Scale and Its Variables - A Comparative Study
F A C U L D A D E D E M E D I C I N A D A U N I V E R S I D A D E D O P O R T O
2 º S E M E S T R E M A R Ç O D E 2 0 1 2
Abel Trzan
Ana Fonseca
Ana Filipa Ferreira
Ana Isabel Maduro
Catarina Tinoco
Guilherme Amorim
Joana Antunes
João Dias
José Carneiro
Maria João Bastos
Maria João Costa
Pedro Barros
Pedro Santos
Likert Scale and Its Variables - A Comparative Study