GALA 14 th INTERNATIONAL CONFERENCE
description
Transcript of GALA 14 th INTERNATIONAL CONFERENCE
GALA 14th INTERNATIONAL CONFERENCE
Advances in Research on Language Acquisition and Teaching
Thessaloniki, 14-16 December 2007
“ “ Reflections in the mirror: Reflections in the mirror: the contribution of self and peer assessment in the contribution of self and peer assessment in
the teaching of speaking skills”the teaching of speaking skills”
AEGINITOU V., AEGINITOU V.,
NTELIOU E., NTELIOU E.,
VLAHOYANNI N.VLAHOYANNI N.
CHAROKOPIO UNIVERSITYCHAROKOPIO UNIVERSITY
ATHENSATHENS
Overview1. Introduction
1.1. Peer assessment: Benefits1.2. Self-assessment
2. Our study2.1 Background2.2 Context and purpose2.3 Methodology2.4 Discussion and results 2.4.1 Students’ presentations 2.4.2 Questionnaires 2.4.3 Tutorials
3. Implications – Conclusions4. Bibliography
1.1 Peer assessment: Benefits
a. development of professional skillsb. students’ involvement in the learning processc. better rapport between speaker and audienced. increased objectivity of results
Boud & Holmes, 1995; Stefani, 1998; Lejk et al, 1999; Magin & Helmore, 2001; Falchikov, 1986, 1995; Magin & Churches, 1989; Mockford, 1994; Lynch, 1988
1.2 Self assessment: Benefits
a. Monitoring of learning and progress
b. Setting goals for the future
c. Encouraging responsibility for learning
d. Promoting critical thinking
e. Constructing and reconstructing knowledge
f. Bridging the gap between high and low achievers
(Carr, 2002; Harlen & Winter, 2004)
2. Our study
2.1 Background: Pilot study 2005
• Subjects: EAP/ESP under graduates
• Purpose
2.2 Context and purpose
Research Questions
1. Is there a significant level of agreement between the tutors’ and the students’ assessment of oral presentation skills?
2. Is peer evaluation motivating and useful?
3. To what extent is self assessment enhanced by peer assessment?
2.3 Methodology
• prior training- presentation of their own strengths and weaknesses
- assessment checklists (different fortutors and students),
- audio-taped sample presentations• Students’ presentations & Questionnaire completion• Tutorials • Statistical tools: (SPSS-Matlab) Cohen’s Kappa statistic, Spearman correlation, Mc
Nemar- Bowker test
Assessment criteria• CONTENT: content relevant to title / clear central idea /
topic well supported / proper use of sources• ORGANIZATION: clear introduction / main points
coherently stated / main points cohesively stated / relevant conclusion
• LANGUAGE: accurate and clear/ voc. appropriate to topic / technical vocabulary clearly explained / use of transitions / comprehensible pronunciation
• PRESENTATION TECHNIQUES: speed / loudness of voice / eye contact
• VISUAL AIDS: clarity / length
Adapted from: Rignall, M. and Fourneaux, C. 1997. Speaking (English for Academic Studies Series). UK: Prentice Hall.
2.4. Discussion and results
2.4.1. Analysis of students’ presentations
Intermediate level
Variables Kappa kappa p-valueKappa -weighted
Kappa-weighted p-value
Topic support 0.483 0.001 0.503 0.014
Cohesion 0.637 0.000 0.637 0.000
Clarity of visual aids
0.543 0.000 0.550 0.017
Speed 0.787 0.000
Loudness 0.653 0.033
Eye contact 0.386 0.001 0.479 0.004
Variables Kappa kappa p-valueKappa -weighted
Kappa-weighted p-value
Content relevant to title
-0.031 0.517 -0.031 0.512
Sources -0.021 0.591 0.105 0.265
Technical vocabulary
0.108 0.182 0.164 0.1921
Advanced Level
Variables Kappa kappa p-valueKappa -weighted
Kappa-weighted p-value
Topic support 0.380 0.006 0.353 0.049
Clarity of visual aids
0.540 0.000 0.580 0.001
Speed 0.382 0.004
Variables Kappa kappa p-valueKappa -weighted
Kappa-weighted p-value
Sources 0.146 0.131 0.293 0.063
Cohesion -0.019 0.546 0.031 0.454
Technical vocabulary
0.160 0.092 0.133 0.227
Loudness 0.079 0.408
Eye contact 0.192 0.072 0.289 0.066
Problematic variables
Intermediate Level Advanced Level
1. Use of sources 1. Use of sources
2. Technical vocabulary 2. Technical vocabulary
3. Content relevant to title
3. Cohesion
4. Eye contact
5. Loudness
Symmetric Measures
,456 ,128 3,349 ,001
,488 ,137 3,112 ,004c
,436 ,127 2,700 ,011c
33
Kendall's tau-b
Spearman Correlation
Ordinal byOrdinal
Pearson's RInterval by Interval
N of Valid Cases
ValueAsymp.
Std. Errora
Approx. Tb
Approx. Sig.
Not assuming the null hypothesis.a.
Using the asymptotic standard error assuming the null hypothesis.b.
Based on normal approximation.c.
Sources (Intermediate)
Sources (students) * Sources (professors) Crosstabulation
Count
3 0 0 3
13 2 1 16
5 7 2 14
21 9 3 33
No
Quite
Yes
Sources(students)
Total
No Quite Yes
Sources (professors)
Total
Sources (Intermediate)
Chi-Square Tests
22,500 3 ,000
33
McNemar-Bowker Test
N of Valid Cases
Value dfAsymp. Sig.
(2-sided)
Sources (Intermediate)
Symmetric Measures
,475 ,127 3,534 ,000
,530 ,137 3,477 ,002c
,534 ,116 3,520 ,001c
33
Kendall's tau-b
Spearman Correlation
Ordinal byOrdinal
Pearson's RInterval by Interval
N of Valid Cases
ValueAsymp.
Std. Errora
Approx. Tb
Approx. Sig.
Not assuming the null hypothesis.a.
Using the asymptotic standard error assuming the null hypothesis.b.
Based on normal approximation.c.
Sources (Advanced)
Sources (students) * Sources (professors) Crosstabulation
Count
5 0 0 5
5 5 3 13
1 10 4 15
11 15 7 33
No
Quite
Yes
Sources
(students)
Total
No Quite Yes
Sources (professors)
Total
Sources (Advanced)
Chi-Square Tests
9,769 3 ,021
33
McNemar-Bowker Test
N of Valid Cases
Value dfAsymp. Sig.
(2-sided)
Sources (Advanced)
Symmetric Measures
,326 ,152 2,135 ,033
,345 ,161 2,045 ,049c
,347 ,161 2,062 ,048c
33
Kendall's tau-b
Spearman Correlation
Ordinal byOrdinal
Pearson's RInterval by Interval
N of Valid Cases
ValueAsymp.
Std. Errora
Approx. Tb
Approx. Sig.
Not assuming the null hypothesis.a.
Using the asymptotic standard error assuming the null hypothesis.b.
Based on normal approximation.c.
Technical vocabulary (Intermediate)
Technical Vocabulary (students) * Technical Vocabulary (professors)Crosstabulation
Count
10 5 3 18
4 4 7 15
14 9 10 33
Quite
Yes
Technical Vocabulary(students)
Total
No Quite Yes
Technical Vocabulary (professors)
Total
Technical vocabulary (Intermediate)
Technical vocabulary (Advanced)
Symmetric Measures
,145 ,167 ,863 ,388
,155 ,179 ,873 ,389c
,152 ,186 ,859 ,397c
33
Kendall's tau-b
Spearman Correlation
Ordinal byOrdinal
Pearson's RInterval by Interval
N of Valid Cases
ValueAsymp.
Std. Errora
Approx. Tb
Approx. Sig.
Not assuming the null hypothesis.a.
Using the asymptotic standard error assuming the null hypothesis.b.
Based on normal approximation.c.
Symmetric Measures
-,031 ,022 -,730 ,466
-,031 ,022 -,174 ,863c
-,031 ,022 -,174 ,863c
-,031 ,022 -,180 ,858
33
Kendall's tau-b
Spearman Correlation
Ordinal by Ordinal
Pearson's RInterval by Interval
KappaMeasure of Agreement
N of Valid Cases
ValueAsymp.
Std. Errora
Approx. Tb
Approx. Sig.
Not assuming the null hypothesis.a.
Using the asymptotic standard error assuming the null hypothesis.b.
Based on normal approximation.c.
Content relevant to title (Intermediate)
Content relevant to topic (students) * Content relevantto topic (professors) Crosstabulation
Count
33 33
33 33
YesContent relevant totopic (students)
Total
Yes
Contentrelevant to
topic(professors)
Total
Content relevant to title (Advanced)
Symmetric Measures
,085 ,167 ,501 ,617
,089 ,176 ,496 ,623c
,119 ,161 ,670 ,508c
33
Kendall's tau-b
Spearman Correlation
Ordinal byOrdinal
Pearson's RInterval by Interval
N of Valid Cases
ValueAsymp.
Std. Errora
Approx. Tb
Approx. Sig.
Not assuming the null hypothesis.a.
Using the asymptotic standard error assuming the null hypothesis.b.
Based on normal approximation.c.
Cohesion (Advanced)
Loudness (Advanced)
Symmetric Measures
,194 ,234 ,761 ,447
,196 ,236 1,110 ,275c
,117 ,193 ,656 ,517c
33
Kendall's tau-b
Spearman Correlation
Ordinal byOrdinal
Pearson's RInterval by Interval
N of Valid Cases
ValueAsymp.
Std. Errora
Approx. Tb
Approx. Sig.
Not assuming the null hypothesis.a.
Using the asymptotic standard error assuming the null hypothesis.b.
Based on normal approximation.c.
Eye contact (Advanced)
Symmetric Measures
,458 ,124 3,477 ,001
,496 ,134 3,184 ,003c
,496 ,123 3,176 ,003c
33
Kendall's tau-b
Spearman Correlation
Ordinal byOrdinal
Pearson's RInterval by Interval
N of Valid Cases
ValueAsymp.
Std. Errora
Approx. Tb
Approx. Sig.
Not assuming the null hypothesis.a.
Using the asymptotic standard error assuming the null hypothesis.b.
Based on normal approximation.c.
Eye contact (students) * Eye contact (professors) Crosstabulation
Count
4 0 0 4
7 7 2 16
2 7 4 13
13 14 6 33
No
Quite
Yes
Eye contact(students)
Total
No Quite Yes
Eye contact (professors)
Total
Eye contact (Advanced)
Chi-Square Tests
11,778 3 ,008
33
McNemar-Bowker Test
N of Valid Cases
Value dfAsymp. Sig.
(2-sided)
Eye contact (Advanced)
2.4.2. Questionnaire analysis
1. While listening to the presentations of your classmates, have you learned anything new on the
topics under discussion?
No
Little
Quite
A lot
Definitely yes
Q1
Pies show counts
1,52%
18,18%
34,85%
33,33%
12,12%
2. Has the organization of the presentations helped you in the way you will organize your future
presentations?
Little
Quite
A lot
Definitely yes
Q2
Pies show counts
9,09%
24,24%
46,97%
19,70%
3. Have you learned useful words / expressions in your subject area?
No
Little
Quite
A lot
Definitely yes
Q3
Pies show counts
7,58%
19,70%
42,42%
24,24%
6,06%
4. Were the visual aids helpful in your future selection of relevant graphics?
No
Little
Quite
A lot
Definitely yes
Q4
Pies show counts
1,52%
13,64%
34,85%31,82%
18,18%
5. Did you find evaluating your classmates interesting?
No
Little
Quite
A lot
Definitely yes
Q5
Pies show counts
3,03%
12,12%
37,88%
36,36%
10,61%
2.4.3. Tutorials
Questions asked in the tutorials
1. Were you satisfied with your presentation?
2. In which aspects of your presentation you feel you need more practice? Why?
3. In which aspects of your presentation you feel you performed well and you would not change?
4. Is it easy for you to assess yourself?
Students’ comments
“I did not do any presentations at school. I am not quite sure what I have to do”.
“This is not my job. The teacher should do that”.
“Before the training I did not know how to assess myself or my classmates. Now I think I can”.
Analysis of students' comments
• Weaknesses more easily identified than strengths
• Participation in self-assessment procedures can facilitate better judgement on performance levels
• More realistic goals are set for future presentations
Bachman & Palmer 1989; Ready-Morfitt, 1991; Dickinson, 1987; Oscarson, 1997
3. Implications
• Number of subjects
• Absence from training session
• Future design of self-assessment practice
3. Conclusions
• Prior training positively modified the results• The beneficial combination of peer and
self-assessment processes• Two problematic areas: technical
vocabulary and reference to sources• Self-reflective practices should be
introduced in the early stages of instruction• Future research
4. Bibliography• Altman DG. 1991. Practical Statistics for Medical Research. London: Chapman & Hall.• Bland JM & Altman DG. 1986. “Statistical methods for assessing agreement between
two methods of clinical measurement”. Lancett 32, pp. 307-10.• Boud, D. & Holmes, H. 1995. “Peer and self marking in a large technical subject”. In: D.
Boud (Ed.) Enhancing learning through self assessment, London, Kogan, 63-78.• Brindley, G. 2001. ‘Assessment’. In Carter, R. & Nunan, D. (Eds.). The Cambridge
Guide to Teaching English to Speakers of Other Languages. Cambridge: CUP. pp. 137-143.
• Carr, S.C. 2002. “Self-evaluation: involving students in their own learning”. Reading and Writing Quarterly, 18, pp.195-199.
• Cohen, JA. 1968. Weighted Kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 70, pp. 213-20.
• Falchikov, N. 1986. “Product comparisons and process benefits of collaborative peer group and self assessments”. Assessment and evaluation in Higher education, 11, 146-166
• Falchikov, N. 1995. Peer feedback marking: developing peer assessment, Innovations in Education and Training International, 32, 175-187.
4. Bibliography• Fleis JL & Cohen JA. 1973. “The equivalence of weighted Kappa and the intraclass correlation coefficient as
measures of reliability”. Educational Psychology Measurements 33, pp. 613-9.• Harlen, W. & Winter, J. 2004. “The development for assessment for learning: learning from the case of science
and mathematics”. Language Testing, 21(3), pp.390-408.• Hughes, I.E. & Large, B.J. 1993. ‘Staff and peer-group assessment of oral communication skills’. Studies in
Higher Education, 18(3), 379-385.• Landis, JR, Kock, GG. 1977. “The measurement of observer agreement for categorical data”. Biometrics 33,
pp.159-74 • Lejk et al. 1999. “Group assessment in systems analysis and design : a comparison of the performance of streamed
and mixed-ability groups”. Assessment and evaluation in Higher education, 24, 5-14.• Lynch, T. 1988. Peer evaluation in practice, in: A. Brookes and P. Grundy (Eds.) Individualisation and autonomy
in language learning. ELT Documents 131 • Magin, D. & Churches, A. 1989. “Using self and peer assessment in teaching design”, Proceedings, World
conference on Engineering Education for Advancing Technology, Institution of Engineers, Australia, 89/1, 640-644
• Magin, D. & Helmore, P. 2001. “Peer and Teacher Assessments of Oral Presentation Skills: how reliable are they?”. Studies in Higher Education, 26/3, 287-298.
• Mockford, C. 1994. “The use of peer group review in the assessment of project work in higher education”, Mentoring and Tutoring, 2, 45-52.
• Rignall, M. & Fourneaux, C. 1997. Speaking (English for Academic Studies Series). UK: Prentice Hall.• Stefani, L. 1998. “Assessment in partnership with learners”, Assessment and evaluation in Higher education, 23,
339-350.• Streiner, DL & Norman, GR. 1995. Health Measurement Scales: A practical Guide to their Development and
Use, 2nd edn. Oxford: Oxford University Press.
MERRY CHRISTMAS