GALA 14th INTERNATIONAL CONFERENCE

Advances in Research on Language Acquisition and Teaching

Thessaloniki, 14-16 December 2007

“ “ Reflections in the mirror: Reflections in the mirror: the contribution of self and peer assessment in the contribution of self and peer assessment in

the teaching of speaking skills”the teaching of speaking skills”

AEGINITOU V., AEGINITOU V.,

NTELIOU E., NTELIOU E.,

VLAHOYANNI N.VLAHOYANNI N.

CHAROKOPIO UNIVERSITYCHAROKOPIO UNIVERSITY

ATHENSATHENS

Overview1. Introduction

1.1. Peer assessment: Benefits1.2. Self-assessment

2. Our study2.1 Background2.2 Context and purpose2.3 Methodology2.4 Discussion and results 2.4.1 Students’ presentations 2.4.2 Questionnaires 2.4.3 Tutorials

3. Implications – Conclusions4. Bibliography

1.1 Peer assessment: Benefits

a. development of professional skillsb. students’ involvement in the learning processc. better rapport between speaker and audienced. increased objectivity of results

Boud & Holmes, 1995; Stefani, 1998; Lejk et al, 1999; Magin & Helmore, 2001; Falchikov, 1986, 1995; Magin & Churches, 1989; Mockford, 1994; Lynch, 1988 

1.2 Self assessment: Benefits

a. Monitoring of learning and progress

b. Setting goals for the future

c. Encouraging responsibility for learning

d. Promoting critical thinking

e. Constructing and reconstructing knowledge

f. Bridging the gap between high and low achievers

(Carr, 2002; Harlen & Winter, 2004)

2. Our study

2.1 Background: Pilot study 2005

• Subjects: EAP/ESP under graduates

• Purpose

2.2 Context and purpose

Research Questions

1. Is there a significant level of agreement between the tutors’ and the students’ assessment of oral presentation skills?

2. Is peer evaluation motivating and useful?

3. To what extent is self assessment enhanced by peer assessment?

2.3 Methodology

• prior training- presentation of their own strengths and weaknesses

- assessment checklists (different fortutors and students),

- audio-taped sample presentations• Students’ presentations & Questionnaire completion• Tutorials • Statistical tools: (SPSS-Matlab) Cohen’s Kappa statistic, Spearman correlation, Mc

Nemar- Bowker test

Assessment criteria• CONTENT: content relevant to title / clear central idea /

topic well supported / proper use of sources• ORGANIZATION: clear introduction / main points

coherently stated / main points cohesively stated / relevant conclusion

• LANGUAGE: accurate and clear/ voc. appropriate to topic / technical vocabulary clearly explained / use of transitions / comprehensible pronunciation

• PRESENTATION TECHNIQUES: speed / loudness of voice / eye contact

• VISUAL AIDS: clarity / length

Adapted from: Rignall, M. and Fourneaux, C. 1997. Speaking (English for Academic Studies Series). UK: Prentice Hall.

2.4. Discussion and results

2.4.1. Analysis of students’ presentations

Intermediate level

Variables Kappa kappa p-valueKappa -weighted

Kappa-weighted p-value

Topic support 0.483 0.001 0.503 0.014

Cohesion 0.637 0.000 0.637 0.000

Clarity of visual aids

0.543 0.000 0.550 0.017

Speed 0.787 0.000

Loudness 0.653 0.033

Eye contact 0.386 0.001 0.479 0.004

Variables Kappa kappa p-valueKappa -weighted

Kappa-weighted p-value

Content relevant to title

-0.031 0.517 -0.031 0.512

Sources -0.021 0.591 0.105 0.265

Technical vocabulary

0.108 0.182 0.164 0.1921

Advanced Level

Variables Kappa kappa p-valueKappa -weighted

Kappa-weighted p-value

Topic support 0.380 0.006 0.353 0.049

Clarity of visual aids

0.540 0.000 0.580 0.001

Speed 0.382 0.004

Variables Kappa kappa p-valueKappa -weighted

Kappa-weighted p-value

Sources 0.146 0.131 0.293 0.063

Cohesion -0.019 0.546 0.031 0.454

Technical vocabulary

0.160 0.092 0.133 0.227

Loudness 0.079 0.408

Eye contact 0.192 0.072 0.289 0.066

Problematic variables

Intermediate Level Advanced Level

1. Use of sources 1. Use of sources

2. Technical vocabulary 2. Technical vocabulary

3. Content relevant to title

3. Cohesion

4. Eye contact

5. Loudness

Symmetric Measures

,456 ,128 3,349 ,001

,488 ,137 3,112 ,004c

,436 ,127 2,700 ,011c

33

Kendall's tau-b

Spearman Correlation

Ordinal byOrdinal

Pearson's RInterval by Interval

N of Valid Cases

ValueAsymp.

Std. Errora

Approx. Tb

Approx. Sig.

Not assuming the null hypothesis.a.

Using the asymptotic standard error assuming the null hypothesis.b.

Based on normal approximation.c.

Sources (Intermediate)

Sources (students) * Sources (professors) Crosstabulation

Count

3 0 0 3

13 2 1 16

5 7 2 14

21 9 3 33

No

Quite

Yes

Sources(students)

Total

No Quite Yes

Sources (professors)

Total

Sources (Intermediate)

Chi-Square Tests

22,500 3 ,000

33

McNemar-Bowker Test

N of Valid Cases

Value dfAsymp. Sig.

(2-sided)

Sources (Intermediate)

Symmetric Measures

,475 ,127 3,534 ,000

,530 ,137 3,477 ,002c

,534 ,116 3,520 ,001c

33

Kendall's tau-b

Spearman Correlation

Ordinal byOrdinal

Pearson's RInterval by Interval

N of Valid Cases

ValueAsymp.

Std. Errora

Approx. Tb

Approx. Sig.

Not assuming the null hypothesis.a.

Using the asymptotic standard error assuming the null hypothesis.b.

Based on normal approximation.c.

Sources (Advanced)

Sources (students) * Sources (professors) Crosstabulation

Count

5 0 0 5

5 5 3 13

1 10 4 15

11 15 7 33

No

Quite

Yes

Sources

(students)

Total

No Quite Yes

Sources (professors)

Total

Sources (Advanced)

Chi-Square Tests

9,769 3 ,021

33

McNemar-Bowker Test

N of Valid Cases

Value dfAsymp. Sig.

(2-sided)

Sources (Advanced)

Symmetric Measures

,326 ,152 2,135 ,033

,345 ,161 2,045 ,049c

,347 ,161 2,062 ,048c

33

Kendall's tau-b

Spearman Correlation

Ordinal byOrdinal

Pearson's RInterval by Interval

N of Valid Cases

ValueAsymp.

Std. Errora

Approx. Tb

Approx. Sig.

Not assuming the null hypothesis.a.

Using the asymptotic standard error assuming the null hypothesis.b.

Based on normal approximation.c.

Technical vocabulary (Intermediate)

Technical Vocabulary (students) * Technical Vocabulary (professors)Crosstabulation

Count

10 5 3 18

4 4 7 15

14 9 10 33

Quite

Yes

Technical Vocabulary(students)

Total

No Quite Yes

Technical Vocabulary (professors)

Total

Technical vocabulary (Intermediate)

Technical vocabulary (Advanced)

Symmetric Measures

,145 ,167 ,863 ,388

,155 ,179 ,873 ,389c

,152 ,186 ,859 ,397c

33

Kendall's tau-b

Spearman Correlation

Ordinal byOrdinal

Pearson's RInterval by Interval

N of Valid Cases

ValueAsymp.

Std. Errora

Approx. Tb

Approx. Sig.

Not assuming the null hypothesis.a.

Using the asymptotic standard error assuming the null hypothesis.b.

Based on normal approximation.c.

Symmetric Measures

-,031 ,022 -,730 ,466

-,031 ,022 -,174 ,863c

-,031 ,022 -,174 ,863c

-,031 ,022 -,180 ,858

33

Kendall's tau-b

Spearman Correlation

Ordinal by Ordinal

Pearson's RInterval by Interval

KappaMeasure of Agreement

N of Valid Cases

ValueAsymp.

Std. Errora

Approx. Tb

Approx. Sig.

Not assuming the null hypothesis.a.

Using the asymptotic standard error assuming the null hypothesis.b.

Based on normal approximation.c.

Content relevant to title (Intermediate)

Content relevant to topic (students) * Content relevantto topic (professors) Crosstabulation

Count

33 33

33 33

YesContent relevant totopic (students)

Total

Yes

Contentrelevant to

topic(professors)

Total

Content relevant to title (Advanced)

Symmetric Measures

,085 ,167 ,501 ,617

,089 ,176 ,496 ,623c

,119 ,161 ,670 ,508c

33

Kendall's tau-b

Spearman Correlation

Ordinal byOrdinal

Pearson's RInterval by Interval

N of Valid Cases

ValueAsymp.

Std. Errora

Approx. Tb

Approx. Sig.

Not assuming the null hypothesis.a.

Using the asymptotic standard error assuming the null hypothesis.b.

Based on normal approximation.c.

Cohesion (Advanced)

Loudness (Advanced)

Symmetric Measures

,194 ,234 ,761 ,447

,196 ,236 1,110 ,275c

,117 ,193 ,656 ,517c

33

Kendall's tau-b

Spearman Correlation

Ordinal byOrdinal

Pearson's RInterval by Interval

N of Valid Cases

ValueAsymp.

Std. Errora

Approx. Tb

Approx. Sig.

Not assuming the null hypothesis.a.

Using the asymptotic standard error assuming the null hypothesis.b.

Based on normal approximation.c.

Eye contact (Advanced)

Symmetric Measures

,458 ,124 3,477 ,001

,496 ,134 3,184 ,003c

,496 ,123 3,176 ,003c

33

Kendall's tau-b

Spearman Correlation

Ordinal byOrdinal

Pearson's RInterval by Interval

N of Valid Cases

ValueAsymp.

Std. Errora

Approx. Tb

Approx. Sig.

Not assuming the null hypothesis.a.

Using the asymptotic standard error assuming the null hypothesis.b.

Based on normal approximation.c.

Eye contact (students) * Eye contact (professors) Crosstabulation

Count

4 0 0 4

7 7 2 16

2 7 4 13

13 14 6 33

No

Quite

Yes

Eye contact(students)

Total

No Quite Yes

Eye contact (professors)

Total

Eye contact (Advanced)

Chi-Square Tests

11,778 3 ,008

33

McNemar-Bowker Test

N of Valid Cases

Value dfAsymp. Sig.

(2-sided)

Eye contact (Advanced)

2.4.2. Questionnaire analysis

1. While listening to the presentations of your classmates, have you learned anything new on the

topics under discussion?

No

Little

Quite

A lot

Definitely yes

Q1

Pies show counts

1,52%

18,18%

34,85%

33,33%

12,12%

2. Has the organization of the presentations helped you in the way you will organize your future

presentations?

Little

Quite

A lot

Definitely yes

Q2

Pies show counts

9,09%

24,24%

46,97%

19,70%

3. Have you learned useful words / expressions in your subject area?

No

Little

Quite

A lot

Definitely yes

Q3

Pies show counts

7,58%

19,70%

42,42%

24,24%

6,06%

4. Were the visual aids helpful in your future selection of relevant graphics?

No

Little

Quite

A lot

Definitely yes

Q4

Pies show counts

1,52%

13,64%

34,85%31,82%

18,18%

5. Did you find evaluating your classmates interesting?

No

Little

Quite

A lot

Definitely yes

Q5

Pies show counts

3,03%

12,12%

37,88%

36,36%

10,61%

2.4.3. Tutorials

Questions asked in the tutorials

1. Were you satisfied with your presentation?

2. In which aspects of your presentation you feel you need more practice? Why?

3. In which aspects of your presentation you feel you performed well and you would not change?

4. Is it easy for you to assess yourself?

Students’ comments

“I did not do any presentations at school. I am not quite sure what I have to do”.

“This is not my job. The teacher should do that”.

“Before the training I did not know how to assess myself or my classmates. Now I think I can”.

Analysis of students' comments

• Weaknesses more easily identified than strengths

• Participation in self-assessment procedures can facilitate better judgement on performance levels

• More realistic goals are set for future presentations

Bachman & Palmer 1989; Ready-Morfitt, 1991; Dickinson, 1987; Oscarson, 1997

3. Implications

• Number of subjects

• Absence from training session

• Future design of self-assessment practice

3. Conclusions

• Prior training positively modified the results• The beneficial combination of peer and

self-assessment processes• Two problematic areas: technical

vocabulary and reference to sources• Self-reflective practices should be

introduced in the early stages of instruction• Future research

4. Bibliography• Altman DG. 1991. Practical Statistics for Medical Research. London: Chapman & Hall.• Bland JM & Altman DG. 1986. “Statistical methods for assessing agreement between

two methods of clinical measurement”. Lancett 32, pp. 307-10.• Boud, D. & Holmes, H. 1995. “Peer and self marking in a large technical subject”. In: D.

Boud (Ed.) Enhancing learning through self assessment, London, Kogan, 63-78.• Brindley, G. 2001. ‘Assessment’. In Carter, R. & Nunan, D. (Eds.). The Cambridge

Guide to Teaching English to Speakers of Other Languages. Cambridge: CUP. pp. 137-143.

• Carr, S.C. 2002. “Self-evaluation: involving students in their own learning”. Reading and Writing Quarterly, 18, pp.195-199.

• Cohen, JA. 1968. Weighted Kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 70, pp. 213-20.

• Falchikov, N. 1986. “Product comparisons and process benefits of collaborative peer group and self assessments”. Assessment and evaluation in Higher education, 11, 146-166

• Falchikov, N. 1995. Peer feedback marking: developing peer assessment, Innovations in Education and Training International, 32, 175-187.

4. Bibliography• Fleis JL & Cohen JA. 1973. “The equivalence of weighted Kappa and the intraclass correlation coefficient as

measures of reliability”. Educational Psychology Measurements 33, pp. 613-9.• Harlen, W. & Winter, J. 2004. “The development for assessment for learning: learning from the case of science

and mathematics”. Language Testing, 21(3), pp.390-408.• Hughes, I.E. & Large, B.J. 1993. ‘Staff and peer-group assessment of oral communication skills’. Studies in

Higher Education, 18(3), 379-385.• Landis, JR, Kock, GG. 1977. “The measurement of observer agreement for categorical data”. Biometrics 33,

pp.159-74 • Lejk et al. 1999. “Group assessment in systems analysis and design : a comparison of the performance of streamed

and mixed-ability groups”. Assessment and evaluation in Higher education, 24, 5-14.• Lynch, T. 1988. Peer evaluation in practice, in: A. Brookes and P. Grundy (Eds.) Individualisation and autonomy

in language learning. ELT Documents 131 • Magin, D. & Churches, A. 1989. “Using self and peer assessment in teaching design”, Proceedings, World

conference on Engineering Education for Advancing Technology, Institution of Engineers, Australia, 89/1, 640-644

• Magin, D. & Helmore, P. 2001. “Peer and Teacher Assessments of Oral Presentation Skills: how reliable are they?”. Studies in Higher Education, 26/3, 287-298.

• Mockford, C. 1994. “The use of peer group review in the assessment of project work in higher education”, Mentoring and Tutoring, 2, 45-52.

• Rignall, M. & Fourneaux, C. 1997. Speaking (English for Academic Studies Series). UK: Prentice Hall.• Stefani, L. 1998. “Assessment in partnership with learners”, Assessment and evaluation in Higher education, 23,

339-350.• Streiner, DL & Norman, GR. 1995. Health Measurement Scales: A practical Guide to their Development and

Use, 2nd edn. Oxford: Oxford University Press.

MERRY CHRISTMAS