Eaquals International Conference, Lisbon, 21 – 23 April 2016
Assessing Oral Proficiency
Ines Paland-Riedmüller,Deutsch-Uni Online
Claudia Pop,TestDaF-Institut,
g.a.s.t. e.V. Germany
www.eaquals.org
Content of the Workshop
Specifications of speaking The basics of assessment: Purposes,
methods, formats The subjectivity of assessment Quality management tools What is viable for your institution?
Eaquals International Conference, Lisbon, 21 – 23 April 2016
articulating (phonetic)
phrasing / formulating (linguistic)
planning & organising (cognitive)
Specifications of speaking
Complexity Pronun-ciation
Time
Eaquals International Conference, Lisbon, 21 – 23 April 2016
The basics of assessment: Purposes
• Summative assessment• used at end of a certain unit of the learning process• ”assessment of learning”
• Formative assessment• used continuously, during the learning process• ”assessment for learning”
Eaquals International Conference, Lisbon, 21 – 23 April 2016
The basics of assessment: Purposes
• Reference group orientated assessment• comparing the test outcome with the group
outcome• ranking, e.g. course
• Criteria orientated assessment• comparing the outcome directly with the aspects
intended to be measured
Eaquals International Conference, Lisbon, 21 – 23 April 2016
The basics of assessment: Methods
• Holistic/global scoring• assessing the performance as a whole• one general mark
• Analytic/profile scoring• decomposing the performance according to
several aspects of interest• several marks
Eaquals International Conference, Lisbon, 21 – 23 April 2016
The basics of assessment: Formats
Direct assessment face-to-face interview one or more candidates one or more raters
Semi-direct assessment recorded tasks recorded performances
Eaquals International Conference, Lisbon, 21 – 23 April 2016
The basics of assessment: Formats
Eaquals International Conference, Lisbon, 21 – 23 April 2016
situation face to face, media transmittedreal communication, simulated communication
partners interlocutor, assessor, co-candiate(s)real person, avatar
activities production, interaction, spontaneous, reflected
number of partners
monologue, dialogue, multilogue
input material
aural, written, visual, graphical
rating simultaneous, posteriorhuman, (semi-) automated
Which aspects actually influence
our decision making?
An example
Eaquals International Conference, Lisbon, 21 – 23 April 2016
Is the grey surface of object
X in shape and size identical with the grey
surface of object A, object B or
object C?
X
Eaquals International Conference, Lisbon, 21 – 23 April 2016
X
Is the grey surface of object
X in shape and size identical with the grey
surface of object A, object B or
object C?
Eaquals International Conference, Lisbon, 21 – 23 April 2016
X
Is the grey surface of object
X in shape and size identical with the grey
surface of object A, object B or
object C?
Eaquals International Conference, Lisbon, 21 – 23 April 2016
X
Is the grey surface of object
X in shape and size identical with the grey
surface of object A, object B or
object C?
Eaquals International Conference, Lisbon, 21 – 23 April 2016
X
Is the grey
surface of object X in shape and
size identical with the grey
surface of object A, object B or
object C?
Eaquals International Conference, Lisbon, 21 – 23 April 2016
We have built-in perceptions of
geometrical forms.
How much stronger might this effect be in assessing learners´ proficiency?
Eaquals International Conference, Lisbon, 21 – 23 April 2016
The subjectivity of assessment
• Rudolf Weiss, 1965• main question: How large is the variability in
teachers´ assessment?• rating of one student´s essay by 92 teachers
independently, criteria: • correctness• style• content• general mark
Eaquals International Conference, Lisbon, 21 – 23 April 2016
The subjectivity of assessment
Eaquals International Conference, Lisbon, 21 – 23 April 2016
Mark Correctness Style Content
1 7% 24% 26%
2 28% 41% 47%
3 39% 19% 20%
4 22% 14% 7%
5 4% 2% 0%
Weiss, 1965
The subjectivity of assessment
1 2 3 4 5
-10%
0%
10%
20%
30%
40%
50%
60%
10%
45%
35%
10%
0%
General mark
Weiss, 1965
Eaquals International Conference, Lisbon, 21 – 23 April 2016
The subjectivity of assessmentReplication of the Weiss-study: Birkel & Birkel, 2002
1 2 to 3+ 3 to 4+ 4 to 5+ 50
10
20
30
40
50
60
1
18
51
16
2
Birkel & Birkel, 2002
The subjectivity of assessment
Assessment “errors”:• halo effect• contamination effect• serial effect• central/ extreme tendency• personal dispositions
Eaquals International Conference, Lisbon, 21 – 23 April 2016
Serious reliability problem – what can we do?
Standardise!
Eaquals International Conference, Lisbon, 21 – 23 April 2016
Quality management tools
trainings calibration material monitoring system rating grids
Eaquals International Conference, Lisbon, 21 – 23 April 2016
Tools: Trainings
• Face-to-Face-Interview • standardise the procedure• discuss appropriate behaviour• analyse recorded material• practise: Role play
Eaquals International Conference, Lisbon, 21 – 23 April 2016
Tools: Trainings
• Assessment• recollect the goal (construct)• introduce criteria • operationalise the criteria• assess in groups and discuss the
results • raise error awareness• discuss problems
Eaquals International Conference, Lisbon, 21 – 23 April 2016
Tools: Calibration material
• Guidelines• manual, script, interlocutor frame • best practice example of a face to face
interview• Bench marks
• typical performances for each level• calibrated assessment (expert group)• framework of expectation
Eaquals International Conference, Lisbon, 21 – 23 April 2016
Tools: Monitoring
• two or more raters for the same performance
• (peer) observation• video analysis
constructive feedback
Eaquals International Conference, Lisbon, 21 – 23 April 2016
Tools: Rating grids
• “dare” to modify the CEFR according to your context and aim of assessment
• e.g. ALTE-Materials for the Guidance of Test Item Writers (module 4)
Eaquals International Conference, Lisbon, 21 – 23 April 2016
Descriptors should be
• positive • short • accurate/ precise• independent of each other
Tools: Rating grids
• “dare” to modify the CEFR according to your context and aim of assessment
• e.g. ALTE-Materials for the Guidance of Test Item Writers (module 4)
• evaluate and modify your grid before introducing it
Eaquals International Conference, Lisbon, 21 – 23 April 2016
Feasible?
• What is already done in your institution?
• Which additional tools do you think might be suitable?
Eaquals International Conference, Lisbon, 21 – 23 April 2016
Thank you
[email protected]@deutsch-uni.com
Eaquals International Conference, Lisbon, 21 – 23 April 2016