Opening up multiple choice - assessing with confidence
-
Upload
jon-rosewell -
Category
Education
-
view
589 -
download
0
Transcript of Opening up multiple choice - assessing with confidence
eST
EeM
Opening up Multiple Choice: Assessing with confidence
CALRG Conference, June 2011
eST
EeM
Multiple Choice Questions (MCQ)• Basic fare of e-assessment:
– robust– easy to implement– but pedagogically not ideal
• Open questions– pedagogically preferable – but difficult to provide robust and valid marking
eST
EeM
Demo• Standard MCQ• Standard CBM• Open CBM
screen dump
eST
EeM
eST
EeM
Confidence-Based Marking (CBM)• Student selects:
– answer and – level of confidence/certainty
• Incentive is scoring:– Confidently correct: Full marks– A tentative correct answer: Some credit– Cocksure & wrong: Penalty
• Pays to be honest about own knowledge
eST
EeM
CBM scoring
Confidence
Score
Correct Wrong
Low 1 0
Medium 2 -2
High 3 -6
Tentative & correct
Confidently correct
Cocksure – and wrong!
eST
EeM
CBM motivations• Rewards care and effort• Greater engagement• Encourages reflective learning• Encourages self-assessment
eST
EeM
Demo• Standard MCQ• Standard CBM• Open CBM
screen dump
eST
EeM
eST
EeM
Open CBM• Two stage question:
– Initially:• Question (but no MCQ options)• Student sets confidence level
– Next:• MCQ options revealed
• Finally:– Marking and feedback as for normal MCQ / CBM
eST
EeM
Demo• Standard MCQ• Standard CBM• Open CBM
screen dump
eST
EeM
eST
EeM
Open CBM advantages• Mechanically, question remains simple MCQ
– answer matching is trivial– easy to implement– possible to reuse existing question banks
• But transformed closed to open question– student must formulate answer to decide confidence– must decide answer without +ve or –ve clues.– cannot work backward – will not be led into misconceptions.
must answer the question as set.
eST
EeM
MCQ Open CBM• Pros
– objective marking– reliable marking– easy to implement
• Cons– distractors trivial– distractors engender
misconceptions– working backwards
• Pros– open question– reflection– ...as MCQ
• Cons– not always applicable– intimidating?– personality dependent?
eST
EeM
Possible benefits• Students
– better engagement– improved learning– more reflective learners
• Staff– easier question setting
eST
EeM
eSTEeM project plan (revised…)• Implement open CBM question type
– in OU Moodle Quiz and/or OpenMark• Trial in course assessment
– controlled experimental design– pilot (formative) on T216 CISCO networking
• Survey/interview to probe attitudinal aspects• User-lab trials• Log measures
– Assignment scores– Time spent on task