8/8/2019 Ottawa Final

1/16

Student detection of errors in

examinationsDr Christopher Hebbes (cph8@leicester.ac.uk)

Dr Jenny Briggs

Dr Emmanuel DavidsonDr Ricky Bell

Dr Heather Crick

8/8/2019 Ottawa Final

2/16

Introduction

10% of the UK medical student populationstruggle to pass exams

Multiple reasons

Implications for patient safety

Yates J, James D. Risk factors at medical school for subsequent professional misconduct: multicentreretrospective case-control study. BMJ 2010;340:c2040

8/8/2019 Ottawa Final

3/16

Self assessment

Self assessment often inaccurate

Why?

Do students give the same error attributionas their tutors?

I. Colthart, et al. (2008). `The effectiveness of self-assessment on the identification of learner needs, learneractivity, and impact on clinical practice: BEME Guide no. 10'. Medical Teacher30(2):124-145.

= +Score True score Error component

8/8/2019 Ottawa Final

4/16

Aims

Establish a taxonomy of common types oferrors

Determine ability to detect gold standard

errors

Encourage reflection in learners

Student engagement in assessment

8/8/2019 Ottawa Final

5/16

Methods

Pilot study

67 students includedof 84

Failed one or bothsemester exams

Student consent

Mock exam

10 SAQ questions

SAQ Exam

Self-marked Tutor-marked

Errors coded Errors coded

Comparison

Standardisedcoding sheet

8/8/2019 Ottawa Final

6/16

Errors (overall)

Tutor mark = max mark? Student mark = max mark?N

Student mark = max mark

Y

True PositiveFalse Positiv

False Negative

True Negativ

Error No Error

Error

No Error

Tutor

Stud

ent

False Positive

True Negative

N

Y

False Negative

True Positive

Y

N

8/8/2019 Ottawa Final

7/16

Errors (specific)

=Tutor mark max mark? Tutor code = X*N

Exclude

Y

An error has occurred

* Where X is the test

Y

Student code = X*

N

Student code = X*

True Positive

False Negative

Y

N

False Positive

True Negative

Y

N

8/8/2019 Ottawa Final

8/16

Results

8/8/2019 Ottawa Final

9/16

Results

.Cronbach s Alpha 0 81

: .Tutor 29 45: .Student 44 29

< .p 001Eta2 .608

8/8/2019 Ottawa Final

10/16

Error Detection (overall)

2107 Errors*

Students detected74%

Error prevalence 0.88 Sensitivity 0.74

True Positive1560

False Positive102

False Negative547

True Negative173

Error No Error

Error

No Error

Tutor

tudent

*Instances where full marks were not achieved for a part of a question

8/8/2019 Ottawa Final

11/16

Error Type DetectionComparison of tutor and student error attribution

Sensitivity Specificity

A 0.016 0.919

B 0.012 0.908

C 0.166 0.723

D 0.032 0.837

8/8/2019 Ottawa Final

12/16

Summary

Students are able to identify errors withhigh sensitivity

There is a difference in error attribution

between students and their tutors

8/8/2019 Ottawa Final

13/16

Future work

Is there a difference in error attributionbetween first, second and third yearstudents?

What factors are associated withcorrelation between student and tutorerror attribution?

Do interventions to help students attributeerrors improve correlation and improveacademic performance?

8/8/2019 Ottawa Final

14/16

Acknowledgements

Professor Stewart Petersen, Professor of MedicalEducation, University of Leicester

Dr Heather Crick, Lecturer, University of Leicester

Dr Gordon French, Deputy Postgraduate Dean,East Midlands Healthcare Workforce Deanery

Dr Adrian Hastings, Senior Clinical Educator,University of Leicester

Dr David Matheson, Lecturer in MedicalEducation, Medical Education unit, University ofNottingham & East Midlands HealthcareWorkforce Deanery

8/8/2019 Ottawa Final

15/16

?: / / . /ttp tinyurl com hebbes

8/8/2019 Ottawa Final

16/16

References