Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is...
-
date post
15-Jan-2016 -
Category
Documents
-
view
223 -
download
0
Transcript of Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is...
![Page 1: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/1.jpg)
Standard setting Determining the pass mark
![Page 2: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/2.jpg)
The Old Way
……..I think that ..I think that the cut score in the cut score in this is exam is this is exam is probably about probably about here…..here…..
there is a there is a natural lacuna I natural lacuna I can see can see dividing the dividing the bottom from bottom from the restthe rest
![Page 3: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/3.jpg)
Assessment
• Blueprinting Assessments
• Pass or Fail? Methods of Standard Setting
- Angoff - Borderline- Hofstee
………………………………………………………….
• Assessment of Professional Behaviours
![Page 4: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/4.jpg)
Vision StatementThe University of Sheffield strives to produce excellent medical graduates.
The medical curriculum will be outcome focussed where the aim is to produce graduates who are able to fulfil their role as junior doctors in the NHS and who also possess the generic skills expected of students attending a research-led university.
The course will feature increased opportunities to see patients in the community; a high degree of integration; an emphasis on facilitating student learning; and an increase in student choice.
The course will be organised on a body system basis with a progressive emphasis on learning around undifferentiated patient problems.
The instructional approach will consist of a spine of problem, case and patient-based integrated learning activities complemented by a range of other teaching and learning activities. There will be an increase in systematic teaching of some components to ensure competence in key areas.
Students will be expected to become progressively more self-directed, aided by increasing reliance on IT-based and distance learning materials and activities.
Assessment, both formative and summative, will be closely matched to defined outcomes.
The curriculum will be managed centrally by a multidisciplinary team, including those with a stake in the outcome of medical education. The Department of Medical Education, the Administration and an IT-based curriculum management system will provide support.
A monitoring system will be established to evaluate the implementation of the curriculum and to support a process of continuous improvement.
Excellent graduates
Outcome focussed
Integrated programmeStudent choiceUndifferentiated problems
Integrated Learning Activities
Able learners
Assessment matches outcomes
Curriculum management
Quality enhancement
![Page 5: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/5.jpg)
Align
Objectives
Teaching & learning
Assessment
![Page 6: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/6.jpg)
Summary
• Assessments should match
local/national outcomes for Validity• Multiple observations
required for Reliability• Electronic data collection
for optimal Feasibility• Good information to students
and staff to enhance Acceptance
![Page 7: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/7.jpg)
Pass
Competent
Fail
Incompetent
![Page 8: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/8.jpg)
measurementerror
Pass
Fail
![Page 9: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/9.jpg)
Standard setting
Knowledge Skills
![Page 10: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/10.jpg)
Angoff
Hofstee
Borderline method
Knowledge Skills
Standard setting
![Page 11: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/11.jpg)
Angoff (modified)What is the mark that a student who just passes would score for each item?
DISCUSS+/-MODIFY
Mean mark for all items is the pass mark
Written and clinical examinations
![Page 12: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/12.jpg)
The pass mark for this exam using a modified Angoff is 63.7%
Item Number Judge 1 Judge 2 Judge 3 Judge 4 Judge 5 Judge 61 70 75 80 70 70 802 50 70 70 60 50 603 66 80 70 75 70 80
40 60 70 70 60 60 70
overall estimate 57.7 65.7 69.7 64.4 60.2 64.6
mean of all judges 63.7
Angoff
![Page 13: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/13.jpg)
Angoff standard setting
• Determine the pass mark for the ‘just-competent’ student and write it in the box
• Do not assume that the pass mark is 50% or 60% or constant
![Page 14: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/14.jpg)
Borderline procedureBorderline procedure
Used for clinical (OSCE) examinations
![Page 15: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/15.jpg)
![Page 16: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/16.jpg)
Borderline procedureBorderline procedure
Used for clinical (OSCE) examinations
Examiners score the students’s performance at the station e.g 17/20
Examiners judge the overall performance
clear pass / borderline / clear fail
Mark sheets rated borderline identified and the scores of borderline students averaged
Process repeated for each station
Calculate the median borderline score for all stations
![Page 17: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/17.jpg)
Item I Clinical Problem
Type of station/question
Max marks available
Median Borderline score
Station1 Cranial nerves Examination 40 28.75
Station2b Patient education
Communication 20 13.38
Station3 Breathlessness History 20 12.00 Station4 chest Examination 20 11.00
Overall borderline score
510 297.89
Overall borderline score %
100
58.41
Borderline Method
![Page 18: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/18.jpg)
1 2 3
Pass/Fail
Station 15 (static) Oral Lesions
0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00
Score
0
10
20
30
40
50
Co
un
t
![Page 19: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/19.jpg)
Borderlinegroup
pass
fail
Borderline score 58.41
![Page 20: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/20.jpg)
Borderlinegroup
pass
fail
Borderline score 58.41
How wide should this band be?
![Page 21: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/21.jpg)
Borderlinegroup
pass
fail
Borderline score 58.41
How wide should this band be?
+/- 1 - standard deviation- standard error- or what?
![Page 22: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/22.jpg)
The Standard Error of MeasurementThe Standard Error of Measurement
• depends on the reliability of the test (R)
• depends on the standard deviation of the test (SD)
• SEM = SD 1 - R• acts as a confidence interval in high stakes situations
![Page 23: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/23.jpg)
The Standard Error of MeasurementThe Standard Error of Measurement
• depends on the reliability of the test (R)
• depends on the standard deviation of the test (SD)
• SEM = SD 1 - R• acts as a confidence interval in high stakes situations
R = Variance Variance x Error
![Page 24: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/24.jpg)
Borderlinegroup
Pass
Fail
Borderline score 58.41
Pass Score60.93
} - 1 SEM
Fail score 55.88
} + 1 SEM
![Page 25: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/25.jpg)
Station/MEQ ID Competence System Problem Max marksavailable
Median Borderline score
Station1 Phys Exam CNS Abnormal Gait 40 23.50
Station2 (static) Data (x-ray) RS Wheeze 10 6.00
Station3 Phys Exam GI Abdominal Mass 20 11.75
Station4 History RS Haemoptysis 20 12.00
Station6 Pat ed/comms Skin/Misc Rash 20 13.00
Station7 History Endo Weight loss 20 13.50
Station8 (static) Pract skills Onc Pain 10 5.00
Station9 History Eyes/ENT Visual Disturbance 20 12.00
Station10 Phys Exam MSS/CNS Laceration 20 11.00
Station11 Pract Skills CVS Collapse 20 10.50
Station12 Phys exam Onc Breast Lump 20 13.00
Station14b Pat ed/Comms Onc Dying Patient 20 12.50
Station15 (static) Data Skin/Misc Itch 10 6.00
Written1 Prob solving/Dx/Manage CVS Claudication 10 6.00
Written2 Prob solving/Dx/Manage CVS/Endo Palpitations 10 6.00
Written3 Prob solving/Dx/Manage CVS Heart failure 10 6.00
Written4 Prob solving/Dx/Manage Resp Haemoptysis 10 7.00
Written5 Prob solving/Dx/Manage Mental Mania 10 5.00
Written6 Prob solving/Dx/Manage ENT Hearing Difficulty 10 4.00
Written7 Prob solving/Dx/Manage CNS Coma 10 7.00
Written8 Prob solving/Dx/Manage MSS Joint Pain 10 5.00
Written9 Prob solving/Dx/Manage MSS Fracture 10 5.00
Written10 Prob solving/Dx/Manage GI Diarrhoea 10 7.00
Written11 Prob solving/Dx/Manage GI Haematemesis 10 6.00
Written12 Prob solving/Dx/Manage Mental Overdose 10 5.00
Written13 Prob solving/Dx/Manage Endo Weight loss 10 7.00
Written14 Prob solving/Dx/Manage Onc Breast Lump 10 6.00
Written15 Prob solving/Dx/Manage Haem Bleeding 10 5.00
Written16 Prob solving/Dx/Manage Onc Testicular Mass 10 4.00
Written17 Prob solving/Dx/Manage Eyes Red Eye 10 5.00
Written18 Prob solving/Dx/Manage Endo Hairy 10 5.00
Written19 Prob solving/Dx/Manage CNS Weakness 10 5.50
Written20 Prob solving/Dx/Manage GI Vomiting 10 7.00
Static1 Investig /Interpretation CNS Headache 10 6.00
Static2 Investig /Interpretation GI Juandice 10 5.00
Static3 Investig /Interpretation Resp SOB 10 6.00
Static4 Investig /Interpretation Endo Dehydration 10 5.00
Static5 Investig /Interpretation CNS Consciuoness 10 6.00
Static6 Investig /Interpretation ENT Abnormal Hearing 10 6.00
Overall borderline score
510
297.25
Overall borderline score %
100
58.28
![Page 26: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/26.jpg)
Written14 Prob solving/Dx/Manage Onc Breast Lump 10 6.00
Written15 Prob solving/Dx/Manage Haem Bleeding 10 5.00
Written16 Prob solving/Dx/Manage Onc Testicular Mass 10 4.00
Written17 Prob solving/Dx/Manage Eyes Red Eye 10 5.00
Written18 Prob solving/Dx/Manage Endo Hairy 10 5.00
Written19 Prob solving/Dx/Manage CNS Weakness 10 5.50
Written20 Prob solving/Dx/Manage GI Vomiting 10 7.00
Static1 Investig /Interpretation CNS Headache 10 6.00
Static2 Investig /Interpretation GI Juandice 10 5.00
Static3 Investig /Interpretation Resp SOB 10 6.00
Static4 Investig /Interpretation Endo Dehydration 10 5.00
Static5 Investig /Interpretation CNS Consciuoness 10 6.00
Static6 Investig /Interpretation ENT Abnormal Hearing 10 6.00
Overall borderline score
510
297.25
Overall borderline score %
100
58.28
![Page 27: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/27.jpg)
Number of students
Mean Score (%)
Overall Borderline Score (%)
Standard deviation
Standard Error of Measurement
Pass mark (%)
Reliability
04 195 73.25
58.28 5.50 2.40 60.68
0.81
03 215 74.4 58.41 5.77 2.52 60.93
0.81
02 214 73. 0 59. 81 4.95 2.98 62.79
0.64
01 195 71.85
58.90 5.86 3.21 62.11
0.70
![Page 28: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/28.jpg)
Internal Reliability of the ExamCronbach’s Alpha
Cronbach’s alpha
Shows whether randomly split halves of the exam by item vary together
Split halfCorrelations
(SH)
Item / Stations
Population
(students)
![Page 29: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/29.jpg)
Borderline
Grading System
Borderline
Fail
Pass
Good Pass
Top Mark
Borderline Mark
Distinction
Pass Mark
Pass
Fail
![Page 30: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/30.jpg)
Reducing measurement errorReducing measurement error
1. Increase reliability of exam
2. Compromise with feasibility/cost
* Content specificity
* More items in the exam
* Blueprinting
* Quality assurance of item writing
* Training examiners/standardised patients
* Feedback from exam performance
![Page 31: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/31.jpg)
Hofstee procedureHofstee procedure
Used for written examinations
Judges review a copy of the whole exam
Judges indicate:
minimum % failure rate
maximum % failure rate
minimum acceptable cut percentage
maximum acceptable cut percentage
Administer test, plot curve and read off standard
![Page 32: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/32.jpg)
0102030405060708090
100
50 60 70 80
Student score %
% c
andid
ate
s
![Page 33: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/33.jpg)
0102030405060708090
100
50 60 70 80
Student score %
% c
andid
ate
s
![Page 34: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/34.jpg)
0102030405060708090
100
50 60 70 80
Student score %
% c
andid
ate
s
![Page 35: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/35.jpg)
0102030405060708090
100
50 60 70 80
Student score %
% c
andid
ate
s
![Page 36: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/36.jpg)
0102030405060708090
100
50 60 70 80
Student score %
% c
andid
ate
s
![Page 37: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/37.jpg)
0102030405060708090
100
50 60 70 80
Student score %
% c
andid
ate
s
![Page 38: Standard setting Determining the pass mark. The Old Way …..I think that the cut score in this is exam is probably about here….. there is a natural lacuna.](https://reader036.fdocuments.us/reader036/viewer/2022081508/56649d5e5503460f94a3d08c/html5/thumbnails/38.jpg)