Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in...
-
Upload
ronald-willow -
Category
Documents
-
view
221 -
download
1
Transcript of Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in...
![Page 1: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/1.jpg)
XerteXerteStandard SettingStandard Setting
ininPsychiatry Undergraduate Psychiatry Undergraduate
OSCEOSCEDr Ben di MambroDr Ben di Mambro
Clinical Lecturer in PsychiatryClinical Lecturer in Psychiatry
![Page 2: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/2.jpg)
Psychiatry ExaminationPsychiatry Examination
80 EMIs80 EMIs 1 hour 15 minutes1 hour 15 minutes Standard setStandard set
2 OSCE stations2 OSCE stations Each station 10 minutesEach station 10 minutes 3 tasks based on competencies3 tasks based on competencies % given for each task% given for each task
![Page 3: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/3.jpg)
May OSCEMay OSCE
StationStation MaxMax MinMin AllAll
AA60.160.1
(14.3)(14.3)46.446.4
(5.6)(5.6)54.554.5
(10.3)(10.3)
BB60.8 60.8 (6.6)(6.6)
48.248.2
(4.4)(4.4)55.555.5
(9.0)(9.0)
CC66.166.1
(8.5)(8.5)49.049.0
(9.7)(9.7)56.256.2
(9.8)(9.8)
DD67.767.7
(12.8)(12.8)47.247.2
(4.1)(4.1)54.354.3
(10.8)(10.8)
![Page 4: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/4.jpg)
Psychiatry OSCEPsychiatry OSCE
Changes to the OSCEChanges to the OSCE Criterion marking schemeCriterion marking scheme Global clinical assessmentGlobal clinical assessment
HonoursHonours PassPass BorderlineBorderline FailFail
2 examiners were placed in each room2 examiners were placed in each room Standard set pass mark for each taskStandard set pass mark for each task
![Page 5: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/5.jpg)
TaskTask
Develop CAL package:Develop CAL package: Train the examiners in the new Train the examiners in the new
marking schememarking scheme Standardise marking schemeStandardise marking scheme
To allow for multiple sites to To allow for multiple sites to accommodate increasing numbers of accommodate increasing numbers of studentsstudents
Improve inter-rater reliabilityImprove inter-rater reliability
![Page 6: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/6.jpg)
Development of PackageDevelopment of Package
Scripted 10x 2-3 minute OSCE Scripted 10x 2-3 minute OSCE scenariosscenarios Borderline/PassBorderline/Pass Pass/HonoursPass/Honours
2 hours filming in my office2 hours filming in my office 2 lecturers2 lecturers 1 bullied student1 bullied student
![Page 7: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/7.jpg)
Creation of MediaCreation of Media
wmv files converted to flv using Flash wmv files converted to flv using Flash MacromediaMacromedia
Images created utilising Flash Images created utilising Flash MacromediaMacromedia
MP3 creation using AudacityMP3 creation using Audacity
![Page 8: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/8.jpg)
XerteXerte
Template ‘ratings’ created by Julian Template ‘ratings’ created by Julian TenneyTenney
Media was imported into Xerte Media was imported into Xerte through the Wizardthrough the Wizard
![Page 9: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/9.jpg)
![Page 10: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/10.jpg)
The PackageThe Package
An introduction to the new marking An introduction to the new marking schemescheme
Example mark sheetsExample mark sheets Video of example studentsVideo of example students
Rated on criteriaRated on criteria Global impression givenGlobal impression given User then offered feedbackUser then offered feedback
![Page 11: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/11.jpg)
Using the WizardUsing the Wizard
Once Wizard create – efficient Once Wizard create – efficient method of presenting contentmethod of presenting content Define scenarioDefine scenario Define marking sheet, including number Define marking sheet, including number
of criteriaof criteria Define fail/borderline/pass/honoursDefine fail/borderline/pass/honours Import mediaImport media
![Page 12: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/12.jpg)
![Page 13: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/13.jpg)
![Page 14: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/14.jpg)
Sample of PackageSample of Package
![Page 15: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/15.jpg)
Distribution of PackageDistribution of Package Letters and emails were sent to all Letters and emails were sent to all
examining consultants with following examining consultants with following linklink
www.nottingham.ac.uk/~mczbjd/OSCEwww.nottingham.ac.uk/~mczbjd/OSCE.htm.htm
Package ran for ~20 minutesPackage ran for ~20 minutes
![Page 16: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/16.jpg)
Result of Package Result of Package IntroductionIntroduction
30% examiners accessed the 30% examiners accessed the package prior to the OSCEpackage prior to the OSCE
Remainder underwent abridged Remainder underwent abridged version before the OSCEversion before the OSCE
![Page 17: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/17.jpg)
AnalysisAnalysis
ANOVAs undertaken on awarded ANOVAs undertaken on awarded scores by the different examiners on scores by the different examiners on the same stationthe same station
![Page 18: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/18.jpg)
December OSCEDecember OSCE
StationStation MaxMax MinMin AllAll
AA70.370.3
(5.0)(5.0)53.453.4
(8.6)(8.6)62.362.3
(8.6)(8.6)
BB72.672.6
(9.1)(9.1)52.152.1
(8.85)(8.85)63.763.7
(11.5)(11.5)
CC66.166.1
(5.1)(5.1)51.051.0
(7.6)(7.6)56.956.9
(8.6)(8.6)
DD66.166.1
(56.7)(56.7)56.656.6
(5.0)(5.0)60.760.7
(7.4)(7.4)
![Page 19: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/19.jpg)
**p<0.001 *p<0.05**p<0.001 *p<0.05
MayMay DecemberDecember
OSCE 1OSCE 154.5**54.5**
(10.3)(10.3)62.3**62.3**
(8.6)(8.6)
OSCE 2OSCE 256.2**56.2**
(9.77)(9.77)63.7**63.7**
(11.5)(11.5)
OSCE 3OSCE 355.5**55.5**
(9.02)(9.02)56.9**56.9**
(8.6)(8.6)
OSCE 4OSCE 454.3**54.3**
(10.8)(10.8)60.760.7
(7.4)(7.4)
![Page 20: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/20.jpg)
VariationVariationStationStation MayMay DecDec
AA 13.713.7 16.916.9
BB 12.612.6 20.520.5
CC 17.117.1 15.015.0
DD 20.520.5 10.510.5
![Page 21: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/21.jpg)
LimitationsLimitations
Flash Player 7 not installed as standard Flash Player 7 not installed as standard on healthcare trusts’ computerson healthcare trusts’ computers
Minor problems with Xerte - since Minor problems with Xerte - since resolvedresolved
Delay on slow broadband connectionsDelay on slow broadband connections ?Representative videos?Representative videos Any improvement likely to be due to Any improvement likely to be due to
the new marking schemethe new marking scheme
![Page 22: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/22.jpg)
XerteXerte
Once familiar:Once familiar: Easy to useEasy to use Multimedia packages can be create Multimedia packages can be create
in minutesin minutes Easily distributedEasily distributed Much more complicated to create Much more complicated to create
bespoke packagesbespoke packages
![Page 23: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/23.jpg)
FutureFuture
Improve IT access in the healthcare Improve IT access in the healthcare truststrusts
Reduce bitrateReduce bitrate Produce improved examples of Produce improved examples of
borderline studentsborderline students Extend the package to attempt to Extend the package to attempt to
standardise the Formative assessmentsstandardise the Formative assessments ?Make available to students?Make available to students
![Page 24: Xerte Standard Setting in Psychiatry Undergraduate OSCE Dr Ben di Mambro Clinical Lecturer in Psychiatry.](https://reader036.fdocuments.us/reader036/viewer/2022062417/551c59c65503467b488b56f2/html5/thumbnails/24.jpg)