WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2...
-
Upload
caren-perry -
Category
Documents
-
view
224 -
download
0
Transcript of WBAs: the Northern Ireland School of Surgery Experience Richard Mayes 1 Robert Gilliland 1, 2...
WBAs: the Northern Ireland School of Surgery
ExperienceRichard Mayes1 Robert Gilliland1, 2
Jeffrey Campbell1, 2 Helen Holscher3
Department of Surgery, Ulster Hospital Dundonald, Northern Ireland1
School of Surgery, Northern Ireland Medical & Dental Training Agency2,Royal College of Surgeons of England3..
The Royal College of Surgeons of England
Regional Representatives Meeting
Wednesday 10th November 2010
Introduction of a Curriculum
• Web based curricula • Workplace Based Assessments (WBAs)
• Valid and reliable
• Formative and summative
• Clear indicators of professional progress
GMC April 2010
AimsTo determine if WBAs:
• Differentiate between trainees of different abilities (summative)
• Demonstrate progression towards competence (summative)
• Are completed at regular intervals through training (formative)
Methods
• Core Surgical trainees appointed Aug 2008
• Trainees not completing 24 months excluded
• Trainees were ranked according to interview
score (Q1, Q2, Q3 and Q4)
• 4 x 6 month placements (P1, P2, P3 and P4)
• Median (range): ANOVA
Methods
• CBDs, DOPS, Mini-CEXs – score 1-6• 1-2 - Below expectations• 3 - Borderline• 4 - Meets expectations• 5-6 - Above expectations
• PBAs – score 0-4• Level 0 – Insufficient evidence to support judgement• Level 1 – Unable to perform procedure• Level 2 – Able to perform procedure, or part under
supervision• Level 3 – Minimum supervision• Level 4 – Unsupervised, able to deal with complications
Results
• 39 trainees • 12 excluded• n=27 trainees
• 1,256 assessments• 381 CBDs• 408 DOPS• 346 Mini-CEXs• 121 PBAs
Number of assessments
0
50
100
150
200
250
300
350
P1 P2 P3 P4
Placement
Nu
mb
er PBA
CEX
DOPS
CBD
p=0.39
Can WBAs differentiate between trainees of different abilities?
Can WBAs differentiate between trainees of different abilities?
Trainees from Q1 would out perform trainees from Q4
• number of assessments performed • scores obtained
Median (range) Scores in CBD and Mini-CEXs
0
2
4
6
Quartile
p=0.11
Med
ian
Sco
re
Median (range) Number of WBAs Completed per Trainee
0
20
40
60
80
Quartile
p=0.92
Num
ber
of A
sses
smen
ts
Can WBAs demonstrate progression towards competence?
Can WBAs demonstrate progression towards competence?
Scores should show improvement over
4 placements
CBD, DOPS and Mini-CEX Median (range) Scores
P1 P2 P3 P40
2
4
6
DOPS
Mini-CEX
CBD
Placement
Sco
re
CBD p=0.47, DOPS p=0.19, Mini-CEX p=0.96
PBA Median (range) Scores
P1* P2 P3
P4*0
1
2
3
4
I----NS----I I----NS----I I----NS----I
I-------------*p=0.0007-------------I
Placements
Sco
re
Are WBAs being used in formative assessment?
Are WBAs being used in formative assessment?
WBAs should be completed at regular intervals through training placements
Total Number of Assessments per Month
A S O N D J F M A M J J A S O N D J F M A M J J0
50
100
150
200
P1 P2 P3 P4
*
*
*
*
Months of Placement
Nu
mb
er o
f A
sses
smen
ts
Summary
• No inter-quartile differences with regards to the numbers of WBAs performed or scores obtained
• CBD, DOPS, Mini-CEX scores do not improve with time
• PBAs scores improve with time
• WBAs are clustered immediately prior to appraisal
Conclusions
• WBAs do not fulfil GMC requirements
• Need modification
• Assessors require to be more robust in their evaluations
Is one assessor type more robust than the other?
Median (range) Scores in WBAswith HST/Consultant assessors
HSTsCons
HSTsCons
HSTsCons
0
2
4
6
P1 P2 P3
Placements
Sco
re
ANOVA p=0.42
Distribution of assessment scores
1 2 3 4 5 60
200
400
600
Score
Nu
mb
er