Sample Descriptive Statistics This Table Presents Descriptive Statistics
Web viewA descriptive analysis of the use of workplace-based assessments in UK surgical training....
Transcript of Web viewA descriptive analysis of the use of workplace-based assessments in UK surgical training....
A descriptive analysis of the use of workplace-based assessments in UK surgical training
Joseph Shalhoub1, Cristel Santos2, Maria Bussey2, Ian Eardley2, William Allum2
1 Department of Surgery & Cancer, Imperial College London, UK2 Joint Committee on Surgical Training, UK
For Correspondence
Mr William Allum
ISCP Surgical Director
Joint Committee on Surgical Training
35 - 43 Lincoln’s Inn Fields
London WC2A 3PE, UK
E-mail: [email protected]
Tel: +44 (0) 20 8661 3982
Word Count3,266 Words
1
Abstract
Background
Workplace-based assessments (WBAs) were formally introduced in the UK in 2007.
The study aim was to describe use of WBAs by UK surgical trainees and examine
variations by training region, specialty or level of training.
Methods
The database of the Intercollegiate Surgical Curriculum Programme (ISCP) was
interrogated for WBAs between August 2007 and July 2013, with in-depth analysis of
two periods: August 2011 to July 2012, and August 2012 to July 2013.
Results
Numbers of validated WBAs per trainee per year increased more than 7-fold, from
median 6 per trainee in 2007-2008, to 39 in 2011-2012, and 44 in 2012-2013. In
2011-2012, 58.4% of core trainees completed the recommended 40 WBAs; only
38.1% of specialty trainees achieving 40 validated WBAs. In 2012-2013, these
proportions increased to 67.7% and 57.0% for core and specialty trainees,
respectively. Core trainees completed more WBAs per year than specialty trainees in
the same training region. London core trainees completed the highest numbers of
WBAs in both 2011-2012 (median 67) and 2012-2013 (median 74). There was a
peak in WBAs completed by London specialty trainees in 2012-2013 (median 63).
The most validated WBAs were completed by ST1/CT1, with a gradual fall in median
WBAs to ST4, followed by a plateau; in 2012-2013 there was an increase in WBAs at
ST8. Core surgical trainees complete ~50% ‘operative’ (PBA/DOPS) and ~50% ‘non-
operative’ assessments (CBD/CEX). During specialty training, PBAs represented
~46% of WBAs, DOPS 11.2%, CBD ~23% and CEX ~15%.
Conclusions
UK surgical trainees are, on average, undertaking one WBA per week. Variation
exists in use of WBAs between training regions. Core trainees tend to use the
spectrum of WBAs more frequently than their senior colleagues. Further work is
required to examine the role of WBAs in assessment, and engagement and training
of trainers in processes and validation of WBAs.
Key Words
2
Introduction
Workplace-based assessments (WBAs) were formally introduced in the UK in 2007
to accompany the competency-based Modernising Medical Careers postgraduate
training programme 1. WBAs are delivered as part of the Intercollegiate Surgical
Curriculum Programme (ISCP), a joint activity by the four surgical Royal Colleges 2.
The ISCP has provided each surgical specialty with a comprehensive syllabus, a
teaching and learning framework, an assessment system, a repository for evidence
on individual trainee progress and an interactive web platform.
The use of WBAs is a mandatory component of each surgical trainee’s portfolio
through both core surgical training and specialty training. The main WBA assessment
tools include procedure-based assessments (PBA), direct observation of procedural
skills (DOPS), clinical evaluation exercise (CEX) and case-based discussion (CBD),
as well as multi-source feedback (MSF). A glossary of terms used is presented in the
Appendix.
PBA involves direct observation of an index procedure or operation with comments
given on important steps, tasks or skills at the pre-, intra- and post-operative stages
of the procedure, considered to be essential for its safe and successful completion.
The assessment form for a PBA is specific to the procedure or operation being
assessed. Similar to PBA, DOPS are more generic and relate to either less complex
procedures or parts of a larger procedure. The domains of assessment in DOPS are
generic and fixed and considered to be transferable across this spectrum of tasks.
CEX is an assessment of a trainee’s clinical skills, for example history taking, clinical
examination, and information giving. CBD refers to a formalized discussion related to
the care of a patient with a focus on knowledge and attitude. Each of these WBAs is
completed by both trainee and trainer, validated by the trainer and is recorded within
ISCP. MSF is an opportunity for members spread across the multidisciplinary health
care and administrative team to offer anonymous feedback on the trainee. The
results are then discussed at a meeting with the assigned educational supervisor and
the trainee’s annual review of competence progression (ARCP) 3.
At present, the Joint Committee on Surgical Training (JCST) recommends that each
trainee completes a minimum of 40 WBAs per year of training. This approximately
equates to one WBA per week, although trainees and trainers are encouraged to
4
complete more with a spread throughout a training post to demonstrate progression
of clinical and technical skills 5.
The purpose of this study is to report on the use by surgical trainees in the UK of the
largest platform for competence-based training worldwide. We aim to describe the
use of WBAs by surgical trainees and to examine any variations observed by training
region, surgical specialty or level of training.
5
Methods
Every assessment recorded in trainees’ portfolios is recorded with background
information about the individual trainee. These data were collated to facilitate
analysis to quantify the use of the WBAs. The ISCP database was interrogated for
WBAs linked to trainees’ portfolios between August 2007 and July 2013, with an in-
depth analysis of two periods: August 2011 to July 2012, and August 2012 to July
2013. These two time periods represent the most recent two years for which, at the
time of data analysis, complete data was available. Trainees in Core Surgery and all
but one of the surgical specialties were included from 2007. Vascular surgery only
became a separate specialty in 2012 but has been included for the second time
period. Trauma and Orthopaedics trainees were variably included in the first years of
ISCP as they were also able to use the Orthopaedic Competence Assessment
Project (OCAP) system; all had transferred to ISCP by 2012/13.
Medical graduates in the UK commence postgraduate clinical training with a 2-year
generic ‘Foundation Programme’, from which candidates apply through a competitive
national selection system for core surgical training (CST) programmes – previously
termed the senior house officer (SHO) grade. Following successful completion of
CST and the Intercollegiate Membership of the Royal College of Surgeons
examinations, trainees apply through competitive national selection for higher
surgical training (registrar grade) in one of the 10 surgical specialities 6.
The main WBA assessment tools included in the analysis were DOPS, PBA, CEX
and CBD for all trainees in both core and higher specialty training, although there
was some data available for assessment of audit, teaching and multi-source
feedback. All trainees with an ISCP appointment type of core surgery (CT), fixed-
term specialty training appointments (FTSTA), locum appointments for training (LAT),
SHO, specialist registrars (SpR), specialty registrars (StR), and StRs converted from
SpR were included in the analysis. CT1 and CT2, as well as (for ‘run-through’
specialties/programmes) ST1 and ST2 were combined for analysis for consistency of
reporting on the early years of training.
Only WBAs which had been trainer validated were included. Entries with missing
variables were excluded. Data were anonymised and allocated a unique identifier to
avoid duplicate counting, and analysed by training region (Local Education and
Training Board [LETB] in England and Deanery in Wales, Scotland and Northern
6
Ireland), surgical specialty, and level of training at the time of the assessment.
Records without appointment type, training region, specialty or training level were
excluded from further analysis. After data cleaning, data comparison and summary
statistics were performed using STATA version 11 (StataCorp LP, Texas, USA). For
the analysis, medians and percentiles were used to represent the data as these are
not affected by extreme values.
7
Results
Records without appointment type, training region, specialty or training level totalled
1,433 for 2011-2012 and 1,355 for 2012-2013; these records were excluded from
further analysis. A total of 754,165 ISCP WBAs were validated by UK surgical
trainees between August 2007 and July 2013 (Table 1). Approximately two thirds of
trainees included are male (Table 1). There has been an increase in both the
number of WBAs validated using the ISCP and the number of surgical trainees using
the ISCP for WBAs year on year between 2007 and 2013. The number of validated
WBAs per trainee has increased more than 7-fold, from a median of 6 per trainee in
2007-2008, to 39 per trainee in 2011-2012, and 44 per trainee in 2012-2013.
In 2011-2012, 55.5% of core surgical trainees had completed the JCST-
recommended 40 WBAs, with only 35% of specialty trainees in surgery achieving 40
validated WBAs. In 2012-2013, these proportions increased to 65.5% and 53.6% for
core and specialty trainees, respectively Figure 1 illustrates the number of WBAs
being completed by core and specialty trainees for these two time periods.
WBAs across training regions
The use of WBAs across training regions for 2011-2012 and 2012-2013 can be seen
in Table 2. For the year 2011-2012, the median number of WBAs undertaken by
trainees varied from 30 to 50 (Figure 2a). For 2012-2013 this variation was between
39 and 67, with the peak number undertaken by London trainees. After separating
core and specialty trainees (Figures 2b and 2c), core trainees tended to complete
more WBAs per year than specialty trainees in the same training region. London core
surgical trainees completed the highest numbers of WBAs in both 2011-2012
(median 67) and 2012-2013 (median 74). There was also a peak in WBAs completed
by London specialty trainees in 2012-2013 (median 63).
WBAs across core surgical training and surgical specialties
The use of WBAs across core surgical training and individual surgical specialties for
2011-2012 and 2012-2013 can be seen in Table 3. The median number of WBAs
completed by core surgical trainees increased from 43 in 2011-2012 to 48 in 2012-
2013. Excluding vascular surgery, which became a recognised surgical specialty in
2012, the median number of WBAs completed by specialty trainees in 2011-2012
ranged from 21 in trauma and orthopaedic surgery, to 41 in neurosurgery, oral and
maxillofacial surgery, and otolaryngology. By 2012-2013, this range increased to
8
between 39 in paediatric surgery, and 45 in otolaryngology and vascular surgery. For
trauma and orthopaedic surgery, the median number of validated WBAs almost
doubled from 21 in 2011-2013 to 41 in 2012-2013 reflecting the change from OCAP
to ISCP.
WBAs across surgical training levels
The use of WBAs across surgical training levels for 2011-2012 and 2012-2013 can
be seen in Table 4. The most validated WBAs were completed by ST1/CT1 trainees,
with a gradual fall in the median number of WBAs to ST4. This is followed by a
plateau, however in 2012-2013 there was an increase in WBA numbers at the ST8
training level.
Types of WBAs being used
Analysis of the type of WBA shows core surgical trainees complete approximately
50% ‘operative’ or interventional (PBA and DOPS) and 50% ‘non-operative’
assessments (CBD and CEX), with the remainder (namely assessment of audit,
observation of teaching and multi-source feedback) contributing a small proportion
(Figure 3a). During core training, PBAs contributed approximately 14% of validated
WBAs, whilst DOPS represented about 31%; CBDs and CEXs were equally split at
about a quarter of validated WBAs each. During specialty training, PBAs represented
approximately 46% of WBAs, DOPS 11.2%, CBD about 23% and CEX approximately
15%; the remaining WBA types again contributing a small proportion (Figure 3b).
The proportion of WBA types remained relatively stable from 2011-2012 to 2012-
2013. Variability in the type of WBAs performed across the specialties was observed.
9
Discussion
The ISCP has offered a curriculum for surgical training and a platform for monitoring
of the competency-based training that has accompanied the Modernising Medical
Careers era of postgraduate medical training 1. At present, competence in this
system is being assessed primarily through the use of WBAs. Few would disagree
that the introduction of WBAs to support competency-based training is one of the
most significant changes to educational policy in medical and surgical training in
recent years.
The ISCP is designed to allow the trainee to demonstrate progression in knowledge
and clinical and technical skills, as well as achieving professional capabilities using
WBAs as formative tools. Engagement with this approach to training has required
faculty and trainee reinforcement of its principles and, while acknowledging a specific
number should not be an absolute, the figure of 40 has been selected as it
approximates to a minimum of one WBA per week. For each training year, there are
expected levels of competence defined within the curriculum which are reviewed on
an annual basis by a team comprising a programme director, independent
educational supervisors, the postgraduate Dean and external specialty advisors. At
this review the formative assessments undertaken by clinical supervisors, together
with an overview report from an assigned educational supervisor, are considered in a
summative way to confirm the appropriateness of a trainee proceeding onto the next
year of training.
Whilst WBAs have been a part of surgical training since 2007, from the 2012-2013
academic year the American Board of Surgery has required the completion of in-
program assessments 4. These are broadly divided into operative performance
assessments and clinical performance assessments. At the time of implementation in
2012-2013, two of each of these assessments was required, increasing to six of each
from the 2015-2016 academic year 4. The Operative Performance Rating System
(OPRS) is employed for operative performance assessment. OPRS WBAs are
broadly similar to PBAs in that they are operation-specific, examine and assess
defined steps of a procedure, with each step scored against a 5-point Likert (as
compared with ‘satisfactory’ and ‘needs development’ used for PBAs). For clinical
performance assessment, similar to CEX in the UK, the American Board of Surgery
uses mini-CEX and Clinical Assessment and Management Exam – Outpatient
(CAMEO) for the direct observation of clinical assessment of patients.
10
The purpose of this study was to examine the use of WBAs across a number of
denominators in UK surgical training. This is the first time that data describing the
use of WBAs and their validation has been released by the ISCP. The data reflects
that there has been a good uptake with regards the use of WBAs since the
commencement of their use in 2007, with a steady rise in WBAs per trainee per year
from 2007 to 2013.
However, the proportions of trainees who are meeting the JCST’s set minimum for
validated WBAs of 40 per annum – although rising – remains below 70% and 60% for
core and specialty trainees, respectively, for 2012-2013. It may be that these trainees
are completing 40 WBAs, but these are not being validated. At present there is a
perceived emphasis on the numbers of WBAs completed, as compared with their
content; this may be driven, at least in part, by the minimum numbers requirement.
While this may show engagement with training it is essential that quantity does not
trump quality 7.
One of the implications of this study is to highlight that additional support is
necessary for trainees to facilitate their use of WBAs, but importantly there is
undoubtedly a need to ensure that trainers are engaged, trained and supported by
Deaneries / LETBs in the processes related to WBA completion and validation.
Mechanisms for this include through accredited courses, for example Training the
Trainers (TtT), Training and Assessment in Practice (TAiP) and Training and
Assessment in the Clinical Enviroment (TrACE), and through the Faculty of Surgical
Trainers (affiliated with the Royal College of Surgeons of Edinburgh).
Core trainees are completing more WBAs than specialty trainees but it is expected
that as trainees who have ‘grown up’ with WBAs become the majority within specialty
training, this difference will be minimised. There is no clear difference with regards
the uptake of WBAs by different specialties, in particular the numbers for trauma and
orthopaedic surgery have come up to the level of other surgical specialties in 2012-
2013 following the transition onto ISCP from OCAP.
The variation in completed WBAs by training region highlights differences in
approach across the UK by Deanery / LETB. For example in London there was a
requirement in February 2012 for trainees to complete 80 per annum 8. This
requirement has resulted in a peak in WBAs in London from both core and specialty
11
trainees in 2012-2013. Despite this peak, the medians remained below 80 per annum
at a median of 74 and 63 per annum for core and specialty trainees in surgery,
respectively. Powell and colleagues specifically evaluated trainee and trainer opinion
with regards increasing the number of WBAs per trainee per year 7. Several concerns
were highlighted including a reduction in efficacy with increasing number, an
increased burden on trainers and the development of trainees focussed on
quantitative rather than qualitative outcomes. There is however no clear guidance in
the optimum number and further evaluation is required.
The dominance of ‘operative’ or interventional WBAs, particularly PBAs, amongst
specialty trainees is perhaps unsurprising. Work examining WBAs undertaken in
Wales emphasised that usage amongst specialty trainees is skewed towards PBAs,
with the authors concluding that the annual incremental uptake of WBAs between
2007 and 2013 “is reflective of the acceptability and reliability of the PBA and the
increasing confidence of trainers and trainees with WBAs” 9. However WBAs should
also be used to evaluate non-operative skills such that the more equitable proportion
of the different WBA tools seen in Core training should be re-established in Specialty
training.
The use of WBAs in surgical training is an evolutionary process. Although the
educational value of PBAs is supported by an evidence base 3,11 there is uncertainty
about the reliability of some of the other tools 3 10. The existing evidence base for
WBAs is related to their formative rather than their summative use, and trainee and
trainer survey data underlines the desire at present that their use reflects this7. The
approach to the introduction of WBAs has somewhat inadvertently resulted in them
being used as assessments of learning. If WBAs are used as assessments for
learning additional work is required to support the educational value of the other key
WBAs, namely DOPS, CEX and CBD. Indeed the GMC has highlighted the need to
separate the use of WBAs into Supervised Learning Events (SLEs, an assessment
for learning, formative) and Assessment of Practice (AoP, an assessment of learning,
summative) 11.
Study limitations
This work represents a preliminary, purely descriptive analysis, which will form the
basis for further analyses of the rich dataset. The analysis looks at number of WBAs
completed, not how WBAs are being used nor the content or outcomes of the
assessment, i.e. whether trainees are performing appropriately for their stage in
12
training or if they are showing an improvement. Furthermore, no qualitative data has
been analysed to date. Given this analysis focuses on validated WBAs, it is
questionable that the mechanisms that trainers have for validating WBAs are
universally uniform.
Future work
Future work will aim to address the shortcomings of this present analysis, including
an examination of the qualitative elements within WBAs, how WBA numbers relate to
outcomes in terms of WBA final ratings and ARCP outcomes, and an analysis of
index surgical procedures numbers against PBAs through integration of the e-
logbook. With regards examining trainers’ input, an analysis of the quality of
feedback is planned as well as a review of individual trainers’ training activities.
There is an aim to introduce an automated report within ISCP, which may act as a
data source on training for all stakeholders.
13
Conclusions
This study confirms quantitatively the uptake of WBAs in surgical training in the UK. It
identified that surgical trainees are on average undertaking one WBA per week. The
majority of WBAs have assessed operative or interventional skills with lesser
emphasis on clinic-based interactions.
This study has shown some variation in the use of WBAs between the training
regions. Core trainees tend to use the spectrum of WBAs more frequently than their
senior colleagues. As trainees progress through training, operative or interventional
assessments predominate with limited use of non-operative assessments. Further
work is required to examine the role of WBAs in assessment, and the engagement
and training of trainers in the processes and validation of WBAs.
14
References
1. Health Do. Modernising Medical Careers, 2003.2. McKee RF. The Intercollegiate Surgical Curriculum Programme (ISCP). Surger
(Oxford) 2008;26(10):411-16.3. Shalhoub J, Vesey AT, Fitzgerald JE. What Evidence is There for the Use of
Workplace-Based Assessment in Surgical Training? J Surg Educ 2014;71(6):906-15.
4. Surgery ABo. Training & Certification - Resident Performace Assessments. Secondary Training & Certification - Resident Performace Assessments. http://www.absurgery.org/default.jsp?certgsqe_resassess.
5. Marriott J, Purdie H, Crossley J, et al. Evaluation of procedure-based assessment for assessing trainees' skills in the operating theatre. Br J Surg 2011;98(3):450-7.
6. Fitzgerald JE, Giddings CE, Khera G, et al. Improving the future of surgical training and education: Consensus recommendations from the Association of Surgeons in Training. Int J Surg 2012;10(8):389-92.
7. Powell HRF, DiMarco AN, Saeed SR. Trainee and trainer opinion on increasing numbers of workplace-based assessments. Bulletin of the Royal College of Surgeons of England 2014;96(5):160-62.
8. Khera G, Baird E. Annual Workplace-Based Assessments. In: Standfield NJ, ed. London, 2012.
9. Fishpool SJC, Stew B, Roberts C. Otolaryngology WBAs in the Wales Deanery: the first six years. Bulletin of the Royal College of Surgeons of England 2014;96(5):164-66.
10. O'Kane R. Workplace Based Assessments: The Association of Surgeons in Training, 2010.
11. Council GM. Learning and assessment in the clinical environment: the way forward: General Medical Council, 2011.
15
Tables
Table 1The adoption of workplace-based assessment (WBA) tools. Data represent trainer-validated WBAs between 2007 and 2013.
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012 2012-2013 TOTAL 2007-2013
WBAs total 11,153 41,685 273.8% 82,627 98.2% 141,708 71.5% 210,249 48.4% 266,743 26.9% 754,165
Number of trainees 1,549 3,042 3,851 4,705 5,459 5,808
Female 415 26.8% 773 25.4% 1,037 26.9% 1,258 26.7% 1,497 27.4% 1,618 27.9%
Male 1,121 72.4% 2,249 73.9% 2,799 72.7% 3,437 73.0% 3,943 72.2% 4,172 71.8%
No information 13 0.8% 20 0.7% 15 0.4% 10 0.2% 19 0.3% 18 0.3%
Median WBAs per trainee 6 13 20 29 39 44
16
Table 2Workplace-based assessments by training region. HEE, Health Education England; WBA, workplace-based assessment.
Deanery / Local Education & Training Board
Mean number ofWBAs per trainee
Median number ofWBAs per trainee Inter-quartile range
2011-12 2012-13 2011-12 2012-13 Change 2011-12 2012-13East Midlands HEE 35.0 38.0 35.0 41.0 6 22-45 23 27-48 21
East of England HEE 33.4 37.9 34.0 39.0 5 17-47 30 24-51 27
KSS HEE 44.5 47.8 49.5 52.0 3 24-65 41 28-66 38
London HEEs 45.3 61.3 40.0 67.0 27 20-68 48 37-84 47
North West (Mersey Sector) HEE 33.2 41.1 34.0 43.0 9 22-44 22 30-51 21
North West (North West Sector) HEE 38.8 44.2 41.0 45.0 4 34-48 14 39-54 15
Northern East HEE 38.9 41.2 40.0 41.0 1 31-48 17 33-48 15
Northern Ireland 34.9 40.2 38.0 41.0 3 23-43 20 36-47 11
Scotland 32.9 37.2 38.0 39.0 1 16-45 29 26-46 20
South West HEE 32.2 36.9 33.0 39.0 6 17-45 28 23-48.5 26
Thames Valley HEE 36.7 38.7 40.0 42.0 2 17-50 33 27.5-49 22
Wales 30.9 37.1 30.0 40.0 10 19-42 23 19-52 33
Wessex HEE 33.5 37.4 37.0 39.0 2 17-47 30 26-47 21
West Midlands HEE 41.3 47.1 42.0 48.0 6 22-55 33 34-61 27
Yorkshire and the Humber HEE 31.0 37.5 32.0 39.0 7 17-44 27 24-48 24
17
Table 3Workplace-based assessments by surgical specialty. Core surgical training takes into account all trainees at core level (CT1/ST1 and CT2/ST2)
including neurosurgery, trauma and orthopaedic surgery, and Scottish trainees. The deadline for trauma and orthopaedic trainees to register
with ISCP was the 4th of April 2011. WBA, workplace-based assessment.
SpecialtyMean number of
WBAs per traineeMedian number ofWBAs per trainee Inter-quartile range
2011-12 2012-13 2011-12 2012-13 Change 2011-12 2012-13Core Surgical Training 43.0 47.8 43.0 48.0 5 28-56 28 32-63 31
Cardiothoracic Surgery 36.8 43.5 34.0 41.0 7 20-47 27 26-54 28
General Surgery 37.3 44.4 38.0 42.0 4 25-46 21 31-57 26
Neurosurgery 43.0 45.8 41.0 42.0 1 28-52 24 32-59 27
Oral and Maxillofacial Surgery 41.0 45.8 41.0 42.0 1 30-50 20 33-58 25
Otolaryngology 41.1 46.6 41.0 45.0 4 26-54 28 35-58 23
Paediatric Surgery 34.2 34.1 37.0 39.0 2 23-47 24 20.5-47 27
Plastic Surgery 38.0 44.4 35.0 42.0 7 21.5-48.5 27 28.5-56 28
Trauma and Orthopaedic Surgery 24.6 41.9 21.0 41.0 20 12-35 23 28-54 26Urology 33.8 38.9 36.0 39.0 3 23-43 20 27-48 21Vascular Surgery - 48.6 - 45.0 45 - - 34-59.5 26
18
Table 4Workplace-based assessments by level of training. ST8 level does not include oral and maxillo-facial surgery, and urology as trainees in these
specialties finish their training at ST7. It is also noted that the first cohort of Modernising Medial Careers trainees are finishing their training in
August 2013. CT, core surgical training; ST, specialty training in surgery; WBA, workplace-based assessment.
Training levelMean number of
WBAs per traineeMedian number ofWBAs per trainee Inter-quartile range
2011-12 2012-13 2011-12 2012-13 Change 2011-12 2012-13
ST1/CT1 41.7 47.8 44.0 49.0 5 21-57 36 26-65 39
ST2/CT2 37.5 40.8 38.0 43.0 5 17-51 34 20-57 37
ST3 26.8 33.3 23.0 32.0 9 10-41 31 12-48 36
ST4 24.9 32.0 21.0 31.0 10 9-38 29 13-45 32
ST5 25.6 30.6 23.0 29.0 6 9-39 30 10-44 34
ST6 24.6 30.4 21.0 28.0 7 9-38 29 12-44 32
ST7 27.4 30.3 25.0 28.0 3 13-39 26 12-43 31
ST8 26.7 35.5 21.5 33.0 12 5-40 35 20-45 25
19
Figure Legends
Figure 1Distribution of frequency of validated workplace-based assessments (WBAs). (a)
Core surgical trainees 2011-2012. (b) Specialty trainees in surgery 2011-2012. (c)
Core surgical trainees 2012-2013. (d) Specialty trainees in surgery 2012-2013.
Figure 2Median numbers of validated workplace-based assessments by training region. (a)
For all trainees 2011-2012 (dark blue) and 2012-2013 (light blue). (b) For core
surgical trainees 2011-2012 (blue) and 2012-2013 (red). (c) For specialty trainees in
surgery 2011-2012 (blue) and 2012-2013 (red).
Figure 3Break down of number of workplace-based assessment per type. (b) For core
surgical trainees 2011-2012 (pink) and 2012-2013 (red). (c) For specialty trainees in
surgery 2011-2012 (light blue) and 2012-2013 (dark blue). PBA, procedure-based
assessment; DOPS, direct observation of procedural skill; CBD, case-based
discussion; CEX, clinical evaluation exercise; OT, observation of teaching; MSF,
multi-source feedback.
20
Appendix
Glossary of terms presented in alphabetical order. Adapted from 3.
AoP Assessment of PracticeARCP Annual Review of Competence ProgressionCAMEO Clinic Assessment and Management Examination - OutpatientCBD Case-Based Discussion
Formalized discussion related to the care of a patient with a focus on knowledge and attitude
CEX Clinical Evaluation ExerciseAn assessment of a trainee’s clinical skills (e.g., history, examination, and information giving)
CST Core Surgical TrainingCT Core TraineeDOPS Direct Observation of Procedural Skill
Similar to PBA, but more generic and relate to either less complex procedures or parts of a larger procedure. The domains of assessment in DOPS are generic and fixed and considered to be transferable across this spectrum of tasks
FTSTA Fixed-Term Specialty Training AppointmentGMC General Medical CouncilHEE Health Education EnglandISCP Intercollegiate Surgical Curriculum ProgrammeJCST Joint Committee on Surgical TrainingLAT Locum Appointment to TrainingLETB Local Education and Training BoardMSF Multi-Source Feedback
An opportunity for members spread across the multidisciplinary health care and administrative team to offer anonymous feedback on the trainee. The results are then discussed at a meeting with the assigned educational supervisor and the ARCP
OCAP Orthopaedic Competence Assessment ProjectOPRS Operative Performance Rating SystemOT Observation of Teaching
Workplace-based assessment permitting the documentation of an observed teaching episode carried out by the trainee
PBA Procedure-Based AssessmentDirect observation of an index procedure or operation with comments given on important steps, tasks or skills at the pre-, intra- and post-operative stages of the procedure, considered to be essential for its safe and successful completion
SHO Senior House OfficerSLE Supervised Learning EventST Specialty TraineeSpR Specialist RegistrarStR Specialty RegistrarTAiP Training and Assessment in PracticeTrACE Training and Assessment in the Clinical EnvironmentTtT Training the TrainersWBA Workplace-Based Assessment
21