SpecialtySelec+on&Future Na+onalEvalua+onof ... ·

24/02/2012

1

Specialty*Selec+on*&*Future*Developments*

Dr*Alison*S*Carr*Senior*Clinical*Advisor*

Medical*Educa+on*&*Training*Programme*

What’s'in'this'presenta-on'

•  Context*• Na+onal*Evalua+on*of*Specialty*Selec+on*(NESS)*report*

•  Shortlis+ng*•  Communica+on*skills*

•  Reliability*• What*areas*of*selec+on*require*further*work*and*what*are*the*priori+es?*

Medical Education & Training Programme

Context'

Improvements'to'Specialty'Selec-on

To*pilot*new*ways*of*selec+ng*doctors*thoroughly*

Stakeholder*engagement*

To*develop*a*clear*evidence*base*before*introducing*new*methods*of*selec+on*

Small*incremental*changes*made*that*are*carefully*evaluated*


What'are'we'aiming'to'achieve'in'

Specialty'Selec-on?'

Cost- Effectiveness

Validity Acceptability Reliability

What'are'we'aiming'to'achieve?'

Recruitment*&*Selec+on*being:*• *Fair:*improved*transparency*&*equity*• *Valid*• *Reliable*• *CostNeffec+ve*with*limited*resources*• *Feasible*• *Defensible/*equal*opportuni+es*compliant*• *Collabora+ve*between*Colleges,*Deaneries*&*DH*

Recruitment*&*selec+on*based*on*bestNprac+ce*and*sound*governance*resul+ng*in*improved*quality*&*safety*for*pa+ents*through*selec+on*of*trainees*more*likely*to*be*effec+ve*doctors,*high*fill*rates*and*minimal*risk.*

Medical education & training programme

24/02/2012

2

The'Na-onal'Evalua-on'of''

Specialty'Selec-on'Project'

An*overview*and*presenta+on*of*selected*sensi+vi+es*within*the*NESS*report*

The'NESS'Project'

•  Scope*o  Selec+on*into*14*special+es*(mainly*at*ST1/CT1)*

o  In*depth*analysis*in*5/13*deaneries*

o  Evalua+on*mainly*in*2009*

www.mmc.nhs.uk

•  Commissioned*in*2008*•  Report*in*2010*

o  Evaluate*Specialty*Selec+on*Pilots*(Wave*1*&*2)*in*2009*

o  Addi+onal*brief:*report*on*na+onally*coordinated*recruitment*


NESS*covers*the*selec+on*processes*for*71%*of*4,060*hospital*training*posts*(England*only)*and*2,751*General*Prac+ce*(GP)*training*posts*(UK)*available*at*ST1/CT1*level*in*2009**

Aimed*to*answer:**

*•  Acceptability:*Do*candidates*and*assessors*have*

confidence*in*the*selec+on*process?**•  Reliability:*Would*the*outcomes*of*the*selec+on*

process*be*the*same*if*it*were*repeated*on*a*different*day,*with*different*ques+ons*and*different*assessors?**

•  Validity:*Do*selec+on*scores*predict*performance*in*speciality*training?**

•  CostHeffec-veness:'Does*the*selec+on*process*provide*value*for*money*as*determined*by*an*economic*evalua+on?**

Pilots'for'NESS'evalua-on'

'Wave'' Organisa-on' Specialty'and'level' Methodology'

1*(extended)* South*West*Peninsula*Deanery*

Anaesthesia,*ACCS*and*CMT,*CT1*

Machine*marked*test,*selec+on*centre*

1* RCP/KSS*Deanery* CMT,*CT1* Clinical*problem*solving*test*

1* Trauma*and*Orthopaedic*SAC*

Trauma*and*Orthopaedic,*CT1*

10*sta+on*selec+on*centre,*clinical*problem*solving*tests*

1*(extended)* RCS* Core*Surgical*Training,*CT1/2*

Selec+on*centre*

2** Mul+Ndeanery* Neurosurgery,*ST1*–*4* New*sta+ons*for*selec+on*centre*

2** West*Midlands* GP,*ST1*–*3* Cogni+ve*ability*test*

2** KSS*Deanery* GP,*ST1*–*3* Regional*selec+on*centres*

2** Mul+Ndeanery* General*Surgery,*ST3* Regional*selec+on*centre*

2** Mul+Ndeanery* Paediatrics,*ST1* Targeted*assessment*of*porfolio*

2** London* Paediatrics,*ST1* Knowledge*test*

NESS:'SpecialtyHDeanery'Matrix'

Specialty

Eas

t M

idla

nds

Eas

t of

Eng

land

K

SS

Lond

on

Mer

sey

Nor

th

Wes

tern

N

orth

ern

Oxf

ord

Pen

insu

la

Sev

ern

Wes

sex

Wes

t M

dlan

ds

York

s &

H

umbe

r

ACCS/Anaesthesia

�*

Cardiothoracic Surgery

�* �* �* �* �* �* �* �* �* �* �* �* �*

Core Medical Training

�* �* �* �* �* �* �* �* �* �* �* �* �*

Core Surgical Training

�* �* �* �* �* �* �* �* �*

Emergency Medicine

�* �* �* �* �* �* �* �* �* �* �* �* �*

General Practice �* �* �* �* �* �* �* �* �* �* �* �* �*General Surgery �* �* �*Histopathology �* �* �* �* �* �* �* �* �* �* �* �* �*Neurosurgery �* �* �* �* �* �* �* �* �* �* �* �* �*Obs & Gynae �* �* �* �* �* �* �* �* �* �* �* �* �*Paediatrics �* �* �* �* �* �* �* �* �* �* �* �* �*Psychiatry �* �* �* �* �* �* �* �* �* �* �* �* �*

24/02/2012

3

Data*sources*considered:*

!  Documents*(academic*literatures,*specialty/deanery/College/MMC/DH*documents*and*websites*and*pilot*reports);**

!  Surveys*(online*applicant*ques+onnaire,*selec+on*day*ques+onnaires*completed*by*candidates*and*assessors);**

!  Interviews*(with*candidates,*assessors*and*specialty/deanery*representa+ves);**

!  Score*data.**

Acceptability:*

Overall,*high*degree*of*trainee*sa+sfac+on*in*all*3*domains*surveyed:*

o Fairness*o FitnessNforNpurpose*o Organisa+on**

Reliability'H'Internal'consistency:'

!  Ness*comment**Cronbach�s*alpha*is*widely*reported*as*a*measure*of*reliability,*despite*concerns*of*many*sources*of*variance,*e.g.*between*assessors,*not*considered*in*calcula+on*

!  NESS*regard*an*alpha*>*0.7*as*a*necessary*but*not*sufficient*condi+on*for*high*reliability.**

!  Cronbach�s*alpha*was*calculated*for*all*the*selec+on*systems*with*obtainable*data.**

!  7/*10*special+es*(centres*and*special+es*named)*had*a*mean*overall*alpha*lower*than*0.7,*which*is*a*cause*for*concern.*

How*reliable*is*the*decision*on*whether*to*offer*a*candidate*a*post?**

!  The*fundamental*purpose*of*selec+on*is*to*have*confidence*that*the*candidates*offered*posts*are*beker*than*those*who*are*not.**

!  Across*the*eight*special+es*where*this*was*es+mated,*there*was*considerable*uncertainty*as*whether*posts*should*or*should*not*have*been*offered*to*between*28*and*54%*of*candidates.**

Validity:*

!  In*terms*of*the*predic+ve*validity*of*shortlis+ng,*aler*correc+ng*for*restric+on*of*range,*the*Pearson*correla+on*coefficients*for*the*specialty/*deanery*combina+ons*for*which*data*was*available*range*from*0.33*(CST*in*the*West*Midlands)*to*0.82*(O&G*in*North*Western),*with*a*mean*of*0.55.**

!  None*of*these*assessments*of*validity*address*long*term*predic+ve*validity*and*the*issue*of*whether*the*selec+on*systems*iden+fy*those*who,*in*the*long*term,*are*bestNsuited*to*be*consultants*in*the*specific*specialty.**

Cost*effec+veness*

•  Per*applicant*costs*ranged*from*£160*to*£1,100.**

24/02/2012

4

ST1/CT1'Shortlis-ng'in'2009'

Additional)Qualifications

Additional)Achievements

Achievements)Outside)Medicine

Activities)and)Achievements)showing)commitment)to)specialty

Commitment)to)Specialty

Clinical)Skills

Research)Skills/Management

Presentations

Publications

Clinical)Audit/Clinical)gov

Teaching)experience

Training)courses)attended

Team)working

Information/Technology

Career)progression

Skills)in)written)English

Personal)statement

Overall)excellence)of)application

Additional)Qualifications

Additional)Achievements

Achievements)Outside)Medicine

Activities)and)Achievements)showing)commitment)to)specialty

Commitment)to)Specialty

Clinical)Skills

Research)Skills/Management

Presentations

Publications

Clinical)Audit/Clinical)gov

Teaching)experience

Training)courses)attended

Team)working

Information/Technology

Career)progression

Skills)in)written)English

Personal)statement

Overall)excellence)of)application

419

2.511

3.528

3.518

4.514

2.517

ACCS

Anaesthesia

Microbiology

Ophthalmology

Radiology

Surgery

Additional2Qualifications

Additional9QualificationsMAXAdditional9QualificationsMIN

ST1'Regional'Recruitment'2009'

Scores*are*%*Total**Applica+on*Form*Score*

411

37

57

5&&&&&&&12.5

39

514

ACCS

Anaesthesia

Microbiology

Ophthalmology

Radiology

Surgery

Additional&Achievements

Additional9AchievementsMAXAdditional9AchievementsMIN





What'developments'have'been'made?'

24/02/2012

5

30

44

70

85

95

2007 2008 2009 2010 2011

Nationally2co4ordinatedrecruitment2(%)

125,000

15,540

85,000

10,009

33,202

7,073

24474

6799

16,044

6,697

2007 2008 2009 2010 2011

CT1/ST10Applications0per0year

CT1/0ST10Posts0per0year

Effects'of'Na-onal'Specialty'Recruitment' What'developments'have'been'made?'

•  Collabora-on'with'Colleges,'Deaneries'&'

BMA/JDC'

•  Electronic'applica-ons'via'College'or'Deanery'IT'

systems'

•  Offers'period'•  Evalua-on'of'selec-on'

processes'&'further'

refinements'

www.mmc.nhs.uk

Na-onal'recruitment'

•  Standardisa+on*o  Na+onal*Person*Specifica+ons*o  Applica+on*form*o  Scoring*matrices*o  Selec+on*centre*

•  Cost*effec+veness*" Applica+ons*" Interviews*per*specialty*


Pilots'of'selec-on'into''

Specialty'Training'(2008H10)'

Na+onal*Evalua+on*of*Specialty*Selec+on*(NESS),*University*of*Birmingham*– Wave*1*&*2*pilots*– Na+onal*processes*

South'West'Peninsula'Pilot'ST1:'

CPS/'SJT'

• *Anaesthesia*• *ACCS*Selec-on'Centre*• Anaesthesia*

CPS*Test**

What'are'we'trying'to'achieve?'

Predic-ve'Validity'

Significant'predictors'of:' GP' Anaes'

CT1'

CMT'

CT1'

CST'

CT1'

CPS/SJT* Selec+on*performance* �* �* �* �**

Exam*results* �* �* �*

InNtraining*performance* �*

EndNof*training*competence* �**

SC* Selec+on*performance* �* �*

Exam*results* �* �*

InNtraining*performance* �* �*

EndNof*training*competence* �**

Medical Education & Training Programme CPS*Clinical*Problem*Solving*Test*SJT*Situa+onal*Judgment*Test*SC*Selec+on*Centre

Predic-ve'validity'and*incremental'validity'of*selec+on*tools*in*predic+ng*performance*in*MRCP'

Exam' Predictor' R'values' P'value'

MRCP Part 1*

GP CPS SJT CMT shortlisting, CMT interview CMT final total*

r=.73; range across diets r=.53 to r=.79, r=.41 mean r=.50 mean r=.48 mean r=.53 *

All <.001**

MRCP Part 2*

GP CPS SJT CMT shortlisting, CMT interview and CMT final total*

r=.58; range across diets r=.52 to r=.74, r=.40 mean r=.32 mean r=.37 mean r=.38 *

< 0.05*All*<.001*

PACES* the CMT interview SJT CPS CMT shortlisting*

mean r=.44 r=.39, r=.38, mean correlation of r=.35*

<.001 <.001 <.001 <.001

Validation study used data from 2009 CMT selection process, incl. GP machine-markable tests, and MRCP outcome data (Part 1, Part 2 and PACES) since 2008

Patterson F, Carr V, Burr WA, Berkin L, Dacre J March 2011

What'further'developments'are'needed?'

24/02/2012

6

Interdeanery CMT Raw Interview Score Correlations 2009 Deanery sites

1 2 3 4 5 6 7 8 10

2 .74** N= 41

5 .76** N= 163

.59** N= 175

7 .72** N= 30

.68** N= 134

8 .69** N= 34

9 .50** N= 93

10 .46* N= 26

11 .41* N= 26

12 .62** N= 24

.74** N= 28

13 .77** N= 20

.76** N= 40

.73** N= 39

.57** N= 28

14 .72** N= 82

.89**

* Correlation significant at 0.05 level (2-tailed), ** Correlation significant at 0.01 level (2-tailed).

Pat

ters

on e

t al

Anaesthesia selection scores differ by deanery (Kruskal Wallis test, p<0.001)

16151413121110987654321

120

110

100

90

80

70

60

50

40

30

Deanery

Sele

ctio

n sc

ore

Scores are particularly high in deaneries 4 & 13 and particularly low in 5 & 9.

Interview scores CT1 Anaesthesia 2011

•  For*applicants*interviewed*in*2*UoAs,*r=0.39*between*1st*and*2nd*na+onal*score*totals.**

•  Corresponding*correla+ons*between*1st*and*2nd*structured*interview,*presenta+on*and*porfolio*scores*are*0.16,*0.18*and*0.41.***

There*is*a*slight*learning*effect*amongst*twiceNinterviewed*candidates.**

Ronny*Cheung1*Toni*Ardolino1*Alison*Carr2*

1Clinical*Advisor,*DH***2Senior*Clinical*Advisor,*METP*

Communica+on*skills**in*level*1*specialty*recruitment*in*2010'

Context*&*Methods*

•  Disparity*of*assessment*methods*and*endpoints*

•  Ques+onnaire*survey*to*12*special+es*

Current*prac+ces*•  Scoring**•  Assessment*methodology*

•  Ongoing*evalua+on*•  Examples*of*best*prac+ce*

•  Assessment*stage*– Applica+on*stage:*3/12*–  Interview*stage:*all*special+es*

•  Separate*score?*–  8/12;*Range*10%*(CMT)*N*35%*(Paediatrics)*

•  Separate*sta+on?*–  2/12*use*a*communica+onNonly*sta+on*

•  10*/12*special+es*would*not*reject*an*applicant*solely*based*on*an*unsa+sfactory*score*in*communica+on*skills**

Key*Findings:*Scoring*

24/02/2012

7

How*are*communica+on*skills*assessed?*

Anaesthetics/

AC

CS

CM

T

General P

ractice

Histopathology

Microbiology

Neurosurgery

Ophthalm

ology

Paediatrics

Psychiatry

Public H

ealth

Radiology

Surgery

Academic summary

# #

Handwriting #

Written exercise

# # #

Portfolio interview

# # #

Panel interview

# # #

Presentation # # # #

Clinical interview

# # # #

Simulated clinical scenario

# # # # #

Team exercise

#

Anaesthetics/

AC

CS

CM

T

General P

ractice

Histopathology

Microbiology

Neurosurgery

Ophthalm

ology

Paediatrics

Psychiatry

Public H

ealth

Radiology

Surgery

Academic summary

# #

Handwriting #

Written exercise

# # #

Portfolio interview

# # #

Panel interview

# # #

Presentation # # # #

Clinical interview

# # # #

Simulated clinical scenario

# # # # #

Team exercise

#

•  Design*–  7/12*outlined*the*basis*for*design*–  Expert*consensus*most*common*(6/7)**

•  Evalua+on*–  Informal,*itera+ve*appraisals*–  5/12*conduc+ng*formal*evalua+on*

Key*Findings:*Evidence*base*

• Explicit scoring for communication skills

• Minimum standard for recruitment • Formal evaluation processes • Best practices shared

–  E.g. simulated clinical scenarios, use of lay interviewers

Recommendations for ST1/CT1 in 2013

Likely'areas'of'future'development'

•  Need*to*address:'•  Fairness*•  Consistency*•  Standardisa+on*•  Content*development*•  U+lity*

•  Focusing*on*costN*effec+veness*

•  Future*improvements*•  Implement*standards*for*recruitment*&*selec+on*across*all*special+es,*levels*and*locali+es*

• Develop*consistency*• Validate*processes*• Train*raters*• Benchmarking*• Evaluate*processes*effec+vely*


UK'MaDRaS'

•  Quality*SubNcommikee*•  3*areas*for*explora+on*in*2012/3:*

o Promote selection pilots to improve validity & reliability of specialty selection.

•  include studies of predictive validity •  further developing a multi-specialty MMT for selection into

ST1.*

o  Analysis*of*training*and*benchNmarking*of*selectors*and*determina+on*of*minimum*requirements*

o  Determining*the*minimum*evalua+on*of*na+onal*specialty*selec+on*


Summary'


Any'Ques-ons?'

•  Context*• Na+onal*Evalua+on*of*Specialty*Selec+on*(NESS)*report*

•  Shortlis+ng*•  Communica+on*skills*

•  Reliability*• What*areas*of*selec+on*require*further*work*and*what*are*the*priori+es?*

Specialty*Selec+on*&*Future* Na+onal*Evalua+on*of ... ·

Documents

Transcript of Specialty*Selec+on*&*Future* Na+onal*Evalua+on*of ... ·

SpecialtySelec+on&Future Na+onalEvalua+onof ... ·

Transcript of SpecialtySelec+on&Future Na+onalEvalua+onof ... ·