How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An...
Transcript of How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An...
![Page 1: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/1.jpg)
Kevin Fox
Lead Task Force Certification at ESC
ESC Education Committee
How to Evaluate Clinical Performance
October 2017
![Page 2: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/2.jpg)
Introduction
• Why evaluate clinical performance
• Performance v Competence
• Knowledge - exams
• Skills and Professionalism – multiple methods
![Page 3: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/3.jpg)
Embedding the importance of going beyond knowledge to train well performing doctors
![Page 4: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/4.jpg)
Testing knowledge - exams still best
![Page 5: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/5.jpg)
An Effective Exam Process – a 1 year cycle - The EEGC (a UEMS / ESC collaboration)
• Write Questions - SBA MCQs - group review
• Select Questions
• Set Standard – score each item individually, then as a group (2d)
• Final Review
• Candidates take test
• Analysis of test results
• Agree a pass mark – different techniques (Angoff v Hofstee)
• Distribution of exam results
![Page 6: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/6.jpg)
Analysis of pass mark – Hofstee method
![Page 7: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/7.jpg)
Comments
The validity of any exam is influenced by
• Candidates perception of the exam
• Candidates level of training
• Wider stakeholder perceptions eg National and Trainer
• Familiarity with exam format (trainers and candidates)
• The challenge
• Contextualising for individual countries (eg NICE in the UK)
![Page 8: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/8.jpg)
Skills and Professionalism
![Page 9: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/9.jpg)
Methods to assess performance
Skills and Professionalism
![Page 10: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/10.jpg)
• The development of DOPS
Skills and Professionalism
![Page 11: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/11.jpg)
• Introduced (in UK) 2005
• There is scanty psychometric data on DOPS perhaps due to the fact that direct
observation is carried out informally. However intrinsically, in terms of
“competency level” it is seen as a high quality instrument as it tests at the “does”
level. Authors have commented on the lack of studies accessing the validity and
reliability of DOPS, despite it being fairly widely used to assess competency of
surgical trainees
DOPS
![Page 12: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/12.jpg)
• Train the trainers
• Standardise the assessment framework
• Use lots of items
Increasing reliability
![Page 13: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/13.jpg)
• The OSCE can be a reasonably reliable, valid and objective method of
assessment , but its main drawback is that it is resource intensive
OSCE assessment - assesses competence
![Page 14: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/14.jpg)
Increasing reliability
Train the assessors
Increasing the number of stations
Standardise assessment tool
OSCE assessment
![Page 15: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/15.jpg)
Simulation – for individuals and teams
Recommendation: The BCS, SAC and Deaneries should work together to
develop simulator based assessment of practical and professional
skills for cardiology trainees. Initially this will probably be for
formative assessment but could progress to summative assessment if
robust data on reliability, validity and utility can be provided
![Page 16: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/16.jpg)
Relationship Between Peer Assessment and Later PDs’ Reports
• There were 43 students from the class of 2004 for whom we received
internship directors’ ratings (response rate = 44%). Overall ratings were
significantly correlated with both second- and third-year WH scores
(r = .32 [P = .015] and r = .43 [P = .004], respectively). Interpersonal
attributes scores were not correlated with later PDs’ ratings (r = .15 and
−.09, respectively).
360 degree assessment
![Page 17: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/17.jpg)
Skills tracking (longitudinal performance)
– use of portfolios /logbooks
![Page 18: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/18.jpg)
General Logbook Process – EACVI TTE
![Page 19: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/19.jpg)
Grader can mark the cases and can chose to: Send the logbook to back office agent Fail Pass
Process re-started
Logbook review by graders
![Page 20: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/20.jpg)
Team Performance – the wider context
![Page 21: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/21.jpg)
Conclusion
Assessing performance is vital
But accurate reliable, reproducible assessment is extremely challenging
Exams remain effective for knowledge
A variety of reasonably validated techniques exist for (competence) and performance
These methods themselves require quality assessment
![Page 22: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/22.jpg)
Thank you
![Page 23: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/23.jpg)
![Page 24: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/24.jpg)
Standard Setting (2-day) Meeting
• Scored each item difficulty in advance
• Discussed each item
• Rescored each item difficulty at meeting
• This data used to derive the boundaries for the acceptable pass mark
![Page 25: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/25.jpg)
Set the pass mark – Hofstee method
54.1% 66.4%
![Page 26: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/26.jpg)
Graders activity and performance
![Page 27: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/27.jpg)
Appeals against assessments
• Appeals and complaints will happen
• They are difficult stressful and time consuming and potentially expensive
![Page 28: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/28.jpg)
![Page 29: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/29.jpg)
Examples of challenges
I always wear my noise reduction earphones for exams but they wouldn’t let me
How clear are the rules?
I spoke to a woman at the Heart House and she said that it wouldn’t matter if my log book was a few weeks late
Is every contact documented?
![Page 30: How to Evaluate Clinical Performance - uems.eu · How to Evaluate Clinical Performance ... An Effective Exam Process ... The BCS, SAC and Deaneries should work together to](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b4fb10f7f8b9a5a6f8ce8d9/html5/thumbnails/30.jpg)
EEGC