Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical...
Transcript of Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical...
![Page 1: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/1.jpg)
Accreditation Council for Graduate Medical Education
© 2014 Accreditation Council for Graduate Medical Education
Assessment in Residency Training:
Global Insights
Eric Holmboe
![Page 2: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/2.jpg)
Disclosures
• Eric Holmboe works for the ACGME and
receives royalties from Mosby-Elsevier for a
textbook.
![Page 3: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/3.jpg)
© 2014 Accreditation Council for Graduate Medical Education
Outline
Major trends - summary
CBME and system outcomes
Overview of competencies, milestones and
EPAs
Importance of informed judgment
Group process
![Page 4: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/4.jpg)
© 2014 Accreditation Council for Graduate Medical Education
Major Trends in Assessment
Developmental and competency-based
Greater focus on connecting educational and
clinical outcomes
Moving away on over-reliance of
psychometrically based assessment (e.g. tests)
Concept of entrustment for supervision
Operationalizing CBME through Milestones and
entrustable professional activities (EPAs)
Programs of assessment
Group process in judgments of competence and
professional development
![Page 5: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/5.jpg)
Accreditation Council for Graduate Medical Education
© 2014 Accreditation Council for Graduate Medical Education
Why These Changes in
Assessment?
![Page 6: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/6.jpg)
![Page 7: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/7.jpg)
© 2015 Accreditation Council for Graduate Medical Education
CBME: Start with System Needs
7Frenk J, et al. Health professionals for a new century: transforming
education to strengthen health systems in an interdependent world.
Lancet. 2010
![Page 8: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/8.jpg)
© 2015 Accreditation Council for Graduate Medical Education
What Are The Outcomes?
• A competent (at a minimum) practitioner
aligned with:
CMS Triple Aim
![Page 9: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/9.jpg)
© 2015 Accreditation Council for Graduate Medical Education
How is Sweden doing as a nation on the
triple aim?
What is the responsibility and role of the
medical education system in achieving
the triple aim?
Pair and Share
![Page 10: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/10.jpg)
© 2014 Accreditation Council for Graduate Medical Education
Diagnostic Errors
IOM Report
Released September 2015
• At least 5 percent of U.S.
adults who seek outpatient
care each year experience a
diagnostic error.
• Postmortem examination
research shows diagnostic
errors consistently contribute
to ~ 10 percent of patient
deaths.
• Diagnostic errors account for
6 to 17 percent of hospital
adverse events.
![Page 11: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/11.jpg)
© 2015 Accreditation Council for Graduate Medical Education
CBME and Assessment
11Frenk J, et al. Health professionals for a new century: transforming
education to strengthen health systems in an interdependent world.
Lancet. 2010
Assessment and
Curriculum must be
integrated:
Assessment drives
learning; learning
drives assessment
![Page 12: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/12.jpg)
Early Principles: CBME
• World Health Organization (1978):
• “The intended output of a competency-
based programme is a health
professional who can practise medicine
at a defined level of proficiency, in accord
with local conditions, to meet local
needs.”
McGaghie WC, Miller GE, Sajid AW, Telder TV. Competency-based
Curriculum Development in Medical Education. World Health
Organization, Switzerland, 1978.
![Page 13: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/13.jpg)
© 2015 Accreditation Council for Graduate Medical Education
CBME Today
An outcomes-based approach to the
design, implementation, assessment
and evaluation of a medical education
program using an organizing framework
of competencies1
1Frank, JR, Snell LS, ten Cate O, et. al. Competency-based medical
education: theory to practice. Med Teach. 2010; 32: 638–645
![Page 14: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/14.jpg)
Accreditation Council for Graduate Medical Education
© 2014 Accreditation Council for Graduate Medical Education
Competencies
![Page 15: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/15.jpg)
Competency: Definition
Competency: An observable ability of a
health professional, integrating multiple
components such as knowledge, skills,
values and attitudes.
The International CBME Collaborators, 2009
![Page 16: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/16.jpg)
Competency-frameworks*
CanMeds
Medical expert
Communicator
Collaborator
Manager
Health advocate
Scholar
Professional
ACGME Medical knowledge
Patient care
Practice-based learning& improvement
Interpersonal and communication skills
Professionalism
Systems-based practice
GMC Good clinical care
Relationships with patients and families
Working with colleagues
Managing the workplace
Social responsibility and accountability
Professionalism
*From CPM Van Der Vleuten
![Page 17: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/17.jpg)
© 2015 Accreditation Council for Graduate Medical Education
Linking Clinical and Educational Outcomes
National Health Service – UK.
http://www.wipp.nhs.uk/tools_gpn/unit6_education.php
Competencies
Triple Aim
![Page 18: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/18.jpg)
Accreditation Council for Graduate Medical Education
© 2014 Accreditation Council for Graduate Medical Education
Milestones
![Page 19: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/19.jpg)
© 2015 Accreditation Council for Graduate Medical Education
PC1. History (Appropriate for age and impairment)
Level 1 Level 2 Level 3 Level 4 Level 5Acquires a
general medical
history
Acquires a basic
clinical history
including
medical,
functional, and
psychosocial
elements
Acquires a
comprehensive
clinical history
integrating medical,
functional, and
psychosocial
elements
Seeks and obtains
data from secondary
sources when needed
Efficiently acquires
and presents a
relevant history in a
prioritized and
hypothesis driven
fashion across a
wide spectrum of
ages and
impairments
Elicits subtleties and
information that may
not be readily
volunteered by the
patient
Gathers and
synthesizes
information in a
highly efficient
manner
Rapidly focuses on
presenting problem,
and elicits key
information in a
prioritized fashion
Models the
gathering of subtle
and difficult
information from the
patient
Competency
Developmental
Progression or Set of
Milestones Sub-competency
Specific
Milestone
![Page 20: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/20.jpg)
Purposes and Implications
ACGME
• Accreditation – continuous monitoring of programs; lengthening of site visit cycles
• Public Accountability – report at a national level on competency outcomes
• Community of practice for evaluation and research, with focus on continuous improvement
Training Programs
• Framework for CCC• Guide curriculum development• More explicit expectations of trainees• Support better assessment• Enhanced opportunities for early
identification of under-performers
Certification Boards
• Research for CBME
Residents and Fellows
• Increased transparency of performance requirements
• Encourage informed self-assessment and self-directed learning
• Better feedback
Milestones
20
Milestones are a Formative Assessment Framework
![Page 21: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/21.jpg)
© 2015 Accreditation Council for Graduate Medical Education
Milestones as Roadmap
Observations:
1) Journey not a
straight line
2) More than one
path (but not
infinite paths)
3) “If you don’t know
where you are
going, any road
will get you there”
![Page 22: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/22.jpg)
© 2015 Accreditation Council for Graduate Medical Education
Dreyfus & Dreyfus Development Model
Dreyfus SE and Dreyfus HL. 1980
Carraccio CL et al. Acad Med 2008;83:761-7
Time, Practice, Experience
Novice
Advanced Beginner
Competent
Proficient
Expert/
Master
MILESTONESCurriculum
Assessment
Curriculum
Assessment
Curriculum
Assessment
Curriculum
Assessment
Curriculum
Assessment
Development is a
non-linear phenomenon
![Page 23: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/23.jpg)
In general, Milestones are …
Describe the educational outcomes of the
individual
Developmental in nature
Grounded in Dreyfus model of expertise
development
Behaviorally-based
Describe a learner using narratives
Independent of level of training
Overarching trajectory of training
![Page 24: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/24.jpg)
Accreditation Council for Graduate Medical Education
© 2014 Accreditation Council for Graduate Medical Education
Entrustable Professional Activities
![Page 25: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/25.jpg)
© 2015 Accreditation Council for Graduate Medical Education
Video Exercise
What was this intern entrusted to do
without direct supervision?
![Page 26: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/26.jpg)
Entrustable Professional Activities
EPAs represent the routine professional-life
activities of physicians based on their specialty
and subspecialty
The concept of “entrustable” means:
‘‘a practitioner has demonstrated the necessary
knowledge, skills and attitudes to be trusted to
perform this activity [unsupervised].’’1
1Ten Cate O, Scheele F. Competency-based postgraduate
training: can we bridge the gap between theory and
clinical practice? Acad Med. 2007; 82(6):542–547.
![Page 27: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/27.jpg)
An Entrustable Professional Activity
ten Cate et al.
Acad Med 2007; 82: 542-47
Part of essential work for a qualified professional
Requires specific knowledge, skill, attitude
Acquired through training
Leads to recognized output
Observable and measureable, leading to a
conclusion
Reflects the competencies expected
EPA’s together constitute the core of the
profession
![Page 28: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/28.jpg)
“Patients’ and instructors’ … entrustment
of responsibility to a trainee is an essential
concept in this approach…”
Question: What do you currently
entrust your learners to do with only
indirect, reactive supervision?
Entrustable Professional Activities
ten Cate et al.
Acad Med 2007; 82: 542-47
![Page 29: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/29.jpg)
Attending physicians assess a multi-dimensional construct of “trustworthiness” when deciding a level of supervision
Entrustment implies a level of competence
Entrustment in Residency Training
Kennedy, et. al.
Acad Med 2008; 83(10 Suppl): S89-92
![Page 30: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/30.jpg)
Accreditation Council for Graduate Medical Education
© 2014 Accreditation Council for Graduate Medical Education
How Does All of This Fit
Together?!?
![Page 31: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/31.jpg)
EPAs, Competencies, and Milestones
ten Cate, O. (February 2015) Entrustable Professional Activities as a Framework for
Assessment, Presentation given at the 2015 ACGME Annual Educational Conference.
Date and graphic from ten Cate, et al. 2015 (under review).
![Page 32: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/32.jpg)
Competency Milestones EPA in Training EPA in
Practice
Medical
Knowledge
MK1
MK2 “Lead” a
care team
Lead & work
within IP health
care teams.Patient Care PC1
PC2
Professionalism Prof1
Care for patients
with chronic
illness with
indirect
supervision
Manage care of
patients with
chronic
diseases
Prof2
Interpersonal
Skills
ISC1
ISC2
Systems-based
Practice
SBP1
SBP2 Participate in QI
and pt. safety
initiatives
Enhance patient
safety.
Improve quality
of health care
Practice-based
learning
PBLI1
PBLI2
Analyze to
Understand
Synthesize to
Educate and
Evaluate
Shared Mental Models and Frameworks
Physicians competent to
meet the health care
needs of the population
![Page 33: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/33.jpg)
© 2015 Accreditation Council for Graduate Medical Education
U.S. Model of Integration
COMPETENCY
Entrustable Professional Activity1
MILESTONES
COMPETENCY
MILESTONES
COMPETENCY
MILESTONES MILESTONES
COMPETENCY
COMPETENCY COMPETENCY COMPETENCY COMPETENCY
MILESTONESMILESTONESMILESTONESMILESTONES
Entrustable Professional Activity2
“White space” “White space”
![Page 34: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/34.jpg)
EPA’s as an assessment strategy
Faculty “get it” (or do they???)
EPA provides meaningful context
Synthetic work-based assessment
Vehicle for faculty to provide honest and specific feedback for growth
Reflect desired outcomes of profession, public and policy makers
![Page 35: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/35.jpg)
Accreditation Council for Graduate Medical Education
© 2014 Accreditation Council for Graduate Medical Education
Programmatic Assessment
![Page 36: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/36.jpg)
© 2015 Accreditation Council for Graduate Medical Education
The Professional Self-regulatory “System”
Assessments within
Program:
• Direct observations
• Audit and
performance data
• Multi-source FB
• Simulation
• ITExam
Qual/Quant
“Data”
Synthesis:
Committee
Residents
Faculty, PDs
and others
Milestones and EPAs
as Guiding Framework and Blueprint
Accreditation
Unit of Analysis:
Program
Certification and
Credentialing
Unit of Analysis:
Individual
J
U
D
G
M
E
N
T
D
FB
FB
DD FB
P
U
B
L
I
C
![Page 37: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/37.jpg)
© 2015 Accreditation Council for Graduate Medical Education
Structured Portfolio
Medical record audit andQI project
MSF: Directed per protocol
Twice/year
Practice-based learning and improvement
Systems-based prac
Mini-CEX:10/year
Interpersonal skills and Communication
ITE:1/year
Patient care
Faculty Evaluations
EBM/Question Log
Medical knowledge
Professionalism
“Sources” of Assessment
■ Learner-directed ■ Direct observation
![Page 38: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/38.jpg)
Time
AssessmentActivities
TrainingActivities
SupportingActivities
v v v v v v
Inte
rme
dia
te E
va
l
Inte
rme
dia
te E
va
l
Fin
al E
va
lua
tio
n
= learning task
= learning artifact
= single assessment data-point
= single certification data point for mastery tasks
= learner reflection and planning
= social interaction around reflection (supervision)
= learning task being an assessment task also
Model For Programmatic Assessment(With permission from CPM van der Vleuten)
Committee
![Page 39: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/39.jpg)
© 2015 Accreditation Council for Graduate Medical Education
Group Decision Making
Key Issues
What is the environment in which the committee
performs its work?
What is the local culture?
Groups within groups
What is the medical culture of your institution?
What are the effects of hierarchy on group
decision making?
Berg: Medicine one of the most hierarchical of all
professions
Single variable of effectiveness: extent to which
people are willing to say “positive” and “negative”
comments and observations in a group
![Page 40: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/40.jpg)
© 2015 Accreditation Council for Graduate Medical Education
Group Process in U.S.
Institutional Culture
Info Sources:
• Faculty Evals
• Direct Obs
• Multisource FB
• Patient surveys
• ITExams
• +/- Simulation
• Critical events
• Informal (e.g.
“hallway talks”)
Pre
-me
etin
g D
ata
Pre
pa
ratio
n
Group ProcessKnown Variables:
• Group composition
• Info presentation
• Evidence vs.
verdict
• Hierarchy
• Info context
• Time pressures
• Additional info
Judgment
Program
Culture
Feedback
“Filter”
Institutional Culture
Learner
![Page 41: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/41.jpg)
“Wisdom of the Crowd”
• Hemmer (2001) – Group conversations more
likely to uncover deficiencies in professionalism
among students
• Schwind, Acad. Med. (2004) –
• 18% of resident deficiencies requiring active remediation became apparent only via group discussion.
• Average discussion 5 minutes/resident (range 1 – 30 minutes)
![Page 42: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/42.jpg)
Narratives and Judgments
Pangaro (1999) – matching students to a
“synthetic” descriptive framework (RIME) reliable
and valid across multiple clerkships
Regehr (2007) – matching students to a
standardized set of holistic, realistic vignettes
improved discrimination of student performance
Regehr (2012) – faculty created narrative
“profiles” (16 in all) found to produce consistent
ranking of excellent, competent and problematic
performance
![Page 43: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/43.jpg)
Rules for future assessments of trainee:
Be crystal clear on the purpose of a curricular
activity/experience
Measure the expected outcome(s) for a defined
activity or rotation (based on purpose)
Provide meaning to faculty
Provide meaning to trainee
Provide meaning to CCC (in time)
Reflect needs of our health delivery system
![Page 44: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/44.jpg)
Will necessitate educational redesign
Direct observation of trainees
Purposeful assessment at times of entrustment
Capture what you already do!
Coordination and cooperation between training programs and healthcare professions
Detailed mapping between assessments and milestones in electronic evaluation system
![Page 45: Assessment in Residency Training: Global Insights · Accreditation Council for Graduate Medical Education ... Assessment in Residency Training: Global Insights ... education to strengthen](https://reader030.fdocuments.us/reader030/viewer/2022020303/5b29e17e7f8b9ad6458b568b/html5/thumbnails/45.jpg)
Faculty Development
It is very unlikely that you can create a
successful assessment system without faculty
training
Evaluation tools are only as good as the person
using them
At the present time – still need faculty judgment
Need shared mental models and an
understanding of the expected outcomes