®

Development of Technology Standards for the Performance Framework: Progress Report by the Competencies Working Group

Rosalyn P. Scott, M.D., M.S.H.A.Wright State University Department of Veterans Affairs

Tim Willett, M.D.Sim-ONE

Valerie SmothersMedBiquitous

®

Competencies Working Group• Co-Chairs• Rosalyn Scott, Department of Veteran's Affairs• Tim Willett, Sim-ONE• Members• Susan Albright, Tufts University• Mary Pat Aust, American Association of Critical-Care Nurses• Dana Bostrom, Association of American Medical Colleges• Terri Cameron, Association of American Medical Colleges• Mary Jo Clark, American Association of Colleges of Nursing• Stephen Clyman, National Board of Medical Examiners• Matthew Cownie, University of the West of England• Robert Englander, Association of American Medical Colleges• Ted Hanss, University of Michigan• Sean Hilton, St. George's University of London• Joshua Jacobs, National University of Singapore• PJ Kania, Association of American Medical Colleges• David Kiger, Tufts University• JB McGee, University of Pittsburgh• David Price, American Board of Family Medicine• Isarin Sathitruangsak, Tufts University• Lesley Southgate, St. George's University of London• Nabil Zary, Karolinska Institute

• Invited Experts• Connie Bowe, Partners HealthCare International• Kelly Caverzagie, University of Nebraska Medical

Center• Allan Cumming, University of Edinburgh• Rachel Ellaway, Northern Ontario School of

Medicine• Simon Grant, JISC-CETIS• Linda Gwinn, Georgetown University• Robyn Herring, University of Mississippi• Kevin Krane, Tulane University• Steven Lieberman, University of Texas Medical

Branch• Linda Lewin, University of Maryland• Melanie Loving, University of Texas Medical

Branch• Karen Macauley, University of San Diego Hahn

School of Nursing and Health Science • Chandler Mayfield, University of California, San

Francisco• Daniel Rehak• Juliane Schneider, Harvard University

®

ANSI/MEDBIQ CF.10.1-2012, Competency Framework• Learner and educators can search for

learning resources addressing a particular competency.

• Educators can determine where specific competencies are addressed in a curriculum.

• Boards and hospitals can track and manage competency data for the professional.

• Administrators can map one competency framework to another.

®

Competency Objects and Frameworks

The use of outcome and competency frameworks is a growing part of health professions education and regulation in many countries.

To implement competency-based education and assessment, competency frameworks must be connected to curricula, learning resources, and assessment data.

®

Unfinished Business….

• Assessing Competence• Performance Levels• How do detailed competencies fit together to

do the work we do on a daily basis?• What does competence look like in the real

world?• We all know it when we see it!!

®

Performance Framework• Requirements and XML data specifications

for the exchange of performance levels and performance frameworks

• Initial focus on undergraduate and graduate medical education

• North American and International paradigms for medical education should inform the structure of the specification

• Other healthcare professionals critical to development

®

CompetencyFramework.xml

Professionalism.xml

A competency object . . .

Relates to other competency objects . . .

xml xml

xml xml xml xml

xml xml xml xml xml xml

Which can relate to resources, other data,

assessment . . .

Learning Object

Educational Activity

Educational Achievement

Data

Current Specifications

Performance Framework

®

Next Accreditation System:ACGME Competencies and Outcomes• Evaluate programs based on reporting of

outcomes through educational milestones• Patient care• Medical Knowledge• Professionalism• Systems-based practice• Practice-based learning and improvement• Interpersonal and communication skills

• Foster innovation and improvement by programs

®

What ACGME Plans……

• Continuous oversight not episodic sampling • Milestone reports (semi-annual), Resident &

Faculty questionnaires (annually), Case Logs (annually), Program & Institutional Infrastructure (annually)

• Program Site Visits at ~10 year intervals • Emphasis on program effectiveness as

evidenced by resident advancement through the milestones

®

Assessing Competence• Milestones: Granular

• Developmental framework from less to more advanced

• Drill down to smaller elements of behavior that provide the substance of formative feedback, remediation, and a learning roadmap

• Entrustable Professional Activities: Holistic • Integrate competencies within a clinical context• Assess clusters of behaviors required to care for

patients

®

ACGME Milestones: A Developmental Roadmap• Milestones are knowledge, skills, attitudes,

and other attributes for each of the ACGME competencies organized in a developmental framework. They are descriptors and targets for resident performance as a resident moves from entry into residency through graduation.

®

Orthopedic Surgery Milestone

®

Framing ACGME Competencies in Entrustable Professional Activities• Competencies are no longer abstractions• Reflect complex integration of competencies

needed in actual day-to-day practice • Address authentic practice • Entrustable designates that “a practitioner

has demonstrated the necessary knowledge, skills and attitudes to be trusted to independently perform this activity.”

®

Level of Mastery of an EPA

Five stages of decreasing supervision:

(1) no task execution

(2) task execution under direct supervision on site

(3) task execution with supervision quickly available on call

(4) unsupervised practice

(5) supervision may be provided to juniors

®

How it all works together…….

®

How are use cases addressed?

• Map performance frameworks to competencies• YES (CompetencyReference)

• Publish a performance framework for use in curriculum planning and assessment• YES (Requires IDs for Components and groups of

PerformanceLevels so that assessments may be tied to performance levels)

• Describe an individual’s current performance for purposes of formative or summative assessment• YES (a score and a reference to a group of

PerformanceLevels should do the trick)

®

How are use cases addressed?

• View data regarding the performance levels in a program for both external and internal evaluation • YES (would require a spec that allows for the

exchange of aggregate data)

• Publish program data for public viewing• YES (would require a spec that allows for the

exchange of aggregate data

• Define level of performance required to progress to the next phase or block within a curriculum• YES (will require updates to curriculum inventory)

®

How do you know a trainee is qualified and what supervision is required?

Trainee

Professional Profile

Is competent in

Competency Framework

At what level

Performance Framework

Is entrusted to doCompetency Framework