1

BLUEPRINT PROJECT

September 14, 2014

Beyond the Point-in-time Assessments for High-Stakes Decision Making

MCC Annual General Meeting

2

Plenary Objectives

• Outline the progress of the Blueprint project since AGM 2013 – Claire Touchie

• Review assessment and validity frameworks for high-stakes decision making – Dame Lesley Southgate

• Propose models for combining formative and summative assessments for high-stakes decision making – André De Champlain

The New BlueprintChallenging the Comfort Zone

4

Purpose of the Blueprinting

• … is to assure the public that physicians licensed to practice medicine have the required knowledge, skills and attitudes for safe and effective patient care.

• Only those who meet this standard are qualified to enter professional practice

5

• ensure that critical core competencies, knowledge, skills and behaviors required of a physician entering supervised and unsupervised practice are being appropriately assessed

The process will

• ensure that MCC assessments continue to fulfill all the requirements and standards for credentialing examinations

• provide a clearly documented and deliberate process to

• update exam specifications • respond to ongoing developments in the profession

Project Objectives

Stakeholder Consultations Overview

7

Recommended Common Blueprint

Dimensions of Care

Health Promotion and

Illness Prevention

Acute Chronic Psychosocial Aspects

Physicia

n Activitie

s

Assessment/Diagnosis

Management

Communication

Professional Behaviors

8

External ConsultationsGovernance Board July 5, 2013

UGME Meeting June 20, 2013

 CaRMS

 August 22, 2013

 Annual General Meeting

Sept. 15-17, 2013

AFMC – CACMS September 23, 2013(11h45 – 15 mins)

AFMC – Council of Deans October 30, 2013(in am – 15 mins)

 Collège des médecins du Québec

(CMQ)

 October 31st, 2013

RCPSCAssessment Committee

November 7, 2013 

 UBC

 November 12, 2013

 Student & Residents

(2 meetings) 

November 18, 2013 (Meeting 1 of 2)

November 25, 2013 (Meeting 2 of 2)

 FMRAC

 November 26, 2013

(9:00 a.m. – 2:00 p.m.)

 CFPC

 

November 27th, 2013(a.m.)

RCPSCEducation Committee

November 27, 20131:00 – 2:00 p.m.

 PGME

November 27th, 2013(7 – 9 p.m.)

IMG Symposium January 14, 2014

9

Initial Reactions

What have we heard?

• Overall support for the Blueprint dimensions and definitions as proposed

• Importance of language: – supervised means different things to different folks– Residents do certain activities unsupervised

• Need to make explicit certain terms:– Teamwork, patient safety, inter- and intra-professionalism, leadership– Concerns about the term for and placement of the dimension

“Psychosocial Aspects”

• Relevance of the D2 (and part II) when certification is the final step to licensure

• All CanMEDS roles can be mapped to each dimension• Focused on challenges and opportunities

10

MCC Common BlueprintIncorporating Consultation Feedback

12

Recommended Common Blueprint

Dimensions of Care

Health Promotion and

Illness Prevention

Acute Chronic Psychosocial Aspects

Physicia

n Activitie

s

Assessment/Diagnosis

Management

Communication

Professional Behaviors

Definitions

Dimensions of Care Focus of care for the patient, family, community and/or population.

Health Promotion and Illness Prevention

The process of enabling people to increase control over their health and its determinants, and thereby improve their health. Illness prevention covers measures not only to prevent the occurrence of illness such as risk factor reduction but also to arrest its progress and reduce its consequences once established. This includes but is not limited to screening, periodic health exam, health maintenance, patient education and advocacy, and community and population health.

AcuteBrief episode of illness, within the time span defined by initial presentation through to transition of care. This dimension includes but is not limited to urgent, emergent, and life-threatening conditions, new conditions, and exacerbation of underlying conditions.

Chronic Illness of long duration that includes but is not limited to illnesses with slow progression.

Psychosocial AspectsPresentations influenced by the social and psychological determinants of health and how these can impact on wellbeing or illness. The determinants include but are not limited to life challenges, income, culture, and the impact of the patient’s social and physical environment.

DefinitionsPhysician Activities Reflects the scope of practice and behaviors of a physician practicing in

Canada

Assessment/ Diagnosis

Exploration of illness and disease through gathering, interpreting and synthesizing relevant information that includes but is not limited to history taking, physical examination and investigation.

ManagementProcess that includes but is not limited to generating, planning, and organizing safe and effective care in collaboration with patients, families, communities, populations, and other professionals (e.g., finding common ground, agreeing on problems and goals of care, time and resource management, roles to arrive at mutual decisions for treatment, working in teams).

Communication

Interactions with patients, families, caregivers, other professionals, communities and populations. Elements include but are not limited to relationship development, intraprofessional and interprofessional collaborative care, education, verbal communication (e.g.: using the patient-centered interview and active listening), non-verbal and written communication, obtaining informed consent, and disclosure of patient safety incidents.

Professional Behaviors

Attitudes, knowledge, and skills based on clinical and/or medical administrative competence, communication ,ethics, as well as societal and legal duties . The wise application of these behaviors demonstrates a commitment to excellence, respect, integrity, empathy, accountability and altruism within the Canadian health-care system. Professional behaviors also include but are not limited to self- awareness, reflection, life-long learning, leadership, scholarly habits and physician health for sustainable practice.

15

Dimensions of Care

Health Promotion and Illness Prevention

Acute Chronic Psychosocial Aspects

Row Percent

Physician

Activities

Assessment/Diagnosis 30±5

Management 20±5

Communication 30±5

Professional Behaviors 20±5

Column Percent 20±5 30±5 30±5 20±5 100

Assessment leading up to Decision 1: Entry into Residency

16

Dimensions of Care

Health Promotion and Illness Prevention

Acute Chronic Psychosocial Aspects

Row Percent

Physician

Activities

Assessment/Diagnosis 25±5

Management 35±5

Communication 20±5

Professional Behaviors 20±5

Column Percent 20±5 25±5 35±5 20±5 100

Assessment leading up toDecision 2: Entry into Independent Practice

TransitionsMCCQE Part I and Part II

18

Classification of MCC Content

3 days and 26 physicians

• 3514 Multiple-choice items

• 1321 Clinical Decision Making items

• 103 OSCE cases

19

Gap Analysis Results

MCCQE Part IDIMENSIONS OF CARE    

Health promotion and Illness Prevention

Acute Chronic Psychosocial Aspects

TOTAL

PHYSICIAN ACTIVTIES

Assessment/ Diagnosis

151 1654 715 33 2555

3.1% 34.2% 14.8% 0.7% 52.9%

Management 235 1068 467 27 1797

4.9% 22.1% 9.7% 0.6% 37.2%

Communication 34 53 22 28 137

0.7% 1.1% 0.5% 0.6% 2.8%

Professional Behaviours

60 82 130 57 329

1.2% 1.7% 2.7% 1.2% 6.8%

  TOTAL480 2857 1335 145 4834

9.9% 59.1% 27.6% 3.0% 100%

Gap Analysis Results

MCCQE Part II

20

DIMENSIONS OF CARE    

Health promotion and Illness Prevention

Acute Chronic Psychosocial Aspects

TOTAL

PHYSICIAN ACTIVTIES

Assessment/ Diagnosis

0.2% 37.2% 8.4% 2.5% 48.3%

Management 2.8% 7% 2.3% 0.6% 12.7%

Communication 6.5% 23% 2.9% 2.0% 34.4%

Professional Behaviours

0.6% 3.2% 0.3% 0.5% 4.5%

  TOTAL10.1% 70.4% 13.9% 5.6%

21

Transition of Exams

• MCC Part I– MCQs and CDM items– Spring 2017

• MCC Part II– New station types being piloted– Focus on communication and professional

behaviors within a clinical context– Still will expect components of history-taking and

physical examinations– Fall 2018

Our Work Going ForwardAssessment Evolution – Challenging the Comfort Zone

23

To meet overall Blueprint

• Will need additional assessments– Opportunities– Innovations– Collaborations

Consultation Input - Opportunities

Harmonize with surgical foundations exam

A UG ITER

Influence accreditation standards wrt assessment expectations

Self assessment

Standardize mini-CEX across schools

Encourage discussions with both colleges and related initiatives

Common technical skills assessment tools

Standardize ITER/FITER to include as assessment tool

E-portfolio rollout

Standardize faculty of medicine OSCEs

Harmonize with specialty examinations at the Royal College

0 5 10 15 20 25 30 35 40 45 50

0

1

1

2

5

10

15

19

23

28

45

24

25

Assessment Evolution - Opportunities

26

UGME Retreat

4 priority areas to be explored:• Assessment Design & Creation – Content• Assessment Design & Creation – Design• Item Banks• E-portfolios

27

Moving forward

• Meeting with “opportunity leads” to move forward with the UGME recommended opportunities

• Environmental scans– UGME assessments– Assessment Evolution meeting/FMEC-PG

recommendation #6 meeting

Tomorrow’s workshops

29

Workshop 1

An Assessment E-Portfolio across the Continuum: A Discussion of the “What”

 • Discuss the types of assessments to be

included in an e-portfolio for the two MCC decision points

• Propose mechanisms for this to be feasible and acceptable for schools, candidates and regulators

30

Workshop 2

Feeding Forward: From A Student-Centered to Patient-Centered Approach

 • Define the meaning and understanding of

“feeding forward” • Discuss opportunities and barriers to sharing

of assessment information within and between stakeholder organizations for the purpose of quality improvement and learning

31

Blueprint Project Team

The Blueprint Project Team

Dr. Claire Touchie Chief Medical Education Advisor – Project co-lead

Cindy Streefkerk Consultant – Project co-lead

Tanya Rivard Senior Test Development Officer and Special Projects

Anna Di Medio Project Administrator

Dr. Sydney Smee Manager, Strategic Initiatives and Clinical Skills Assessment

Jessica Hertzog-Grenier Director of Communications

Ingrid de Vries Associate Director – Evaluation Bureau

Becca Carroll Manager - Computer-Based Testing

Dr. Andrea Gotzmann Research Psychometrician

Dr. Marguerite Roy Medical Education Researcher

Sasha Papayanis Business Analyst – Information Technology

Alex Clay Project Coordination Manager

Thank You!