Defining the New MCC Blueprint

1

BLUEPRINT PROJECT

Blueprint Project Team September 2013

Defining the New MCC Blueprint

Consultation with Council

Defining the New Blueprint

• Background and Context• Process• Information sources overview• Candidates, the Blueprint & Test

Specifications• Next Steps

2

ARTF Recommendations

3

1. LMCC becomes ultimate credential (legislation issue)

2. Validate and update the blueprints for all MCC examinations

3. More frequent scheduling of the exams and associated automation (and harmonization of MCCQE II)

4. IMG assessment enhancement and national standardization (NAC & Practice Ready Assessment)

5. Revalidation of physicians (FMRAC will lead this one)

6. Implementation oversight, including the R&D Committee priorities and R&D Budget

Recommendations to focus on MCC’s reassessment and realignment of exams:

Core vs. Discipline-Specific Competencies

Assessment Review Task Force:

• There is general consensus that the current MCC examinations should concentrate on the assessment of those core competencies, including knowledge, skills, attitudes and behaviours, required of every physician entering independent practice

4

5

Item Development Scoring

Fairness

Standard SettingLongitudinal Outcomes

Test Content/ Blueprinting

R&D Validity Agenda

Purpose of the Blueprinting

• … is to assure the public that physicians licensed to practice medicine have the required knowledge, skills and attitudes for safe and effective patient care.

• Only those who meet this standard are qualified to enter professional practice

6

MCC’s Present Blueprints

• MCCQE Part I– Equal distribution of questions based on discipline

• Medicine, Pediatrics, PHELO, Psychiatry, Obstetrics/Gynecology, Surgery

• MCCQE Part II– Distributed by discipline

– Also by skill

• History, Physical Examination, Management, Counseling/Education, Patient Interaction

7

Project Objectives

• ensure that critical core competencies, knowledge, skills and behaviors required of a physician entering supervised and unsupervised practice are being appropriately assessed

The process will

• ensure that MCC assessments continue to fulfill all the requirements and standards for credentialing examinations

• provide a clearly documented and deliberate process to

• update exam specifications • respond to ongoing developments in the profession

10

This afternoon…

• “Competency-based Assessment: The Good, The Bad, and The Puzzling”

Dr. Kevin Eva

• “Defining the New MCC Blueprint”

Dr. Claire Touchie

• “MCC Blueprint: Building Consensus”

Blueprint Project Team

11

Defining the New Blueprint

Claire Touchie, MD, FRCPC

for the Blueprint Project team

Purpose of this session

• Provide information about the process

• Review the blueprint and test specs

• Provide an opportunity for consultation over the next two days

13

Purpose & Format

1. To engage Council members in a consultation about the proposed Blueprint and Test Specifications

2. Formata. This afternoon• Gather initial feedback on the blueprint and test

specifications

b. Tomorrow• Linkages with CanMEDS

• Workshop #1 – explore feedback from afternoon

• Workshop #2 – explore opportunities & next steps

14

Defining the Blueprint

15

Documents used

Current Issues in Health Professional and Health

Professional Trainee Assessment

Supervising PGY-1 Residents

Incidence and Prevalence

National Survey

16

Supervising New PGY-1 Residents: A Case Study of Supervisors expectations vs. Residents’ perceptions

Ten EPAs defined

1. Recognition and initial management of a critically ill patient

2. Disclosure of medical errors

3. Interpretation of investigations (laboratory, ECG, radiographs) with proper communication of results to patients

4. Management of intravenous fluids

5. Handover of patient care to colleagues/other service

18

Ten EPAs defined

6. Discharge prescription writing including medication reconciliation

7. Coordination of patient discharge/transfer (including counseling of patient, organizing follow-up and completing discharge summary)

8. Completion of admission and/or post-operative orders

9. Obtaining informed consent

10. Obtaining advanced directives/goals of care (code status)

19

PGY1 CS PGY1 CS PGY1 CSIVF Informed consent Goals of care

No supervisionIndirect supervisionDirect supervisionNot performed

EPAs that varied betweenSUPERVISORS and RESIDENTS

20

EPAs Resident Responses: DAYTIME vs. NIGHTTIME

Day Night Day Night Day NightCritically ill Critically ill Handover Handover Patient D/C Patient D/C

No supervision

Indirect supervision

Direct supervision

Not performed

21

How does this help in setting the blueprint?

• Helps to define who is the candidate at Decision point 1 – entry to supervised practice

• Use this information to ensure that the knowledge, skills and attitudes associated with the EPAs are assessed prior to entering residency

22

Incidence and Prevalence ofDiseases and other Health Related Issues in Canada

Incidence and Prevalence Data

• Determine what Canadian physicians see in their practice

• E.g.: Frequency of clinical presentations

• Review certain areas of importance for the practice of medicine in Canada

• Other needed specific competencies• Care of Elderly• Population Health• Care of Aboriginal people• Patient safety

24

Inpatient Hospitalization(Excluding Maternal Cases)

Emergency Department Outpatient care/Clinics

1 Appendicitis Abdominal/ Pelvic Pain Anxiety

2 Gallstones Chest/ Throat Pain Supervision of Normal Pregnancy

3Fracture of Lower Leg, Including Ankle

Open Wound, Wrist/ Hand Depressive Disorder

4 Abdominal/ Pelvic Pain Back Pain Backache

5 Schizophrenia Other Medical Care Contraception Counsel/ Advice

6Excessive and Irregular Menstruation

Urinary Tract Infection Abdominal Pain

7 Convalescence Sore Throat Upper Respiratory Infection

8Mental/ Behavioural DisorderDue to Alcohol

Upper Respiratory Infection Urinary Tract Infection

9 Complications of Procedures Diarrhea and Gastroenteritis Essential Hypertension

10 Pancreatitis Sprain/ Strain of Ankle/ Foot Acute Pharyngitis

Main Diagnosis – 19 to 44

25

Main Diagnosis – 65+

26

Hosp. Inpt Emergency Ambulatory (AB) Ambulatory (NS)

COPD Chest/throat pain UTI HTN

Heart failure Other med care Chemotx DM II

ACS UTI Chest pain COPD

Pneumonia Abdo/pelvic pain Surg dressing Backache

Femur # COPD Other med care UTI

Knee arthrosis Pneumonia HTN Anxiety

Other med care Cellulitis Repeat prescription Pneumonia

Atrial flutter/fib. Heart failure Pneumonia Dementia

National Survey of the Physicians, Pharmacists, Nurses and Public in Canada: 2013

• Provide the judged importance of the knowledge, skills and attitudes (KSAs)

• Different stakeholders: Physicians, Pharmacists, Nurses, and the public

28

Purpose of the National Survey

• Based on the MCC Objectives◦ Medical expert: expert

◦ Non-medical expert: communicator, collaborator, health advocate, manager, scholar, and professional

• Two decision points◦ Supervised: for a physician starting residency training who

is assessing a patient at the initial presentation.

◦ Unsupervised: for any newly licensed physician entering unsupervised practice who is assessing a patient at the initial presentation.

29

National Survey Design-Physicians

unsupervised

Overlap between supervised and unsupervised decision points

n = 327

n = 122

30

Importance of Roles

31

Supervised Unsupervised0

10

20

30

40

50

60

70

80

Medical ExpertNon Medical Expert

Pe

rce

nta

ge

of

Ve

ry/E

xtr

em

ely

Imp

or-

tan

t q

ue

sti

on

s

Medical Expert Questions

32

Determine Cause Initiate Management0

20

40

60

80

100

120

140

SupervisedUnsupervised

Nu

mb

er

of Q

ue

stio

ns

Non Medical Expert

Collabora

tor

Comm

unicato

r

Health A

dvocate

Manager

Profe

ssional

Scholar

0

10

20

30

40

50

60

70

80

90

100

SupervisedUnsupervised

Pe

rce

nt

of

tota

l su

rve

y q

ue

sti

on

s

33

Public – open ended question

As a person who used services provided by the Canadian health care system, what are the most important competencies that a physician should have?

34

Public Survey

Concerns around time

Centered on the patient

Knowledge/Credibility

Doctor Characteristics

0 200 400 600 800 1000 1200 1400

Public Survey

35

Conclusions

1. Complete overlap of survey questions “Very/Extremely Important” across supervised and unsupervised decision points

2. Non-medical expert questions were proportionally as important as the medical expert questions (i.e., collaborator, communicator, professionalism) at the first decision point

3. At supervised level

– Determine Cause slightly more important

4. At unsupervised level

– Initiate Management slightly more important

36

MCC Blueprint SME Consultation

Defining the Blueprint

38

Who were the SMEs?

Blueprint

39

SME Panel Meeting – Defining the Proposal

Candidate Descriptions

(D1 & D2)Blueprint

Test Specifications (D1 & D2)

40

Blueprint

Undifferentiated physician at 2 decision points

Decision Point 1Entering supervised practice

Decision Point 2Entering unsupervised practice

41

MCC Common Blueprint

Proposed Common Blueprint

43

Dimensions of Care

Health Promotion and Illness Prevention

Acute Chronic Psychosocial Aspects

Physician

Activities

Assessment/Diagnosis

Management

Communication

Professional Behaviors

Definitions

Dimensions of Care

Focus of care for the patient, family, community and/or population.


The process of enabling people to increase control over their health and its determinants, and thereby improve their health. Illness prevention covers measures not only to prevent the occurrence of illness such as risk factor reduction but also to arrest its progress and reduce its consequences once established. This includes but is not limited to screening, periodic health exam, health maintenance, patient education and advocacy, and community and population health.

Acute

Brief episode of illness, within the time span defined by initial presentation through to transition of care. This dimension includes but is not limited to urgent, emergent, and life-threatening conditions, new conditions, and exacerbation of underlying conditions.

Chronic Illness of long duration that includes but is not limited to illnesses with slow progression.

Psychosocial Aspects

Presentations rooted in the social and psychological determinants of health that include but are not limited to life challenges, income, culture, and the impact of the patient’s social and physical environment.

44

Definitions

Physician Activities

Reflects the scope of practice and behaviors of a physician practicing in Canada

Assessment/ Diagnosis

Exploration of illness and disease through gathering, interpreting and synthesizing relevant information that includes but is not limited to history taking, physical examination and investigation.

Management

Process that includes but is not limited to generating, planning, organizing care in collaboration with patients, families, communities, populations, and health care professionals (e.g., finding common ground, agreeing on problems and goals of care, time and resource management, roles to arrive at mutual decisions for treatment).

Communication

Interactions with patients, families, caregivers, other professionals, communities and populations. Elements include but are not limited to active listening, relationship development, education, verbal, non-verbal and written communication (e.g. patient centered interview, disclosure of error, informed consent).


Attitudes, knowledge, and skills based on clinical and/or medical administrative competence, communication skills, ethics, societal, and legal duties resulting in the wise application of behaviors that demonstrate a commitment to excellence, respect, integrity, empathy, accountability and altruism. (e.g., self- awareness, reflection, life-long learning, scholarly habits and physician health for sustainable practice).

45

Consultations to date

• General comfort with and support of the proposed Blueprint

• Varied responses for Psychosocial Aspects– Name itself may be seen as pejorative

– Having it explicit may socialize it

– Should be incorporated in the other 3 Dimensions of Care

• Initial considerations regarding weightings between decision points

• Patient-safety is not explicit

46

Refreshment Break

47

MCC Blueprint: Building Consensus

Practice Poll

• Übermeetings tool– Web: mcc.ubermeetings.com– Text: 613-519-1313

• Who would you have liked to meet?

49


50

Dimensions of Care



Physician

Activities


Management

Communication


Consultation

• What was your first reaction to the Blueprint?– One word….

51

Consultation

• How well do the dimensions and titles resonate with you?

– Do the Dimensions of Care resonate positively with you?

– Do the Physician Activities resonate positively with you?

– Should Psychosocial Aspects of care be used as a title?

52

Consultation

• When you think of being a physician, what key words are missing in the definitions?

53

Test Specifications


55

Dimensions of Care



Physician

Activities


Management

Communication


Dimensions of Care



Row Percent


Assessment/Diagnosis 30±5

Management 20±5

Communication 30±5

Professional Behaviors 20±5

Column Percent 20±5 30±5 30±5 20±5 100

Assessment leading up to Decision 1: Entry into Supervised Practice

56

ConstraintsDecision 1 – Entry into Supervised Practice

Specification 1 – Constraints

CONSTRAINT CATEGORY DESCRIPTION CONDITION

Complexity multiple morbidities at least 10%

AgeNeonate, infant/child, adolescent, adult, adult women of childbearing age, frail elderly

sample across the age categories including adult woman of childbearing age and the frail elderly

Gender male, female balance evenly (minimum of 40% each)

Special populations

Included but not limited to immigrant, LGBT, rural, disabled and First Nation populations; end of life patients; refugees; inner city poor, the addicted and the homeless

sample across categories

SettingIncluded but not limited to rural or remote settings, long term care institutions and home visits


57

Consultation

For Decision 1 (supervised practice)• Do you agree with the weightings?

58

Dimensions of Care

Health Promotion and

Illness Prevention


Row Percent



Management 20±5

Communication 30±5


Column Percent 20±5 30±5 30±5 20±5 100


59

Dimensions of Care


Illness Prevention


Row Percent



Management 35±5

Communication 20±5


Column Percent 20±5 25±5 35±5 20±5 100

Assessment leading up to Decision 2: Entry into Unsupervised Practice

60

ConstraintsDecision 2 – Entry into Unsupervised Practice

Specification 2 – Constraints

CONSTRAINT CATEGORY DESCRIPTION CONDITION

Complexity multiple morbidities at least 20%

AgeNeonate, infant/child, adolescent, adult, adult women of childbearing age, frail elderly

sample across the age categories including adult woman of childbearing age and the frail elderly

Gender male, female balance evenly (minimum of 40% each)

Special populations

Included but not limited to immigrant, LGBT, rural, disabled and First Nation populations; end of life patients; refugees; inner city poor, the addicted and the homeless


SettingIncluded but not limited to rural or remote settings, long term care institutions and home visits


61

Consultation

For Decision 2 (unsupervised practice):• Do you agree with the weightings?

62

Dimensions of Care



Row Percent

Physician

Activities


Management 35±5

Communication 20±5


Column Percent 20±5 25±5 35±5 20±5 100


63

Comparison between Two Decision points – Dimensions of Care

Health Promotion & Illness Prevention

Acute Chronic Pyschosocial Aspects0

5

10

15

20

25

30

35

40

20

30 30

2020

25

35

20

Test Specification Weightings between Decision 1 and 2

D1-Entry into Supervised Practice D2-Entry into Unsupervised PracticeDimensions of Care

Wei

gh

tin

g %

64

Comparison between Two Decision points – Physician Activities

Assessment/Diagnosis Management Communication Professional Behaviors0

5

10

15

20

25

30

35

40

30

20

30

20

25

35

20 20


D1-Entry into Supervised Practice D2-Entry into Unsupervised PracticePhysician Activities

Wei

gh

tin

g %

65

Consultation

• Do the differences between the two specifications make sense?

66

Consensus Scale 1

Activity

1. Think about how comfortable you are with the blueprint & test specifications?

2. Identify where you are on the scale?

1 2 3 4 5

I hate it!I don’t like it

I can live with & support it I like it I love it!

1. Source: Facilitators Guide to Participatory Decision Making by Sam Kaner

Do we have consensus to move forward?

67

Next Steps

Next Steps for MCCBlueprint & Specifications

Stakeholder consultations

Input & Impact

Approval

Current QEs

Gap Analysis

Changes

New content

Structures

Future View

Research & Analysis

Assessment strategies & tools

Partnership opportunities

Jan. 2014

Future

2016-2017

69

Next Steps

• Planning for implementation underway – pending approved Blueprint and Specifications

• Preliminary impacts identified– Transition QEI and QEII to the blueprint, with a focus on

• Content– Complex cases with multi-morbidity; frail elderly cases;

psychosocial cases

• Assessment Tools Adjustments– Unfolding CDM cases – New OSCE item formats etc.

– Assessment Evolution will be required to meet the blueprint envisioned

70

Tra

nsi

tio

n Q

EI

& Q

EII

to

BP

- C

on

ten

tT

ran

siti

on

QE

I &

QE

II t

o B

P -

Ass

ess

me

nt

To

ols

Tim

eli

ne

July 2013 May 2017

Oct 2013 Jan 2014 Apr 2014 Jul 2014 Oct 2014 Jan 2015 Apr 2015 Jul 2015 Oct 2015 Jan 2016 Apr 2016 Jul 2016 Oct 2016 Jan 2017 Apr 2017

Classification finalized with CEC

Configure Item Bank

(report & structure)

Reclassify, migrate & clean

QEI content

Analyze Pool Pilot

Define pilot requirements /

strategy

Translate content

Implementation

Identify potential item type innovation (tweak & improve) Investigate

Unfolding cases in CDM

Item formats (i.e. new OSCE stations) Design

Establish Test Specs for QEI & QEII(interim targets for current exams)

Test Committees to develop content for known gaps Test Committees to develop content based on pool analysis

TC Committee Structure & Content Development Review

(i.e. committees, process etc.)

Other

Develop

Reclassify, migrate & clean QEII content

Transition QEI and QEII to the blueprint

71

Asse

ssm

en

t E

vo

luti

on

Tim

eli

ne

TBD

July 2013 May 2017

Oct 2013 Jan 2014 Apr 2014 Jul 2014 Oct 2014 Jan 2015 Apr 2015 Jul 2015 Oct 2015 Jan 2016 Apr 2016 Jul 2016 Oct 2016 Jan 2017 Apr 2017

Identify options, needs assessments, scope of

opportunities & prioritizeEstablish/foster relationships

Design, develop, implement assessment strategies

Establish timeframe targets (i.e. years)

Establish supporting business model

Opportunities would span relationships, needs assessment, potential solutions, design/development, business modeling/delivery and implementation / monitoring activities:· E-Portfolio (University Assessment Tool support) could include

· accrediting school OSCEs at graduation (i.e. survey / needs assessment)· Mini-CEX as an assessment for leading up to decision 1· Technical skills/procedure assessment tools (DOPS)· ITER & FITER

· Harmonize with the RCPSC

Timeline to be established

Assessment Evolution through Partnerships

72

Assessment Evolution Opportunities

73

Consultation

• As we continue discussion tomorrow, what could be better?

74

Until Tomorrow

Day 2 – Workshop Consultations


Dimensions of Care



Physician

Activities


Management

Communication


77


Dimensions of Care


Illness Prevention


Row Percent



Management 20±5

Communication 30±5


Column Percent 20±5 30±5 30±5 20±5 100

78


Dimensions of Care


Illness Prevention


Row Percent



Management 35±5

Communication 20±5


Column Percent 20±5 25±5 35±5 20±5 100

79

Workshop Mapping CanMEDs

• Based on consultations-to-date, there has been a view to map the MCC Assessment Blueprint to CanMEDs roles to ensure alignment considering a comprehensive view of physician assessment

• Activity

– Map CanMEDs roles to the Blueprint components

• Dimensions of Care

• Physician Activities

80


Dimensions of Care



Physician

Activities


Management

Communication


81

Mapping CanMEDs Roles

• For the specific dimension select all the CanMEDS roles that can be mapped

82

Mapping CanMEDs





83



Management


Mapping CanMEDs

84



Communication


Mapping CanMEDs

85





Mapping CanMEDs

86

Dimensions of Care



Illness Prevention


Mapping CanMEDs

87

Dimensions of Care


Acute

Brief episode of illness, within the time span defined by initial presentation through to transition of care. This dimension includes but is not limited to urgent, emergent, and life-threatening conditions, new conditions, and exacerbation of underlying conditions.

Mapping CanMEDs

88

Dimensions of Care


ChronicIllness of long duration that includes but is not limited to illnesses with slow progression.

Mapping CanMEDs

89

Dimensions of Care




Mapping CanMEDs

90

Mapping CanMEDs Roles

• Discussion…

91

Practice Poll Prep

• What is the best thing to do in Ottawa?– Brainstorm– Vote

92

Workshop #1Gaps and Issues: Dimensions, Definitions and Specifications

Workshop #1Gaps & Issues

• Explore in more detail feedback from yesterday’s initial consultation to establish key take-aways in consultation with Council members

94


Dimensions of Care



Physician

Activities


Management

Communication


95

Overall Blueprint - Feedback

• What key words are missing?

– Add key words on Sunday pm

96

Overall Blueprint - Feedback

• Insert responses from Consensus scale if not all 3-5’s

97

Overall Blueprint

• What are the critical take-aways we need to consider as part of this consultation from your perspective?

98

Definitions

Dimensions of Care



Illness Prevention


AcuteBrief episode of illness, within the time span defined by initial presentation through to transition of care. This dimension includes but is not limited to urgent, emergent, and life-threatening conditions, new conditions, and exacerbation of underlying conditions.

Chronic Illness of long duration that includes but is not limited to illnesses with slow progression.



99

Blueprint – Dimensions of Care

• What are the critical take-aways we need to consider as part of this consultation?

100

Definitions





Management


Communication




101

Blueprint – Physician Activities


102

Comparison between Two Decision points – Dimensions of Care

Health Promotion & Illness Prevention

Acute Chronic Pyschosocial Aspects0

5

10

15

20

25

30

35

40

20

30 30

2020

25

35

20


D1-Entry into Supervised Practice D2-Entry into Unsupervised Practice

Dimensions of Care

Wei

gh

tin

g %

103

Comparison between Two Decision points – Physician Activities

Assessment/Diagnosis Management Communication Professional Behaviors0

5

10

15

20

25

30

35

40

30

20

30

20

25

35

20 20


D1-Entry into Supervised Practice D2-Entry into Unsupervised Practice


Wei

gh

tin

g %

104

Blueprint – Test Specifications

• Do the differences between the two specifications make sense?

– Why did it resonate? Or why not?

– If you where unsure has anything to date changed your mind?

105

Blueprint – Test Specifications


106

Workshop #2Blueprint Impacts, Challenges and Opportunities

DRAFT - for discussion only

Workshop #2Challenges & Opportunities

• To explore the impacts, challenges and opportunities of the proposed Blueprint and Test Specifications for an undifferentiated physician

108

Blueprint – Challenges

• Based on your understanding of MCC qualifying examinations what will be some of the challenges for MCC to implement this proposed blueprint?

– Is it possible to meet the proposed blueprint with the tools we have for Decision Point 1 and Decision point 2?

– No challenges? How do you propose that we fulfill 50% communication/professional behavior blueprint requirement for decision point 1 with a written exam? Can we cover the entire BP with our present OSCE for decision point 2?

109


110


• Can you see the BP fit along the Assessment Continuum? What are the challenges for using this Blueprint in the Assessment Continuum?

111

Assessment Evolution Opportunities

112

Blueprint – Opportunities

• What other opportunities exist to collaborate in assessment leading to the two decision points?

– Standardize faculty of medicine OSCEs

– Standardize mini-CEX across schools

– Common technical skills assessment tools

– Harmonize with specialty exams of RC

– Standardize ITER/FITER to include as assessment tool

– Other suggestions…

113

Blueprint – Opportunities

• What would you consider to be your personal “top 3” collaboration opportunities for what ever reason?

114

Thank You

Defining the New MCC Blueprint

Health & Medicine

Transcript of Defining the New MCC Blueprint