Blueprinting and Choosing Appropriate Tools for Assessment

of Student Performance: A Start to the Charting of the C’s

Claire Touchie, MD, FRCPC

Associate Professor, University of Ottawa

Chief Medical Education Advisor

Medical Council of Canada

Property of the Medical Council of Canada

Conflict of Interest

None

2

ObjectivesBy the end of the workshop, participants will be able to:• Define blueprinting in the context of specific

assessment needs• Design an appropriate blueprint for the purpose of a

specific assessment• Describe available tools for student performance

assessments• Construct an assessment tool box to achieve

assessment goals

3

Agenda• Assessment and blueprinting• Exercise #1: Designing a blueprint• Group Discussion• Leg stretch• Assessment tool boxes• Exercise #2: Assembling a tool box• Group Discussion• Wrap-up and Evaluations

4

5

Microsoft clip-art, 2012

Professional Competence

Developmental

Impermanent

Context dependent

6

Dimensions of Professional Competence

ProfessionalCompetence

Epstein and Hundert, JAMA 2002; 287:226-235 7

The CanMEDS Seven C’s

8

Royal College of Physicians and Surgeons of Canada

Charting the Seven C’s

Seven C’s

Epstein and Hundert, JAMA 2002; 287:226-235 9

Purposes of Assessment• Certify the competence of future practitioners

• Minimal level of competence

• Discriminate among candidates for advanced training

• Rank competitively

• Provide motivation and direction for learning• Provide formative feedback

• Judge the adequacy of a training program• Evaluate teacher or courses

Epstein and Hundert, JAMA 2002; Crossley et al, Medical Education, 2002 10

11

Purposes of Assessment

Learning

AssessmentFeedback

• To discover the worth• To improve the quality

AssessmentDoes

Shows How

Knows How

Knows

Modification of Miller from Crossley et al. Medical Education, 2002 12

Professional Authenticity

Cognition

Behavior

“If you don’t know where you’re going, you probably ain’t going to get

there”

13

Microsoft clip-art, 2012

Blueprinting

Defining the “what” should be represented on your assessment• For licensure or certification

• Practice analysis

• For a course or rotation• Objectives

14

Blueprint“Specifies all the elements of performance relevant to the assessment so that appropriate samples of activity and corresponding methods can be selected according to their relative importance to the overall assessment process.” Newble & Dawson, 1994

“When the items of a test are judged to adequately represent well-defined domains of content…generalizable samples…” Cronbach, 1971

15

16

Domains to be tested

Items tested

Blueprint

Valid assessment: interpretation of the results reflect what was meant to be tested• Appropriate blueprint

• Defines the content of the assessment• Provides validity evidence

17

How do I blueprint for my assessment?

1. Define the purpose of your assessment• Summative or Formative?• To decide on competency or to rank?• To provide feedback to the student or to the

teacher?

2. Tabulate curricular content• Topics vs clinical presentation vs objectives

18

Example of curricular content• Community acquired pneumonia• Hospital acquired pneumonia• Otitis media• Urinary tract infections• Meningitis• Endocarditis• Osteomyelitis/septic arthritis• Septicemia• Cellulitis/skin infections• HIV

19

How do I blueprint for my assessment?3. Provide relative weighting of the content

Importance X frequency• Importance/impact

1. Non-urgent, little prevention potential

2. Serious, but not immediately life threatening

3. Life threatening emergency and/or high potential for prevention impact

• Frequency1. Rarely seen

2. Relatively common

3. Very common

20

How do I blueprint for my assessment?

4. Sample opinion on weighting• Involve course chairs, teachers, evaluation

coordinators, previous learners• Consensus method• Readjust weighting if necessary

21

Example of weightingContent presentation Importance Frequency I X F Weight

CAP 3 3 9 0.22

HAP 3 1 3 0.073

Otitis media 1 3 3 0.073

UTI 2 3 6 0.15

Meningitis 3 1 3 0.073

Endocartitis 3 1 3 0.073

Osteo/Septic arthritis 2 1 2 0.049

Septicemia 3 2 6 0.15

Cellulitis/Skin infn 2 2 4 0.098

HIV 2 1 2 0.049

Total 41

22

How do I blueprint for my assessment?

5. Decide total number of items on your assessment• Based on your weighting, you can decide

how many items per content area

23

How do I blueprint for my assessment?

6. Decide how you would like to break it down?

• Tasks• Diagnosis/Investigation/Treatment/Counseling

• Context of care• ED/Inpatient/Outpatient

• Competencies

24

How do I blueprint for my assessment?Content No. of items Diagnosis Investigation Treatment

CAP 11 4 4 3

HAP 3.65 1-2 0 1-2

Otitis media 3.65 1-2 0 1-2

UTI 7.5 2 3 2

Meningitis 3.65 1 1 1-2

Endocarditis 3.65 1 1-2 1

Osteo/Septic arthritis 2.45 1 1 0

Septicemia 7.5 2-3 1-2 3

Cellulitis/Skin infection 4.9 2-3 0 2

HIV 2.45 1 1-2 0

25

How do I blueprint for my assessment?

Final blueprint often looks like a matrix• Patient group X Task• Clinical presentation X Context of Care• Clinical Activities X Competencies

26

Exercise #1• Using worksheet #1 start working on a

blueprint for your purpose• Get together with colleagues and choose

one blueprint to work on• Large group discussion

27

28

Microsoft clip-art, 2012

Assessment Tool Boxes

What will I use to assess my blueprint?

29

How do I assess what I want to?

Blueprint • Establishes what you wish to test

Assessment tool box• Helps to determine how you will test

30

Assessing the 7 C’s

31

Assessing the Competencies

Professional Competence• More than a demonstration of isolated

competence Eraut, 1994

• “…when we see the whole, we see its parts differently than when we see them in isolation” Polanyi, 1969

32

Assessing the Competencies• Context specific• What may be considered “competent” in

one context may not be in a different context.

33

Assessment tools for CompetenciesRCPSC suggest:• In-training evaluations (ITERs)• MCQs• SAQs• Essays• Oral examinations• OSCEs• Simulations• Log books

34

Microsoft clip-art, 2012

CanMEDS Evaluations and PGE

Med expert • ITER>Oral>MCQ>SAQ

Communicator • ITER>Oral>OSCE>sim

Other roles• ITER>Oral>OSCE

Chou et al. Medical Education, 2008 35

CanMEDS Evaluation and PGEPG directors levels of satisfaction with CanMEDs evaluations• Medical Expert:

• satisfied to very satisfied

• Communicator, collaborator, scholar, and professional

• Neutral to satisfied

• Manager and health advocate• Dissatisfied to neutral

Chou et al. Medical Education, 2008 36

ACGME Competencies• Patient care• Medical knowledge• Practice-based learning and improvement• Inter-personal and communication skills• Professionalism• Systems-based practice

37

Assessment tools for ACGME Competencies

ACGME tool box:Record review Chart stim. Recall

Checklist Global rating

SP OSCE

Simulations 3600 global rating

Portfolios MCQs

Oral exam Log books

Patient survey

38

Assessment Challenges when Charting the 7 C’s

• Socially negotiated educational competencies• Accreditation agencies mandate that competencies

be assessed• Tools don’t match single competencies

independently except for medical knowledge/expert• Other competencies/roles

• Reflect personal attributes?

• What is desirable VS what is observable?

Lurie, Medical Education, 2012 39

Potential solutions?• Linking abstract competencies to

observable behaviors• Key competencies/Enabling competencies

• Mapping competencies to observable behaviors• Milestones, Entrustable Professional Activities

(EPAs)

40

Word of caution

“…a tendency to describe general competencies in exhaustive detail, leading to bulky, fragmented documents that lose practical value for education as they become less and less connected with the real world.” ten Cate et al. Medical Teacher 2010

41

Building your own assessment toolbox• Define the purpose of your assessment• Define the population you are assessing• From your blueprint, what is best

assessed with what tool• Is it going to be reliable? Feasible?

Acceptable?• Will the result interpretation be valid?

42

Exercise #2• Create your own toolbox using

worksheet #2• Discuss your toolbox with a colleague• Group discussion

• Challenges?

43

44

Microsoft clip-art, 2012

In Summary• Define what you wish to assess by

designing a blueprint• Decide how you will assess• Understand the challenges and

limitations to assessing the 7 C’s• Stay tuned to further developments

45

Thank You!

Questions?

Please give me feedback at ctouchie@mcc.ca

46

Useful references1. Epstein RM and Hundert EM. Defining and assessing

professional competence. JAMA 2002;287:226-235

2. Crossley J, Humphris G, Jolly B. Assessing health professionals. Medical Education 2002;36:800-804

3. Lurie SJ. History and practice of competency-based assessment. Medical Education 2012;46:49-57

4. Coderre S, Woloschuk W, McLaughlin K. Twelve tips to blueprinting. Medical Teacher 2009;31:322-324

47