“Show Me How to Get Past MCQs: Emerging Opportunities in Measurement ” Carol O’Byrne, PEBC...

“Show Me How to Get Past MCQs:

Emerging Opportunities in Measurement ”

Carol O’Byrne, PEBC Karen S. Flint and Jaime Walla, AMP

Drs. Frank Hideg, Paul Townsend, & Mark Christensen, NBCE

Alison Cooper, CAPRLila Quero-Munoz, Consultant

Presented at the 2004 CLEAR Annual

ConferenceSeptember 30 – October 2 Kansas City,

Missouri

Presented at the 2004 CLEAR Annual ConferenceSeptember 30 – October 2 Kansas City, Missouri

Goals1. Gain an overview of performance assessment

2. Observe and try out electronic & standardized patient simulations

3. Consider exam development, implementation and administration issues

4. Consider validity questions & research needs

5. Create computer-administered & standardized patient simulations with scoring rubrics

6. Set passing standards


Part 1 - Presentations

Introduction to performance assessment– Purposes and objectives– Models– Issues, successes and challenges

15-minute presentations– Four models, including their unique aspects

with two participatory demonstrations– Developmental and ongoing validity

issues and research studies


Part 2 - Break-out Sessions

1. Identify steps in development and implementation of a new performance assessment and develop a new station

2. Create a new electronic simulation and set passing standards

3. Create a new standardized patient simulation and scoring rubrics

4. Participate in a standard setting exercise using the ‘Competence Standard Setting Method’

and all the while, ask the ‘hard questions’


Performance Assessment - WHY?

To assess important problem solving, critical thinking, communications, hands-on and other complex skills that:

– Impact clients' safety and welfare if not performed adequately and

– Are difficult to assess in a multiple choice question format


HOW?• ‘Pot luck’ direct observation (e.g., medical

rounds, clerkships and internships)• Semi-structured assessments (e.g. orals

and Patient Management Problems) • Objective, Structured Clinical Examinations

(OSCEs) (combining standardized client interactions with other formats)

• Other standardized simulations (e.g., airline pilots' simulators)

• Electronic simulations (e.g., real estate, respiratory care, architecture)


Does it really work?

Links in the Chain of Evidence to Support the Validity of Examination Results:ℴ Job Analysisℴ Test Specificationsℴ Item Writingℴ Examination Constructionℴ Standard Settingℴ Test Administrationℴ Scoringℴ Reporting Test Results


PEBC Qualifying Examination

• Based on national competencies

• Two parts: MCE & OSCE

• Must pass both to be eligible for pharmacist licensure in Canada

• Offered spring and fall in multiple locations

• 1400+ candidates/year

• $1350 CDN

• 15-station OSCE– 12 client interactions

(SP or SHP) + 3 non-client stations

– 7 minute stations

• One expert examiner• Checklist to

document performance

• Holistic ratings to score exam

• Standard Setting• Reports – results and

feedback


Competencies Assessed by PEBC’s MCE and OSCE

COMPETENCIES MCE%

OSCE %

1. Practise pharmaceutical care 56.5 29

2: Assume ethical, legal and professional responsibilities

8.5 9

3: Access, retrieve, evaluate and disseminate relevant information

6.5 5

4:4: Communicate and educate effectivelyCommunicate and educate effectively 1.51.5 4343

5: Manage drug distribution 23 9

6: Apply practice management knowledge and skills

4 5


Comparing PEBC’s OSCE (PS04) and MCE (QS04)

ScoresPaired Samples Statistics

473.7354 711 95.14097 3.56807

481.2897 711 96.62295 3.62364

PS04

QS04

Pair1

Mean N Std. DeviationStd. Error

Mean

Paired Samples Correlations

711 .607 .000PS04 & QS04Pair 1N Correlation Sig.


Comparing PEBC’s OSCE and MCE scores

Paired Samples Statistics

500.0402 560 68.96394 2.91426

502.4589 560 85.94416 3.63180

353.7158 114 109.35265 10.24181

387.7982 114 88.43879 8.28305

445.4000 37 99.66310 16.38451

448.9459 37 95.69075 15.73146

PS04

QS04

Pair1

PS04

QS04

Pair1

PS04

QS04

Pair1

OriginS041

2

USA

Mean N Std. DeviationStd. Error

Mean

Paired Samples Correlations

560 .445 .000

114 .571 .000

37 .688 .000

PS04 & QS04Pair 1

PS04 & QS04Pair 1

PS04 & QS04Pair 1

OriginS041

2

USA

N Correlation Sig.


Holistic Rating ScalesCOMMUNICATION Skills (1)

– Rapport– Organization– Verbal and nonverbal

expression

Problem-solving OUTCOME (2)– Information processing– Decision making– Follow-up

Overall PERFORMANCE (3)– Comm & Outcome – Thoroughness (checklist)– Accuracy (misinformation)– Risk

Analytical Score Means

0

20

40

60

80

1 2 3 4 5 6

CompetenciesP

erc

en

t

Holistic Scale Means

0

20

40

60

80

100

1 2 3

Holistic Scale

Pe

rce

nt


PEBC QUALIFYING EXAMINATION - PART II (OSCE) EXAMINATION FEEDBACK REPORT

Candidate ID Status: Fail

Table 1 RATINGS

Communication Rating # of Stations (of 12)Your average 3.10 4=Acceptable 4Group average 3.67 3=Marginally acceptable 6

2=Marginally unacceptable 11=Unacceptable 1

Outcomes Rating # of Stations (of 15)Your average 2.33 4=Problem solved 2Group average 2.90 3=Solved marginally 5

2=Uncertain/marginally unsolved 41=Problem unsolved 4

Overall Performance Rating # of Stations (of 15)Your average 2.51 4=Acceptable 3Group average 3.01 3=Marginally acceptable 5


Table 2 MISINFORMATION AND RISK/INEFFECTIVE THERAPY

# of Stations/Instances (of 15) Group AverageMisinformation 3 1.31Risk or ineffective therapy 2 0.63

Table 3 COMPETENCY SCORES

Competency Your Score (%) Group Average (%)1 Practise pharmaceutical care 47 592 Assume ethical, legal and professional responsibilities 39 563 Access, retrieve, evaluate, disseminate relevant information 44 474 Communicate and educate effectively 35 535 Manage drug distribution 80 746 Apply practice management knowledge and skills 50 63

PEBC QUALIFYING EXAMINATION - PART II (OSCE) EXAMINATION FEEDBACK REPORT

Candidate ID Status: Fail

Table 1 RATINGS

Communication Rating # of Stations (of 12)Your average 3.10 4=Acceptable 4Group average 3.67 3=Marginally acceptable 6


Outcomes Rating # of Stations (of 15)Your average 2.33 4=Problem solved 2Group average 2.90 3=Solved marginally 5

2=Uncertain/marginally unsolved 41=Problem unsolved 4

Overall Performance Rating # of Stations (of 15)Your average 2.51 4=Acceptable 3Group average 3.01 3=Marginally acceptable 5


Table 2 MISINFORMATION AND RISK/INEFFECTIVE THERAPY

# of Stations/Instances (of 15) Group AverageMisinformation 3 1.31Risk or ineffective therapy 2 0.63

Table 3 COMPETENCY SCORES

Competency Your Score (%) Group Average (%)1 Practise pharmaceutical care 47 592 Assume ethical, legal and professional responsibilities 39 563 Access, retrieve, evaluate, disseminate relevant information 44 474 Communicate and educate effectively 35 535 Manage drug distribution 80 746 Apply practice management knowledge and skills 50 63


Validity – an ascent from Practice Analysis to Test

ResultsJob/practice analysis

– Who/what contexts?– How?

Test specifications & sampling– Which competencies?– Which tasks/scenarios?– Other parameters?

Item writing and review– Who and how?

Scoring– Analytic (checklists) &/or holistic (scales)?


Detect and minimize unwanted variability, e.g.:– Items/tasks – does the mix matter?– Practice effect – how can we avoid it?– Presentation/administration – what is the impact of

different SPs, computers, materials/equipment?– Scores – how do we know how accurate and dependable

they are? What can we do to improve accuracy?

Set Defensible Pass-fail Standards– How should we do this when different standard setting

methods -> different standards?– How do we know if the standard is appropriate?

Report Results– Are they clear? Interpreted correctly?– Are they defensible?

Validity – an ascent from Practice Analysis to Test

Results


Validity – flying highEvidence

– Strong links from job analysis to interpretation of test results

– Relates to performance in training & other tests

Reliable, generalizable & dependable– Scores– Pass-fail standards & outcomes

Feasible– Large & small scale programs– Economic, human, physical, technological

resources

Ongoing Research


Wild LifeCandidate diversity

– Language– Training– Format familiarity,

e.g. computer skills– Accommodations

Logistics– Technological requirements– Replications (fatigue, attention

span)

Security

“Computer-Based

Simulations”Karen S. Flint

Director, Internal Development & Systems Integration

Applied Measurement Professionals, Inc.

Presented at the 2004 CLEAR Annual Conference

September 30 – October 2 Kansas City, Missouri


Evolution of Simulation Exam Format

• AMP’s parent company, NBRC, provided oral exams from 1961 to 1978

• Alternative sought due to:– Limited number of candidates that

could be tested each administration– Cost to candidates who had to travel

to location– Concern about potential oral

examiner bias


Evolution of Simulation Exam Format

• Printed simulation exam format introduced in 1978 using latent image technology

• Latent image format used by NBRC from 1978 to 1999

• NBRC decision to convert all exams to computer-based testing

• Proprietary software developed by AMP to administer simulation exams in comparable format via computer – introduced in 2000

• Both latent image test booklets & computerized format being used


How Simulation Exams Differ from MCQs

• Provides accurate assessment of higher order thinking related to a content area of interest (testing more than just recall)

• Challenge test takers beyond complexity of MCQs

• Simulation problems allow test takers to assess their skills against test content drawn from realistic situations or clinical events


Note: n = 68, r = .48, p <.001

Scaled Simulation Score

5045403530252015

Sca

led

Mu

ltip

le C

ho

ice

Sco

re

150

145

140

135

130

125

120

115

110

105

100

95

90

Passers

Failers

Sample relationship between multiple-choice and simulation scores assessing

similar content


Simulation Utility

• Continuing competency examinations

• Self-assessment/practice examinations

• High-stakes examinations– Psychometric characteristics

comparable to other assessment methodologies

– That is, good reliability and validity


Professions Using This Simulation Format

• Advanced-Level Respiratory Therapists• Advanced-Level Dietitians• Lighting Design Professionals • Orthotist/Prosthetist Professionals • Health System Case Management

Professionals (beginning 2005)• Real Estate Professionals• Candidate fees range from $200 to

$525 for full-length certification/licensure simulation exam


Structure of Simulations

• Opening Scenario• Information Gathering (IG) Sections• Decision Making (DM) Sections

– Single or multiple DM

• All choices are weighted (+3 to –3)• Passing scores relate to judgment

of content experts on ‘minimal competence’


Simulation Development(Graphic depiction of path through a simulation problem)


IG Section Details

• IG section– A section in which test takers choose

information that will best help them understand a presenting problem or situation

– Facilitative options may receive scores of +3, +2, or +1

– Uninformative, wasteful, unnecessarily invasive, or potentially illegal options may receive scores of –1, –2, or –3

– Test takers who select undesirable options accumulate negative section points


IG Section Details

• IG Section Minimum Pass Level (MPL)– Among all options with positive scores

in a section, some should be designated as REQUIRED for minimally competent practice

– The sum of points for all REQUIRED options in a section equals MPL


DM Section Details

• DM section– A section of typically 4-6 options in which

the test taker must make a decision about how to handle the presenting situation

– Facilitative options may receive scores of +3, +2, or +1

– Harmful or potentially illegal options may receive scores of –1, –2, or –3

– Test takers who select undesirable options accumulate negative section points and are directed to select another option


DM Section Details

• DM Section Minimum Pass Level (MPL)– May contain two correct choices, but

one must be designated as REQUIRED for minimally competent practice

– The REQUIRED option point value in the section equals MPL


Minimum Passing Level• DM MPL

– The sum of all DM section MPLs

• IG MPL– The sum of all IG section MPLS

• Overall Simulation Problem MPL– Candidates must achieve MPL in both

Information Gathering and Decision Making


Simulation Exam Development

• 8 to 10 simulation problems per examination

• Each problem assesses different situation typically encountered on the job


Let’s Attempt A Computerized Simulation

Problem!!!


• Karen S. Flint, Director, Internal Development & Systems IntegrationApplied Measurement Professionals, Inc.8310 Nieman RoadLenexa, KS 66214913.541.0400

(Fax – 913.541.0156)[email protected]

“Practical

Testing”

Dr. Frank Hideg, DCDr. Mark Christensen, PhD Dr. Paul Townsend, DC

Presented at the 2004 CLEAR Annual

ConferenceSeptember 30 – October 2 Kansas City,

Missouri


NBCE History

• The National Board of Chiropractic Examiners was founded in 1963

• The first NBCE exams were administered in 1965

• Prior to 1965 chiropractors were required to take chiropractic state boards and medical state basic science boards for licensure


NBCE Battery of Pre-licensure Examinations

• Part I – Basic Sciences Examinations

• Part II – Clinical Sciences Examinations

• Part III – Written Clinical Competency

• Part IV – Practical Examination for Licensure


Hierarchy of Clinical Skills

DO

SHOW HOW

KNOW HOW

KNOWLEDGE PARTS I & II

PART III

PART IV

PRACTICE


NBCE Practical Examination

Content Areas• Diagnostic Imaging• Chiropractic Technique• Chiropractic Case Management


Content Weighing

DIMCAMTEC

CAM 67%

TEC 17%

DIM 16%


Diagnostic Imaging

• 10 Four-minute Stations• Candidate identifies radiological

signs on plain film x-rays• Candidate determines most likely

diagnoses• Candidate makes most

appropriate initial case management decisions


Chiropractic Technique

• 5 five-minute stations• Candidate demonstrates two

adjusting techniques per station• Cervical spine• Thoracic spine• Lumbar spine• Sacroiliac articulations• Extremity articulations


Chiropractic Case Management

• 10 five-minute patient encounter stations• 10 linked post-encounter probe (PEP)

stations • Candidate performs focused case

histories• Candidate performs focused physical

examinations• Candidate evaluates patient clinical

database• Candidate makes differential diagnoses• Candidate makes initial case

management decisions


Key Features of NBCE Practical Examination

• Use of standardized patients• Use of OSCE format and protocols


Case History Stations

• Successful candidates use organized approach while obtaining case history information

• Successful candidates communicate effectively with patients

• Successful candidates respect patient dignity

• Successful candidates elicit adequate historical information


Perform a Focused Case History


Post-Encounter Probe Station


Part IV Candidate Numbers

0

500

1000

1500

2000

2500

3000

3500

4000

4500

1996 1997 1998 1999 2000 2001 2002 2003


Part IV State Acceptance

38

47

7

27

454342

36

46

Column 1 7 27 36 38 42 43 45 46 47

1996 1997 1998 1999 2000 2001 2002 2003 2004


All shaded areas are those states that ACCEPT or REQUIRE successful completion of the NBCE Part IV at the NBCE recommended passing level of 375 as one portion of that state’s licensure evaluation process. Those states are:

Alabama Louisiana Ohio Alaska Maine Oklahoma Arizona Maryland Oregon Arkansas Massachusetts Pennsylvania California Minnesota Rhode Island Colorado Mississippi South Carolina Connecticut Missouri South Dakota Delaware Montana Tennessee Georgia Nebraska Texas Hawaii Nevada Utah Idaho New Hampshire Vermont Indiana New Jersey Virginia Iowa New Mexico Washington Kansas New York Wisconsin Kentucky North Dakota Wyoming

* Those states requiring a passing score other than the NBCE recommended score are not listed.


Candidate Qualifications

• Candidates must pass all basic science and clinical science examinations before applying

• Candidates must be within 6 months of graduation from an accredited chiropractic college

• $1,075 examination fee


Contact Information• National Board of Chiropractic

Examiners 901 54th Avenue

Greeley, CO 80634970-356-9100, [email protected]

Station Development

Alison Cooper Manager of Examination

OperationsCanadian Alliance of

Physiotherapy Regulators




First Principles

• If it’s worth testing, it’s worth testing well– it is possible to test anything badly– this is more expensive

• Some things are not worth testing– trivia– infrequently used skills


Overview

• Write• Review• Dry run• Approve


Write

• Focus of station• SP portrayal - general• Checklist & scoring• Instructions to candidate• Details of SP instructions• Review everything• References


Focus of Station

• Each station must have a clear focus– establish the focus in one sentence– take time to get this right– you can’t write a good station without

a clear focus

• Example: Perform passive range of motion of the arm for a client who has had a stroke.


SP Portrayal - General

• Consider SP movement, behaviour– a picture in your head– use real situations to guide you

• Not detailed yet• Example: Client is 55 years old, is

disoriented, and has no movement in the left arm or leg.


Checklist & Scoring

• What is important to capture• Consider the level of the

candidates• Group items logically• Assign scores to items• Scoring scales


Checklist Example

• Explains purpose of interaction 1• Corrects client’s position 2• Performs passive ROM of scapula 1• Performs passive ROM of shoulder 1• Performs passive ROM of elbow 1• Performs passive ROM of wrist 1• Performs passive ROM of hand & fingers 1• Performs passive ROM of thumb 1• Uses proper body mechanics 3• Uses proper handling 3


Instructions to Candidate

• Information the candidate needs– age and sex of client– pertinent information and

assumptions

• The task for the candidate– exactly what they are to do and not

do


Example

Eric Martin55 years oldThis client had a right middle cerebral artery

haemorrhage resulting in a left sided hemiplegia two (2) weeks ago.

The client presents with confusion and left sided flaccidity. His cardiovascular status is stable.

• Perform passive range of motion on the client’s left upper extremity.

• Perform only one (1) repetition of each movement.

• Assume that you have the client’s consent.


Details of SP Instructions

• History, onset, changes• Initial position, movements,

demeanor, must say/ask– anticipate strong AND weak

candidates

• Cover the checklist and candidate instructions

• SP prompts


SP Instructions...

• Use plain language• Include

– what to wear/not wear– features of the SP (height, scars)

• Diagrams are often helpful


Example

• Presenting complaint• Initial position, general mobility, affect• Comments you must make• Medical, social history• Medications• Activities and areas affected• Sensation• Pain• Muscle quality• Responses to candidate• Emotions


Check Everything

• Go back and check– does it make sense?– is there still a clear focus?– is anything missing?

• Edit/revise as needed– add notes to examiner for

clarification

• Check for plain language


References

• Use references you expect candidates to know

• Umphred, 2nd edition, page 681


Next Steps

• Review by others• Dry run• Approve for use


Thank you

Canadian Alliance of Physiotherapy Regulators1243 Islington Ave., Suite 501Toronto, ON, Canada M8X 1Y9(W)416-234-8800, (F)[email protected]

“OSCE Research: The Key to a Successful

Implementation”

Lila J Quero Muñoz, PhD Consultant




Prior to the OSCE: CPBC and PEBC

• Need for assessing communication, counseling, and interpersonal skills to provide pharmaceutical care to patients

• PEBC MC examination was not assessing the full scope of pharmacy practice as profiled by NAPRA (National Association Pharmacy Regulatory Authorities of Canada)


Generalizability: Data Analyses

• Psychometrically, OSCEs, are complex phenomena, producing scores with potential errors from multiple sources, including: – Examiners (pharmacists and non-

pharmacists)– Cases (context, complexity, # of

stations)– Scoring methods (global vs. checklists)– Standard setting– Differential grading practices


Research Question # 1

• How many examiners are required to obtain consistent and dependable candidates’ scores?


Results #1-1998

• 1 examiner per case yielded similar consistency as 2 (G=.82, .81, D=.81, .79) indicating that examiners agreed highly on their scores

• Examiners contributed little to the scoring errors of candidates’ performance


1 vs. 2 Global -1999Global Grading Roving & Assessor G and D Reliability as a

Function of Stations & RatersMean of 3 Sets of Stations

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

3 8 10 13 15 17 20

# of Stations

G &

D R

elia

bili

ty E

sti

ma

tes

G Rel 1Rater

G Rel 2 Raters

D Rel 1 Rater

D Rel 2 Raters



• How many cases are required to maintain consistency, validity and generalizability of scores? – Adequate and representative sampling of

professional practice are necessary to capture a candidate’s abilities.

– Multiple observations of abilities yield more consistent and content valid inferences.

– Logistical constraints restrict the number of cases that are timely and economically feasible to administer within one OSCE examination.


Results # 2-1998

• 15 cases reduced the candidate’s score error due to sampling variability of the cases dramatically from 5 or 10 cases and improved the consistency of scores from G=.60 to .81

• 15 cases reduced the cases and raters interaction variance as an indication that raters agreed on their scores across cases


Results # 2-1998• Candidates’ scores varied mostly due

to their differential performance across cases.

• Sampling of the cases might affect the candidates’ performance on an OSCE.

• We suggest, however, that differential performance across cases might be due to candidate’s differential levels of skills across the pharmacy competencies assessed


Profile of Sources of Errors in %-1998

0

5

10

15

20

25

30

35

40

45 True score variance

Rater variance

Case variance

Interaction can. &rater

Interaction can. &case

Interaction rater &case



• How do different scoring methods such as checklists or global grading affect candidates’ scores?


Results # 3-1998

• Low correlations between checklist and global scores suggest both methods might not be interchangeable

• If used in isolation they would yield different end results, particularly for borderline candidates

• Global grading yields higher mean scores than checklist grading (p values.81 and .59)


Global vs. Checklist-1999

Compar ing Hol isti c(Global ) & Checkl ist Grading G and D

Rel iabi l i ty (Assessor & Standardized Patient)/ One Rater

0.3

0.5

0.7

0.9

Stations

G &

D

Estim

ates

G1 CheckNest Rel.D1 CheckNest Rel.G1 GlobalNest Rel.D1 GlobalNest Rel.



• What is the validity and defensibility of standard-setting procedures and pass/fail decisions


Results # 4-1998

• SME’s agreed highly on the minimum standard necessary for safe pharmacy practice for the borderline qualified pharmacists

• On different occasions, SME’s had similar standards for entry-to-practice for the core cases

• Standards varied little between 26 & 20 cases and were consistent enough with 15 cases (G=.74, .74, .71)


Results # 4-2003Standard Setters (SS) Variation Within Station and Mean

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

Stations

SS M

ean

Scor

es

ss1

ss2

ss3

ss4

ss5

ss6

ss7

ss8

ss9

ss10

MEAN



• Are there differential grading practices among Canadian Provinces?

• Are candidates’ pass/fail decisions affected by provincial differences on scoring practices?


Results # 5-Videos 2003• Variability in scores between sites are due

mostly to true score variance• Differences between exam sites are in

magnitude of scores but not in pass/fail status

• Differences between assessors are mostly of magnitude of scores but not in pass/fail status

• Pass/Fails decisions did not vary between sites and assessors

• There is more variance between assessors than between exam sites


Results # 5-2003

Stations J03 & J29 Spring 03

1

1.5

2

2.5

3

3.5

4

101 103 401 402 701

OSCE Sites and Assessors

As

se

ss

ors

' Me

an

S

co

res

J03

J29


Results # 5-2003Global Grading Spring Exam Sites 10, 40

Global Grading Spring Exam Sites 10, 40

J03 J29

Facets df var SE

%var

Facets df var SE

%var

P18

0.150

0.077 25% P

14

0.267

0.104 69%

L 10.05

80.06

9 10% L 10.00

00.01

5 0%

R:L 20.02

90.03

5 5% R:L 20.03

80.03

0 10%

PL18

0.000

0.066 0% PL

14

0.026

0.021 7%

PR:L36

0.368

0.085 61% PR:L

28

0.057

0.015 15%

Total 0.60

6 100

% Total 0.38

8 100

%


Results # 5-2003

124 Videos (6 Stations Spring 03 Exam) Scored by 35 Raters from 9 Exam Sites

(Passing Score 2.786)

11.21.41.61.8

22.22.42.62.8

33.23.43.63.8

4

MJ33 MRa02 MJ38 MJ45 MJ03 MJ29

Stations

As

se

ss

ors

' Me

an

Sc

ore

s


Conclusions 1998-2004

• Development of cases should follow templates, guidelines and a detailed blueprint

• Selection of cases must follow a detailed blueprint to mirror OSCE forms between exam administrations to control for differences in cases such as complexity and content



• Multiple sources of errors in OSCEs forces us to do more extensive and non-traditional research than for MC exams

• OSCEs require continuous vigilance to assess the impacts of the many sources of errors

• OSCE research must be planned and implemented beyond exam administrations



• OSCE infrastructure must support both design research and exam administration research

• Successful implementation and continuous improvements of OSCE go hand and hand with research

• More collaborative efforts among OSCE users are needed to built on each other’s success and avoid pitfalls



• Although OSCE research is costly it is a deterrent to litigation and wasted exam administration resources

• Similar conclusions may apply to other performance assessments


• Carol O’Byrne, BSP, OSCE Manager• John Pugsley, PharmD, Registrar, PEBC

[email protected] 416-979-2431, 1-416-260-5013 Fax

• Lila J. Quero-Muñoz, PhD, Consultant 787-431-9288, 1-888-663-6796 Fax

[email protected]

“Show Me How to Get Past MCQs: Emerging Opportunities in Measurement ” Carol O’Byrne, PEBC...

Documents

Transcript of “Show Me How to Get Past MCQs: Emerging Opportunities in Measurement ” Carol O’Byrne, PEBC...