Writing Multiple Choice Questions

Amy S. Oxentenko, MD, FACP, FACG, AGAFMayo Clinic, Rochester

• Relevant Relationships:

Consulting work for MKSAP 14-17

On the IM-ITE exam committee

Disclosures

Outline/Objectives

• Define the test taker level before writing

• List components of a multiple-choice question Objective, stem, lead line, options Annotated description of answers

• Evaluate your questions

Who Are You Writing For?

• Med school GI pathophysiology course? More details; physiology on USMLEs

• Resident end-of-rotation exam? Presentation, testing, management

• GI Fellow board review course? More advanced understanding of above

• IM or GI CME course? Very general vs very specific

What are the Stakes of the Questions?

• Board certification?

• Passing a class/rotation?

• Simple knowledge assessment?

• Solely for interaction?

The Stakes

ReproducibleAccurate

Learning

What Should Be Tested?

• For a clerkship/CME course Match goals/objectives

• For a standardized exam Match the targeted learner level

How Often Will Learner Encounter Information

• Frequent More relevant to test often Atypical manifestations

• Infrequent Test the “can’t miss” things Life threatening Typical manifestations

Types of Questions

• Single best answer With clinical scenario

• Factoid questions• All of the following EXCEPT• Matching• True/False• A and B, A and C, all of the above

What are the Components of a Multiple Choice Question (MCQ)?

STEM

OPTIONS

LEADLINE

Question Components

EXPLANATION

OBJECTIVEANSWER

Question Components

PLEASE TAKE THE PRETEST.

Steps to Writing the MCQ

1. Determine the objective

2. Create the lead line and options

3. Draft the stem

4. Write the explanation, if required

The Objective

Example: Objective

• Recall alcoholic hepatitis

What is wrong with this?

Example: Objective

• Treat a patient with alcoholic hepatitis with corticosteroids in the absence of contraindications

More focused

The Objective

• Succinct sentence• Specifies a clinical skill to be learned• Uses action verbs to delineate a goal

Recognize, diagnose, treat, manage• Avoid vague action verbs

Remember, recall, know• Allows you to match curriculum

VERB+

TOPIC+

MODIFIER

Objective Format

Action Verbs for Objectives

Examples of Objectives:The Good, the Bad, the Ugly

• Good: Diagnose niacin deficiency in a patient

with carcinoid syndrome• Bad:

Recall features of niacin deficiency• Ugly:

Understand niacin deficiency

Other Objective Examples

• Select appropriate testing needed before beginning a biologic agent on an IBD patient.

• Evaluate for splenic vein thrombosis in a patient with isolated gastric varices.

• Manage a patient with celiac disease who has persistent symptoms on a GFD.

• Identify peptic ulcers with stigmata for rebleeding that require endoscopic therapy.

Lead Line

Example: Lead Line A 56-year old alcoholic stumbles into the ER, ataxic,

with nystagmus on examination. Although he appears confused and intoxicated, his blood alcohol level is normal. You start IV glucose and symptoms worsen.

The next step would be to administerA. NiacinB. CT scan of the headC. ThiamineD. InsulinE. Dismissal to home

What iswrong

with this?

Example: Lead Line A 56-year old alcoholic stumbles into the ER, ataxic,

with nystagmus on examination. Although he appears confused and intoxicated, his blood alcohol level is normal. You start IV glucose and symptoms worsen.

What is the next best step in management?A. NiacinB. CT scan of the headC. ThiamineD. InsulinE. Dismissal to home

Keep lead line general; avoid syntax errors

Lead Line

• Most commonly used lead lines: Which of the following is the most likely diagnosis? What is the next best step in management?

Half treatment options, half diagnostic options

• Avoid negative lead lines: All of the following EXCEPT Which of the following is LEAST likely Which would you NOT do

Lead Line• Keep the lead line generic

“Next step” or “management” Allows diagnostic & therapeutic options

• Avoid writing syntax or grammatical errors that help exclude answers

• “Cover the options” rule Should be able to guess the answers

The Options

Example #1: Options

What is the next best step in management?A. Add IV metronidazoleB. Continue po vancomycinC. Start cholestyramineD. Begin rifaximinE. Stop the po vancomycin

What is wrong

with this?

Example #1: Options

What is the next best step in management?A. Add IV metronidazoleB. Continue po vancomycinC. Start cholestyramineD. Begin rifaximinE. Stop the po vancomycin

Mutually exclusive

Example #2: Options

What is the next best step in management?

A. Metronidazole 500 mg po TID x 14 daysB. Vancomycin 250 mg IV QID x 10 daysC. Metronidazole 500 mg IV QID x 14 daysD. Vancomycin 125 mg po QID x 14 days

What is wrong with this?

Example #2: Options

What is the next best step in management?

A. Metronidazole 500 mg po TID x 14 daysB. Vancomycin 250 mg IV QID x 10 daysC. Metronidazole 500 mg IV QID x 14 daysD. Vancomycin 125 mg po QID x 14 days

Testing too many concepts!Drugs, dose, route, duration

Example #3: OptionsThe cause of diarrhea in most patients with Zollinger-Ellison syndrome is:

A. Antacid useB. High glucagon levelsC. Fat malabsorption from pancreatic lipase

inactivation by acid productionD. Bacterial overgrowth What is wrong

with this?

Example #3: OptionsThe cause of diarrhea in most patients with Zollinger-Ellison syndrome is:

A. Antacid useB. High glucagon levelsC. Fat malabsorption from pancreatic lipase

inactivation by acid productionD. Bacterial overgrowth Correct answer

longer, detailed

The Options

• 4 or 5 options (less may be better)• One is not right and the rest 100% wrong

C E A B DWrong Right

The Options

• 4 or 5 options (less may be better)• One is not right and the rest 100% wrong• One is correct, the rest plausible

C E A B DLeast correct Most correct

Options

• Avoid mutually exclusive options Increases/decreases/same Always/never; stop/continue

• Each option should test one concept Drug OR dose OR route OR duration

• Keep options brief; similar in length• Options should be homogeneous

Length, complexity

Stem

The Stem

• Chronologic order of presentation Age, gender (avoid race unless needed) Site of visit (ER, clinic, hospital) Chief complaint (add features) PMH/Meds/FH/SH (relevant or distracter) Vitals/Exam/Labs/Tests (pertinent)

The Stem

• Fine to have details to make the other options attractive; avoid extraneous info

• It takes time to read; make sure it is fitting in with time to administer test

• Organize labs if many

The Explanation

The Explanation

• Explain why right answer correct first• Explain why wrong answers incorrect

– Do this chronologically

• Example for explanation order:– If correct answer is B– B, A, C, D, E

How Do You Analyze Your Questions?

Question Performance

• Goal for correct answer 50-70% 40-90% may be allowed at times Difficulty score of > 90% (too easy) Difficulty score of < 40% (too hard)

• See if it discriminates: Top test takers Bottom test takers

Performs well; difficulty ok and discriminating

(A) (B) (C) (D)

Answer *

Top 50% 7 80 9 4

Bottom 50% 19 44 28 9

Total 13 62 18 7

Option 0.62

Performs poor; difficulty ok, non-discriminating

Option (A) (B) (C) (D)

Answer *

Top 50% 62 0 7 31

Bottom 50% 80 0 7 13

Total 71 0 7 22

Difficulty 0.71

Performs poor; difficulty low (hard), non-discriminating

Option (A) (B) (C) (D)

Answer *

Top 50% 3 45 2 50

Bottom 50% 6 51 5 38

Total 5 48 3 44

Difficulty 0.44

http://www.nbme.org/publications/item-writing-manual-download.html

The Pretest: How Did You Do?

Avoid Duplicating Terms

1. The primary role of the denter is to move the a) ormeb) lattenc) denter pind) stagle

Avoid Repeat Digits, Look at Pattern

2. How many steps are in an average carrum?a) 3.7b) 370c) 240d) 37

Avoid Absolute Terms:Never, Always, Every, All, None

3. The grotto is frequently humid becausea) every ictal is fabliterousb) langers are always ganteringc) reslins are often dewinestd) culers are never liblis

Avoid Inhomogeneous Options

4. Rigniter is aa) form of glatnickb) prestigious subset of the family of honorary gloutninsc) signal by a sellantd) crown of slessings

Avoid Repeating the Correct Option More Than Others

5. Which of the following pairs contributed the Sterlik track?

a) Smith & Johnsonb) Jones & Smithc) Johnson & Thomsond) Smith & Davis

Avoid Syntax or Grammatical Errors

6. The tarnon is proceeded by aa) oflerb) arglingc) isterind) glantern

Questions?

Summary

• Know your learner level

• Writing order: objective, lead line, options, stem

• Avoid errors in MCQ writing

• Evaluate the effectiveness of your MCQs

Thank you!

oxentenko.amy@mayo.edu