3/7/2013

1

Examination Item Development

Bonnie Rush

Kansas State University

Multiple-choice Questions

• Advantages – Rapid automated scoring

– Item analysis for question quality

– Objective and reliable

– Prep for future standardized tests

• Disadvantages – Difficult to prepare

– Time consuming (20 – 60 min)

– Examination security

Item Writing Flaws Threats to Validity

• Students guess correctly

– Grammatical clues

– Distracter errors

• Students fail to demonstrate knowledge

– Awkward stem structure

– Erroneous complexity

– Insufficient complexity

3/7/2013

2

Item Writing Flaws

• Medical educational literature – 35-55% examination items flawed

– Lower order questions (knowledge) predominate (90%)

– More flaws in lower order questions

– 15% of failing students would have passed

“Academic institutions have an ethical and legal responsibility to

ensure faculty provide assessments are valid and of high quality.”

You are examining a group of newly weaned 5 to 6 month old foals with fever, depression, and serous nasal discharge.

Which of the following vaccine strategies would have prevented or attenuated this outbreak?

a) Intranasal influenza vaccine starting at 3 month of age.

b) Killed EHV-1/EHV-4 vaccine, starting at 4 months of age.

c) Killed influenza vaccine starting at 3 months of age.

d) Modified live EHV-1 vaccine, starting at 3 mo of age.

e) Vaccination for common pathogens in 10th month of pregnancy.

Stem errors

• Hypoxic vasoconstriction is: • ____is the term for abdominal pain. • Negative stem

– Which of the following is NOT true? – The following statements are true EXCEPT: – Which of the following clinical signs is INCORRECTLY…? – Which of the following medications is CONTRAINDICATED?

• Erroneous or misleading information

3/7/2013

3

Distracter errors • Series of true/false

– Disadvantages all students.

• All of the above

– Students only need to identify two.

• None of the above

– Student did not demonstrate knowledge of the correct answer.

• Two of the above

– Students might not select the two correct answers.

• A and C

– Unnecessarily complex.

Distracter errors

• Always, never, only, all – Obviously incorrect, students eliminate

• Frequently, occasionally, commonly, often – Ambiguous, interpretation

• Overuse of “C” (27%) – Underuse of “E” (5%)

• Mutually exclusive distracters – Narrows to two

• Repeated words in stem and response – Highlights correct answer

Which of the following is NOT a Halstead principle?

1. Gentle tissue handling

2. Meticulous hemostasis

3. Sepsis

4. Obliterate dead space

5. None of the above

3/7/2013

4

In a Type II hiatal hernia….

1. The gastric cardia invaginates into distal esophagus.

2. Gastroesophageal sphincter and other abdominal viscera lie cranial to the hiatus.

3. Gastroesophageal sphincter is found cranial to esophageal hiatus.

4. The gastroesophageal sphincter occupies a normal position while the fundus or other abdominal viscera herniate through the phrenicoesophageal ligament.

Winter dysentery often presents as an explosive outbreak of watery, possibly hemorrhagic, diarrhea most severely affecting older cattle. The disease often has a self limiting clinical course of 4-5 days in the individual and the outbreak is over within a month. What is thought

to be the causative organism of winter dysentery?

A. BVD type III

B. Coccidiosis

C. Coronavirus

D. Campylobacter

E. Rotavirus

According to Anderson et al in Veterinary Record, traction/avulsion in combination with

oral prednisolone at 1-2 mg/kg/day taper for 2-4 weeks resulted in ___% recurrence rate?

A: 29%

B: 59%

C: 10%

D: 100%

3/7/2013

5

The particle size of aerosolized medications should be ______ for ideal pulmonary drug

distribution.

1. 3 to 5 μm

2. 8 to 10 μm

3. 16 to 18 μm

4. 20 to 25μm

5. 30 to 35 μm

Pick the correct statement in regards to parasites associated with equine

dermatologic conditions.

A. Habronemiasis is spread via culicoides.

B. Skin biopsy is helpful in diagnosis of culicoides hypersensitivity.

C. Lice results in intense pruritus of the lower limb in draft horses.

D. Treatment for onchocerciasis includes ivermectin.

E. Both A & D.

Which of the following is an example of a xenograft?

1. Harvested from a rib of dog “A” and placed into the ilium of dog “A”.

2. Harvested from a rib of dog “A” and placed into the ilium of dog “B”.

3. Harvested from a rib of dog “A” and replacing a segment of femur of dog “B”.

4. Harvested from a rib of cow “A” and placed into the ilium of dog “A”.

5. Harvested from cloned sheep “Dolly” and placed into identical clone “Dolly 2”.

3/7/2013

6

Recall (Lower-level)

• Limit the use of these questions (30%)

– 90% of questions in human medical survey

• Factual knowledge

– Which of the following species does not have a gall bladder?

• Conceptual knowledge

– Which of the following is an example of….?

What is the calculated blood volume of a 500 kg horse?

a. 20L

b. 40 L

c. 100 L

d. 200 L

e. 400L

89.5% Correct

0.22 Discrimination Factor

You are treating a herd outbreak of foal diarrhea at a QH farm. Young children and an elderly

grandparent live on the premises. Which differential diagnoses for equine diarrhea are

zoonotic?

a. Rotavirus and Coronavirus

b. E coli and Lawsonia

c. Giardia and Toxoplasmosis

d. Salmonella and C. difficile

3/7/2013

7

Higher-level • Interpret the stem using recall facts. • Apply those facts to select a response.

• Application – Which diagnostic test would you perform next?

• Analyzing conceptual knowledge – Based on evaluation of the blood work, radiograph, video. – Provide normal values

• Evaluating procedural knowledge – Case study: What should have been done?

If Australia’s horse population was naïve to equine herpesvirus and EHV-4 was introduced into the population, which of the following outcomes would you predict based on your knowledge of EHV-4?

a) Eradication attempts using a cross-protective MLV EHV-1

vaccination buffer zone and restricted movement would be successful.

b) Foals under 9 months of age would develop respiratory disease; older horses are resistant to EHV-4 infection.

c) Abortion storms and neonatal illness would decimate the foal crop during the year of primary infection.

d) Eradication attempts using an EHV-4/EHV-1 vaccination buffer zone and restricted movement would be unsuccessful.

e) Naïve horses would develop neurologic signs, characterized by ataxia, ascending paralysis, and urinary incontinence.

You are evaluating a 10 yr old donkey with a 2 day history of left-sided epistaxis. The blood is fresh, bright red, and non-

malodorous. The client occasionally observes blood at the right nostril. Based on the owner’s description, which of the following

anatomic structures of the left nasopharyngeal tract is of particular interest?

a) nasomaxillary aperture

b) upper arcade

c) ethmoid turbinate

d) arytenoid cartilage

e) guttural pouch

3/7/2013

8

You are evaluating a 19 year old Thoroughbred gelding with fever, weight loss, cough, and dyspnea at rest. He has a prolonged

inspiratory phase of respiration and a shortened expiratory phase. You obtain a thoracic radiograph: Which of the following diagnostic

tests would you like to perform next?

A. Pleuroscopy

B. Broncholalveolar lavage for EHV-5 PCR testing

C. Lung biopsy to diagnose disseminated neoplasia

D. Transtracheal wash for fungal culture

E. Endoscopic examination for endobronchial biopsy

Discriminating Questions

• Discrimination Index > 0.20

• Hi 27 and Low 27

• Distracters not selected should be replaced

Easy Questions

First order: knowledge recall questions

3/7/2013

9

Faulty Questions

< 0.2 Index with < 75% correct

Subtract points from the denominator

You are examining a group of newly weaned 5 to 6 month old foals with fever, depression, and serous

nasal discharge. Which of the following vaccine strategies would have prevented or attenuated this

outbreak?

a) Intranasal influenza vaccine starting at 3 month of age.

b) Killed EHV-1/EHV-4 vaccine, starting at 4 months of age.

c) Killed influenza vaccine starting at 3 months of age.

d) Modified live EHV-1 vaccine, starting at 3 mo of age

e) Vaccination for common pathogens in 10th month of pregnancy.

If Australia’s horse population was naïve to equine herpesvirus and EHV-4 was introduced into the population, which of the following outcomes would you predict based on your knowledge of EHV-4?

• a) Eradication attempts using a cross-protective MLV EHV-1

vaccination buffer zone and restricted movement would be successful.

• b) Foals under 9 months of age would develop respiratory disease; older horses are resistant to EHV-4 infection.

• c) Abortion storms and neonatal illness would decimate the foal crop during the year of primary infection.

• d) Eradication attempts using an EHV-4/EHV-1 vaccination buffer zone and restricted movement would be unsuccessful.

• e) Naïve horses would develop neurologic signs, characterized by ataxia, ascending paralysis, and urinary incontinence.

3/7/2013

10

Overall Examination Statistics

• Target class average = 83% correct

• At minimum 15% of students > 90% correct

• Average discrimination index = 0.18 to 0.21

• Test too challenging → identify extra credit

Extra Credit

Review the exam

• Do the questions reflect the course objectives?

• Do the questions test clinically-relevant principles?

– Disease recognition, diagnosis, treatment, prognosis

• Use of medical terminology

• Generic drug names

• Unnecessary abbreviations (ADR, BDLD)

3/7/2013

11

Ask someone to review the exam for you.

• Numbering errors

• Duplicate questions

• Inappropriate clues

• Awkward grammar and syntax

• Punctuation problems

Classroom Management

• Proctor present at all times

• No electronic devices, no backpacks

• Images and videos on a rolling PPT

• Share responses with the whole class

• Announce 30 min, 15 min, 5 min

Post-examination management

• Review results for mis-keyed items.

• Review item analysis for poor performing questions.

• Post grades ASAP: average, range, and adjustments.

• Post examination when all have taken.

• Post examination in a controlled environment.

• Argue for points.

3/7/2013

12

Resources

http://www.nbme.org/publications/item-writing-manual.html

Tarrant M: The frequency of item writing flaws in multiple choice questions used

in high stakes nursing assessments. Nurse Educ Pract. 2006;6:354-363.

Downing SM: The effects of violating standard item writing principles on tests and students: The consequences of using flawed test items on achievement examination in medical education. Adv Health Sci Educ. 2005;10:133-134.

Burton et al: How to prepare better multiple-choice test items: Guidelines for university faculty. Brigham Young University Testing Services. http://testing.byu.edu/info/handbooks/betteritems.pdf