© Courseware Interactive Ltd., 2003

PresentationMay 22nd 2003

Designing Effective Multiple Choice Questions

ILTA,Waterford

Niall Watts

Courseware Interactive Ltd

© Courseware Interactive Ltd., 2003

Evaluation - content & teaching

Retention

Reinforcement

Diagnosis - feedback

Assessment

Engagement - Interaction

Why ask questions?

© Courseware Interactive Ltd., 2003

Interactivity

Students interacting with content

Make them thinkCheck their knowledgeCognitive activity

Not necessarily:FlashMenusDatabases

© Courseware Interactive Ltd., 2003

Types of Question

Choosing - Multiple Choice, Matching, Selecting

Open - Text Entry, Short Answer, Fill in the Blank, Completion

© Courseware Interactive Ltd., 2003

Choosing

Advantages Easy to Mark Usability

Disadvantages Hard to set Guessing answer Recall of factual information

Popular with teachers & designers

© Courseware Interactive Ltd., 2003

Short Answer

Advantages Easy to Set No guessing

DisadvantagesHard to Mark

Less Popular

© Courseware Interactive Ltd., 2003

Multiple Choice

Multiple-choice questions are often criticised for being easy to guess and for emphasising factual information. We contend that well-written multiple-choice questions can do a good job of assessing comprehension, problem solving and other higher order skills.

Alessi & Trollip, Multimedia for Learning 3rd Ed.,2001

© Courseware Interactive Ltd., 2003

Multiple Choice Terminology

Anatomy StemOptionsDistractors (Foils)

© Courseware Interactive Ltd., 2003

Multiple Choice Clarity

Most text in stem

Easy to ReadFocused on questionNo additional or supplementary

informationAvoid scrolling

© Courseware Interactive Ltd., 2003

Multiple Choice Clarity

Clear InstructionsNo ambiguityTell them what is expected

e.g Number of Correct OptionsNo tricks

Just checking the student’s knowledge and understanding of the course

© Courseware Interactive Ltd., 2003

Multiple Choice Clarity

AvoidNegativesConditionals

© Courseware Interactive Ltd., 2003

Multiple Choice Distractors

Hard to setThe secret of good multiple choice question

design is well-written distractorsA distractor must never suggest the correct

answer

© Courseware Interactive Ltd., 2003

Multiple Choice Issues

Plausible distractors

more

Option order

Number of options

Accidental hints

© Courseware Interactive Ltd., 2003

Distractors - Plausibility Sensible

All options should sound plausible No ‘Mickey Mouse’ unless it’s a Disney Quiz

Hint Write question as open Use popular wrong answers as distractors

© Courseware Interactive Ltd., 2003

Testing Comprehension

Avoid repeating lesson text verbatim students may recognise key words students may recall context

Paraphrase for comprehension Use different examples or case studies Ask student to apply a rule or principle

© Courseware Interactive Ltd., 2003

Higher Order Learning

Problem solvingGive several answers to a problem

AnalysisChoose between interpretations Select reasons for choice

© Courseware Interactive Ltd., 2003

Distractors – Accidental Hints

ConditionsGrammatical & Linguistic ConsistencyOption LengthRepeating TextPrecisionIntersecting OptionsOverlapping OptionsAll of the AboveNone of the Above

© Courseware Interactive Ltd., 2003

Distractors – Accidental Hints

Correct answers must always be correct Distractors must always be incorrect Negatives can be confusing Conditionals can obscure the answer or give it away

Exercise

© Courseware Interactive Ltd., 2003

Multiple Choice Distractors

Avoid ambiguityCorrect answers must always be correctDistractors must always be incorrectE.g. Word Processing Question

– Complete the following sentence correctly. A Word processing program is best defined as software used to:

– .Publish newsletters– .Produce documents– .Design graphics– .Send mailshots

– Option 2 is probably ‘the best definition’ but options 1 and 4 could also be

correct. more

© Courseware Interactive Ltd., 2003

Multiple Choice Distractors

Avoid ambiguityE.g. Word Processing Question

– The question could be better written as:

– Complete the following sentence correctly. A Word processing program is best defined as software used to:

– 1. Send e-mail messages– 2. Produce documents– 3. Produce graphics– 4. Produce accounts

more

© Courseware Interactive Ltd., 2003

Multiple Choice Distractors

Avoid ambiguityAvoid Negatives. They can confuse a student.

E.g.– Which of the following is not a fraction?

– 0.5– 1/2

• This could be better written as:– Which of the following is a fraction?

– 0.5– 1/2

more

© Courseware Interactive Ltd., 2003

Multiple Choice Distractors

Avoid ambiguity

Negatives in true/false questions can be particularly confusing. E.g.– London is not the capital of England. True or False.

• In particular avoid negatives in both the stem and the options

more

© Courseware Interactive Ltd., 2003

Multiple Choice Distractors

Avoid Ambiguity - Conditions

Avoid conditions such as ‘often’, ‘sometimes’,’usually’,’always’,’never’ or ‘all’.

Avoid the use of the verb ‘may’. Conditions may make the answer unclear or give it away altogether.

True or False. Metals are usually solids at room temperature.

The word usually should be avoided as it hints that the correct answer is true.

True or False. All Metals are solids at room temperature.

The word all should be avoided as it hints that the correct answer is false.

© Courseware Interactive Ltd., 2003

Multiple Choice Distractors

Grammatical & Linguistic Consistency

Sentence structure reveals answer where language flows from stem to correct answer. Example:Second option below

more

Which option completes the following sentence? In 1492 Christopher Columbus…

is the discoverer of North Americalanded in the New Worldis buried in Seville cathedralwas beaten by the Vikings by 500 years

© Courseware Interactive Ltd., 2003

Multiple Choice Distractors

Length of Options

Where one answer is much longer than the others it may well be correct

more

Which of the following best defines a bird?

A feathered animalA warm-blooded animal with feathersA warm-blooded animal with feathers that lays eggs

© Courseware Interactive Ltd., 2003

Multiple Choice Distractors

Repeating Text Where text in the stem is repeated in one of the options

that option is likely to be correct. Example

more

Why was the Financial Services Authority founded?

To prevent money launderingTo regulate financial servicesTo provide consumer informationTo control pension sales

© Courseware Interactive Ltd., 2003

Multiple Choice Distractors

Precision Where one option is more precise it hints that it may be

correct

more

The height of Ben Nevis is:

1000 metres1343 metres1500 metres4000 metres

© Courseware Interactive Ltd., 2003

Multiple Choice Distractors

Intersecting Options An option covering more than one point is more likely to be

correct. Here, option 2 covers both eating & time.

more

Medication to lower blood pressure should be taken once a day. Should it be taken?

At a mealtimeBefore breakfastIn the morning

© Courseware Interactive Ltd., 2003

Multiple Choice Distractors

Overlapping Options Be careful that you do not choose overlapping distractors. At

first glance they may seem correct but potentially there are multiple correct answers

more

Due to cutbacks the budget was reduced to less than:

£10,000 £8,000 £5,000

Suppose the budget was reduced to £4,500!

© Courseware Interactive Ltd., 2003

Multiple Choice Distractors

All of the Above Avoid ‘all of the above’

If one distractor is wrong it cannot be the answer If two options are correct it must be the answer (1 correct only)

more

© Courseware Interactive Ltd., 2003

Multiple Choice Distractors

None of the Above ‘None of the above’ can be used safely provided

Sometimes it is the correct answer Provided ‘the best answer’ is not required

more

© Courseware Interactive Ltd., 2003

Multiple Choice Distractors

Ambiguity Negatives Conditions Grammatical & Linguistic Consistency Option Length Repeating Text Precision Intersecting Options Overlapping Options All of the Above None of the Above

© Courseware Interactive Ltd., 2003

Multiple Choice Options

How many options?

4 is popularHow many correct?Higher number of options reduces guessing riskThree options may be appropriate e.g.

– At constant temperature with increasing pressure the volume of a fixed mass of gas

– 1. Decreases– 2. Increases

– 3. Stays the Same

© Courseware Interactive Ltd., 2003

Multiple Choice Options

Option Order

3rd position correct is popular1st is unpopular!Position choice:

Correct answer must be randomly positioned Or options in order e.g alphabetical

Position must NOT suggest a correct choice

© Courseware Interactive Ltd., 2003

True/False Questions

Multiple Choice with 2 options 50% chance of correct guess Easy to Mark Hard to Set Can be disguised e.g

select an object on a diagram

© Courseware Interactive Ltd., 2003

CourseBuilder

Used to create learning interactions Extension for Dreamweaver Free Register with Macromedia Download from www.macromedia.com/ Follow instructions on screen

© Courseware Interactive Ltd., 2003

CourseBuilder

Make a multiple choice question

Insert - CourseBuilder Interaction

Multi Choice Radios with Defaults

Red button to edit

© Courseware Interactive Ltd., 2003

CourseBuilder

Customising multiple choice question

Multi Choice Radios with Set Text of Layer

Multi Choice Checkboxes with Layers

© Courseware Interactive Ltd., 2003

ResourcesInstructional Design

IFETS - ifets.ieee.orgITFORUM - University of GeorgiaERICAssessment – Question Mark

Web developmentMacromediaCoursebuilder ListServRapidIntake – www.rapidintake.comJakob Nielsen Usability - www.useit.com

© Courseware Interactive Ltd., 2003

Contact Niall Watts Courseware Interactive Ltd

niall@courseware.ie www.courseware.ie (353-1) 295 2484