2. Format of stem

1. Balance of content level

What is a “good” (or “bad”) multiple-choice item? (Trade secrets from a professional)

Psychology and applied psychometricsRationale for multiple-choice testing

Item quality that can be assessed before the

test:

Item quality that can be assessed after the test:1. How difficult was the item?

2. How well did the item discriminate?!The Item Characteristic Curve (ICC)

The point-biserial coefficient (r)

a. It can have extreme emotional content.b. It has only a single function.c. It cannot be varied or stopped once triggered.d. It is relatively independent of learning or

experience.e. All of the above do not follow from that fact.

What statement below does not follow from the fact that an FAP is ballistic?

Wall’s Bad Example 1(too many negative stems)

In accordance with a decay theory of forgetting, if a participant learning a list of words was then subjected to one of the following procedures, he would forget the greatest number of words if_______

a. he slept for four versus two hoursb. he learned other lists for four versus two hoursc. he performed arithmetic problems for four versus two hoursd. he performed crossword puzzles for four versus two hourse. All of the above would be comparable.

Wall’s Bad Example 2(complex fill-in-the-blank formats (“unquestionized”))

Of the five properties listed below, which are the most important in relation to the function of the plasma membrane in living cells?

i. Selective permeabilityii. Strengthiii. Elasticityiv. Hydrophilicityv. Fluidity

a. i, ii, and iiib. i and vc. ii and iiid. iii and ive. iv and v

Wall’s Bad Example 3(Compound questions)

Scatterplot for TT3 (W’04)

0

10

20

30

40

50

60

0 5 10 15 20

r = +0.65

Score on essay

Sco

re o

n m

ult

iple

-ch

oic

e

Cognitive Levels

Student Activity Words in Item Stems

Judging based on established set of criteria

Appraise, evaluate, justify

Producing something new from component parts

Design, develop, create, formulate, construct

Breaking down material to see relationships, hierarchy

Differentiate, compare-contrast, relate

Using a concept to solve a problem

Solve, show, demonstrate, compute

Explaining-interpreting Explain, predict, infer, account for, summarize

Remembering facts, terms, concepts, definitions

Define, list, state, identify, label

Evaluation

Synthesis

Analysis

Application

Comprehension

Knowledge

“QUESTIONIZED”Why does damage in the language center of a left-handed person often have less drastic consequences than in a right-handed person?

FILL-IN-THE-BLANK Damage in a left-handed person in the language center often has less drastic consequences than in a right-handed person because _______

a. left-handers are slower to develop language function.

b. in left-handers, recovery in the left hemisphere is more rapid.

c. highly developed visual-spatial abilities compensate for language loss.

d. in left-handers, the hemispheres are not as functionally lateralized.

e. left-handers have several language centers.

Multiple-Choice Item Analysis

I tem Key a b c d e Pass % Biserial 1 b 21 275 8 4 27 82% 0.38 2 d 47 51 8 178 49 53% 0.26 3 e 116 25 26 48 120 36% 0.37 4 a 272 4 28 12 18 81% 0.38 5 b 17 297 16 4 0 89% 0.31 6 c 5 11 302 10 7 90% 0.21 7 c 13 85 203 28 6 61% 0.53 8 b 38 179 44 42 32 53% 0.50 9 a 279 18 11 16 11 83% 0.47 10 e 96 17 23 59 140 42% 0.10

. . . . . . . . .. . . . . . . . .. . . . . . . . .

N = 335

Index of “difficulty” or “easiness”

A hypothetical item analysis for a single multiple-choice item:

The item characteristic curve (ICC)

F D C B A

1.0

0.9

0.8

0.70.6

0.5

0.40.3

0.2

0.10

Student-performance groups

Pro

port

ion

gett

ing

ite

m c

orr

ect

Chance performance

Perfect performance

F D C B A

1.0

0.9

0.8

0.70.6

0.5

0.40.3

0.2

0.10

Student-performance groups

Pro

port

ion

gett

ing

ite

m c

orr

ect

Chance performance

Perfect discriminator(between below-C andabove-C students)

A hypothetical item analysis for a single multiple-choice item:

The item characteristic curve (ICC)

F D C B A

1.0

0.9

0.8

0.70.6

0.5

0.40.3

0.2

0.10

Student-performance groups

Pro

port

ion

gett

ing

ite

m c

orr

ect

Ogive

?!!

A hypothetical item analysis for a single multiple-choice item:

(Possible item characteristic curves (ICC’s) )

F D C B A

1.0

0.9

0.8

0.70.6

0.5

0.40.3

0.2

0.10

Student-performance groups

Pro

port

ion

gett

ing

ite

m c

orr

ect

Actual Item Analyses: good discriminators

Item 11

Item 7

Item 8

Item 43

F D C B A

1.0

0.9

0.8

0.70.6

0.5

0.40.3

0.2

0.10

Student-performance groups

Pro

port

ion

gett

ing

ite

m c

orr

ect

Actual Item Analyses: poor discriminators

Item 6

Item 12

Item 10

Item 42

The issue of discriminability: Biserial statistics

I tem Key a b c d e Pass % Biserial r

1 b 21 275 8 4 27 82% 0.38 2 d 47 51 8 178 49 53% 0.26 3 e 116 25 26 48 120 36% 0.37 4 a 272 4 28 12 18 81% 0.38 5 b 17 297 16 4 0 89% 0.31 6 c 5 11 302 10 7 90% 0.21 7 c 13 85 203 28 6 61% 0.53 8 b 38 179 44 42 32 53% 0.50 9 a 279 18 11 16 11 83% 0.47 10 e 96 17 23 59 140 42% 0.10

. . . . . . . . .. . . . . . . . .. . . . . . . . .

N = 335

?

Illustration of obtaining the point-biserial correlation for a single item:

A 0 54 B 1 76 C 1 87

D 0 58

etc.

N

Y)()Y(

N

X)()X(

N

Y)X)((XY

22

22

r

StudentScore on item

(0 or 1)Overall scoreon test

X Y