Chapter7 Item Analysis
-
Upload
lnl-stainless-steel -
Category
Documents
-
view
220 -
download
0
Transcript of Chapter7 Item Analysis
-
7/24/2019 Chapter7 Item Analysis
1/41
Chapter 7 Item Analysis In constructing a new test (or shortening or
lengthening an existing one), the final set of
items is usually identified through a process
known as item analysis.
Linda Croker
-
7/24/2019 Chapter7 Item Analysis
2/41
Both the alidity and the relia!ility of any
test depend ultimately on the characteristics of
its items.
-
7/24/2019 Chapter7 Item Analysis
3/41
Two Approaches of Item Analysis
Qualitative Analysis
Quantitative Analysis
-
7/24/2019 Chapter7 Item Analysis
4/41
Qualitative Analysis
includes the consideration of content
validity (content and form of items), as well as
the evaluation of items in terms of effective
item-writing procedures.
-
7/24/2019 Chapter7 Item Analysis
5/41
Quantitative Analysis
includes principally the measurement of
item difficulty and item discrimination.
-
7/24/2019 Chapter7 Item Analysis
6/41
1 Item ifficulty
1. Definition
!he item difficulty for item i, pi , is defined
as the proportion of e"aminees who get that
item correct.
-
7/24/2019 Chapter7 Item Analysis
7/41
"hough the proportion of examinees passing
an item traditionally has !een called the item
difficulty, this proportion logically should !e
called item easiness, !ecause the proportion
increase as the item !ecomes easier.
-
7/24/2019 Chapter7 Item Analysis
8/41
#. Estimation Methods
$ethod for ichotomously %cored Item
$ethod for &olytomously %cored Item
'rouping $ethod
-
7/24/2019 Chapter7 Item Analysis
9/41
Method for Dichotomously Scored
Items
(.1)
p is the difficulty of a certain item.
# is the numer of e"aminees who get that item
correct.$ is the total numer of e"aminees.
N
RP=
-
7/24/2019 Chapter7 Item Analysis
10/41
Example 1
"here are %& high school students attending a
science achieement test, and ' students pass
item , * students pass item &. +lease calculate
the difficulty for item and & separately.
-
7/24/2019 Chapter7 Item Analysis
11/41
Method for Polytomously Scored
Items
(.#)
max
X
XP=
X , the mean oftotal e"aminees* scores on one item
maxX , the perfect scores of that item
-
7/24/2019 Chapter7 Item Analysis
12/41
Example 2
"he perfect scores of one open ended item is*& points, the aerage score of total examinees
on this item is points. -hat is the item
difficulty
/ey0 .11
-
7/24/2019 Chapter7 Item Analysis
13/41
'rouping $ethod (+se of "treme 'roups)
2pper (2) and Lower (L) Criterion groups are selected
from the extremes of distri!ution of test scores or 3o! ratings.
!. . elley (1/0/) proposed that upper and lower
# could lead to the optimal point when the total test
scores are normally distriuted.
-
7/24/2019 Chapter7 Item Analysis
14/41
2
LU PP
P
+
=
UP is th proportion for e"aminees of upper groupwho get the item correct.
(.0)
LP is the proportion for e"aminees of lower groupwho get the item correct.
-
7/24/2019 Chapter7 Item Analysis
15/41
Example 3 "here are 4& examinees attending a language
test. /nown that '5 examinees of *46 upper
extreme group pass item 1, and examinees of *46
lower extreme group pass the same item. +lease
compute the difficulty of item 1.
/ey 0 .57
-
7/24/2019 Chapter7 Item Analysis
16/41
3. Correct Chance Effects on Item
Difficulty for Multiple-Choice Item
(.2)1
1
= K
KP
CP
CP ,corrected item difficulty
P ,uncorrected item difficulty
K , the number of choices for that item
-
7/24/2019 Chapter7 Item Analysis
17/41
Example 4
"he diffuculty of one fiechoice item is .1&,
the difficulty of another fourchoice item is .1.
-hich item is more difficulty
-
7/24/2019 Chapter7 Item Analysis
18/41
ANSWER
%o, the four-choice item is more difficulty.
38.0
15
15.05
1
11 =
=
=
K
KPCP
37.014
153.04
1
12 =
=
=
K
KPCP
-
7/24/2019 Chapter7 Item Analysis
19/41
4. Item Difficulty and Discrimination
iscrimination
ifficulty
-
7/24/2019 Chapter7 Item Analysis
20/41
If there are && persons in one population ,then ,we can calculate the discriminations as
following0
P=.01, 1 !! = !!P=.02, 2 !8 = 1!"
P=.3, 30 70 = 2100
P=.5, 50 50 = 2500
-
7/24/2019 Chapter7 Item Analysis
21/41
5. est difficulty and the
Distri!ution of est Scores 3ow to 4alculate the !est ifficulty 5
Two Methods
A calculate the mean of all item difficultiesof the test
B compute the ratio of mean of test scoresto perfect test scores
-
7/24/2019 Chapter7 Item Analysis
22/41
!est difficulty and the istriution of !est %cores
(a)
&ositive %6ewed istriution()
7egtive %6ewed istriution
-
7/24/2019 Chapter7 Item Analysis
23/41
# Item iscrimination
-hen the test as a whole is to !e ealuated !y means of
criterionrelated alidation, the items may themseles !e
ealuated and selected on the !asis of their relationships to
the external criterion.
-hen we identify an item for which high scoring
examinees hae a high pro!a!ility of answering correctly
and lowscoring examinees hae a low pro!a!ility ofanswer correctly, we would say such an item can
discriminates or differentiates the examinees.
-
7/24/2019 Chapter7 Item Analysis
24/41
1. Interpretation
Item discriminationrefers to the degree to
which an item differentiates correctly among
test takers in the !ehaior that the test is
designed to measure.
-
7/24/2019 Chapter7 Item Analysis
25/41
". Estimation Methods
Inde" of iscrimination
(used for dichotomously scored items)
D = PH- PL#7.5$
-e need to set one or two cutting scores to diide the examinees into
upper scoring group and lower scoring group.
+8 is the proportion in the upper group who answer the item
correctly and +L is the proportion in the lower group who answer theitem correctly.
9alues of : may range from .&& to .&&.
-
7/24/2019 Chapter7 Item Analysis
26/41
Example 1
"here are 5& students attending a world history test.
() If we use the ratio *46 to determine the upper andlower group, then how many examinees are there in the
upper and lower group separately (*)If % examinees
in upper group answer item 1 correctly, and ' examinees
in lower group answer it correctly, then calculate thediscrimination index for item 1.
-
7/24/2019 Chapter7 Item Analysis
27/41
Example 2 89 "aminees* !est ata on :-Item %cale Aout ;o%tress.
Item 1 # 0 2 8 < :
+8
+L
.82 .:1 .2 .0# .81 .1: .
-
7/24/2019 Chapter7 Item Analysis
28/41
'uidelines for Interpretation of =alue
:;.5&, the item is functioning
-
7/24/2019 Chapter7 Item Analysis
29/41
Correlation Indices of Item
Discrimination
(1) &earson &roduct $oment 4orrelation
4oefficient
"his formula is commonly used to estimate the degree of the
relationship !etween item and criterion scores
YX
XYsNs
xyr =
-
7/24/2019 Chapter7 Item Analysis
30/41
(#) &oint >iserial 4orrelation
If we use the total test score as the criterion, and test itemis scored & to , then we can use the following formula0
qp
s
XXr
t
tp
pbi %
= (.
-
7/24/2019 Chapter7 Item Analysis
31/41
Example 3
the !est ata of 18 "aminees"aminees 1 2 3 4 5 6 7 8 9 10 11
12 13 14 15
!est score
Item score
/9 :1 :9 : 9
-
7/24/2019 Chapter7 Item Analysis
32/41
72.2115
15%88258!3" 2=
=ts
48.4""7.%5333.72.21
80.585."8 =
=pbir
80.5815
10...8081!0=
++++=tX
-
7/24/2019 Chapter7 Item Analysis
33/41
!ransformation of ?ormula .iserial 4orrelation 4oefficient
or
Y
p
s
XXr
t
tp
b
=
Y
pq
s
XXr
t
qp
b
=
-
7/24/2019 Chapter7 Item Analysis
35/41
(2) 4orrelation >etween Items
a) Tetrachoric Correlation Coefcient
>ach aria!le is created through dichotomi?ing an
underlying normal distri!ution
$180cos# +
=BCAD
ADrt
(.:)
A >
4
Item i
9 1
Item
3
1
9
@AC BA:
@AB
CA:
-
7/24/2019 Chapter7 Item Analysis
36/41
b) PHI4oefficient
$$#$#$## DBDADCBAADBCr
++++
= (./)
-
7/24/2019 Chapter7 Item Analysis
37/41
#ariance for item
2
12
# $n
ij i
j
i
X X
sn
=
=
(.19)
-
7/24/2019 Chapter7 Item Analysis
38/41
ifficulty and iscrimination
&
1.99 9.99 9./9 9.#9
9.9 9.
-
7/24/2019 Chapter7 Item Analysis
39/41
0 Application 4ase of Item
Analysis
1. Procedures %elect a representative sample of e"aminees and
administer the test@
ifferentiate the e"aminees into upper # (or 09etc.) group and lower # group according to theirtest scores@
4alculate &+and &-, then estimate & and for each
item@
4ompare the responses on different choices for eachitem etween the upper group and lower group@
Bevise items.
-
7/24/2019 Chapter7 Item Analysis
40/41
". $nalysis Case
&
9.1 9.2# 9.8#
9.2# 9.0# 9.00
9.01 -9.9< -9.92
9.1# 9.92 9.9:
Item 'roup 7umer of "aminees on ach4hoice
ey
A > 4 Cmit
1 +pper 8 /# 1 # 9 >
ower ## 89 1# 1< 9
# +pper 8: 19 18 1< 1 A
ower #< #1 18 0< #
0 +pper 1 18 #: #: 1#
ower #8 11 1/ 02 11
2 +pper 1 22 12 0< 8 4
ower 1 8< 19 #: 8
br
-
7/24/2019 Chapter7 Item Analysis
41/41
Choice Analysis Dhether the e"aminees who choose the correct choice
is more than those who choose the wrong choices Dhether a lot of e"aminees choose the wrong choices
Dhether the e"aminees of upper group who choose
the correct choice is more than the e"aminees of lower
group Dhether the e"aminees of upper group who choose the
wrong choice is more than those of lower group
Dhether there is any choice that few e"aminees choose
Dhether there is any item that Euite a numer of
e"aminees ma6e no choices