NRTs and CRTs
-
Upload
samson-lowery -
Category
Documents
-
view
60 -
download
1
description
Transcript of NRTs and CRTs
![Page 1: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/1.jpg)
NRTs and CRTs
Group members: Camila, Ariel, Annie, William
![Page 2: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/2.jpg)
NRTs—Norm-referenced Item Analysis
Item facility Item discrimination
![Page 3: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/3.jpg)
Why do we do item analysis?
Assemble a large number of items of the type you want on the test
make sure the items are well written and clear
Pilot the items Analyze the results of the pilot testing Select the most effective items/get rid
of the ineffective items or revise the weak ones
![Page 4: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/4.jpg)
The basic purpose of NRTs
To spread students out along a general continuum of language abilities
making aptitude, proficiency, or placement decisions
![Page 5: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/5.jpg)
Two item statistics used in IA of NRTS
Item facility(IF): the proportion of students who answered a particular item correctly.
45/50=.90---90% of the students answered the item correctly—the item is very easy
Item discrimination(ID):Caculate IF for the upper group and the lower group using AVERAGE (C2:C6) and AVERAGE (C15:C19) ID=IFu-IFl
ex: IFu-IFl=.20
![Page 6: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/6.jpg)
Item Discrimination (ID)
can be calculated by first figuring out who the upper and lower students are on the test
using their total scores to sort them from the highest score to the lowest
Equal numbers of students in three groups—high/middle/low
ID=Ifupper - Iflower
![Page 7: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/7.jpg)
Ideal items in an NRT
Should have an average IF of .50 50% of the students answered correctly
and 50% of them answered incorrectly In reality, items rarely have an IF of
exactly .50 Those that fall in a range between .30
and .70 are usually considered acceptable for NRT purpose
![Page 8: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/8.jpg)
Items within .30 to .70 range
The items among them that have the highest IDs should be further selected for inclusion in the revised test.
This process would help the test designer to keep only those items that are well centered and discriminate well between the high and the low scoring students
![Page 9: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/9.jpg)
Conclusion for NRT
IF and ID are only appropriate for developing and analyzing norm-referenced tests
Used at the institutional level overall English language proficiency tests or placement
tests
Not appropriate for developing and analyzing classroom oriented criterion-referenced tests like the diagnostic, progress, and achievement tests
![Page 10: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/10.jpg)
CRTs-Criterion-referenced Item Analysis
Purpose of CRTs—to measure the amount (or percent) of material in a course or program of study that students know
Usually for purposes of making diagnostic, progress, or achievement
![Page 11: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/11.jpg)
Two item statistics used in IA of CRTs
1.The difference index(DI): the item facility on the particular item for the posttest minus the item facility for that same item on the pretest
pretest:10/50=.20; posttest:45/50=.90 DI=.90-.20=.70 DI tells how much the students are
improving between the pretest and posttest on each item
The higher the value of the DI, the better. A value of 1.00 is a perfect difference
index
![Page 12: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/12.jpg)
Two item statistics used in IA of CRTs
2.The B-index: the item facility on the particular item for the students who passed the test minus the item facility for the students who failed
The B-index show how well each item is contributing to the pass/fail decisions that are often made with CRTs
B-index=IFpass - IFfail=14/14-0/6=1.0-.00=1
![Page 13: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/13.jpg)
P.21
The B-index tell us how well each item is contributing to the pass/fail decision on this test at the cut-point.
Like ID and DI statistics, the higher the B-index, the better.
A perfect B-index would be 1.00.
![Page 14: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/14.jpg)
Conclusion-CRTs
When should these indices be used? To analyze the items on a CRTs for
purposes of revising the test The items with the highest values
should generally be kept in both cases, DI and BI
Making these decisions not as simple as it is for NRT development, because a CRT item may not be performing well in terms of these statistics for many reasons.
![Page 15: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/15.jpg)
Reasons that a CRT item may not be performing well
The item is written/ working poorly The objective the item is testing is vague Ss are not ready to learn this particular
objective One/ all of the teachers did not teach this
particular objective or teaching it poorly The materials are confusing with regard to
this particular objective The statistics cannot tell you exactly what
is wrong even though they can point you to places in your curriculum where something is not working well
![Page 16: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/16.jpg)
Some common-sense analysis of the entire situation needs to be done
To revise the CRT or other aspects of your curriculum such as the objectives, the materials, the teaching, etc.,
![Page 17: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/17.jpg)
What the statistics help you?
Help you figure out where to focus your energies
The DI will tell you how well each item fits the objectives of your curriculum
The BI will tell you how each item is contributing to the pass/fail decision that you must make at whatever cut-point you are using.
![Page 18: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/18.jpg)
2 Definitions for “Criterion” in CRT
1.Refers to the material being taught in the course
--CRT would assess the particular learning points of a particular course or program
This definition fits very well with the difference index, which indicates how well each item fits objectives of the curriculum
![Page 19: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/19.jpg)
2 Definitions for “Criterion” in CRT
2. The standard of performance (or cut-point for decision making) that is expected for passing the test/course
--CRTs –used to assess whether students pass or fail at a certain criterion level (or cut-point)
Fits very well with the B-index, which indicates how well each item is contributing to the pass/fail decision that you must make at whatever cut-point your are making
![Page 20: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/20.jpg)
What should you focus?
If you are interested in the degree to which your items are reflecting the material in your course….DI
If you are interested in the degree to which your items are helping you make decisions at a certain cut-point… BI
If you are interested in both statistics… DI and BI
![Page 21: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/21.jpg)
What should they not be used?
Not used to analyze the effectiveness of norm-referenced items
![Page 22: NRTs and CRTs](https://reader033.fdocuments.us/reader033/viewer/2022042504/568131a5550346895d9814cb/html5/thumbnails/22.jpg)
What is the Ultimate Goal?
To produce a curriculum and CRTs that match each other such that you get high difference indexes and high B-indexes.