Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module,...
Transcript of Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module,...
![Page 1: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/1.jpg)
Dr Luning Sun
Psychometrics Module, Lecture 3
Social Sciences Research Methods Centre | SSRMC
![Page 2: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/2.jpg)
Introduction to CAT
CAT in R
CAT in Concerto
![Page 3: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/3.jpg)
Some materials and examples come from previous workshops run by:
Michal Kosinski (Stanford University)
David Stillwell (University of Cambridge)
Chris Gibbons (Harvard University)
![Page 4: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/4.jpg)
Standard test is likely to contain questions that are too easy or too difficult◦ Classical Test Theory
◦ Item Response Theory
Adaptively adjusting the level of the test to individual participant:◦ Increases the accuracy
◦ Saves time / money
◦ Prevents boredom / frustration
![Page 5: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/5.jpg)
Item bank and calibration (IRT model)
Starting point
Item selection algorithm (CAT algorithm)
Scoring on-the-fly method
Termination rules
And
Item bank protection / overexposure
Content Balancing
![Page 6: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/6.jpg)
0.0
0.2
0.4
0.6
0.8
1.0
-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0
Probab
ility
Theta
Start the test:1. Ask first question, e.g.
of medium difficulty2. Correct!3. Score it4. Select next item with a
difficulty around the most likely score (or with
the max information)
5. And so on…. Until the stopping rule is reached
Most likely score
Difficulty
Correct responseIncorrect response
Normal distribution
![Page 7: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/7.jpg)
Maths ability
Standard test to assess Kumamon
= A question from our test
2+2 1134 x 16
![Page 8: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/8.jpg)
Maths ability
= A question from our test
Kumamon’s ability
Standard test to assess Kumamon
![Page 9: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/9.jpg)
Maths ability
Computer adaptive test to assess Kumamon
= A question from our test
8 x 4
![Page 10: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/10.jpg)
Maths ability
Computer adaptive test to assess Kumamon
= A question from our test
![Page 11: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/11.jpg)
Maths ability
Computer adaptive test to assess Kumamon
= A question from our test
182 + 427
![Page 12: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/12.jpg)
Maths ability
Computer adaptive test to assess Kumamon
= A question from our test
![Page 13: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/13.jpg)
Maths ability
Computer adaptive test to assess Kumamon
= A question from our test
1134 x 16
![Page 14: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/14.jpg)
Maths ability
Computer adaptive test to assess Kumamon
= A question from our test
![Page 15: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/15.jpg)
Maths ability
Computer adaptive test to assess Kumamon
= A question from our test
1712 + 3218
![Page 16: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/16.jpg)
Maths ability
Computer adaptive test to assess Kumamon
= A question from our test
![Page 17: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/17.jpg)
Maths ability
Computer adaptive test to assess Kumamon
= A question from our test
204 x 16
![Page 18: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/18.jpg)
Maths ability
Computer adaptive test to assess Kumamon
= A question from our test
![Page 19: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/19.jpg)
Maths ability
Computer adaptive test to assess Kumamon
= A question from our test
Kumamon’s ability
![Page 20: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/20.jpg)
Maximum Fisher information (MFI)◦ Obtain a current ability estimate
◦ Select next item that maximises information around the current ability estimate
Urry’s method (bOpt; in 1PL equals MFI)◦ Obtain a current ability estimate
◦ Select next item with a difficulty closest to the current one
Other methods:◦ Minimum expected posterior variance (MEPV)
◦ Maximum likelihood weighted information (MLWI)
◦ Maximum posterior weighted information (MPWI)
◦ Maximum expected information (MEI)
![Page 21: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/21.jpg)
Randomesque approach (Kingsbury & Zara, 1989)◦ Select >1 next best item
◦ Randomly choose from this set
Embargo on overexposed items
Location / Name / IP address rules
Large item bank
Regularly updated item bank
Kingsbury, G. G., and Zara, A. R. (1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2, 359-375.
![Page 22: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/22.jpg)
Ascertain that all subgroups of items are used equally
Example: ◦ Arithmetic, Algebra and Geometry in a math test◦ Different domains in an intelligence test◦ Emotion recognition test
Multidimentional CAT
![Page 23: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/23.jpg)
Test length (e.g.., 20 items, 15 items)
Test time (5 minutes)
Reliability of theta estimate (standard error)
Other, clever stuff
![Page 24: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/24.jpg)
𝑟𝑒𝑙𝑖𝑎𝑏𝑖𝑙𝑖𝑡𝑦 = 1 − 𝑆𝐸2
![Page 25: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/25.jpg)
Alpha(0.90) = SE(0.32)
Alpha(0.80) = SE(0.45)
Alpha(0.70) = SE(0.55)
![Page 26: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/26.jpg)
1. The pool of available items is searched for the optimal item, based on the current estimate of the examinee's ability
2. The chosen item is presented to the examinee, who then answers it correctly or incorrectly
3. The ability estimate is updated, based upon all prior answers
4. Steps 1–3 are repeated until a termination criterion is met
![Page 27: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/27.jpg)
• Efficiency – how many items do I need to ask before I get to a certain level of precision
• Precision – How precise can my measurement be
![Page 28: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/28.jpg)
• What do we need for CAT –
Item information (questions, scoring keys)Item parameters Item selection method Scoring algorithm Stopping rule Others ……
![Page 29: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/29.jpg)
catR package
![Page 30: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/30.jpg)
Women’s Mobility◦ Item 1Go to any part of the village/town/city.
◦ Item 2Go outside the village/town/city.
◦ Item 3Talk to a man you do not know.
◦ Item 4Go to a cinema/cultural show.
◦ Item 5Go shopping.
◦ Item 6Go to a cooperative/mothers' club/other club.
◦ Item 7Attend a political meeting.
◦ Item 8Go to a health centre/hospital.
![Page 31: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/31.jpg)
library(ltm)
my2pl<-ltm(Mobility~z1)
plot(my2pl,type="IIC")
![Page 32: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/32.jpg)
![Page 33: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/33.jpg)
require(catR)c<-coef(my2pl)itemBank <- cbind(c[,2], c[,1], 0, 1)
![Page 34: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/34.jpg)
Choose the item to start with:
max info around average?plot(my2pl, type = "IIC")
plot(my2pl, type = "IIC", items=4)
Random one?
![Page 35: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/35.jpg)
items_administered<-c(4)
responses<-c(1)
it<-itemBank[items_administered, 1:4,drop=F]
theta<-thetaEst(it, responses)
sem<-semTheta(theta,it)
q<-nextItem(itemBank, theta=theta,out=items_administered)
q$item
![Page 36: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/36.jpg)
![Page 37: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/37.jpg)
Introduction
Items
Feedback
Bank
Parameters
Theta
SEM
LogicResponses
![Page 38: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/38.jpg)
Concerto hosting website◦ https://hosting.concertoplatform.com/user/registration
Sign up and log in
Create your own server
Start your Concerto experience
![Page 39: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/39.jpg)
Name
URL
Node:◦ info
◦ questionnaire
◦ CAT
◦ form (save_data)
◦ feedback
![Page 40: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/40.jpg)
Basic questionnaire
CES-D scale (The Center for Epidemiologic Studies Depression Scale; Radloff, 1977)◦ 20 items
◦ 4 response options
◦ Score above 16 indicates depression
https://concertotest.com/luning/SSRMC/test/cesd
Radloff, L. S. (1977). The CES-D scale: A self-report depression scale for research in the general population. Applied psychological measurement, 1(3), 385-401.
![Page 41: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/41.jpg)
![Page 42: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/42.jpg)
CAT – dichotomous
Women’s Mobility◦ 8 items in the item bank
◦ Item selection: MFI
◦ Scoring: BM
◦ Stopping: 3 items
◦ Randomesque: 1
◦ Content balancing: no
◦ Feedback:
score$score<-round(score$theta*15+100,0)
![Page 43: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/43.jpg)
faceiq.icar-project.com◦ Adaptive face detection test
◦ Adaptive emotion recognition test
◦ Adaptive abstract reasoning test
◦ And more ……
![Page 44: Dr Luning Sun Psychometrics Module, Lecture 3 Social ... · Dr Luning Sun Psychometrics Module, Lecture 3 Social Sciences Research Methods Centre | SSRMC ... Introduction Items Feedback](https://reader030.fdocuments.us/reader030/viewer/2022040116/5ee07c40ad6a402d666ba928/html5/thumbnails/44.jpg)
Any questions?