INTRO 2 IRT

Tim Croudace

2

Descriptions of IRT• “IRT refers to a set of

mathematical models that describe, in probabilistic terms, the relationship between a person’s response to a survey question/test item and his or her level of the ‘latent variable’ being measured by the scale”

• Fayers and Hays p55– Assessing Quality of Life in

Clinical Trials. Oxford Univ Press: – Chapter on Applying IRT for

evaluating questionnaire item and scale properties.

• This latent variable is usually a hypothetical construct [trait/domain or ability] which is postulated to exist but cannot be measured by a single observable variable/item.

• Instead it is indirectly measured by using multiple items or questions in a multi-item test/scale.

3

The data:0000100000010010100110100011101101001100010101101101111001111111

n4776312

1507

32114

2319413

37812

1694531

Sources of knowledge : q1 radio q2 newspapers q3 reading q4 lectures A single latent dimension Z Normal (mean 0; std dev =1 ) so Var= 1 too!

logit {πhi} = αh 0 + αh 1zi

αh0 α10

αh0 α40

αh1 α21

4

Simple sum scores (n=1729 new individual values)

0 0 0 0 [n] Total score 0 0 0 0 477 0 477 zeros added to data set (new column) 1 0 0 0 63 1 0 0 0 1 12 1 0 0 1 0 150 1 1 0 0 1 7 2 1 0 1 0 32 2 0 0 1 1 11 2 1 0 1 1 4 3 0 1 0 0 231 1 1 1 0 0 94 2 0 1 0 1 13 2 0 1 1 0 378 2 1 1 0 1 12 3 1 1 1 0 169 3 0 1 1 1 45 3 1 1 1 1 31 4

5

Binary Factor / Latent Trait Analysis Results: logit-probit model

F

U1 U2 U3 Up. . .

Warming up to this sort of thing … soon ….

2 items with similar thresholds and similar slopes3 items with different thresholds but similar slopes

6

The key concept … latent factor models for constructs underpinning multiple

binary (0/1) responses• … based on innovations in educational testing

and psychometric statistics > 50 years old• Same models used in educational testing with

correct incorrect answers can be applied to symptom present / absent data (both binary)

• Extensions to ordinal outcomes (Likert scales)• Flexibility in parametric form available• Semi- and non-parametric approaches too…

7

Binary IRT : The A B C D of it

8

Linear vs non-linear regression of response probability on latent variable

x-axis score on latent construct being measured

y-axis

prob

of

response

(“Yes”)

on a

simple

binary

(Yes/No)

scale

item

Adapted without permissionfrom aslide by Prof H Goldstein

9

Ordinal IRT : The A B C D of GRM

10

IRT models• Simplest case of a latent trait analysis…– Manifest variables are binary: only 2 distinctions are made

• these take 0/1 values– Yes / No– Right / Wrong– Symptom present / absent

• Agree / disagree distinctions for attitudes more likely to be ordinal [>2 response categories] .. see next lecture IRT 2 on Friday

• For scoring of individuals – (not parameter estimation for items)

• it is frequently assumed that the UNOBSERVED (latent) variable < the latent factor / trait> • is not only continuous but normally distributed

– [or the prior dist’n is normal but the posterior dist’n may not be]

11

IRT for binary data The most commonly used model was developed by

Lord-Birnbaum model (Lord, 1952; Birnbaum, )

2-parameter logistic [a.k.a. the logit-probit model; Bartholomew (1987)]

• The model is essentially a non-linear single factor model– When applied to binary data, the traditional linear factor model is only an

approximation to the appropriate item response model• sometimes satisfactory, but sometimes very poor (we can guess when)

• Some accounts of Item Response Theory make it sound like a revolutionary & very modern development

• this is not true!– It should not replace or displace classical concepts, and has suffered from

being presented and taught as disconnected from these– A unified treatment can be given that builds one from the other (McDonald,

1999) but this would be a one term course on its own

12

What IRT does IRT models provide a clear statement [picture!] of the performance of each item in the scale/test

and

how the scale/test functions, overall,for measuring the construct of interest in the study population

The objective is to model each item by estimating the properties describing item performance characteristics

hence Item Characteristic Curve or Symptom Response Function.

13

Very bland (but simple) example

• Lombard and Doering (1947) data• Questions on cancer knowledge with four addressing

the source of the information• Fitting a latent variable model might be proposed as

a way of constructing a measure of how well informed an individual is about cancer

• A second stage might relate knowledge about cancer to knowledge about other diseases or general knowlege

14

Very bland (but simple) example

• Lombard and Doering (1947) data• Questions on cancer knowledge with four

addressing the source of the information– radio– newspapers– (solid) reading (books?)– lectures

• 2 to the power 4 i.e. 16 possible response patterns from 0000 to 1111

15

Data

• Lombard and Doering (1947) data

• 2 to the power 4 – i.e. 16 possible response

patterns (all occur)– with more items this is neither

likely nor necessary

– frequency shown for• 0000 to 1111• frequency is the number with

each item response pattern

0000100000010010100110100011101101001100010101101101111001111111

n4776312

1507

32114

2319413

37812

1694531

16

The data:0000100000010010100110100011101101001100010101101101111001111111

n4776312

1507

32114

2319413

37812

1694531

Sources of knowledge : q1 radio q2 newspapers q3 reading q4 lectures A single latent dimension Z Normal (mean 0; std dev =1 ) so Var= 1 too!

logit {πhi} = αh 0 + αh 1zi

αh0 α10

αh0 α40

αh1 α21

17

Basic objectives of modelling• When multiple items are applied in a test / survey can use

latent variable modelling to– explore inter-relationships among observed responses– determine whether the inter-relationships can be explained by a small

number of factors

– THEN , to assign a SCORE to each individual each on the basis of their responses

– Basically to rank order (arrange) or quantify (score) survey participants, test takers, individuals who have been studied» CAN BE THOUGHT OF AS ADDING A NEW SCORE TO YOUR DATASET FOR

EACH INDIVIDUAL• this analysis will also help you to understand the properties of each

item, as a measure of the target construct (what properties?)» GRAPHICAL REPRESENTATION IS BEST

18

Item Properties that we are interested in are captured graphically by so called Item Characteristics

Curves (ICCs)

19

Item/Symptom & Test/Scale INFORMATION– is useful and necessary to examine score precision (the

accuracy of estimated scores)– we are interested in this for different individuals

(individuals with different score values) – by inspecting the amount of information about each

score level, across the score range (range of estimated scores) we are identifying variations in measurement precision (reliable of individual’s estimated scores)

– this enables us to make statements about the effective measurement range of an instrument in an population

20

e.g. Item Characteristics Curves

21

Item information functions- add them together to get TIF

beware y axis scaling : not all the same

22

Test Information Function

23

Item information functions- shown alongside their ICCs

beware y axis scaling : not all the same

1111

Item Characteristics CurvesItem Characteristics Curves

0.14

0.40

3.0

0.14

24

1 / Sqrt [Information] = s.e.mInfo Sqrt(Info) 1/(sqrt(Info)1 1.0 1.02 1.4 0.73 1.7 0.64 2.0 0.55 2.2 0.46 2.4 0.47 2.6 0.48 2.8 0.49 3.0 0.310 3.2 0.311 3.3 0.312 3.5 0.3

Standard error of measuremenr is not constant (U-shaped, not symmetrical)

25

Approximate reliability

• Reliability= 1 – 1/[Info]

= {1 – 1 / [1 / (s.e.m ^2) }

s.e.m. = standard error of measurement

26

Back to the Data

• Lombard and Doering (1947) data

• 2 to the power 4 – i.e. 16 possible response

patterns (all occur)– with more items this is neither

likely nor necessary

– frequency shown for• 0000 to 1111• frequency is the number with

each item response pattern

0000100000010010100110100011101101001100010101101101111001111111

n4776312

1507

32114

2319413

37812

1694531

What would be the easiest thing to do with these numbers; to score the patterns..?

27

Answer ..0000100000010010100110100011101101001100010101101101111001111111

What would be the easiest thing to do with these numbers; to score the patterns..?

• Simply add them up

28

Simple sum scores (n=1729 new individual values)

0 0 0 0 [n] Total score 0 0 0 0 477 0 477 zeros added to data set (new column) 1 0 0 0 63 1 0 0 0 1 12 1 0 0 1 0 150 1 1 0 0 1 7 2 1 0 1 0 32 2 0 0 1 1 11 2 1 0 1 1 4 3 0 1 0 0 231 1 1 1 0 0 94 2 0 1 0 1 13 2 0 1 1 0 378 2 1 1 0 1 12 3 1 1 1 0 169 3 0 1 1 1 45 3 1 1 1 1 31 4

29

Weighted [by discriminating power] scores 0 0 0 0 [n] Total Factor Component [weighted by alpha h 1]

score score score 0 0 0 0 477 0 -0.98 0 = 0 1 0 0 0 63 1 -0.68 0.720.72 0.720.72 0 0 0 1 12 1 -0.67 0.770.77 0.770.77 0 0 1 0 150 1 -0.46 1.341.34 1.341.34 1 0 0 1 7 2 -0.41 0.720.72++ 0.770.77 1.481.48 1 0 1 0 32 2 -0.23 0.720.72 +1.341.34 2.062.06 0 0 1 1 11 2 -0.22 1.341.34++ 0.770.77 2.102.10 1 0 1 1 4 3 0.0 0.720.72++ 1.341.34++ 0.770.77 2.822.82 0 1 0 0 231 1 0.16 3.403.40 3.403.40 1 1 0 0 94 2 0.42 0.720.72++3.403.40 4.124.12 0 1 0 1 13 2 0.43 3.403.40++ 0.770.77 4.164.16 0 1 1 0 378 2 0.66 3.403.40++ 1.341.34 4.744.74 1 1 0 1 12 3 0.72 0.720.72++ 3.403.40++ 0.770.77 4.884.88 1 1 1 0 169 3 0.99 0.720.72++ 3.403.40++1.341.34 5.46 0 1 1 1 45 3 1.02 3.403.40++1.341.34++ 0.770.775.505.50 1 1 1 1 31 4 1.41 0.720.72++3.403.40++1.341.34++0.770.77 6.226.22

0.720.723.403.401.341.340.770.77

3737

Mplus version 4.1Mplus version 4.1 ML Estimate S.E. ML Estimate S.E.

Z by Q1 alpha h 1 0.721 0.093Z by Q1 alpha h 1 0.721 0.093Z by Q2 alpha h 2 3.358 1.035Z by Q2 alpha h 2 3.358 1.035Z by Q3 alpha h 3 1.344 0.167Z by Q3 alpha h 3 1.344 0.167Z by Q4 alpha h 4 0.769 0.145Z by Q4 alpha h 4 0.769 0.145

Variances Z 1Variances Z 1

Compare with Bartholomew (1987) p160Compare with Bartholomew (1987) p1600.72 (0.09) 0.72 (0.09) 3.40 (1.14)3.40 (1.14)1.34 (0.17)1.34 (0.17)0.77 (0.15)0.77 (0.15)

30

The data:0000100000010010100110100011101101001100010101101101111001111111

n4776312

1507

32114

2319413

37812

1694531

Sources of knowledge : q1 radio q2 newspapers q3 reading q4 lectures A single latent dimension Z Normal (mean 0; std dev =1 ) so Var= 1 too!

logit {πhi} = αh 0 + αh 1zi

αh0 α10

αh0 α40

αh1 α21

31

Weighted [by discriminating power] scores 0 0 0 0 [n] Total Factor Component [weighted by alpha h 1]

score score score 0 0 0 0 477 0 -0.98 0 = 0 1 0 0 0 63 1 -0.68 0.720.72 0.720.72 0 0 0 1 12 1 -0.67 0.770.77 0.770.77 0 0 1 0 150 1 -0.46 1.341.34 1.341.34 1 0 0 1 7 2 -0.41 0.720.72++ 0.770.77 1.481.48 1 0 1 0 32 2 -0.23 0.720.72 +1.341.34 2.062.06 0 0 1 1 11 2 -0.22 1.341.34++ 0.770.77 2.102.10 1 0 1 1 4 3 0.0 0.720.72++ 1.341.34++ 0.770.77 2.822.82 0 1 0 0 231 1 0.16 3.403.40 3.403.40 1 1 0 0 94 2 0.42 0.720.72++3.403.40 4.124.12 0 1 0 1 13 2 0.43 3.403.40++ 0.770.77 4.164.16 0 1 1 0 378 2 0.66 3.403.40++ 1.341.34 4.744.74 1 1 0 1 12 3 0.72 0.720.72++ 3.403.40++ 0.770.77 4.884.88 1 1 1 0 169 3 0.99 0.720.72++ 3.403.40++1.341.34 5.46 0 1 1 1 45 3 1.02 3.403.40++1.341.34++ 0.770.775.505.50 1 1 1 1 31 4 1.41 0.720.72++3.403.40++1.341.34++0.770.77 6.226.22

0.720.723.403.401.341.340.770.77

3737

Mplus version 4.1Mplus version 4.1 ML Estimate S.E. ML Estimate S.E.

Z by Q1 alpha h 1 0.721 0.093Z by Q1 alpha h 1 0.721 0.093Z by Q2 alpha h 2 3.358 1.035Z by Q2 alpha h 2 3.358 1.035Z by Q3 alpha h 3 1.344 0.167Z by Q3 alpha h 3 1.344 0.167Z by Q4 alpha h 4 0.769 0.145Z by Q4 alpha h 4 0.769 0.145

Variances Z 1Variances Z 1

Compare with Bartholomew (1987) p160Compare with Bartholomew (1987) p1600.72 (0.09) 0.72 (0.09) 3.40 (1.14)3.40 (1.14)1.34 (0.17)1.34 (0.17)0.77 (0.15)0.77 (0.15)

32

Something a little more subtle

• Simple sum scores assumes all item responses equally useful at defining the construct– may not be the case

• If items are differentially important– different discriminating power with respect to what we

are measuring, we might want to take that into accounf• How? Weighted sum scores [Component scores]

– weighted by what?» weighted by the estimates (factor loading type parameter) from

a latent variable model» [latent trait model with a single latent factor]

33

Weightedscores

3737

Mplus version 4.1Mplus version 4.1 ML Estimate S.E. ML Estimate S.E.

Z by Q1 alpha h 1 0.721 0.093Z by Q1 alpha h 1 0.721 0.093Z by Q2 alpha h 2 3.358 1.035Z by Q2 alpha h 2 3.358 1.035Z by Q3 alpha h 3 1.344 0.167Z by Q3 alpha h 3 1.344 0.167Z by Q4 alpha h 4 0.769 0.145Z by Q4 alpha h 4 0.769 0.145

Variances Z 1Variances Z 1

Compare with Bartholomew (1987) p160Compare with Bartholomew (1987) p1600.72 (0.09) 0.72 (0.09) 3.40 (1.14)3.40 (1.14)1.34 (0.17)1.34 (0.17)0.77 (0.15)0.77 (0.15)

2626

CancerKnowledge zi

The data:0000100000010010100110100011101101001100010101101101111001111111

n4776312150732114

2319413378121694531

Sources of knowledge : q1 radio q2 newspapers q3 reading q4 lectures A single latent dimension Z Normal (mean 0; std dev =1 ) so Var= 1 too!

logit {πhi} = αh 0 + αh 1zi

αh0 α10

αh0 α40

αh1 α21

WeightsWeightsalpha h 1 alpha h 1 parametersparameters

Q1Q1 0.720.72Q2Q2 3.403.40Q3Q3 1.341.34Q4Q4 0.770.77These numbers These numbers related to the related to the slopes of the S’sslopes of the S’s

2020

Item information functionsItem information functions-- shown alongside their shown alongside their ICCsICCs

beware y axis scaling : not all the same

1111

Item Characteristics CurvesItem Characteristics Curves

0.14

0.40

3.0

0.14

34

Estimated component scores (weighted values)

0 0 0 0 [n] Total Factor Component [weighted by alpha h 1]

score score score 0 0 0 0 477 0 -0.98 0 = 0 1 0 0 0 63 1 -0.68 0.720.72 0.720.72 0 0 0 1 12 1 -0.67 0.770.77 0.770.77 0 0 1 0 150 1 -0.46 1.341.34 1.341.34 1 0 0 1 7 2 -0.41 0.720.72++ 0.770.77 1.481.48 1 0 1 0 32 2 -0.23 0.720.72 +1.341.34 2.062.06 0 0 1 1 11 2 -0.22 1.341.34++ 0.770.77 2.102.10 1 0 1 1 4 3 0.0 0.720.72++ 1.341.34++ 0.770.77 2.822.82 0 1 0 0 231 1 0.16 3.403.40 3.403.40 1 1 0 0 94 2 0.42 0.720.72++3.403.40 4.124.12 0 1 0 1 13 2 0.43 3.403.40++ 0.770.77 4.164.16 0 1 1 0 378 2 0.66 3.403.40++ 1.341.34 4.744.74 1 1 0 1 12 3 0.72 0.720.72++ 3.403.40++ 0.770.77 4.884.88 1 1 1 0 169 3 0.99 0.720.72++ 3.403.40++1.341.34 5.46 0 1 1 1 45 3 1.02 3.403.40++1.341.34++ 0.770.775.505.50 1 1 1 1 31 4 1.41 0.720.72++3.403.40++1.341.34++0.770.77 6.226.22

??????????0.720.723.403.401.341.340.770.77

35

But the bees knees are..

• The estimated factor scores from the model• Not just some simple sum or unweighted or

weighted items• Takes into account the proposed score distribution

(gaussian normal) and the estimated model parameters (but not the fact that they are estimates rather than known values) and more besides (when missing data are present)

… the estimated factor scores

36

A graphical and interactiveintroduction to IRT

• Play with the key features of IRT models

• www2.uni-jena.de/svw/metheval/irt/VisualIRT.pdf

37

a b (see) [2 parameter IRT model]

• VisualIRT (pdf)– Page

• VisualIRT (pdf)– Page

Individual’s score = new ruler valueAny hypothetical latent variable [factor/trait] continuum expressed in a z-score metric (gaussian normal (0,1)

Item propertiesslope = item discriminationlocation = item commonality [difficulty/prevalance/ severity]

38

IRT Resources• A visual guide to Item Response Theory

– I. Partchev• Introduction to RIT,

– R.Baker• http //ericae.net/irt/baker/toc.htm

• An introduction to modern measurement theory– B Reeve

• Chapter in Fayers and Machin QoL book– P Fayers

• ABC of Item Response Theory– H Goldstein

• Moustaki papers, and online slides (FA at 100)• LSE books (Bartholomew, Knott, Moustaki, Steele)

39

Item Response Theory Books

Applications of Item Response Theory to Practical Testing Problems Frederick M. Lord. 274 pages. 1980. Applying The Rasch Model Trevor G. Bond and Christine M. Fox 255 pages. 2001. Constructing Measures: An Item Response Modeling Approach      Mark Wilson. 248 pages. 2005. The EM Algorithm and Related Statistical Models      Michiko Watanabe and Kazunori Yamaguchi. 250 pages. 2004. Essays on Item Response Theory Edited by Anne Boomsma, Marijtje A.J. van Duijn, Tom A.A. Snijders. 438 pages. 2001. Explanatory Item Response Models: A Generalized Linear and Nonlinear Approach      Edited by Paul De Boeck and Mark Wilson. 382 pages. 2004. Fundamentals of Item Response Theory Ronald K. Hambleton, H. Swaminathan, and H. Jane Rogers. 184 pages. 1991. Handbook of Modern Item Response Theory Edited by Wim J. van der Linden and Ronald K. Hambleton. 510 pages. 1997. Introduction to Nonparametric Item Response Theory Klaas Sijtsma and Ivo W. Molenaar. 168 pages. 2002. Item Response Theory Mathilda Du Toit. 906 pages. 2003. Item Response Theory for Psychologists Susan E. Embretson and Steven P. Reise. 376 pages. 2000. Item Response Theory: Parameter Estimation Techniques (Second Edition, Revised and Expanded w/CD) Frank Baker and Seock-Ho Kim. 495 pages. 2004. Item Response Theory: Principles and Applications Ronald K. Hambleton and Hariharan Swaminathan. 332 pages. 1984. Logit and Probit: Ordered and Multinomial Models Vani K. Borooah. 96 pages. 2002. Markov Chain Monte Carlo in Practice      W.R. Gilks, Sylvia Richardson, and D.J. Spiegelhalter. 512 pages. 1995. Monte Carlo Statistical Methods      Christian P. Robert and George Casella. 645 pages. 2004. Polytomous Item Response Theory Models      Remo Ostini and Michael L. Nering. 120 pages. 2005. Rasch Models for Measurement David Andrich. 96 pages. 1988. Rasch Models: Foundations, Recent Developments, and Applications Edited by Gerhard H. Fischer and Ivo W. Molenaar. 436 pages. 1995. The Sage Handbook of Quantitative Methodology for the Social Sciences Edited by David Kaplan. 511 pages. 2004. Test Equating, Scaling, and Linking: Methods and Practices (Second Edition) Michael J. Kolen and Robert L. Brennan. 548 pages. 2004.