How Should We Assess the Fit of Rasch -Type Models?
-
Upload
kimberly-weaver -
Category
Documents
-
view
29 -
download
1
description
Transcript of How Should We Assess the Fit of Rasch -Type Models?
![Page 1: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/1.jpg)
How Should We Assess the Fit of Rasch-Type Models?
Approximating the Power of Goodness-of-fit Statistics in Categorical Data Analysis
Alberto Maydeu-OlivaresRosa Montano
![Page 2: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/2.jpg)
OutlineIntroductionRasch-Type Models for Binary DataRationale of Goodness-of-Fit Statistics
◦Full Picture◦M2, R1 and R2
Estimating the PowerEmpirical Comparison of R1, R2 and M2Numerical ExamplesDiscussion and Conclusion
![Page 3: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/3.jpg)
IntroductionTwo properties of Rasch-Type models
◦ Sufficient statistics◦ Specific objectivity
Estimation methods◦ Specific for Rasch-Type models (CML)◦ General procedures (MML via EM)
Goodness-of-fit testing procedures◦ Specific to Rasch-Type models◦ General to IRT or multivariate discrete data
models
![Page 4: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/4.jpg)
IntroductionCompare the performance of certain
goodness-of-fit statistics to test Rasch-Type models in MML via EM◦ Binary data◦ 1PL (random effects)
R1 and R2 for 1PLM2 for multivariate discrete data
![Page 5: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/5.jpg)
Rasch model and 1PL
Fixed effects◦ The distribution of ability is not specified
Random effects◦ Specify a standard normal distribution for
ability◦ The less restrictive definition of specific
objectivity still hold
![Page 6: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/6.jpg)
Rationale
(000)
(100)
(010)
(001)
(110)
(101)
(011)
(111)
1 0 1 0 0 0 0 0 0
2 0 0 0 0 0 1 0 0
3 0 0 0 0 1 0 0 0
Marginal Total for each cell > 5
1. High-dimensional contingency table
C = 2^n cells which n is the number of items.For example, 20 items testC = 2^20 = 1048576 cellsTo fulfill the rule of thumb >5, at least 1048576*5 sample size is needed.
![Page 7: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/7.jpg)
(000)
(100)
(010)
(001)
(110)
(101)
(011)
(111)
1 0 1 0 0 0 0 0 0
2 0 0 0 0 0 1 0 0
…
Marginal Total 10 17 21 32 15 8 12 19 134
Observed proportion 0.07
ProbabilityUnder Model
0.11
2.
![Page 8: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/8.jpg)
When order r = 2, Mr -> M2M2 used the univariate and bivariate
informationThe degree of freedom is It is statistics of choice for testing IRT models
3. Limited information approach (M2)
Pooling cells of the contingency table
![Page 9: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/9.jpg)
Degree of freedom is n(n-2)Specific to the monotone increasing and
parallel item response functions assumptions
3. Limited information approach (R1 and R2)
Degree of freedom is (n(n-2)+2)/2Specific to the unidimensionality assumption
![Page 10: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/10.jpg)
Estimating the Asymptotic Power RateUnder the sequence of local
alternatives◦The noncentrality parameter of a chi-
square distribution can be calculated given the df for M2, R1 and R2
The Kullback-Leibler discrepancy function can be used◦The minimizer of DKL is the same as
the maximizer of the maximum likelihood function between a “true” model and a null model
![Page 11: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/11.jpg)
Study 1: Accuracy of p-values under correct model
df = Mean; df = ½ Var Another Study by Montano (2009), M2 is better than
R1 and the discrepancies between the empirical and asymptotic rate were not large.
Group the sum scores ->
![Page 12: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/12.jpg)
The degree of freedom is also adjustAn iterative procedureWhen appropriate score ranges are used, the
empirical rejection rate of R1 should be closely match the theoretical rejection rates.
This should be also done in R2
![Page 13: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/13.jpg)
Study 2: Asymptotic Power to reject a 2PL
![Page 14: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/14.jpg)
Study 3: Empirical Power to reject a 2PL
![Page 15: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/15.jpg)
Study 4: Asymptotic Power to reject a 3PL
![Page 16: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/16.jpg)
Study 5: Asymptotic Power to reject a multidimensional model
![Page 17: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/17.jpg)
Empirical Example 1: LSAT 7 Data
The agreement in ordering between value/df ratio and power
![Page 18: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/18.jpg)
Empirical Example 2: Chilean Mathematical Proficiency Data
![Page 19: How Should We Assess the Fit of Rasch -Type Models?](https://reader031.fdocuments.us/reader031/viewer/2022020102/568132ef550346895d99a9e9/html5/thumbnails/19.jpg)
Discussion and ConclusionsGenerally, M2 is more powerful
than R1, R2.That is, the R1 and R2 which
developed specific to Rasch-type models is not superior than the general M2