FIT ANALYSIS IN RASCH MODEL

University of OstravaCzech republic

26-31, March, 2012

Two questions relating fit analysis:

How to assess fit the real data to the chosen model of measurement?

What should we do if the data don’t fit the model?

3

Two approaches to assessing fit

• Fit statistics, based on standardized residuals

• Chi – square criteria, assesssing the closeness of model and empirical characteristic curves

Let ani be a scored response for the interaction of

the person n, n=1,…,N, and the item i, i=1,…,I (the 1’s and 0’s in dichotomous case);

xni – standardized residual;

Pni – the probability of a correct response for person n on item i.

( )

( )ni ni ni ni

ni

ni ni ni

a M a a Px

D a P q

( ) , ( )ni ni ni ni niM a P D a P q

Properties of standardized residuals xni

M(xni)=0, D(xni)=1; In theory values vary in range (-∞,+∞); in practice values

usually range from -10 to +10. If Pni=0,99 and ani=0 than xni≈-9,94 ; similarily if Pni=0,01 and ani=1, than xni≈ 9,94 ;

Positive values represent correct responses: xni>0 if ani=1;

Negative values represent incorrect responses: xni<0 if ani=0; The values are assumed to have normal distribution N

(0,1).

Dependence of the standardized residual on

the difference θn - δi

Statistics xni can have positive and negative values, so summing residuals across items and persons is informative. The solution of this problem is squaring standadized residuals:

22 ( )ni ni

ni nini ni

a Py x

P q

Properties of yni

Statistics yni has only non negative values;

In theory values yni vary in range (0,+∞); in practice values usually range from 0 to 100, at that most of values are in range (0,2);

The expected value and variance of yni are: M(yni)=1, D(yni)=2.

Statistics yni = xni2 can be evaluated as having χ2

distribution with one degree of freedom.

Distribution features of statistics yni

These squared standardized residuals are only approximate chi-squares

Statistics yni would have exact χ2 distribution, if the following

conditions were completed:

1) ani was continuous variable (rather than discrete);

2) exact values of the possibity Pni were known (indeed only

estimates of this possibility are known that are based on parameter estimates);

3) The data fit the measurement model.

Person fit statistics for an examinee n:

This statistics is a sum of all values yni for the examinee across all items. It has approximately χ2 distribution with df=I.

A problem: for each degree of freedom there is a different crirical value, so no single critical value can be used

2

1

I

n nii

y

Item fit statistics for an item i:

This statistics is a sum of all values yni for the item across all examinees. It has approximately χ2 distribution with df=N.

The same problem: for each degree of freedom there is a different crirical value, so no single critical value can be used.

2

1

N

i nin

y

A possible solution of the critical value problem

Transformating the chi-squre statistics into a mean –square by dividing the chi-square by its degrees of freedom (Outfit MNSQ in Winsteps):

Person-fit statistics for an examinee n:

Item-fit statistics for an item i :

2(1)

1 1

1 1 ( )I Ini ni

n nii i ni ni

a PU y

I I P q

2(1)

1 1

1 1 ( )N Nni ni

i nin n ni ni

a PU y

N N P q

Properties of mean square statistics

Un(1) and Ui

(1)

Statistics Un(1) и Ui

(1) vary in range from [0,+∞);

Expected value is 1: M(Un(1))=M(Ui

(1))=1 .

Statistics Un(1) и Ui

(1) are very sensitive to outliers (unexpected correct or incorrect responses).

To counteract this sentivity to outliers the weighted versions of person-fit and item-fit statistics were developed:

2

(2) 1 1

1 1

( ) ( )I I

ni ni ni nii i

n I I

ni ni ni nii i

y D a a PU

P q P q

2

(2) 1 1

1 1

( ) ( )N N

ni ni ni nin n

i N N

ni ni ni nin n

y D a a PU

P q P q

Properties of weighted fit statistics

Each squared standardized residual yni is weighted by

the dispersion D(ani)=Pni·qni before it is summed. The

value D(ani) is the least for the items difficulty of which

don’t correspond to ability level of the examinee. Thus, contribution of these items to statistics Un

(2) and Ui(2) will

be reduced. Statistics Un

(2) and Ui(2) vary in range [0,+∞) and have

expected value of 1.

Total fit statistics

The MNSQ statistics Un(1), Un

(2), Ui(1) и Ui

(2) are called total fit statistics.

Observed values of MNSQ statistics Un(1), Un

(2), Ui(1) and Ui

(2) are the more closed to expected value 1, the more the real data fir the Rasch model.

If the real data don’t fit the model, observed values of MNSQ statistics will differ from 1.

The problem with critical values of total fit statistics

Critical values of mnsq statistics are different for different samples and different tests

The distributions of mnsq statistics are approximate and, as a rule, empirical distributions differ from the theoretical ones

So we can not use the same critical values for mnsq statistics defined from their theoretical distribution.

Interpretation of total fit statistics values

The value of item-fit ststistics of 1.3 can be interpreted as indicating noise in the data in the item response pattern: there is 30% more variation in the data than it was predicted by the modal (underfit)

The value of item-fit ststistics of 0.8 can be interpreted as indicating Guttman pattern: there is 20% less variation in the data than it was predicted by the modal (overrfit)

Recomendations on interpretation of fit statistics

Transformating the mnsq statistics to standardized form (zstd in Winsteps)

There are two kinds of transformation that converts the mean-square to an approximate t-statistics:

Logarithm transformation

Cube-root transformation

1(ln 1)

8

kt U U

3 3 ( )( 1)

( ) 3

D Ut U

D U

Properties of standardized fit statistics

Standardized fit statistics t have approximately normal distribution N(0,1),

So with this statistics common critical values can be developed: for significance level 0,05 the acceptable values are in the range of (-2,+2)

Simulation studies have shown that the standardized fit statistics have more consistent distributional properties in the face of varying sample size than do the mnsq ststistics

Statistics for Item Fit Analysis

Total item-fit statistics Ui(1) (Mnsq Outfit)

Standardized item-fit statistics ti(1) (Zstd Outfit)

Weighted total item-fit statistics Ui(2) (Mnsq Infit)

Standardized weighted item-fit statistics ti(2) (Zstd Infit)

A combination of item-fit statistics provides the best opportunity to detect poor fit items

Recommended ranges for the total item-fit statistics for different test types

Some reasons for poor item-fit (underfit - mnsq fit-statistics values are more than 1.2; zstd fit-statistics values are more than +2)

Test is not unidimensional Bed items (mistakes with keys, bed distractors in

MC items, item mistakes, etc.) Particular features of examinee behavior

(guessing, carelessness, etc.)

About items that are overfit (mnsq fit-statistics values are less than 0.8; zstd fit-statistics values are less than -2)

Patterns of these items are too perfect (Guttman). It is not in agreement with a probabilistic nature of the model

Too perfect patterns of these items can be consequence of more high discrimination of these items.

A possible reason for it is violation of local independence: examinee’s response to one item affects his response to another one. Such items don’t contribute into measurement of ability.

26

The second approach to item fit: chi-square statistics

2

2

1

( )( )

sn ni

i nn i n n

l lf

f l

df = s - 1

Problems with chi-square approach use

How many points for sample dividing should we take?

3 or 5 or 10 or more?

In many test situations this statistics has low efficiency

In Rasch measurement the preference gives to item-fit statistics described above. In addition software Winsteps produces confidence intervals for ICC based on chi-square approach

Example of fit analysis: test description

Test contains 50 items which are divided into three parts: part 1 has 32 MC items with 4 options (А1 – А32); part 2 has 12 opened items with a short answer (В1 – В12); part 3 has 6 items with free-constructed response (С1 – С6)/

Most of items were scored dichotomously, and a few items were scored polytomously.

The total sample size is 655.

Item statistics

ICC of a poor fit item

Dicriminating power of this item is lower than other items have

ICC of a good item

ICCs of the first 9 items (А1-А9)

ICC of an item with two fit statistics with values below the left critical value

Discriminating power of the item is higher than other items have

Statistics for Person Fit Analysis

Total person-fit statistics Un(1) (Mnsq Outfit)

Standardized person-fit statistics tn(1) (Zstd Outfit)

Weighted total person-fit statistics Un(2) (Mnsq Infit)

Standardized weighted person-fit statistics tn(2) (Zstd Infit)

A combination of person-fit statistics provides the best opportunity to detect poor fit items

Some reasons of poor person fit:

Bed items Personal features of the examinee Gap in the examinee knowledge Violation of test conditions

Example of person analysis

Analysis of examinewe profiles