byby

Dr. Mohammad H. OmarDr. Mohammad H. OmarDepartment of Mathematical SciencesDepartment of Mathematical Sciences

May 16, 2006May 16, 2006

Presented at Statistic Research (STAR) colloquium,Presented at Statistic Research (STAR) colloquium,King Fahd University of Petroleum & Minerals,King Fahd University of Petroleum & Minerals,

Dhahran, Saudi Arabia.Dhahran, Saudi Arabia.

EquatingEquating

Seminar

Dept of Mathematical Sciences King Fahd University of Petroleum and Minerals

Presenter Dr. Mohammad H. Omar Dept of Mathematical Sciences, KFUPM

Title Some Statistics for Equating Multiple Forms of a Test

Topic Statistics in Educational Testing

Audience All KFUPM community are cordially Invited

Date Tuesday, May 16, 2006

Time 1:00 PM

Location Building 5, Smart Classroom # 203

Abstract A single test form can be administered to a group of students. However, with only one

test form administration, less disciplined students are likely to cheat by copying answers from neighbouring student. For test security purposes, many standardised testing programs use more than one test form to circumvent cheating efforts by less than honest students. Although using multiple test forms ensure virtually low scores for cheating students, it introduces one rather potentially big problem of equity in test scores. Equity as defined by Lord (1980) simply means ‘it is a matter of indifference which test form a student take’. That is, which test form an honest student take, his test score will be unaffected. To ensure this, multiple test forms are (1) built to the same test content specifications and (2) built to the same test parameter specifications.

If multiple test forms meet these content and test parameter specifications, equity is not

hard to achieve. Despite best efforts, however, some variations from equity occur in practice. Thus, scores arising from multiple test forms can disadvantage some honest students unless some adjustments in scores are formally made through an educational measurement procedure called ‘test equating’. This procedure attempts at equalizing two score distributions. In this talk, we will discuss what is equating, some equating designs, commonplace equating methods including some statistical procedures, embedded in these procedures, and advantages and disadvantages of these procedures.

Refreshments will be provided

Brief overview of TalkBrief overview of Talk• Test administration usingTest administration using

– Only one formOnly one form– More than one formMore than one form

• Test EquityTest Equity• Steps to ensuring equitySteps to ensuring equity• Conditions for Equated ScoreConditions for Equated Score• Data Collection DesignsData Collection Designs• Equating proceduresEquating procedures

– Illustration of the Equipercentile Equating processIllustration of the Equipercentile Equating process– Use of smoothing techniquesUse of smoothing techniques– Application of equipercentile equating to data collection Application of equipercentile equating to data collection

designdesign– Standard errors of equipercentile equatingStandard errors of equipercentile equating

• Linear equatingLinear equating– Illustration of the Linear Equating processIllustration of the Linear Equating process– Application of linear equating to data collection designApplication of linear equating to data collection design– Standard errors of linear equatingStandard errors of linear equating

• Comparison of equating methodsComparison of equating methods

Test Administration using only one Form

1) Score means the same thing for every student

(1) Dishonest students can copy answers from neighbouring

students.

(2) Scores of dishonest students can be unreliably high

(3) Honest students are disadvantaged by acts of dishonest students.

If cheating doesn’t occur

Test Administration using more than one form

1) Substantially reduce chance for dishonesty & cheating

(1) Some equity issues if test equating is not carried out

2) Honest students are not disadvantaged by acts of dishonest students.

3) Scores of dishonest students are reliably low if cheating occurs

Test EquityTest Equity

Definition (laymen’s definition) :Definition (laymen’s definition) :

Equity Equity ""It is a matter of It is a matter of indifference which test indifference which test

form a student tookform a student took""

Steps to ensuring EquitySteps to ensuring Equity

1.1. Building test forms to the same test Building test forms to the same test content specificationscontent specifications

- Test forms should be interchangeable.Test forms should be interchangeable. No one form should have different No one form should have different

content specifications content specifications than others.than others.

- Test length should be the same.Test length should be the same. No one form should be longer than No one form should be longer than another another Students should not be disadvantaged by Students should not be disadvantaged by taking a longer test form than their peers.taking a longer test form than their peers.

Interchangeable Content?

  Form X

Form Y

Differentiation 80% 20%

Integration 20% 80%

Same length?

 Time Form X Form Y

Required to finish

2 hr 1 hr

Allotted for Administration

1 hr 30 min

Steps to ensuring EquitySteps to ensuring Equity continued…//continued…//

2.2. Building test forms to the same Building test forms to the same test parameter specificationstest parameter specifications

- Test forms should be equally difficultTest forms should be equally difficultStudents should not be disadvantaged by taking Students should not be disadvantaged by taking

test forms that are very difficult test forms that are very difficult compared to compared to what their peers take in the what their peers take in the same same administration.administration.

- Test forms should be equally Test forms should be equally reliable.reliable.

Same Difficulty?

  Form X

Form Y

Percent of student below median of X

50% 70%

Same consistency?

  Form X Form Y

Coefficient alpha

0.70 0.90

Conditions For Equated ScoresConditions For Equated Scores

The purpose of equating is to establish, as nearly as possible, an The purpose of equating is to establish, as nearly as possible, an effective equivalence between raw scores on two test forms. effective equivalence between raw scores on two test forms. Because equating is an empirical procedure, it requires a design Because equating is an empirical procedure, it requires a design for data collection and a rule for transforming scores on one test for data collection and a rule for transforming scores on one test form to scores on another. form to scores on another. Many practitioners would agree with Lord (1980) that scores on Many practitioners would agree with Lord (1980) that scores on test test XX and test and test YY are equated if the following four conditions are are equated if the following four conditions are met:met:1)1) Same AbilitySame Ability – – the two tests must both be measures of the same characteristic (latent trait, the two tests must both be measures of the same characteristic (latent trait,

ability or skill).ability or skill).

2)2) EquityEquity – – for every group of examinees of identical ability, the conditional frequency for every group of examinees of identical ability, the conditional frequency distribution of scores on test distribution of scores on test YY, after transformation, is the same conditional frequency , after transformation, is the same conditional frequency distribution of scores on test distribution of scores on test XX..

3)3) Population InvariancePopulation Invariance – – the transformation is the same regardless of the group from of the transformation is the same regardless of the group from of which it is derived.which it is derived.

4)4) SymmetrySymmetry – – the transformation is invertible, that is, the mapping scores from formthe transformation is invertible, that is, the mapping scores from form X X to form to form

Y Y is the same as the mapping of scores from form is the same as the mapping of scores from form Y Y toto formform X X

Conditions For Equated Scores Conditions For Equated Scores continued…//continued…//

The equity condition is unlikely to be precisely satisfied in The equity condition is unlikely to be precisely satisfied in practice. practice.

Although it might be possible to build two forms of a test Although it might be possible to build two forms of a test that measured the same characteristic and were equally that measured the same characteristic and were equally reliable generally, it is highly unlikely that one could ever reliable generally, it is highly unlikely that one could ever build two forms that were equally reliable at every ability build two forms that were equally reliable at every ability level, let alone that which can produce the same conditional level, let alone that which can produce the same conditional frequency distributions. frequency distributions.

Data Collection DesignsData Collection Designs

Equating Data Collection DesignsEquating Data Collection Designs

No statistical procedure can provide completely No statistical procedure can provide completely appropriate adjustments when non-equivalent or appropriate adjustments when non-equivalent or naturally occurring groups are used, naturally occurring groups are used,

but but adjustments based on an another test that is as adjustments based on an another test that is as close as possible to the tests to be equated are close as possible to the tests to be equated are much more satisfactory than those based on much more satisfactory than those based on nonparallel tests.nonparallel tests.

Equating ProceduresEquating Procedures

Can regression be used to equate Can regression be used to equate scores?scores?

No. Because Y = a+bX does not give us No. Because Y = a+bX does not give us the same conversion function as X = the same conversion function as X = c+mYc+mY

To ensure equity, the conversion To ensure equity, the conversion functions need to be the same. functions need to be the same.

^

^

Equating ProceduresEquating Procedures

Pre-EquatingPre-Equating– Equating done on sections of a test, not the final test Equating done on sections of a test, not the final test

bookletsbooklets– Scores are not counted for studentScores are not counted for student

Post-EquatingPost-Equating– Equating done on final test booklets, not sections of a testEquating done on final test booklets, not sections of a test

Equipercentile EquatingEquipercentile Equating– Equates percentiles of two score distributions for two test Equates percentiles of two score distributions for two test

formsforms

Linear EquatingLinear Equating– Equates means and standard deviations of two score Equates means and standard deviations of two score

distributions distributions for two test formsfor two test forms

Illustration of the Equipercentile Illustration of the Equipercentile Equating ProcessEquating Process

Equipercentile equating can be thought as a two-stage process Equipercentile equating can be thought as a two-stage process (Kolen, 1984). (Kolen, 1984).

FirstFirst, , the relative cumulative frequency (i.e. percentage of cases the relative cumulative frequency (i.e. percentage of cases below a score interval) distributions are tabulated or plotted below a score interval) distributions are tabulated or plotted for the two forms to be to be equated. for the two forms to be to be equated. SecondSecond, , equated scores (e.g. scores with identical relative cumulative equated scores (e.g. scores with identical relative cumulative frequencies) on the two forms are obtained from these frequencies) on the two forms are obtained from these cumulative frequency distributions.cumulative frequency distributions.

Illustration of the Equipercentile Illustration of the Equipercentile Equating ProcessEquating Process continued…//continued…//

A graphical method for equipercentile is A graphical method for equipercentile is illustrated in Figure 6.4. illustrated in Figure 6.4.

FirstFirst, , the relative cumulative frequency distributions, each the relative cumulative frequency distributions, each based on 471 examinees, for two forms (designated based on 471 examinees, for two forms (designated X X andand Y)Y) of a 60-item number-right-scored test were plotted. of a 60-item number-right-scored test were plotted. The crosses (and stars) represent the relative cumulative The crosses (and stars) represent the relative cumulative frequency (i.e., percent below) at the lower real limit of frequency (i.e., percent below) at the lower real limit of each integer score interval (e.g, at each integer score interval (e.g, at ii-0.5, for -0.5, for i=i=1, 2, ……, 1, 2, ……, n, where n is the number of items). n, where n is the number of items).

NextNext, , the crosses (stars) were connected with straight line the crosses (stars) were connected with straight line segments. segments. Graphs constructed in this manner are referred to as Graphs constructed in this manner are referred to as linearly interpolated relative cumulative frequency linearly interpolated relative cumulative frequency distributionsdistributions. . The line segments connecting the crosses (stars) need not The line segments connecting the crosses (stars) need not be linear. be linear. Methods of curvilinear interpolation, such as the use of Methods of curvilinear interpolation, such as the use of cubic splines, could also be employed.cubic splines, could also be employed.

Illustration of the Equipercentile Illustration of the Equipercentile Equating ProcessEquating Process continued…//continued…//

Let the form-Let the form-X X equipercentile equivalent of equipercentile equivalent of yyii,, be denoted ebe denoted exx(y(yii). ). The calculation of the form The calculation of the form X X equipercentile equipercentile equivalent eequivalent exx(18) of a number-right score of (18) of a number-right score of 18 on form Y is illustrated in Figure 6.4. 18 on form Y is illustrated in Figure 6.4. The left-hand vertical arrow indicates that the The left-hand vertical arrow indicates that the relative cumulative frequency for a score of relative cumulative frequency for a score of 18 on form 18 on form Y Y is 50. is 50. The short horizontal arrow shows the point The short horizontal arrow shows the point on the curve for form on the curve for form XX with the same relative with the same relative cumulative frequency (50). cumulative frequency (50). The right-hand vertical arrow indicates that a The right-hand vertical arrow indicates that a score of 30 on form score of 30 on form XX is associated with this is associated with this relative cumulative frequency. relative cumulative frequency. Thus, a score of 30 on form Thus, a score of 30 on form X X is considered is considered to be equivalent to a score of 18 on form to be equivalent to a score of 18 on form YY. . A plot of the score conversion (equivalent) is A plot of the score conversion (equivalent) is given in Figure 6.5.given in Figure 6.5.

The equipercentile transformation The equipercentile transformation between two forms, between two forms, X X and and Y,Y, of a of a test will usually be curvilinear. test will usually be curvilinear.

If form If form X X is more is more difficultdifficult than form than form Y, Y, the conversion line will tend to be the conversion line will tend to be concave downward. concave downward. If the distribution of scores on form If the distribution of scores on form X X is flatter, more is flatter, more platykurticplatykurtic, than , than that on form that on form YY, the conversion will , the conversion will tend to be S-shaped. tend to be S-shaped.

If the If the shapesshapes of the score of the score distributions on the two forms are the distributions on the two forms are the samesame (i.e., have the same moments (i.e., have the same moments except for the first two), the except for the first two), the conversion line will be linear.conversion line will be linear.

Use of Smoothing TechniquesUse of Smoothing Techniques

Unsmoothed equipercentile Unsmoothed equipercentile equating uses straight linear equating uses straight linear interpolation for the ogivesinterpolation for the ogivesSmoothing techniques can be Smoothing techniques can be used with curvilinear used with curvilinear interpolation such as cubic interpolation such as cubic splines with different splines with different parametersparametersSmoothing on ogives is Smoothing on ogives is known as pre-smoothing known as pre-smoothing methodmethodSmoothing on conversion Smoothing on conversion functions is known as post-functions is known as post-smoothing methodsmoothing method

Application of Equipercentile-Equating Application of Equipercentile-Equating to Data Collection Designsto Data Collection Designs

Equipercentile equating can also be carried out for Equipercentile equating can also be carried out for the anchor-test-random-groups design in the the anchor-test-random-groups design in the following manner:following manner:

1.1. Using the data for the group taking tests X and V Using the data for the group taking tests X and V (the anchor test), for each raw score on test V, (the anchor test), for each raw score on test V, determine the score on test X with the same determine the score on test X with the same percentile rank.percentile rank.

2.2. Using the data group taking tests Y and V, for each Using the data group taking tests Y and V, for each raw score on test V, determine the score on test Y raw score on test V, determine the score on test Y with the same percentile rank.with the same percentile rank.

Application of Equipercentile-Equating Application of Equipercentile-Equating to Data Collection Designsto Data Collection Designs continued…//continued…//

3.3. Tabulate pairs of scores on tests X and Y that correspond to Tabulate pairs of scores on tests X and Y that correspond to the same raw score on test V.the same raw score on test V.

4.4. Using data from step 3, for each raw score on test Y, Using data from step 3, for each raw score on test Y, interpolate to determine the equivalent score on test X.interpolate to determine the equivalent score on test X.

The last procedure uses the data on test V to adjust for The last procedure uses the data on test V to adjust for differences in ability between the two groups. This differences in ability between the two groups. This procedure really involves two equatings, instead of just one, procedure really involves two equatings, instead of just one, and therefore doubles the variance of equating error. and therefore doubles the variance of equating error.

Standard Errors of Equipercentile Standard Errors of Equipercentile EquatingEquating

Standard Errors of Equipercentile Standard Errors of Equipercentile EquatingEquating

continued…//continued…//

Another procedure that may be used to estimate the Another procedure that may be used to estimate the standard error of an equipercentile equating is the standard error of an equipercentile equating is the

bootstrap method (Efron 1982).bootstrap method (Efron 1982).

Linear EquatingLinear Equating

When tests X and Y are not equally reliable, true score x’ and y’ are used instead

Illustration of Illustration of thethe Linear Linear Equating ProcessEquating Process

Linear equating, like equipercentile equating, can be thought Linear equating, like equipercentile equating, can be thought of as two-stage process. of as two-stage process.

FirstFirst, compute the sample means (m) and standard deviations , compute the sample means (m) and standard deviations (s) of scores on the two forms to be equated. (s) of scores on the two forms to be equated.

SecondSecond, obtain equated scores on the two forms by , obtain equated scores on the two forms by substituting these values into linear equating equation. substituting these values into linear equating equation. – For example, suppose the raw-score means and the standard deviations For example, suppose the raw-score means and the standard deviations

for two-forms, X and Y, of a 60-item number-right-scored test for two-forms, X and Y, of a 60-item number-right-scored test administered to a single group of 471 examinees areadministered to a single group of 471 examinees are

Illustration of Illustration of thethe Linear Equating Linear Equating ProcessProcess continued…//continued…//

Application of Linear Equating to Data Application of Linear Equating to Data Collection DesignsCollection Designs

Linear equating can be carried out for the anchor-test-Linear equating can be carried out for the anchor-test-random-groups design in the same manner as for the random-groups design in the same manner as for the equivalent-group design, in which case, the data on anchor-equivalent-group design, in which case, the data on anchor-test V are ignored. test V are ignored.

However, even when the groups are chosen at random, it is However, even when the groups are chosen at random, it is inevitable that there will be some differences between them, inevitable that there will be some differences between them, which, if ignored, will lead to bias in the conversion line. which, if ignored, will lead to bias in the conversion line.

The data on test V can be used to adjust for differences The data on test V can be used to adjust for differences between groups by means of the maximum-likelihood between groups by means of the maximum-likelihood approach (Lord, 1955a). approach (Lord, 1955a).

Maximum-likelihood estimates of the population means and Maximum-likelihood estimates of the population means and standard deviations on forms X and Y are as follows:- standard deviations on forms X and Y are as follows:-

Application of Linear Equating to Data Application of Linear Equating to Data Collection DesignsCollection Designs continued..//continued..//

Standard Errors of Linear Standard Errors of Linear EquatingEquating

Comparison of equating methodsComparison of equating methods

Equipercentile Equipercentile EquatingEquating– Adjust for differences Adjust for differences

in difficulty of test in difficulty of test formsforms

– Can equate up to the Can equate up to the fourth moments of the fourth moments of the score distributionscore distribution

– Percent of students Percent of students below a particular below a particular score is equated score is equated

Linear EquatingLinear Equating

– Adjust for differences Adjust for differences in difficulty of test in difficulty of test formsforms

– Only equates up to the Only equates up to the first two moments of first two moments of the score distributionthe score distribution

– Percent of students Percent of students scoring below an scoring below an equated score is not equated score is not equatedequated

ReferencesReferences

Kolen and Brennan (1995) Test equating, Kolen and Brennan (1995) Test equating, springer verlagspringer verlag

Kolen, Peterson, & Hoover’s chapter on Kolen, Peterson, & Hoover’s chapter on test equating in Linn (1993) Educational test equating in Linn (1993) Educational Measurement, Ace-Oryx publishingMeasurement, Ace-Oryx publishing

Thank YouThank You