UNIT IV UNIT IV ITEM ANALYSIS ITEM ANALYSIS IN TEST DEVELOPMENTIN TEST DEVELOPMENT

CHAP 14: ITEM ANALYSISCHAP 14: ITEM ANALYSISCHAP 15: INTRODUCTION CHAP 15: INTRODUCTION TO ITEM RESPONSE TO ITEM RESPONSE THEORYTHEORYCHAP 16: DETECTING CHAP 16: DETECTING ITEM BIASITEM BIAS

1

CHAPTER 14 ITEM ANALYSIS**The goal of The goal of test construction test construction is to create a is to create a test with test with minimum length minimum length and and good good reliability and validity.reliability and validity. *Item Analysis *Item Analysis is the is the computationcomputation and and examination of any statistical property of an examination of any statistical property of an item response distribution.item response distribution.*Item Analysis *Item Analysis is a is a processprocess that we go that we go through when constructing a new test or through when constructing a new test or subtests from a subtests from a pool of itemspool of items with with good good reliability and validity. reliability and validity. 2

CHAPTER 14 ITEM ANALYSIS*Categories of *Categories of Item ParameterItem Parameter **Item parameters Item parameters fall intofall into 3 3 categories or categories or

indices.indices.1. 1. IndicesIndices that that describedescribe the distribution of the distribution of

responsesresponses to a singleto a single item item (e. g. mean and(e. g. mean and variancevariance of item responses).of item responses).

2. 2. IndicesIndices that that describedescribe the degree of the degree of relationshiprelationship between thebetween the response to the response to the item item and some and some criterion criterion of interest. of interest.

Ex. next

3

CHAPTER 14 ITEM ANALYSISEx. The Ex. The relationshiprelationship between the questions between the questions

(items) (items) and the and the criterion of interest criterion of interest i.e., i.e., depression in Factor Analysis.depression in Factor Analysis.

3. 3. IndicesIndices that are a function of that are a function of bothboth, , meaning, meaning, relationship relationship to to item variance/meanitem variance/mean and a and a criterioncriterion of of interest.interest.

Ex. Ex. First, First, find the find the variancevariance/mean for your items /mean for your items then, then, calculate the calculate the relationshiprelationship between these between these items items variance variance and the and the criterioncriterion of interest (i.e., of interest (i.e., depression) for two groups..depression) for two groups..

4

CHAPTER 14 ITEM ANALYSISItem Difficulty “P”Item Difficulty “P” P= P= ff//N N or Number of examinees or Number of examinees who answered an item who answered an item correctlycorrectly // Total number of participants Total number of participants ((See your midterm item analysis and Chap 5).See your midterm item analysis and Chap 5).

The The higherhigher the P value the the P value the easiereasier the the itemitem 5

6

CHAPTER 14 ITEM ANALYSIS *Steps in Item Analysis*Steps in Item Analysis In a typical item analysis In a typical item analysis

the test developer will take the test developer will take 7 steps 7 steps (they are similar to (they are similar to the process of test the process of test construction in Chapter 4). construction in Chapter 4). Next Slide

7

FYI FYI PROCESS OF TEST CONSTRUCTION PROCESS OF TEST CONSTRUCTION CHAP CHAP IV IV

1-Identifying 1-Identifying purposespurposes of test scores of test scores useuse2-2-IdentifyingIdentifying behaviorsbehaviors to represent to represent the constructthe construct3- 3- Preparing Preparing test specification test specification i.e., i.e., Bloom Taxonomy Bloom Taxonomy 4- 4- Item constructionItem construction5- 5- Item Review Item Review

8

PROCESS OF TEST CONSTRUCTION

6- 6- PreliminaryPreliminary item tryoutsitem tryouts7- 7- Field testField test8- 8- Statistical Statistical AnalysisAnalysis9- 9- Reliability and ValidityReliability and Validity10- 10- GuidelinesGuidelines

9

CHAPTER 14 ITEM ANALYSIS *7 Steps in Item Analysis*7 Steps in Item Analysis1. Describe what proportions of the 1. Describe what proportions of the

test score are of test score are of greatest greatest important. important.

Ex. when I select questions for your Ex. when I select questions for your midterm/final exam I look for the midterm/final exam I look for the similaritiessimilarities of the of the questions with questions with those of qualifying/comprehensive or those of qualifying/comprehensive or EPPP examEPPP exam.

10

CHAPTER 14 ITEM ANALYSIS

*Steps in Item Analysis*Steps in Item Analysis 2. Identify the 2. Identify the item parameters item parameters (e.g. (e.g.

mean, variance) most relevant to mean, variance) most relevant to these proportions.these proportions.

3. 3. AdministerAdminister the items to a sample the items to a sample of examinees representative of those of examinees representative of those for whom the for whom the test is intended.test is intended.

Ex. IQ test for Ex. IQ test for childrenchildren or or depression test for depression test for adultsadults..

11

CHAPTER 14 ITEM ANALYSIS Steps in Item AnalysisSteps in Item Analysis4. 4. EstimateEstimate for each item the for each item the

parametersparameters identified in step 2 identified in step 2 i.e., variance).i.e., variance).

5. 5. Establish a plan for Establish a plan for item item selection.selection.

Ex. Using Ex. Using item difficulties (P) item difficulties (P) as as in Item Analysis to select the in Item Analysis to select the items. items.

12

CHAPTER 14 ITEM ANALYSIS Steps in Item AnalysisSteps in Item Analysis6.6. Select Select the final subset of items, or use the final subset of items, or use

the data (Items in your Item Analysis) the data (Items in your Item Analysis) for test revision.for test revision.

Ex. Takeout all questions with very Ex. Takeout all questions with very high or very low item difficulties. high or very low item difficulties.

7. Conduct 7. Conduct a cross validation (validity) a cross validation (validity) study. study.

Ex. Use SPSS and compare the results of 2 Ex. Use SPSS and compare the results of 2 tests or 2 classes (e. g. this year class tests or 2 classes (e. g. this year class and last year class). i.e., Confirmatory and last year class). i.e., Confirmatory Factor Analysis.Factor Analysis. 13

UNIT V UNIT V TEST SCORING AND INTERPRETATIONTEST SCORING AND INTERPRETATION

CHAP 17: CORRECTING FOR GUESSING CHAP 17: CORRECTING FOR GUESSING AND OTHER SCORING METHODSAND OTHER SCORING METHODSCHAP 18: SETTING STANDARDSCHAP 18: SETTING STANDARDSCHAP 19: NORMS AND STANDARD CHAP 19: NORMS AND STANDARD SCORESSCORESCHAP 20: EQUATINGSCORESFROM CHAP 20: EQUATINGSCORESFROM DIFFERENT TESTSDIFFERENT TESTS

14

UNIT VUNIT VTEST SCORING AND INTERPRETATIONTEST SCORING AND INTERPRETATION

CHAPT 19CHAPT 19NORMS AND STANDARDS NORMS AND STANDARDS

SCORESSCORES

15

CHAPTER 19NORMS AND STANDARD SCORES

*Alfred Binet (1910)*Alfred Binet (1910)RatioRatio IQ IQ = = RatioRatio of of MA/CA MA/CA

*Louis Terman *Louis Terman RatioRatio IQ = IQ = Ratio Ratio of of MA/CA MA/CA X 100 X 100 standardized it.standardized it.

*Deviation *Deviation IQ IQ = Uses = Uses NormsNorms to estimate the IQto estimate the IQ We use Norms when we want to compare an We use Norms when we want to compare an

examinee’s score examinee’s score (raw score) or score on a test (raw score) or score on a test to to the distribution of scores the distribution of scores (scaled or standard (scaled or standard scores) scores) for a sample from a well-defined for a sample from a well-defined population. population. Ex. nextEx. next 16

CHAPTER 19NORMS AND STANDARD SCORES

Ex. When we want to estimate the IQ of Ex. When we want to estimate the IQ of a 20 year-old person, We compare a 20 year-old person, We compare his/her his/her raw score raw score on the subtest of an IQ on the subtest of an IQ test with the people of his/her age, test with the people of his/her age, which is his/herwhich is his/her norm (standard scores). norm (standard scores). Using this technique tells us where this Using this technique tells us where this person stands among the people of person stands among the people of his/her age.his/her age.

17

NORMS AND STANDARD SCORESNORMS AND STANDARD SCORES*9 *9 BASIC STEPS IN CONDUCTING A BASIC STEPS IN CONDUCTING A NORMING NORMING

STUDY STUDY (P.432)(P.432)

1. 1. Identify the population of interestIdentify the population of interest Ex. Students, employees of a company, Ex. Students, employees of a company,

inmates, patients, etc.inmates, patients, etc.

2. 2. Identify the most critical statistics that will Identify the most critical statistics that will be computed for the sample data. be computed for the sample data.

Ex. Standard deviation σ, σ² , M, SS, pEx. Standard deviation σ, σ² , M, SS, p 18

NORMS AND STANDARD SCORESNORMS AND STANDARD SCORES*9*9BASIC STEPS IN CONDUCTING A NORMING STUDY (P.432)BASIC STEPS IN CONDUCTING A NORMING STUDY (P.432)

3. 3. Decide on the tolerable amount of Decide on the tolerable amount of sampling errorsampling error

That is the discrepancy between the That is the discrepancy between the sample statistic (M) and population sample statistic (M) and population parameter, (µ) (parameter, (µ) (Central Tendency Central Tendency M=µM=µ). ). The The Central Limit Theorem Central Limit Theorem has 3 has 3 characteristics;characteristics;

1. Central Tendency 1. Central Tendency 2.The Shape of the 2.The Shape of the Distribution (normal) Distribution (normal) and and 3. Variability or 3. Variability or Standard Error of Mean (Standard Error of Mean (σσmm).). M-µ M-µ

19

9BASIC STEPS IN CONDUCTING A NORMING STUDY (P.4329BASIC STEPS IN CONDUCTING A NORMING STUDY (P.432))

4. 4. Device a procedure for Device a procedure for drawing a sample drawing a sample from the from the population of interest.population of interest.

There are 4 types of probability samplingThere are 4 types of probability samplingII Simple Random Sampling Simple Random Sampling Give everyone in the population an equal chance to Give everyone in the population an equal chance to

be selected Ex. Draw names from a hat. be selected Ex. Draw names from a hat. II II Systemic Sampling N/nSystemic Sampling N/n Select every KSelect every Kthth name on the list. name on the list. Ex. Ex. CAU Pop CAU Pop

N=1500 N=1500 and your sample size and your sample size n=150n=150 N/nN/n=1500/150=10 Select every 10=1500/150=10 Select every 10thth student. student.

20

9BASIC STEPS IN CONDUCTING A NORMING STUDY (P.432)

SAMPLING CONT..

IIIIII StratStratified Sampling ified Sampling “Strata” means “Strata” means different layers. different layers. We use Stratified We use Stratified Sampling when we want to compare 2 Sampling when we want to compare 2 different groups different groups (e.g. (e.g. Males and femalesMales and females CAU Doctoral StudentsCAU Doctoral Students).).

First we randomly select males then, First we randomly select males then, randomly select randomly select females. females.

21

9BASIC STEPS IN CONDUCTING A NORMING STUDY(P.432)9BASIC STEPS IN CONDUCTING A NORMING STUDY(P.432)

SAMPLING CONT..SAMPLING CONT..IV IV Cluster Sampling Cluster Sampling We use Cluster We use Cluster

sampling when the population consists of sampling when the population consists of units units not individuals, such as not individuals, such as classes. classes. Ex. Ex. Miami Dade School Districts. If we want Miami Dade School Districts. If we want to conduct a research with the Miami to conduct a research with the Miami Dade 2Dade 2ndnd graders (1000- 2 graders (1000- 2ndnd grade classes). grade classes). We’ll randomly select about 10 of these We’ll randomly select about 10 of these 1000- 21000- 2ndnd grade classes to be in our sample grade classes to be in our sample then we conduct research.then we conduct research.

22

9BASIC STEPS IN CONDUCTING A NORMING STUDY 9BASIC STEPS IN CONDUCTING A NORMING STUDY (P.432)(P.432)

5.5.Estimate the minimum Estimate the minimum sample size (n) sample size (n) required to required to hold the sampling error within the specific limits. hold the sampling error within the specific limits.

There are different statistical procedures to There are different statistical procedures to estimate the estimate the (n). (n). (n) should be ≥30. (n) should be ≥30.

1. n= (σ/d)²1. n= (σ/d)² d=effect size d=M-µ/σd=effect size d=M-µ/σ 2. n= (σ/σ2. n= (σ/σm)m) ² ²

σσmm= σ/√n = σ/√n Standard error of meanStandard error of mean for for pop pop Ex. Ex. Z scoreZ score

SSmm=S=S/√/√n n Estimated Standard Error of the Mean Estimated Standard Error of the Mean for a for a sample. sample. Ex. t-distribution Ex. t-distribution 23

NORMS AND STANDARD SCORESNORMS AND STANDARD SCORES

24

THE EFFECT SIZE THE EFFECT SIZE EX. TWO INDEPENDENT T-TESTEX. TWO INDEPENDENT T-TEST

25

NORMS AND STANDARD SCORESNORMS AND STANDARD SCORES

26

9BASIC STEPS IN CONDUCTING A NORMING STUDY (P.432)

6. 6. Draw the Draw the SampleSample and collect the and collect the DataData7. Compute7. Compute the Values of the Group the Values of the Group

Statistics of interest and their Statistics of interest and their standard standard error. error. SSmm=S/√n=S/√n or or σσm = m = σσ//√n√n

Calculate the Calculate the standard error standard error of of measurement, which is the difference measurement, which is the difference between M andbetween M and µ. Also known as µ. Also known as sampling error.sampling error.

27

9BASIC STEPS IN CONDUCTING A NORMING STUDY (P.432)

8. 8. Identify the Identify the Types of Normative Types of Normative Scores that will be needed, Scores that will be needed, and and prepare the Normative Score prepare the Normative Score Conversion table (see next 2 slide).Conversion table (see next 2 slide).

9. 9. Prepare Prepare written documentation written documentation of of the Normative Scores.the Normative Scores.

28

NORMS AND STANDARD SCORESTypes of Normative ScoresTypes of Normative ScoresRaw ScoreRaw Score Score on a Score on a subtest or a subtest or a

test.test.Scaled ScoreScaled Score Normative score Normative score for for

specific agespecific age..

29

NORMATIVE SCORES

30Wex-ler

*NORMATIVE SCORES*NORMATIVE SCORES

31

NORMS AND STANDARD SCORES

*Usefulness of Scaled Scores*Usefulness of Scaled Scores

Scaled Scores are useful forScaled Scores are useful for two purpose:two purpose:

1. 1. Scaled scores Scaled scores relate relate the examinee’s the examinee’s performance to performance to percentile rank scorespercentile rank scores of the of the norm group and their grade level.norm group and their grade level.

2. 2. In evaluation and research In evaluation and research the mean scaled the mean scaled score score is a is a betterbetter estimation of average group estimation of average group performance than performance than the mean raw score.the mean raw score.

32

33

34

NORMATIVE SCORES

Multiply by 5 to convert to Multiply by 5 to convert to percentile. This means neither percentile. This means neither USA nor Iran are using a USA nor Iran are using a Normal Normal Distribution Distribution in their grading in their grading system. USA is negatively and system. USA is negatively and IRAN is positively skewed. IRAN is positively skewed.

35

CHAPTER 19NORMS AND STANDARD SCORES

*Echternacht (1971) *Echternacht (1971) 3 steps 3 steps Process of Process of Grade Grade and and Age Age Equivalent ScoresEquivalent Scores

1.1. First First we we convert convert the the raw scores raw scores to to scaled scaled scoresscores

2. Second, 2. Second, calculate thecalculate the median median scaled score scaled score for for each grade-leveleach grade-level, and , and plot plot them them on a bivariate on a bivariate scatter plot.scatter plot.

33.Connect.Connect the points the points and draw a smooth and draw a smooth curve.curve.It is similar to Deviation IQ. I.e., Child’s performance compares It is similar to Deviation IQ. I.e., Child’s performance compares

with that of others at a particular age or grade level. with that of others at a particular age or grade level. 36

CHAPTER 19NORMS AND STANDARD SCORES

37

38