Reliability & Validity

Reliability-Having confidence in the consistency of the test results.

Reliability of a test refers to how well it provides a consistent (stable) set of results across similar test situations, time periods, and examiners or the extent to which measurements are repeatable.

Correlation- A statistical method of observing the degree of relationship between two sets of data or two sets of variables (e.g., hours spent studying and grades)

• Correlation coefficient -The numerical representation of the strength and direction of the relationship between two sets of variables or data.

• Represented by a real number between -/+ 1.00 (the stronger the correlation= the more reliable)

* In investigating how data are related, it is important to determine if the two sets of data represent positive, negative, or no correlation.

Positive Correlation

* In a positive correlation, when a student scores high on the first variable or test, the student will also score high on the second measure (e.g. calories eaten and weight)

* The data below illustrate a positive correlation:

Student 1- 75Student 2- 88Student 3- 90Student 4- 63

Student 1- 72Student 2- 89Student 3- 93Student 4- 64

Set 1 Set 2

Negative Correlation

•In a negative correlation, when a student scores high on one variable or test, the student will score low on the other variable or test (e.g. days absent and test scores)•The data below illustrate a negative correlation:

Student 1- 88Student 2- 99Student 3- 56Student 4- 97

Student 1- 32Student 2- 45Student 3- 15Student 4- 12

Set 1 Set 2

Data Set 1947889586277754595

Data Set 2 95 80 90 59 60 78 76 47 97

Two sets of data are presented below. Determine if the datasets represent a positive, negative, or no relationship.

One way to determine thedirection of the relationshipis to plot the scores on aScatter plot.

The data for set 1 and set 2are plotted on the next slide.

*

* *

*

**

*

*

*45 50 55 60 65 70 75 80 85 90 95 100

100 95 90 85 80 75 70 65 60 55 50 45

Set 1

Set 2

The direction of the line plotted on the scatter plot providesa clue about the relationship. If the relationship is positive, the direction of the line looks like this:

If the data represent a negative correlation, the directionof the line in the scatter plot looks like this:

No Correlation• When there is little or no relationship

between scores.• The scattergram does not indicate a

distinct line– Figure 4.4– See scattergrams on p. 121 in Overton text

Methods of Studying Reliability

Test-retest reliability- A study that employs the re-administrationof a single instrument to check for consistency across time.

(Sensitive to practice effects and effects of instruction!)

Equivalent forms reliability-Consistency of a test using like forms that measure the same skill, domain, or trait; also knownas alternate forms reliability.

(Form A & B of a test)

Methods of Studying Reliability, Continued

Internal consistency-Methods to study the reliability across the items of the test (within the test).

• Split-half reliability-studying the reliability across items by comparing the data of the two halves of the test.

• Kuder-Richardson 20- (K-R 20)- A formula used to check consistency across items of an instrument that has items scored as 1 or 0 or right/wrong.

• Coefficient Alpha- A formula used to check the consistency across items of instruments with responses of varying credit. For example, items may be scored as 0,1,2,or 3 points (e.g. a rubric).

Methods of Studying Reliability, Continued

Interrater Reliability- The consistency of a test to measure a skill, trait, or domain across examiners.This type of reliability is most important whenresponses are subjective or open-ended.

Reliability coefficients may vary across age and grade levels of a specific instrument!

Random Error

• Random error describes random events that have nothing to do with the quality being measured but impact the results

• Random error impacts the reliability of a test, and consequently all possible sources of random error must be addressed

Standard Error of Measurement

Each test score is made of two parts: true score and error.

A student’s true score may only be estimated. The standarderror of measurement is a method used to estimate theamount of error of a test. It represents the typical amount oferror of any obtained score. The standard error ofmeasurement is used to estimate a range of scores within which the student’s true scores exists.

Obtained Score = True Score + Error

Standard Error of Measurement, Continued

The standard error of measurement is calculated using the following formula:

1-.r

Where SEM = the standard error of measurementSD = the standard deviation of the norm group of scores obtained during the development of the instrumentr = the reliability coefficient

SEM = SD

Example of Calculating SEM

For a specific test, the standard deviation is 4. Thereliability coefficient is .89. The SEM would be:

4 1- .89

.114

4 x .33 = 1.32

SEM = 1.32

This represents the amount of error on this test instrument.

Applying the SEM

The SEM was 1.32. A student’s obtained score was 89. To determine the range of possible true scores, add and subtract the SEM (a certain factor) from the obtained score of 89.

89 + 1.32(1) = 90.3289 - 1.32(1) = 87.68

The range of possible scores is87.68 - 90.32

Confidence Intervals

• Confidence intervals allow us to make statements about the proximity of the obtained score to the true score.

• A 68% confidence interval allows us to say that even though we don't know the true score, there is only a 32% chance that it is not within the range specified.

• A 95% confidence interval allows us to say that there is only a 5% chance that the person's true score is not within the range specified.

An Example

• Milly's obtained score on the Spelling Accuracy Test was 53. The SEM of the test is 1.85. We wish to be very confident in our estimate of Milly's score and choose a 95% confidence interval.

• See factor list…The factor to multiply the SEM by to achieve a 95% CI is 1.96 so 1.96 x 1.85=3.62.

• 53 + or – 3.62=

• Lower bound= 49.4 and

• Upper bound = 56.7• There is a 95% probability

that Milly's true score on the Spelling Accuracy Test falls in the region between 49.38 and 56.62.

Selecting the Best Test Instruments

When considering which tests will be the mostreliable, it is important to select a test that has the highest reliability coefficient and the smallest standard of error.

This will mean that the results obtained are more likely to be more consistent with the student’s true ability.

The obtained score will contain less error.

Validity indicates the degree of quality of the testinstrument.

Validity coefficient ranges between -/+1.0.

Validity-Having confidence that the test is measuring what it is supposed to measure.

Validity is a relative rather than absolute notion.

Criterion-related Validityestablished by comparing performance on an

accepted standard or criterion

Concurrent-related validity- When a test is comparedwith a similar measure administered within a short periodof time.

There are two ways to study criterion-related validity.

Predictive validity-When a test is compared with ameasure in the future. For example, when collegeentrance exams are compared with student performance inCollege (GPAs).

Content ValidityIn order for a test to have good content validity, it

must have items that are representative of thedomain or skill being assessed.

During the development of the test, items areselected after careful study of the items and thedomain they represent.

To ascertain content validity, submit the test to a panel of expert judges. The judges should examine the test in terms of:

CompletenessAppropriatenessFormatBias

Construct Validity

Construct validity means that the instrument hasthe ability to assess the psychological constructsit was meant to measure.

A construct is a psychological trait or characteristic such as creativity or mathematicalability.

Studying Construct Validity

• Developmental changes• Correlation with other tests• Factor Analysis• Internal consistency• Convergent or discriminant validation• Experimental interventions

Overton, 138 & 139

External, Internal, and Social Validity

• External validity is concerned with generalizability (e.g., Do the results obtained in one particular situation apply in other situations?)

• Internal validity is the fundamental basis for interpreting the results of an intervention (e.g., Did the intervention make a difference in this specific instance?)

• Social validity refers to the social value and acceptability of an educational intervention.