Reliability

Presented by Monika

Singh M.Sc.(H.Sc.)

Introduction

• In the psychometrics, reliability is the overall consistency of a measure.

• A measure is said to have a high reliability if it produces similar results under consistent conditions.

• For example, measurements of people’s height and weight are often extremely reliable

Definition of Reliability

• Reliability is defined (Portney and Watkins, 2000) as:

‘the extent to which a measurement is

consistent and free from error’.

Types of Reliability

Test-retest

reliability

Equivalency

reliability

Inter-rater

reliability

Internal consistenc

y reliability

Test-retest reliability-

• assesses the degree to which test scores are consistent from one test administration to the next.

• It involves:

Administering a test to a group of individuals

Re-administering the same test to the same group at some later time

Correlating the first set of scores with the second

Test-retest reliability-

Assumption of the method

• Number of item in the test are large therefore memory, practice, and carry over will not effect the retest scores.

• Innate ability of an individual remains constant so the growth and maturity will not effect the retest score.

• Time gap of retest score should not be more than six month.

Advantage

• It is the most simplest and easy method of estimating reliability coefficient

• It is the most appropriate method of estimating reliability of speed test but it is adequately used for speeded test.

• It is an economical method of reliability is one form of the test is used on both times.

Limitation

• This method is less accurate

• It is time consuming method of reliability

• The individual΄s health, emotional condition, motivational conditions, mental health and mental set do not remains the same in both the administrations.

Equivalency reliability

• assesses the degree to which test scores are consistent when there is a variation in the methods or instruments used.

• This method of reliability is known by various name or terms such as parallel form of reliability , alternative forms reliability, equivalent forms of reliability, and comparable – forms reliability

Equivalency reliability

• Criteria for Equivalency reliability

The number item should be the same

The item variance should be the same

There should be equivalence with regard to content, types of items, the range of difficulty value.

Instructional procedure of administering the tests should be the same

Assumption

• The memory, practice and carry over factors do not effect the scores

• Two forms of a test are virtually alike with reference to;

a) Difficulty and discrimination indexes

b) Content and type of item

c) Size number of items

d) Time of administration

Advantage

• The method of reliability is an improvement over the test retest method of reliability

• The practice, memory, exercise, and maturity effect are controlled in this method.

• It is an economical and speedy method for calculating reliability coefficient.

• It is useful for the reliability of achievement tests

Limitation

• It is difficult to construct parallel form of test and satisfy all the conditioned mentioned.

• Interval for administration the two forms will not be more than two weeks.

Inter-rater reliability

• Inter rater reliability is the extent to which two or more individuals(raters ) agree.

Internal consistency reliability

• Internal consistency reliability assesses the consistency of results across items within a test.

• The most popular method of testing for internal consistency in the behavioral sciences is coefficient alpha

• There are a wide variety of internal consistency measures that can be used.

a) Average inter-item correlation

b) Split-half reliability

Average inter-item correlation

• Average inter-item correlation is a subtype of internal consistency reliability.

• It is obtained by taking all of the items on a test that probe the same construct (e.g., reading comprehension), determining the correlation coefficient for each pair of terms, and finally taking the average of all of these correlation coefficients.

Average inter-item correlation

Split half method of reliability

• It involves:

Administering a test to a group of individuals

Splitting the test in half in two equal parts

Correlating scores on one half of the test with scores on the other half of the test

• The common way of dividing the test item is the odd and even numbers.

Spearmen –brown prophecy formula-

R= 2 rxy

1+rxy

R=reliability of whole test

rxy= correlation of x (odd) and y (even) series.

Assumption

• It assumes that items of the test is homogeneous. All the items of the test measure the same trait or ability

• The assumption of for splitting test in to two halves that these are equivalent

Advantages

• All the limitation of earlier methods are eliminated in this method- memory, carry over, practice skills, maturity factor.

• When the parallel forms of a test are not advisable and repetition is not possible, this method is employed to compute reliability of test.

Limitation

• Chance factors may effect scores on the two halves of the test in the same way; it tends to make the reliability index too high.

• A test can be divided in two parts in a number of ways, so that the reliability coefficient is not a unique.

• It is not possible to divide the test in two halves in such a way that both are equivalent.

• This method cannot be used in power test and heterogeneous test.

Sources of error

Item Sampling

Construction of the Items

Test administration

Scoring

Difficulty of the Test

Student Factor

Factors that contribute to consistency

• stable characteristics of the individual or the attribute that one is trying to measure.

Factors that contribute to inconsistency:

• Temporary but general characteristics of the individual: health, fatigue, motivation, emotional, strain

• Temporary and specific characteristics of individual: comprehension of the specific test task, specific tricks or techniques of dealing with the particular test materials, fluctuations of memory, attention or accuracy

• Aspects of the testing situation: freedom from distractions, clarity of instructions, interaction of personality, sex, or race of examiner

• Chance factors: luck in selection of answers by sheer guessing, momentary distractions

Satisfactory level of reliability

• A satisfactory level of reliability depends on how a measure is being used.

• The standard is taken from Nunnally (1978), who suggests that in the early stages of research on predictor tests or hypothesized measures of a construct, reliabilities of .70 or higher will be sufficient. scores,

• Nunnally (1978) recommends that a reliability of at least .90 is desirable, because a great deal depends on the exact score made by a person on a test.

Relationship between reliability and validity

• Validity and reliability are closely related

• A test cannot be considered valid the measurements resulting from it are reliable. Likewise, results from a test can be reliable but not necessarily valid.

• While a reliable test may provide useful valid information, a test that is not reliable cannot possibly be valid

Ways to improve reliability

• Write longer tests.

• Pay more attention to the careful construction of the test questions.

• Start planning the test and writing the items well ahead of the time the test is to be given.

• Write clear directions and use standard administrative procedures.