Reliability & Validity Reliability & Validity

ReliabilityReliability ““dependability”dependability” is the indicator consistent?is the indicator consistent? same result every time?same result every time? Does not necessary measure what you think it measures Does not necessary measure what you think it measures

(ex. May consistently measure something other than (ex. May consistently measure something other than the concept)the concept)

ValidityValidity measurement validity - how well the conceptual and measurement validity - how well the conceptual and

operational definitions mesh with each otheroperational definitions mesh with each other does measurement tool measure what we think ?does measurement tool measure what we think ?

Relationship between Measurement Reliability and ValidityRelationship between Measurement Reliability and Validity

reliability necessary for validity but does not reliability necessary for validity but does not

guarantee it (guarantee it (“necessary but not sufficient”)“necessary but not sufficient”) measure can be reliable but invalidmeasure can be reliable but invalid

Examples of Types of ReliabilityExamples of Types of Reliability

stability stability over timeover time

representative representative across different subgroups of a populationacross different subgroups of a population

(ex. of problems: young people may exaggerate (ex. of problems: young people may exaggerate their ages, older people may reduce theirs)their ages, older people may reduce theirs)

equivalence equivalence Comparable results from multiple indicatorsComparable results from multiple indicators intercoder reliabilityintercoder reliability

Improving ReliabilityImproving Reliability

clearly conceptualize constructsclearly conceptualize constructs increase level of measurement increase level of measurement use pretests, pilot studiesuse pretests, pilot studies e.g. use multiple indicators :e.g. use multiple indicators :

DependentVariable Measure

IndependentVariable Measure

Empirical

Association?

a2 a3a1 b1 b2

A B

Specific IndicatorsSpecific Indicators

Measurement Using Multiple Indicators

Neuman (2000: 167)

Selected Types of Validity: Face ValiditySelected Types of Validity: Face Validity

judgement by the scientific community that indicator judgement by the scientific community that indicator measures the conceptmeasures the concept

Construct/concept

MeasureScientific Community

Content ValidityContent Validity measure represents all the aspects of conceptual definition of measure represents all the aspects of conceptual definition of

construct.construct. judgement concerning how adequately a test samples behavior judgement concerning how adequately a test samples behavior

representative of the universe of behavior the test was representative of the universe of behavior the test was

designed to sampledesigned to sample..

Love

Beliefs& Values

CriterionCriterion The validity of indicator is verified by comparing it with another The validity of indicator is verified by comparing it with another

measure of the same constructmeasure of the same construct

PredictivePredictive Relies on occurrence of future event or behavior for Relies on occurrence of future event or behavior for

verification of indicatorverification of indicator ConcurrentConcurrent

relies on pre-existing accepted measure to verify indicatorrelies on pre-existing accepted measure to verify indicator Construct ValidityConstruct Validity

A type of measurement validity that uses multiple indicatorsA type of measurement validity that uses multiple indicators convergent and discriminateconvergent and discriminate

Some types of Measurement Validity Some types of Measurement Validity

Other Dimensions of ValidityOther Dimensions of Validity

Internal ValidityInternal Validity no error of logic internal to research designno error of logic internal to research design

External ValidityExternal Validity results can be generalizedresults can be generalized

Statistical validityStatistical validity correct statistical methodology chosen ?correct statistical methodology chosen ? assumptions fully metassumptions fully met

More terms & concepts for assessing validity (usually in experimental research)More terms & concepts for assessing validity (usually in experimental research)

Threats to Internal ValidityThreats to Internal Validity Selection biasSelection bias History effects (something in context changes)History effects (something in context changes) Maturation Maturation Testing effectTesting effect InstrumentationInstrumentation MortalityMortality Statistical regressionStatistical regression ContaminationContamination Compensatory behaviourCompensatory behaviour Experimenter expectancExperimenter expectanc

Threats to External ValidityThreats to External Validity RealismRealism ReactivityReactivity

Another Aspect of Measurement Design: Levels of MeasurementAnother Aspect of Measurement Design: Levels of Measurement

1-Nominal (ex. Mother tongue)1-Nominal (ex. Mother tongue) different categories (names, labels, images) different categories (names, labels, images) not rankednot ranked

2-Ordinal (county fair prizewinners ranked by first, second & third prize)2-Ordinal (county fair prizewinners ranked by first, second & third prize) different categoriesdifferent categories rank-orderedrank-ordered attributes indicate relatively more or less of that variableattributes indicate relatively more or less of that variable distance between the attributes of a variable is imprecisedistance between the attributes of a variable is imprecise

Levels of Measurement (cont’d)Levels of Measurement (cont’d) 3- Interval Measures (age measured by 5 year age groups)3- Interval Measures (age measured by 5 year age groups)

different categoriesdifferent categories ranked in orderranked in order Can tell amount of difference between categoriesCan tell amount of difference between categories Usually no true zeroUsually no true zero

4- Ratio Measures (age measured by date of birth)4- Ratio Measures (age measured by date of birth) different categoriesdifferent categories ranked in order ranked in order amount of difference between categories amount of difference between categories also possible to state proportion (have a true zero)also possible to state proportion (have a true zero)

Relations between levels --Relations between levels --can collapse from higher can collapse from higher into lower, not vice versainto lower, not vice versa

Babbie (1995: 101)

The Research ProcessThe Research Process