Reliability and Validity - The Basics
-
Upload
sister-edith-bogue -
Category
Education
-
view
16.557 -
download
4
description
Transcript of Reliability and Validity - The Basics
![Page 1: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/1.jpg)
Reliability and Validity
Measurement
![Page 2: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/2.jpg)
Scales (Levels) of measurementScales clarify the characteristics of measurement
processesScales indicate which statistical procedures are
appropriateNominal
• Categories without order
• Colors, gender, political party, nationality
Ordinal
• Categories with order
• Size (S,M,L), Social class, Agreement (strong, some, low, none)
Interval
• Distance is meaningful between categories
• Temperature, ACT scores, shoe size, IQ
Ratio
• Scale of categories has absolute zero
• Age, income, all rates and percents, vacation time
![Page 3: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/3.jpg)
Levels of Measurement: Learn them by playing the
game!
![Page 4: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/4.jpg)
Does my measurement procedure give
the same accurate measurement each time it is used?
Reliability
![Page 5: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/5.jpg)
What is reliability?
Reliability is consistency in measurement
Stern Tone Variator, from The Archives of the History of American Psychology
![Page 6: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/6.jpg)
What is measurement validity?
Lavery Psychograph, from The Archives of the History of American Psychology
Validity is “truth” in measurement
![Page 7: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/7.jpg)
Reliable, but not valid
Reliable
Not valid.
![Page 8: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/8.jpg)
Valid, but not reliableValid
Not reliable
![Page 9: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/9.jpg)
Neither reliable nor validNot reliable
Not valid
![Page 10: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/10.jpg)
Both reliable and valid
Reliable:
Valid.
Bullseye!! by modenadude at http://www.flickr.com/photos/modenadude/3280286776
![Page 11: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/11.jpg)
Why do reliability and validity matter?
All of our research uses data gathered through measurement procedures.
Evaluate reliability and validity of measurement before applying results.
Evaluation usesLogicStatistical techniques
![Page 12: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/12.jpg)
Summary: Reliability and Validity
ReliabilityDoes the value observed and
recorded accurately reflect the “true” value of the object?
Test by measuring the object multiple times or ways.
Every researcher must either use a known instrument, or test and demonstrate the reliability of a new tool. The Literature Search is a
huge labor saving device Using a known instrument
improves research quality
ValidityDoes the value observed and
recorded reflect the concept and dimension of interest?
Test by comparing with other data or similar processes.
Every researcher must either use a known instrument, or test and demonstrate the validity of a new tool. The Literature Search is a
huge labor saving device Using a known instrument
improves research quality.
![Page 13: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/13.jpg)
Existing Measurement ToolsBlackboard: Buros
Blackboard: Social Psych Tools
![Page 14: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/14.jpg)
Reliability = True Score + ErrorUnreliable tools
introduce excess error
Development of new tools requiresTime, often yearsStaffing, often
several professionalsTrial subjects, often
in the hundredsLots of money to pay
for the above
![Page 15: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/15.jpg)
Methods to Establish Reliability
Test-RetestParallel FormsInter-Rater Reliability
Compute correlation of two or more forms, taken under same circumstances.
![Page 16: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/16.jpg)
Reliability – Internal Consistency
Multiple items (questions)
Cronbach’s alpha (α)
![Page 17: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/17.jpg)
Sources of Reduced Reliability
Unclear questions
Untrained raters
Unclear instructions
Outside events
Problems of Effective Range (Scale Attenuation)Ceiling effectThreshold effect
![Page 18: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/18.jpg)
Measurement in research
Real world is messy.
Reliability and Validity issues are intertwined
math problems for girls by woodleywonderworks at http://www.flickr.com/photos/wwworks/3597217248/
![Page 19: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/19.jpg)
W. Andrew Harrell describes his study
Two controversies are connected to
Dr. Harrell’s study:
Do physical traits such as
beauty have an evolutionary
impact (i.e., people are more
likely to have children to pass
along their genes)?
Isn’t beauty a subjective
judgment, not a trait that can be
objectively measured in a
research study.
Dr. Harrell addresses both of those
questions in this 4 minute audio
clip.
Click to listen
University Of Alberta (2005, April 13). Researchers Show Parents Give Unattractive Children Less Attention. ScienceDaily. Retrieved July 25, 2009, from http://www.sciencedaily.com /releases/2005/04/050412213412.htm
![Page 20: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/20.jpg)
Measurement in Harrell’s study
Direct observation of seat belt safety
Previously validated measures of beauty
Two trained researchers evaluate attractiveness(inter-rater reliability)
Two different trained researchers observe safety(inter-rater reliability and avoiding bias)
![Page 21: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/21.jpg)
How to improve reliabilityUse identical instructions.
Use a larger number of items (questions)
Eliminate questions that evoke inconsistent responses
Cover the entire range of the dimension
Reliability and validity must be tested & established in each new situation where the procedure is used.
![Page 22: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/22.jpg)
Does my measurement procedure measure the construct or concept
that I intend? Or is it measuring
something else?
Validity
![Page 23: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/23.jpg)
Two meanings of “validity”Validity is an over-arching concern of research
Measurement – are the observations directly and truly linked to the dimension or concept claimed?
Research design – how well does the experiment or study control the situation so that we are confident that the relationships or results observed were due to the impact of the independent variable?
Operationalization – The entire procedure used to produce the measurement. This includes any instrument, but also instructions for its use.
![Page 24: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/24.jpg)
Operationalization: How the Research is Actually Conducted
Blackboard: Obedience to Authority #2
![Page 25: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/25.jpg)
Methods of Determining Validity
Content validity“Face” validityRange of content
Construct /Internal structureOne or multiple
dimensions?Criterion validity
Predictive (GRE) Concurrent (audition)Convergent
Discriminant validityDivergenceKnown groups validity Manuscripts and checklists by Muffett at
http://www.flickr.com/photos/calliope/173797447/
Milgram’s obedience experiment
![Page 26: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/26.jpg)
Variety of Measurement Situations
Blackboard: The “Strange” Situation
![Page 27: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/27.jpg)
Sources of invalidity
Lavery Psychograph, from The Archives of the History of American Psychology
Incorrect theory
Measurement doesn’t match intention.
Many interpretations of meaning of the measurement.
![Page 28: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/28.jpg)
Relationship of reliability and validity
Both measured on a continuum (low to high)A measurement with low reliability cannot be
valid.Maximum validity is the square root of reliability.
Reliability and validity analysis comes before any statistical analysis.
yReliabilitValidity
![Page 29: Reliability and Validity - The Basics](https://reader036.fdocuments.us/reader036/viewer/2022062303/554934bfb4c9050f4d8b45e5/html5/thumbnails/29.jpg)
Reliability and Validity
Measurement