Measurements and Validity - Division on Addiction...Assessing reliability l Interitem Reliability -...
Transcript of Measurements and Validity - Division on Addiction...Assessing reliability l Interitem Reliability -...
Measurements and Validity
Julia Braverman, PhDDivision on Addictions
Types of measures
Michael John
ANXIETY??
Types of measurement
1. Objective/Physiological measures• Bodily activity, nervous system.• Response time
2. Observational measures• Direct observing participants.
3. Self-report• Participants provide information about
themselves.
Converging operations
Using several measurement approaches to measure a particular variable
Basics of psychometrics: How to build a trait/state assessment measure?
l Concept• Affect
• E.g. I feel sad• Behavior
• E.g. I cannot sleep, I cry a lot• Cognition
• E.g. I think about suicide.
l Question format (Likert scale, yes/no, reverse scale)
Measure quality
1. Reliability2. Validity
Reliability
l The degree of consistency between observations made by the same measurement tool.
Measurement Error. No measure is perfect.
l Observed score = True score + Measurement error.
l True score – is the score that the participant would have obtained if our measure were perfect.
Sources of measurement errors
1. Transient states• Mood, health, anxiety
2. Stable attributes• Suspicious participant may distort their answers
3. Situational factors• Weather outside, baseball game.
4. Characteristics of the measure• E.g. instruction ambiguity
5. Actual mistakes
Theoretical concept of reliability.
Systematic varianceReliability =
Total variance
0 < Reliability < 1
Assessing reliability
1. Test-retest reliability• Measuring the same thing twice.• Reliability = correlation ( r) between
results of the first and the second measurements.
• High reliability > .70
Assessing reliability
1. Test-retest reliability• Problems
•Memory•Experience
Assessing reliability
l Interitem Reliability - Measure of consistency among the items on a scale.1.Item-total correlation
qFor each item how it is correlated with the sum of other items. >. 30
2.Split-item reliabilityqDivide the items on the scale into 2 sets and test
the correlation (instead of test-retest).3.Cronbach’s alpha coefficient
qAverage of all possible split-half reliabilities.
Benevolent sexism scale: 1 (disagree) – 7 (agree)
1. Women should be cherished and protected by men.
2. Women, compared to men, tend to have a superior moral sensibility.
3. Men should be willing to sacrifice their own well-being in order to provide financially for the women in their lives.
4. Many women have a quality of purity that few men possess.
5. A good woman should be set on a pedestal by her man.
6. Men are complete without women.
Made-up table of item-total correlations
Item # r
1 .7
2 .5
3 .9
4 .6
5 .7
6 .4
Made-up table of item-total correlationsChronbach α = .85
Item # r Chronbach α(without the item)
1 .7 .8
2 .5 .8
3 .9 .7
4 .6 .8
5 .7 .8
6 .4 .9
Assessing reliability
l Interrater reliability – consistency between two or more raters or judges who observe the same behavior.• High reliability > .70
Increasing the Reliability Measures
1. Standardize administration of the measure• Same test conditions
2. Clarify instructions and questions. • To reduce ambiguity and misinterpretations.• Pretest questionnaires if possible.
3. Train observers.• To increase interrater reliability.
4. Minimize error in coding data.
Validity
l If the measurement actually measures what it is supposed to measure • Different from reliability• Same measure maybe valid for one purpose
and invalid for another one.
Assessing validity
1. Face validity – if a measure appears to be valid.• Does not mean actual validity. • E.g. SAT reading comprehension test
•Does it measure reading comprehension or common sense? (Katz et al., 1990)
• Affect motivation to participate?
Assessing validity
2. Construct validityn Relation to other measures.
§ Convergent validity§ High correlation with conceptually relevant
measures. n Discriminate validity
• Low correlation with conceptually unrelated constructs
Assessing validity
3. Criterion-Related validity – the correlation between the measure and some current behavior.• E.g. IQ and GPA• Doctor’s productivity
•Peer evaluation•Patient evaluation
Assessing validity
3. Predictive validity – the ability of a measure to predict a certain behavior/situation in a future. • E.g. SAT and GPA or GPA and after-college
salary. • Doctor’s productivity• ?
Test bias
l Test is biased if it is not equally valid for everyone who takes the test. • Groups with the same ability obtain different
scores on the test.
Reliability and Validity
l If reliable• May be valid or not.
l If not reliable • Not valid
Threats to measurement validity
l Using non-validated measures
l Solution• Validate the measure• Use pre-validated measures
Threats to measurement validity
l Loose connection between theory and method. • Disagreement between conceptional and
operational definitions. • E.g. putting more pepper as a measurement of
aggression?
• Solution• Validate your measure with previous
measurements
Threats to measurement validity
l Social desirability (evaluation apprehension) –Desire to look “normal” or to be judged favorably by another person (including the experimenter).
l Solutions• Anonymity• Ask indirect questions
• “How many drinks an average college student have during a party?”
Threats to measurement validity
l Yes-biasl Extreme-score bias
l Solution• Reverse score. • Z-transformation within an individual.
Threats to measurement validity
l Testing effects• Most participants perform better on a test of
personality/behavior/IQ measure the second time they take it.
l Reasons• Learning (e.g. IQ test)• Practice (e.g. physical skills)• Learn the test goal (e.g. personality test)• Attitude polarization
• Thinking about their attitudes
Threats to measurement validity
l Testing effectsl Solutions
• Control group• No pretest• Long waiting period
Validity of experiment
l Internal validity• Extent to which a study provides evidence of
a cause-effect relationship between the variables.
l External validity• The ability to generalize results of the
experiment.
Internal validity
l 3 conditions to determine causality• Covariation• Temporal sequence• No confounds
l Low internal validity – the conclusion that A affects B is wrong.
Threats to internal validity
l Role demands – participants’ expectations to what an experiment requires them to do• Good-subject tendency
• E.g. hypnosis and antisocial acts• Participants reactance
• E.g. What is the weather today?
Threats to internal validity
l Role demands
l Solution• Cover story
• E.g. Independent studies• Add non-relevant tasks, items (For
measurements)
Threats to internal validity
l Experimenter bias• E.g. Gratitude study
l Solution• Double-blind
Threats to internal validity
l Hawthorne effect – Increases in productivity that occur when participants know they are being studied. • Workers responded to any change in working
conditions by working harder than usual. l Solution
• Control group
Common Threats to Internal Validity of Quasi-experimentsl History
• Something occurred between the pretest and posttest.l Maturation
• Normal time changesl Regression to the mean
• If extreme scored Ss. were selected. l Pretest sensitization
• Pretest affects the posttest resultsl Selection bias
• Comparison groups differed from the beginningl Local historyl Contemporary historyl Attrition/mortality
• Only most motivated participants stay• Only participants who experience less adverse effects of treatment stay
External validity
l How well the findings of an experiment generalize to other situations or populations.
Threats to external validity
l Other subjects• Sampling/selection bias
l Other timesl Other settings
Threats to external validity
l Sampling bias• Motivated volunteers• Those available (at home, have phone)
Threats to external validity
l Other setting• Artificial experimental environment
External validity
l External validity - the ability to generalize results of the experiment.
l Tight control - highly specific and artificial situation -> less external validity.
Internal validity External validity
l You are a researcher. In your experiment, you assign the first 20 people in your study to the experimental condition and the second 20 people to your control condition. This could pose a threat to:• Internal validity• Reliability• External validity• Construct validity
l Saying that some measure is ________ definitely means it is also __________.• valid, reliable• reliable, valid• nominal, numerical• observational, self-report• none of the above
l An experimenter wants to examine if a new behavioral intervention program increases compliance among hypertension patients. For this purpose she recruits hypertension patients with low medication compliance and tests their compliance before and after the intervention. What are the potential threats to internal validity: • Regression to the mean• Maturation• History• Pretest sensitization• All of the above
Find a threat/threats to internal validityl The Alzheimers Center wants to evaluate the
effectiveness of their support groups for caregivers of individuals with Alzheimers Disease. The caregivers are given the choice when they first come to the center as to whether they want to join these support groups. The center gives a stress measure to the caregivers that attend these weekly meetings, once they have attended meetings for three months. They also administer the same stress measure to the caregivers who have not attended the support groups, as a control group. Both groups of caregivers are married to the person with Alzheimers disease and both groups have been involved with the center for the same length of time
Any questions?