Threats to ValidityThreats to Validity

overview of threats to validity

• confounding variables

• faulty manipulation

• loose procedures

• order effects

• experimenter effects

• Hawthorne effect

• sensitization

• history and maturation

• regression to the mean

• basement and ceiling effects

• social desirability bias

• selection bias (sampling)

• attrition (mortality)

confounding variables

• confounding variables; may reinforce or suppress the interaction between the independent and dependent variables (p. 267).

• extraneous variables; outside factors, that are not controlled for, that affect the outcome of an experiment (p. 267).

independentvariable

dependentvariable

confoundingvariable

faulty manipulation

• failure to manipulate a variable, or create a stimulus condition as intended– example: Janis & Feshbach's (1953) study on fear appeals

• Remedies:

– include a manipulation check: Teven & Comadena (1996) office aesthetics and teacher credibility

– cover story can disguise true purpose of an experiment

– use neutral observers to judge stimulus materials

loose procedures

• ambiguity or imprecision in the experimental protocol– The experimental procedures changes slightly from one

subject to the next

– Example: observer drift on judgments or ratings.

• Remedies:

– run a pilot study

– script out all instructions to participants

order effects

• the order, sequence, or placement of items in a series can influence participants' perceptions– example: Schuman, Presser, & Ludwig (1981) survey on

attitudes toward abortion

• Remedies:

– shuffle the order of items within a questionnaire, e.g. use multiple versions of the questionnaire

– alter the order if different questionnaires are completed together

experimenter effects

• experimenter expectancies (p. 208)– example: intercessory prayer

• cueing (p. 266)– example: Rosenthal (1966) unintentional

paralinguistic and kinesic cues by experimenters• demand characteristics of the experimental situation• Remedies:

– Have people other than the principle investigator perform the experiment

– Have people other than the principle investigator measure the outcome

Hawthorne effect

• Self-consciousness: the mere knowledge by a participant that he/she is being observed may alter his/her natural behavior (p. 266)– example: Western Electric plant in Illinois (circa late

1970's)

• Remedies:

– use a cover story

– use unobtrusive measures

sensitization

• Also known as “testing effects”

• familiarity with, or practice on, a specific test or task may improve scores.

• pre-test measurements may bias post-test measurements

• Remedies:

– avoid pre-tests, if possible

– use different pre- and post-test measures

history and maturation

• history: events happening outside the experiment that influence participants' responses within the experiment (p. 270)

• maturation: changes within in participants during an experiment (developmental, emotional, etc.) (p. 270)

• Remedies:– keep experiments short– sequester participants if possible

regression toward the mean

• Regression to the mean is a statistical fact of life. Simply stated, things tend to even out over time.– High or low scores are less likely to recur and

more likely to gravitate toward what is normal or average.

– example: Extremely tall parents tend to have children who are taller than other children, but not as tall as the parents.

– example: Sports teams that have a great season are less likely to repeat the following season.

– Remedy: Don’t rely on extreme scores from a nonrandom sample.

ceiling or basement effects

• ceiling effect: responses exceed capacity of measuring instrument; responses are “off the scale.”

• floor or basement effect: responses are below threshold of measuring instrument (see threshold effects, p. 266)

• Remedies:– use sensitive, precise measures– use established scales, instruments

social desirability bias

• Self reports are often distorted because respondents want to present themselves in the best possible light (p. 208)– Examples: in person interviews involving embarrassing

topics; infidelity, spouse abuse, cheating, etc.• Remedies:

– Emphasize anonymity of responses, importance for obtaining genuine responses, no correct answers

– use of indirect questioning: asking respondents to project how they think other persons would answer

selection bias

• failure to gather a random or representative sample, or a reliance on self-selected groups (p. 272)– example: the proverbial "college freshman" used in many

social science studies• Remedies:

– use a random, representative, sufficient sample (easier said than done)

– use random assignment within the experiment

mortality or attrition

• Subjects may drop-out or cease participating in a study. The ones who drop out may be different from the ones who remain (p. 272).– example: long-term “longitudinal” research

– example: studies on intimates

• Remedies:

– keep studies short

– provide incentives for participating