Manipulation and Measurement of Variables

Psych 231: Research Methods in Psychology

Announcements For labs this week you’ll need to download

(and bring to lab):– Class experiment packet

Choosing your independent variable

Choosing the right range (the right levels) of your independent variable– Review the literature– do a pilot experiment– consider the costs, your resources, your limitations – be realistic– pick levels found in the “real world”– pick a large enough range to show the effect

Potential problems

These are things that you want to try to avoid by careful selection of the levels of your IV (issues for your DV as well).

Demand characteristics

Characteristics of the study that may give away the purpose of the experiment

May influence how the participants behave in the study– Examples:

• Experiment title: The effects of horror movies on mood• Obvious manipulation: Ten psychology students looking

straight up• Biased or leading questions: Don’t you think it’s bad to

murder unborn children?

Experimenter Bias

Experimenter bias (expectancy effects)– the experimenter may influence the results

(intentionally and unintentionally)• E.g., Clever Hans

– One solution is to keep the experimenter “blind” as to what conditions are being tested

• Single blind - experimenter doesn’t know the condition

• Double blind - neither the participant nor the experimenter knows the condition

Knowing that you are being measured– just being in an experimental setting, people don’t

always respond the way that they “normally” would.

• Cooperative• Defensive• Non-cooperative

Reactivity

Floor effects

A value below which a response cannot be made– Imagine a task that is so difficult, that none of your

participants can do it. – As a result the effects of your IV (if there are

indeed any) can’t be seen.

Ceiling effects

When the dependent variable reaches a level that cannot be exceeded– Imagine a task that is so easy, that everybody

scores a 100% (imagine accuracy is your measure).

– So while there may be an effect of the IV, that effect can’t be seen because everybody has “maxed out”.

So you want to pick levels of your IV that result in middle level performance in your DV

Measuring your dependent variables:

Scales of measurement - the correspondence between the numbers representing the properties that we’re measuring– The scale that you use will (partially) determine

what kinds of statistical analyses you can perform

Scales of measurement

Categorical variables– Nominal scale

Scales of measurement

Nominal Scale: Consists of a set of

categories that have different names. – Measurements on a nominal scale label and

categorize observations, but do not make any quantitative distinctions between observations.

– Example:• Eye color: blue, green, brown, hazel

Scales of measurement

Categorical variables– Nominal scale– Ordinal scale

Scales of measurement

Ordinal Scale: Consists of a set of categories that are organized in an ordered sequence. – Measurements on an ordinal scale rank

observations in terms of size or magnitude.– Example:

• T-shirt size:

Small, Med, Lrg, XL, XXL

Scales of measurement

Categorical variables– Nominal scale– Ordinal scale

Quantitative variables– Interval scale

Scales of measurement

Interval Scale: Consists of ordered categories where all of the categories are intervals of exactly the same size. – With an interval scale, equal differences between numbers

on the scale reflect equal differences in magnitude. – Ratios of magnitudes are not meaningful.– Example:

• Fahrenheit temperature scale

20º40º“Not Twice as hot”

Scales of measurement

Categorical variables– Nominal scale– Ordinal scale

Quantitative variables– Interval scale– Ratio scale

Scales of measurement

Ratio scale: An interval scale with the additional feature of an absolute zero point.

With a ratio scale, ratios of numbers DO reflect ratios of magnitude.– It is easy to get ratio and interval scales confused– Consider the following example: Measuring your height with

playing cards

Scales of measurementRatio scale

8 cards high

Scales of measurementInterval scale

5 cards high

Scales of measurementInterval scaleRatio scale

8 cards high 5 cards high

0 cards high means ‘no height’

0 cards high means ‘as tall as the table’

Scales of measurement

Categorical variables– Nominal scale– Ordinal scale

Quantitative variables– Interval scale– Ratio scale

“Best Scale?”: • Given a choice, usually prefer highest level of measurement possible

Errors in measurement

Reliability – if you measure the same thing twice (or have two

measures of the same thing) do you get the same values?

Validity – does your measure really measure what it is

supposed to measure? • Does our measure really measure the construct?• Is there bias in our measurement?

Reliability & Validity

Reliability =consistencyValidity = measuring what is intended

unreliable reliable reliableinvalid invalid valid

Example: How can we measure intelligence?

Reliability & Validity

Reliability

True score + measurement error– A reliable measure will have a small amount of

error– Multiple “kinds” of reliability

Reliability

Test-restest reliability– Test the same participants more than once

• Measurement from the same person at two different times

• Should be consistent across different administrations

• Sensitive to type of measure

Reliability

Internal consistency reliability– Multiple items testing the same construct– Extent to which scores on the items of a measure

correlate with each other• Cronbach’s alpha (α)• Split-half reliability

– Correlation of score on one half of the measure with the other half (randomly determined)

Reliability

Inter-rater reliability– Extent to which raters agree in their observations

• Are the raters consistent?

– At least 2 raters observe behavior• Need a second opinion

– Requires some training in judgment

Validity

Does your measure really measure what it is supposed to measure? – There are many “kinds” of validity

VALIDITY

CONSTRUCT INTERNAL EXTERNAL

FACE CRITERION-ORIENTED

PREDICTIVE

CONCURRENT

CONVERGENT

DISCRIMINANT

Construct Validity

Usually requires multiple studies, a large body of evidence that supports the claim that the measure really tests the construct

Face Validity

At the surface level, does it look as if the measure is testing the construct?

“This guy seems smart to me, and he got a high score on my IQ measure.”

External Validity

Are experiments “real life” behavioral situations, or does the process of control put too much limitation on the “way things really work?”

External Validity Variable representativeness

– relevant variables for the behavior studied along which the sample may vary

Subject representativeness – characteristics of sample and target population

along these relevant variables Setting representativeness

– ecological validity

Internal Validity

The precision of the results Did the change result from the changes in the

DV or does it come from something else?

Threats to internal validity History – an event happens the experiment Maturation – participants get older (and other

changes) Selection – nonrandom selection may lead to

biases Mortality – participants drop out or can’t continue Testing – being in the study actually influences

how the participants respond

“Debugging your study”

Pilot studies– A trial run through– Don’t plan to publish these results, just try out the

methods

Manipulation checks– An attempt to directly measure whether the IV

variable really affects the DV.– Look for correlations with other measures of the

desired effects.

Next time Read chapters 8. Remember: For labs this week you’ll need to

download:– Class experiment packet