Research Designs

REVIEW

Review -- research

• General types of research– Descriptive (“what”)– Exploratory (find out enough

to ask “why”)– Explanatory (“why”)

• Unit of analysis: “object, entityor process” under study– Contains the variables

being measured– Case: A single instance

of a unit of analysis

Review - variable• Any characteristic that is (a) measurable (b) can take on different values in the

population or over time– Incarceration rate (no. of persons sent to prison/100,000 population)– Mean number of Boston youth shot dead each month– Period when Boston youth were shot dead (pre- or post-Ceasefire)– Height, weight, gender

• Types of variables– Categorical

• Nominal: mutually exclusive categories (e.g., M/F)• Ordinal: implied ranking (low/medium/high)

– Continuous (e.g., height, weight, scale 1-20)– Can “transform” ordinal into continuous, and continuous into ordinal

• Low/medium/high 1-3 scale• 1-20 scale low (1-5)/medium (6-10)/high (11+)

• Coding– Process of assigning a measurement to a variable– To use certain statistical techniques, nominal variables are sometimes recoded

as “dummy” variables• M/F recoded 0 for male, 1 for female

Review - distributions• An arrangement of cases in a sample or population according to their values or scores on one or more

variables

• Statistics – mean, median, mode, range, standard deviation – summarize distributions

Review - association and causation• Association means that the

values of two or more variableschange together – In Boston, the number of

youth shot dead appearsto be associated with thestudy period

– After invoking Ceasefirethe mean number of youths slain by gunfiredrops

• Causation means that changesin one variable cause corresponding changes in another variable. – The causal variable is called the “independent” variable (here it’s the

time period)– The effect variable is called the “dependent” variable (here it’s the

mean number of monthly deaths) – So, did Ceasefire cause the reduction?

RESEARCH DESIGNS –NON-EXPERIMENTAL

Principles of non-experimental designs• Begin with a hypothesis

– Changes in independent variables(s) changes in dependent variable(s)

– Lower income more crime• Assess the hypothesis by collecting data on variables of interest.

– Data usually reflects the values of variables at one point in time– Data can also be collected in “waves,” meaning at succeeding

points in time– In non-experimental designs investigators only collect data -

they do nothing that might affect the values of the variables • Data sources

– Field observations– Surveys– Official sources (public records, census, etc.)

Non

-exp

erim

enta

l des

igns

Data source: field observations• Research question: do police officers take harsher legal measures if youths display a

bad attitude?• Hypothesis: worse demeanor harsher disposition • Researchers rode along with cops to observe their interactions with youths• Researchers did NOT intervene -- they let things be• Researchers coded...

– Independent variable:youth’s demeanor (2 values)

– Dependent variable:officer disposition (4 values)

• At a later time they used statisticaltechniques to assess whether youth’sdemeanor was associated withofficer disposition in the hypothesizeddirection (the worse the demeanor,the harsher the disposition)

• Depending on the strength of this association they might conclude:– There is a cause-and-effect

relationship between the variables: hypothesis confirmed– The association does not go beyond what could be obtained by chance:

hypothesis rejected

Non

-exp

erim

enta

l des

igns

Non

-exp

erim

enta

l des

igns

Data source: official sources

Panel5

Panel9

Panel10

Non

-exp

erim

enta

l des

igns

Data source: surveys

Panel1

Panel6

Panel4

Non

-exp

erim

enta

l des

igns

Data source: surveys

Panel8

Panel12

Data sources: surveys + official sourcesN

on-e

xper

imen

tal d

esig

ns

Panel3

Panel7

Data sources: surveys + official sourcesN

on-e

xper

imen

tal d

esig

ns

Panel11

• Causal order: Did the change in the independent variable precede (come before) the change in the dependent variable?

Poverty crime OR Crime poverty

• Intervening variables: Could lack of education or living in a violent area be the more proximate (closer) cause of crime?

Poverty poor education crime

Here poverty is still the cause, but it affects crime through intervening variable education, which is the more proximate cause

• Spurious relationship: What seems to be a relationship isn’t - it’s bogus!

― Often caused by a strong association between the independent variable of interest (e.g., poverty) and another independent variable (e.g., poor social controls) which turn out to be the real cause

Poor social controls crime

Poverty

Issues in non-experimental designsN

on-e

xper

imen

tal d

esig

ns

RESEARCH DESIGNS –EXPERIMENTAL

Principles of experimental designs• Purposes–Eliminate other possible “causes” (e.g., that it’s education, not poverty)–Set the causal order (e.g., know you are testing crime poverty)

• Method1.Randomly assign cases to two or more groups. Designate one or more groups as “experimental” and one as “control”2.Measure the dependent variable (time 1) for each group. Random assignment insures that the mean values of the independent variable(s) should be about the same for each group.3.Intervene in the experimental group by adjusting the level of the independent variable of interest4.Post-measure dependent variable (time 2) for each group. If the differences between experimental and control groups are “statistically significant” they can be attributed to the intervention.

• Simple experiment intervention (adjust level of independent variable)

Experimental group ( X ) DVt1……….IV……….DVt2

Control group ( C ) DVt1…..………………DVt2 (no intervention)

Exp

erim

enta

l des

igns

population:200 patrol officers

150 males (75%) 50 females (25%)

Apply the intervention (apply the value of the independent variable – the program.)

NO YES YES NO

CONTROLGROUP

Randomly Assign25 Officers

CONTROL GROUP

Randomly Assign25 Officers

EXPERIMENTALGROUP

Randomly Assign25 Officers

EXPERIMENTALGROUP

Randomly Assign25 Officers

For each group, pre-measure dependent variable officer cynicism

Hypothesis: officers who complete a special training program will be less cynical

For each group, post-measure dependent variable officer cynicism

Also compare within-group changes – what do they tell us?

Exp

erim

enta

l des

igns

Exp

erim

enta

l des

igns

Hypothesis: SOCP reduces recidivism1. Independent (causal) variable: SOCP

(yes/no) (categorical/nominal)

2. Dependent (effect) variable: recidivism (rearrest rate, continuous)

3. Randomly assign youths being released to either X or C– Random assignment makes them

equal overall for background factors such as age, criminal record, etc.

4. X (experimental group) gets intensive supervision (SOCP yes)

5. C (control group) remains with regular supervision (SOCP no)

6. Wait two years, compare recidivism – Does the X group have a significantly

lower rearrest rate?– Does the X group have significantly

lower rates of drug & alcohol use?

Population: youths 12-18 citedduring an 18-month period

Apply the intervention YES NO

CONTROL GROUP

Randomly Assign265 youths

EXPERIMENTALGROUP

Randomly Assign264 youths

pre-measure arrest record, drug and alcohol use, etc.

For each group, post-measure dependent variable measures: arrest

record, drug and alcohol use, etc.

1973 Kansas City Patrol Experiment• Research question: Does routine patrol deter crime?• Hypothesis: Routine patrol reduces crime

1. Independent (causal) variable: Patrol(categorical/ordinal - three levels)

2. Dependent (effect) variable: crime rate(continuous)

3. Randomly divide an area into 15 beats

4. Measure crime in each beat

5. Randomly assign each a different value ofthe independent variable – Five C (control) beats: same patrol

as usual

– Five X1 (experimental) beats: no patrol(“R” - reactive - only answer calls for service)

– Five X2 experimental beats: more patrol than usual (“P” - proactive - more cars cruising, looking for trouble)

6. After one year compare crime rates

Exp

erim

enta

l des

igns

Some issues with experimental designs • According to the Kansas City experimenters, there was no significant

difference in crime rates between the experimental and control groups. – Since neither increasing nor decreasing patrol made a difference, the

hypothesis that random patrol can reduce crime was rejected.• However, the experiment was later criticized:

– Level of the independent variable (amount of patrol) was not sufficiently increased in the proactive beats to be able to demonstrate a statistically significant effect

– Due to contamination by other units, level of patrol was not sufficiently reduced in the reactive beats to be able to demonstrate a statistically significant effect

• Other constraints– Practicality

• Could we experimentally test poverty crime?– Ethics

• Should we experimentally test poverty crime? Can we make some people poor, then see what happens!

Exp

erim

enta

l des

igns

RESEARCH DESIGNS –QUASI-EXPERIMENTAL

Quasi-experimental designs & issues• Experiment that lacks random assignment to groups

– Groups might differ along a key independent variable (“matching” often used to try to make up for this)

• Experiment without a control group

– An extraneous event might be the true cause of the change in the dependent variable

• A non-experimental design thatmimics an experiment

– A known intervention did takeplace (e.g., it’s known that thelevel of the independent variabledid change at a certain time)

– Measures of the dependent variable are available for theperiods before and after theinterventionQ

uasi

-exp

erim

enta

l des

igns

Data source: Vignette

Panel2

• “Vignettes” are brief descriptions of actual or realistic events that are administered to elicit responses by test subjects to key issues of interest to researchers

• In this example a vignette is used to test the hypothesis that police officers with military experience are more likely, in domestic violence situations, to show leniency to other veterans– Officer’s veteran status Officer’s disposition

• Since everyone was administered the vignette (there was no equivalent “control group”) the possibility exists that independent variables other than those tested could explain why officers acted as they did

Qua

si-e

xper

imen

tal d

esig

ns