1

RESEARCH DESIGNRESEARCH DESIGN

A Plan for ResearchA Plan for Research

A RESEARCH DESIGNA RESEARCH DESIGN

■■ A Plan of Action =A Plan of Action =

Design LOGIC &Design LOGIC &

A detailed plan for A detailed plan for data collection and data collection and analysis: “Everything analysis: “Everything but cell entries in the but cell entries in the tables!” tables!”

2

Choosing a Research Design: Choosing a Research Design: LOGICAL considerationsLOGICAL considerations

■■ What type of question am I asking?What type of question am I asking?–– Purpose?Purpose?

–– Exploratory (a case study)Exploratory (a case study)

–– DescriptiveDescriptive

–– ExplanatoryExplanatory

–– PredictivePredictive

–– Policy Evaluation (Pilot study)Policy Evaluation (Pilot study)

What Research Design Logic What Research Design Logic would be most desirable?would be most desirable?

■■ What do I want to explain? What do I want to explain? –– What are my Xs? What are my Xs? YsYs? ? CntsCnts variables?variables?

■■ When (crossWhen (cross--sectional or longitudinal?)sectional or longitudinal?)

■■ About WHOM?About WHOM?–– Units of AnalysisUnits of Analysis

–– Level of Analysis Level of Analysis

■■ How Can I avoid LOGICAL FALLACIES?How Can I avoid LOGICAL FALLACIES?

3

SCOPE of ResearchSCOPE of Research

■■ ““SpacialSpacial--temporal domain” (where & temporal domain” (where & when)when)

■■ What are your “units of analysis”??? What are your “units of analysis”??? individuals or aggregatesindividuals or aggregates

■■ “MICRO” “MICRO” vsvs “MACRO” level“MACRO” level

■■ What domain? population or sampleWhat domain? population or sample

■■ Time Frame?Time Frame?

What measurement What measurement procedures should I use to procedures should I use to

collect data?collect data?■■ Experiment Experiment -- lab, field , quasilab, field , quasi--experimentalexperimental

■■ Participant Observations (unobtrusive)Participant Observations (unobtrusive)

■■ InterviewsInterviews

■■ SurveysSurveys

■■ Secondary Data analysis (existing sources)Secondary Data analysis (existing sources)–– Content Analysis (any trace record)Content Analysis (any trace record)

–– Comparative case studies (library sources)Comparative case studies (library sources)

4

Experiments are the Experiments are the STRONGEST Design Logic STRONGEST Design Logic

■■ The “best” researchThe “best” research

design to determinedesign to determine

CAUSAL RELATIONS CAUSAL RELATIONS

■■ Why not used more Why not used more often in Education?often in Education?

Why Best? Why Best? ---- CONDITIONS CONDITIONS necessary to demonstrate necessary to demonstrate

CAUSALITYCAUSALITY

■■ EMPIRICALEMPIRICAL (*)(*)

■■ *1) Time Order*1) Time Order

■■ *2) Correlation*2) Correlation

■■ *3) No other Xs*3) No other Xs

■■ 4) Plausibility4) Plausibility

■■ 5)Consistency5)Consistency

5

CHARACTERISTICS of an CHARACTERISTICS of an EXPERIMENTEXPERIMENT

■■ RANDOM assignment: 0RANDOM assignment: 01 1 XX 002 (EXP Group)2 (EXP Group)

003 3 004 (CNT Group)4 (CNT Group)

■■ Temporal order 0Temporal order 011 X 0X 022

■■ Physically manipulate X (Treatment)Physically manipulate X (Treatment)

■■ Physical ISOLATION controlled Physical ISOLATION controlled conditonsconditons

■■ Minimizes “confounding” influencesMinimizes “confounding” influences

CHARACTERISTICS of an CHARACTERISTICS of an EXPERIMENT (5)EXPERIMENT (5)

■■ MAXIMIZESMAXIMIZES

■■ Researcher’sResearcher’s

■■CONTROLCONTROL

■■ MINIMIZESMINIMIZES

■■ CONFOUNDINGCONFOUNDING

■■ influencesinfluences

■■ Lab Experiment:Lab Experiment:

■■ 001 1 x 0x 022

■■ Control GroupControl Group

■■ 003 3 0044

6

Use Experimental designs Use Experimental designs whenever possiblewhenever possible

■■ Types of Types of EXPERIMENTALEXPERIMENTAL

DESIGNS:DESIGNS:–– LaboratoryLaboratory

–– Field ExperimentField Experiment

–– QuasiQuasi--ExperimentalExperimental

Advantages of a LAB Advantages of a LAB EXPERIMENT???EXPERIMENT???

■■ INTERNAL INTERNAL

VALIDITY: VALIDITY: HIGHHIGH

■■ “Confounding “Confounding influences influences associated with exp associated with exp conditions” conditions”

7

DISADVANTAGES of a LAB DISADVANTAGES of a LAB EXPERIMENTEXPERIMENT

■■ EXTERNAL EXTERNAL VALIDITY:VALIDITY:

VARIABLEVARIABLE

■■ ““GeneralizabilityGeneralizability

to other to other contexts” may contexts” may be limitedbe limited

Use another EXPERIMENTAL Use another EXPERIMENTAL DESIGN: Field ExperimentsDESIGN: Field Experiments

■■ Useful to evaluateUseful to evaluate

EFFECTIVENESSEFFECTIVENESS

■■ of a public policy of a public policy program.program.

■■ Example: “ Does Example: “ Does HEAD START HEAD START improve improve performance of low performance of low SES kids?”SES kids?”

■■ Exp Group: Exposed Exp Group: Exposed to Head Startto Head Start

■■ 0011 X 0X 022

■■ Control Group: NOTControl Group: NOT

■■ 003 3 0404

■■ ControversialControversial---->>

■■ Random assignment Random assignment to groups!!!to groups!!!

8

Results of Head Start Policy Results of Head Start Policy Experiments?Experiments?

■■ ““Does HEADSTARTDoes HEADSTART

improve performanceimprove performance

of low SES kids?”of low SES kids?”–– YES YES ––l l stst gradegrade

–– Some Some ––2 2 ndnd gradegrade

–– NO DIFFERENCENO DIFFERENCE——3 3 rd graderd grade

■■ EVALUATION? EVALUATION?

Interpreting Policy EvaluationsInterpreting Policy Evaluations

■■ Short Short vsvs Long effectLong effect

■■ Intended Intended vsvsUnintended impactsUnintended impacts

■■ What was purpose What was purpose or INTENT of or INTENT of PROGRAM???PROGRAM???

■■ Note: Politics NOTNote: Politics NOT

science can confuse science can confuse interpretations!!!interpretations!!!

9

Use QUASIUse QUASI--EXPERIMENTALEXPERIMENTALDESIGNSDESIGNS

■■ FIELD FIELD EXPERIMENTSEXPERIMENTS

for policy evaluationsfor policy evaluations

■■ Example: Connecticut Example: Connecticut Crackdown on Crackdown on speedersspeeders

■■ CONNECTICUTCONNECTICUT

■■ 001 1 X 0X 022

■■ 2nd state = 2nd state = CONTROLCONTROL

■■ 0033 004 4

■■ Similar on “ALL Similar on “ALL OTHER Relevant OTHER Relevant Variables” (ZsVariables” (Zs--))

Connecticut Crackdown StudyConnecticut Crackdown Study

■■ Does increased Does increased speed CRACK speed CRACK DOWN (X)DOWN (X)

reduce AUTO reduce AUTO FATALITIES??FATALITIES??

10

Results of CONNECTICUT Results of CONNECTICUT CRACKDOWN STUDY?CRACKDOWN STUDY?

■■ Does X (Crackdown)Does X (Crackdown)

lead to Y (reduced lead to Y (reduced auto fatalities?auto fatalities?

■■ Answer: Answer: YESYES

■■ Confounding influences??Confounding influences??

Use the “logic” of an Use the “logic” of an experiment whenever possibleexperiment whenever possible

■■ Even with nonEven with non--EXPERIMENTAL EXPERIMENTAL Designs,Designs,

■■ Comparative Case Comparative Case StudiesStudies

■■ Surveys, interviewsSurveys, interviews

■■ Aggregate Aggregate (interrupted)(interrupted)

time series analysistime series analysis

11

““Think” like an experimentalist Think” like an experimentalist when DESIGNING researchwhen DESIGNING research

■■ TYPES of TYPES of RESEARCH RESEARCH DESIGNS:DESIGNS:

■■ EXPERIMENTAL EXPERIMENTAL

■■ NONEXPERIMENTALNONEXPERIMENTAL

Try to use EXPERIMENTAL Try to use EXPERIMENTAL Design CHARACTERISTICS Design CHARACTERISTICS

■■ 5 CHARACTERISTICS of an 5 CHARACTERISTICS of an Experiment Experiment

■■ Random assignment of subjects (Random assignment of subjects (ESsESs))

■■ CNT and EXP GroupsCNT and EXP Groups

■■ PRE & POST Measures OPRE & POST Measures O1 1 OO22

■■ Physical manipulation of XPhysical manipulation of X

■■ Reduce impact of “confounding” Reduce impact of “confounding” influences (other variables or “influences (other variables or “ZxZx”) ”)

12

WEAKEST RESEARCH WEAKEST RESEARCH DESIGNDESIGN

■■ ““ONE SHOT”ONE SHOT”

■■ CASE STUDYCASE STUDY

■■WHY???WHY???

““OneOne--shot” Case Study: Preshot” Case Study: Pre--experimental designexperimental design

■■ X 0X 011

■■ Problems:Problems:

■■ PostPost--test only test only

■■ Must infer “X” to be causal agentMust infer “X” to be causal agent

■■ Lots of “confounding” (i.e., unmeasured)Lots of “confounding” (i.e., unmeasured)

variablesvariables

13

““OneOne--shot” Case Study:shot” Case Study:Why Use?Why Use?

■■ EXPLORATORY researchEXPLORATORY research

■■ InIn--depth understanding of processesdepth understanding of processes

hypothesis generation.hypothesis generation.

First stage of research....goal....>First stage of research....goal....>

■■ GENERALIZEGENERALIZE

How to “Strengthen” this How to “Strengthen” this design?design?

■■ Add preAdd pre-- and post and post testtest

■■

■■ Add CASESAdd CASES

■■

■■

■■ Add data pointsAdd data points

■■ QUASIQUASI--ExperimentalExperimental

■■ logic 0logic 01 1 X 0X 022

■■ Case #2 Case #2 003 3 004 4

■■ COMPARATIVE COMPARATIVE CASE STUDIESCASE STUDIES

■■ TIME SERIESTIME SERIES

14

CORRELATION DESIGNS: CORRELATION DESIGNS: 30+ cases 30+ cases

■■ CROSSCROSS--SECTIONAL SECTIONAL correctional designscorrectional designs

■■ Case #1 X1 X2 Y1Case #1 X1 X2 Y1

■■ Case #2 “ “Case #2 “ “

■■ Case #n (30+)Case #n (30+)

■■

■■ at ONE POINT inat ONE POINT in

■■ TIMETIME

■■ TIME SERIES TIME SERIES

■■ CORRELATION CORRELATION DESIGNSDESIGNS

■■ 001 1 002 2 003 3 0044 00n (30+n (30+■■ data data

points)points)

■■ ONE CASE OVER TIMEONE CASE OVER TIME

■■

““STRONGEST” Experimental STRONGEST” Experimental DesignDesign

■■ SOLOMON 4 GROUP DESIGN:SOLOMON 4 GROUP DESIGN:

■■ Group #1Group #1 0011 X 0X 022

■■ Group #2 0Group #2 03 3 0044

■■ Group #3 X 0Group #3 X 055

■■ Group #4 0Group #4 066

■■

■■ Purpose of 4 groups? Purpose of 4 groups?

15

To reduce “TEST EFFECTS”To reduce “TEST EFFECTS”

■■ ““Test Effect”Test Effect”

■■ changes associated changes associated with prewith pre--test”test”

■■

■■ SOLUTIONSOLUTION

Solutions to avoid TEST Solutions to avoid TEST EFFECTSEFFECTS

■■ No preNo pre--testtest

■■ Solomon 4 Group Solomon 4 Group Experimental Experimental DesignDesign

■■ Use Unobtrusive Use Unobtrusive measurementmeasurement

16

THREATS to INTERNAL THREATS to INTERNAL VALIDITYVALIDITY

■■ ““TEST effects” onlyTEST effects” only

■■ 1 of 10 THREATS1 of 10 THREATS

■■ INTERNAL INTERNAL VALIDITY =VALIDITY =

■■ confounding confounding influences influences associated with associated with experimental experimental (research) (research) conditionsconditions

VERY important RULEVERY important RULE

■■ Evaluating ACCURACY of resultsEvaluating ACCURACY of results

■■ You must be sure there were NO You must be sure there were NO THREATS to INTERNAL VALIDITY THREATS to INTERNAL VALIDITY before you can generalize to other before you can generalize to other contextscontexts

■■ (EXTERNAL VALIDITY)(EXTERNAL VALIDITY)

17

THREATS to INTERNAL THREATS to INTERNAL VALIDITY (10)VALIDITY (10)

■■ No.1: Differences in No.1: Differences in Characteristics Characteristics SUBJECTSSUBJECTS

■■ SOLUTION???SOLUTION???

Threat to Internal Validity No. 1:Threat to Internal Validity No. 1:Differences in Differences in SSs s Characteristics Characteristics

■■ SOLUTION???SOLUTION???

■■ RANDOMIZERANDOMIZE

18

Threat to Internal Validity No. 2Threat to Internal Validity No. 2

■■ HISTORY HISTORY EFFECTS:EFFECTS:

■■ events or events or confounding confounding influences occurring influences occurring between prebetween pre-- and and postpost-- testtest

■■ (changes in (changes in environment)environment)

SOLUTION???SOLUTION???

■■ To Avoid HISTORY EFFECTS???To Avoid HISTORY EFFECTS???

19

Threat to INTERNAL Threat to INTERNAL VALIDITY No. 3VALIDITY No. 3

■■ MATURATION:MATURATION:

changes in subjects changes in subjects between prebetween pre-- and and postpost--teststests

■■ SOLUTION???SOLUTION???

Threat to INTERNAL Threat to INTERNAL VALIDITY No. 4VALIDITY No. 4

■■ TEST EFFECTS:TEST EFFECTS:

changes associated changes associated with prewith pre--testtest

■■ SOLUTION???SOLUTION???

20

THREAT to INTERNAL THREAT to INTERNAL VALIDITY No. 5VALIDITY No. 5

■■ INSTRUMENTATION:INSTRUMENTATION:

Changes associated Changes associated with change or decay in with change or decay in test instrumentstest instruments

■■ SOLUTIONS???SOLUTIONS???

THREAT to INTERNAL THREAT to INTERNAL VALIDITY NO. 6 VALIDITY NO. 6

■■ SUBJECT SUBJECT MORTALITY:MORTALITY:

Loss of subjectsLoss of subjects

■■ SOLUTIONS???SOLUTIONS???

21

Threat to INTERNAL Threat to INTERNAL VALIDITY NO. 7VALIDITY NO. 7

■■ REGRESSION REGRESSION towards the meantowards the mean

■■ unreliable measures unreliable measures (values) due to (values) due to choice of extreme choice of extreme values for subjectsvalues for subjects

■■ SOLUTIONSOLUTION

■■ Scores on testScores on test

■■ ff

■■ 100% x Es#1100% x Es#1

■■ xxxxxxxx

■■ 50% 50% xxxxxxxxxx

■■ xxxxxxxxxxxxxxxx

■■ 0% 0% xxxxxxxxxxxxxxxxxxxxxxxx

SOLUTION: RandomizeSOLUTION: Randomize

■■ Best way to Best way to minimize “atypical minimize “atypical values”values”

■■ Can calculate “odds”Can calculate “odds”or “p value” of or “p value” of getting an extreme getting an extreme scorescore

22

Another SolutionAnother Solution

■■ REPLICATION:REPLICATION:

Repeat same study Repeat same study with AT LEAST ONE with AT LEAST ONE new wrinklenew wrinkle

■■ If sample If sample representative representative should get same should get same resultsresults

THREAT to INTERNAL THREAT to INTERNAL VALIDITY NO. 8VALIDITY NO. 8

■■ REACTIVITY REACTIVITY (HAWTHORNE(HAWTHORNE

Effect):Effect):

■■ artificial change artificial change produced by produced by measurement measurement processprocess

■■ SOLUTION???SOLUTION???

23

THREAT to INTERNAL THREAT to INTERNAL VALDITY NO. 9VALDITY NO. 9

■■ EXPERIMENTEREXPERIMENTER

BIASBIAS

■■ SOLUTION???SOLUTION???

INTERNAL THREAT to INTERNAL THREAT to VALIDITY No. 10VALIDITY No. 10

■■ INTERACTIONINTERACTION

EFFECTS:EFFECTS:

■■ Effect caused by Effect caused by any combination of any combination of the above threatsthe above threats

■■ SOLUTION?????? SOLUTION??????

24

INTERACTION effectsINTERACTION effects

■■ Is the “WORST” Is the “WORST” threat to validity:threat to validity:

■■ can’t controlcan’t control

■■ can’t always predictcan’t always predict

■■ must try to avoidmust try to avoid

Threats to EXTERNAL Threats to EXTERNAL VALIDITYVALIDITY

■■ ““GeneralizabilityGeneralizability”...”...

■■ To what extent will To what extent will your findings apply your findings apply to other to other contexts???? contexts????

25

How Representative was How Representative was experiment?experiment?

■■ CRITERIA: JudgmentalCRITERIA: Judgmental

■■ Are SUBJECTS representative?Are SUBJECTS representative?

■■ Is experimental SETTING representative?Is experimental SETTING representative?

■■ Are MANIPULATIONS &MEASURES of X & Are MANIPULATIONS &MEASURES of X & Y representative of all Xs &Y representative of all Xs &YsYs??

■■ Are there UNIQUE circumstances Are there UNIQUE circumstances associated with the EXPERIMENTAL associated with the EXPERIMENTAL CONDITION NOT present in other contexts?CONDITION NOT present in other contexts?

GENERALIZABILITY of GENERALIZABILITY of resultsresults

■■ Internal/External Internal/External Validity are often … Validity are often …

■■ a subjective a subjective judgment… Are you judgment… Are you measure what you measure what you claim to be claim to be measuring???measuring???

26

EXTERNAL VALIDITYEXTERNAL VALIDITY

■■ To what extent can To what extent can your findings be your findings be applied to other applied to other contexts?contexts?

Choose Data Collection Choose Data Collection Method and Analytical Method and Analytical

RoutinesRoutines■■ Data Collection Data Collection

Methods:Methods:

■■ 1+ Case studies1+ Case studies

■■ ExperimentExperiment

■■ Surveys (1 time pt)Surveys (1 time pt)

■■ Content analysisContent analysis

■■ Aggregate data Aggregate data analysis analysis

■■ (cross(cross--sectional)sectional)

■■ (time series)(time series)

■■ Analytic method (stat)Analytic method (stat)

■■ Qualitative (none)Qualitative (none)

■■ ANOVA; xANOVA; x22

■■ Factor AnalysisFactor Analysis

■■ Descrip.stats;pDescrip.stats;p<.05<.05

■■ RegressionRegression

■■ CorrelationCorrelation

■■ Time SeriesTime Series

27

Design Logic... determinesDesign Logic... determinesMeasure & Analysis Measure & Analysis

■■ ExampleExample

■■ Descriptive study of US public Descriptive study of US public opinion. opinion.

■■ CrossCross--sectional (1 pt)sectional (1 pt)

■■ Units = individualsUnits = individuals

■■ Measurement =surveyMeasurement =survey

■■ stats: regression, factor analysisstats: regression, factor analysis

Research QUESTION Research QUESTION determines Research Design determines Research Design

SelectionSelection

■■ i.e., Study political i.e., Study political influence of drug influence of drug merchants in merchants in ColumbiaColumbia

■■ ....>participant ....>participant observation better observation better than interviewsthan interviews

28

““Real World” considerations Real World” considerations as Constraintsas Constraints

■■ Practical considerationsPractical considerations

determine Research determine Research Design & Revised Design & Revised Research questionsResearch questions

Consider FEASIBILITY & Consider FEASIBILITY & PRACTICALITY of Design PRACTICALITY of Design

■■ What’s my motivation?What’s my motivation?

■■ Basic or Applied researchBasic or Applied research

■■ Other ConsiderationsOther Considerations

■■ RESOURCES? Time & MoneyRESOURCES? Time & Money

■■ Can I get the data in allotted time?Can I get the data in allotted time?

■■ A Realistic Work ScheduleA Realistic Work Schedule

■■ Do I have a PUNT Position?????Do I have a PUNT Position?????

29

Remember:Remember:

■■ Double all time Double all time estimatesestimates

■■ Anticipate “No Anticipate “No Significant “ resultsSignificant “ results

■■ Remember (punt Remember (punt positions):positions):

■■ Pilot Studies,Pilot Studies,

■■ Replication &Replication &

■■ Exploratory Exploratory research research

How detailed should I get in How detailed should I get in writing a Research Design?writing a Research Design?

■■ Provide enough Provide enough details so SOME details so SOME ONE ELSE could ONE ELSE could “execute it”...collect “execute it”...collect the data and the data and analyze it according analyze it according to your plan”to your plan”

■■ “All but the cells in “All but the cells in the table”the table”

30

Congratulations!Congratulations!

■■ The hard creative The hard creative work is overwork is over

■■ on to ........on to ........

■■ Implementation of Implementation of your RESEARCHyour RESEARCH

ACTION PLAN!!!ACTION PLAN!!!