Lecture 3: Randomization

Lecture3:RandomizationMaartenVoorsandEGAPLearningDaysInstructors

9April2019— BogotáLearningDaysX

Today• Exercise• Threecoreassumptions• Review:RandomSamplingvs.RandomAssignment• Differentdesigns

• Access• Factorial• Timing(akastepped-wedge)• Encouragement

• StrategiesofRandomization• Simple• Complete• Blocked• Clustered• Factorial• (Twolevel)

• EssentialGoodPractices

Exercise

• Seehandout• 15minsworkinpairs/groupsofthree

• Makenotesforyourself

• 10minplenarydiscussion

Recapthesekeyterms(moretomorrow)

4

• Samplingdistributions• Standarddeviation(andvariation)• Standarderror• Confidenceinterval• Centrallimittheorem• p-value• T-test

Threecoreassumptions

1. Randomassignmentofsubjectstotreatmentso impliesthatreceivingthetreatmentisstatisticallyindependentofsubjects’

potentialoutcomes


2. Non-interference:asubject’spotentialoutcomesreflectonlywhethertheyreceivethetreatmentthemselveso Sounaffectedbyhowthetreatmentshappenedtobeallocatedo i.e.therearenospilloverso orSUTVAholds(stableunittreatmentvalueassumption)


3. Excludability:asubject’spotentialoutcomesrespondonlytothedefinedtreatment,nototherextraneousfactorsthatmaybecorrelatedwithtreatmento Importanceofdefiningthetreatmentpreciselyo Maintainingsymmetry betweentreatmentandcontrolgroups(e.g.,through

blinding,behavioralmeasures,etc)o Noattrition

Absentfromthelistofcoreassumptions…

• Randomsamplingofsubjectsfromalargerpopulationisnotacoreassumption• Thoughrandomassignmentislikerandomsamplingfromtwoalternative

universes

• Theissueofexternalvalidity isaseparatequestionthatrelatestotheissueofwhethertheresultsobtainedfromagivenexperimentapplytoothersubjects,treatments,contexts,andoutcomes

RandomSamplingvs.RandomAssignment

• Randomsampling(from population):selectingsubjectsfromapopulationwithknownprobability

• Randomassignment(to treatmentconditions):assigningsubjectswithknownprobabilitytoexperimentalconditions

RandomSamplingandRandomAssignmentRandomly samplefrom area of interest

Randomly samplefrom area of interest

Randomly assignto treatmentand control

RandomSamplingandRandomAssignment

StrictDefinitionofRandomAssignment

• Everyobservationmusthavethesameknownprobability• between0and1

RandomizationDesigns

1. Access

2. Factorial

3. Waitlist(akastepped-wedge)

4. Encouragement

RandomizationDesignI- Access

• Throughalottery

• Forexample,whenyoudonothaveenoughresourcestotreateveryone,randomlyselectatreatmentgroup

• Thisrandomizesaccesstotheprogram

• Example:HealthinterventionsinSierraLeone

ConsortDiagram

RandomizationDesignI- Access

• Sometimes,someunits(peoples,communities)musthaveaccesstoaprogram.

• EXAMPLE:apartnerorganizationdoesn’twanttoriskavulnerablecommunityNOTgettingaprogram(wantaguaranteethattheywillbealwaysbetreated).

• Youcanexcludethoseunits,anddorandomassignmentamongtheremainingunitsthathaveaprobabilityofassignmentstrictlybetween(andnotincluding)0and1.

RandomizationDesignII:FactorialDesign• Factorialdesignenablestestingofmorethanonetreatment

• Youcananalyzeonetreatmentatatime

• Orcombinationsthereof

T2=0 T2=1

T1=0 25% 25%

T1=1 25% 25%

Example:ColombianBureaucrats(Tara)

• Phoneauditonbureaucratsadministeringsocialprograms(SISBÉNandMás Familias en Acción)inColombianalcaldías

• Highdimensional,twodimensionswere:• Socialclass• Regionalaccent

Bogotá Costeño Paisa

LowerClass(~estratos 1and2)

306calls 306calls 306calls

LowerMiddleClass(~estrato 3)

306calls 306calls 306calls

RandomizationDesignIII-Timingofaccess

• Randomizetimingofaccess totheprogram

• Whenaninterventioncanbeormustberolledoutinstages,youcanrandomizetheorderinwhichunitsaretreated

• Oftenyoudonotthecapacitytoimplementthetreatmentinalotofplacesatonce.

RandomizationDesignIII-Timingofaccess

• Yourcontrolgrouparetheas-yetuntreatedunits• Becareful:theprobabilityofassignmenttotreatmentwillvaryovertime

VargasEGAPpresentation:TheTwistsandTurnsintheRoadtoJusticeandPeaceinColombia

60eligiblemunicipalities,Allshouldbetreated

RandomizationDesignIV- Encouragementdesign

• Randomizesinvitationstosubjectstoparticipateinaprogram.

• Usefulwhenyoucannot‘force’asubjecttoparticipate• andaprogramisONLYavailablethroughtheinvitation.

• Instrumentalvariables,exclusionrestriction• Vouchersforprivateschool,attendingprivateschool,academicperformance

• Wecanlearntheaveragecausaleffectforcompliers:thecausaleffectoftheparticipation(nottheinvitation!)fortheunitsthatparticipatewheninvitedanddon’tparticipatewhennotinvited.

RandomAssignmenttoRelevantUnits

• Treatmentcanbeassignedatmanydifferentlevels:individuals,groups,institutions,communities,timeperiods,ormanydifferentlevels.

• Youmaybeconstrainedinwhatlevelyoucanassigntreatmentandmeasureoutcomes.

• Yourchoiceofanalyticlevelaffectswhatyourstudycandemonstrate.

• Yourdesign?

Controlgroups

• Whattypeofcontrolgroupisneeded?• Nointervention?• Placebointervention?

• Example:• DidanewHausatelevisionstationinnorthernNigeriachangeattitudesaboutviolence,theroleofwomeninsociety,ortheroleofyouthinsociety?

• Doyouwanttolearntheeffectofwatchingafilm+contentofdrama?• Doyouwanttolearntheeffectofthecontentofthedrama,giventhatpeoplearewatchingafilm?

• Orboth?

Implementingrandomizationdesigns

1. Simple

2. Complete

3. Cluster

4. Block

5. Factorial

• Withacomputerinadvance!(ifyoucan)

BasicRandomization

• Excel• Stata• R

SimpleRandomization

• Foreachunit,flipacointoseeifitwillbetreated.Thenyoumeasureoutcomesatthecoin-level.

• Thecoinsdon’thavetobefair(50-50),butyouhavetoknowtheprobabilityoftreatmentassignment.

• Youcan’tguaranteeaspecificnumberoftreatedunitsandcontrolunits.• EXAMPLE:Ifyouhave6unitsandyouflipafaircoinforeach,youhaveabouta3%chanceofgettingallunitsassignedtotreatmentorallunitsassignedtocontrol.

• (1/2)6 +(1/2)6

Example

• Excel

• Stata

• R(inabit)

CompleteRandomization

• Mostcases

• AfixednumbermoutofN unitsareassignedtotreatment.

• Theprobabilityaunitisassignedtotreatmentism/N.

CompleteRandomization

DonebycomputerSimplygivearandomnumbertoeachofN unitsThenselecttheT unitswiththehighestrandomnumber

Blockrandomization

• Wecancreateblocksofthatcategoryandrandomizeseparatelywithineachblock.Youaredoingmini-experimentsineachblock.

• EXAMPLE:block=district,units=communities

• Probabilityoftreatmentassignmentcanbedifferentineachblock• Example:UnconditionalTransfersandDeforestation• Blocks:Chiefdoms,nj=6• Villages:n=68

• 68villages

• 46aid

• 22noaid

Blockrandomization

Blockrandomization

• Advantagestoblockingonfeaturesthatpredicttheoutcome:• Guaranteethatsomeunitsofevery“type”gettreatment,• Treatmentandcontrolgroupsaremoresimilardistributionsofthesetypesthanwithoutblocking

• Iftheblocksarelargeenough:youcanestimatetreatmenteffectsforthosesubgroups

• Usuallyimprovespower – yourprobabilityofdetectingatreatmenteffectifthereisone

• Generally,blockifyoucan.

Clusterrandomization

• Aclusterisagroupofunits,andallunitsintheclustergetthesametreatmentstatus.

• Thisisassigningtreatmentatthecluster-level.

VargasEGAPpresentation:TheTwistsandTurnsintheRoadtoJusticeandPeaceinColombia


• Aclusterisagroupofunits,andallunitsintheclustergetthesametreatmentstatus.

• Thisisassigningtreatmentatthecluster-level.

• Useiftheinterventionhastoworkattheclusterlevel.• Example:Vargas’study.Clustersarethetowns,unitsofanalysisarepeople

• Havingfewerclustershurtsyourpower.Howmuchdependsontheintra-clustercorrelation(rho).

• Higherisworse.


DonebycomputerSimplygivearandomnumbertoeachofN CLUSTERSThenselecttheT CLUSTERSwiththehighestrandomnumber


• Forthesamenumberofunits,havingmoreclustersandsmallerclusterscanhelp.

• Tradeoffspillover andpower

DidRandomizationWork?

• Ofcourse:always

• Makeitreplicable– Setaseed!• Don’tuseexcel

• Sometimesincreasedtransparency>replicability• Preservedistributions• Verify

Goodpractice

• CheckoverallbalancewithanD-squaretest(HansenandBowers2008)

• UseanF-test(treatmentassignmentonLHSandcovariatesonRHS)• Randomassignmentgivesus,inexpectation,overallbalanceonthecovariates.

• Youwillseet-testsofcovariatesonebyone.Justbychance,youmightgetdifferencesononevariable.

Goodpractice

• Afterrandomassignment,don’tmaketheTandCgroupsdifferentbytreatingthemdifferently!– maintainsymmetry

• Don’ttakeextrameasurementsoftheTgroup• Usethesamemeasurementstrategy• Asmuchaspossible,otherpeopleshouldnotknowwhetheraunitisinTorCsotheydon’ttreatthemdifferently

• Enumeratorsdon’tneedtoknowwhethertheyaresurveyingaTorCcommunity

LevelsofrandomizationIIRandomizeinsuchawayastominimizespillovers

• Whenonesubjectlearnsofanothersubject'streatmentstatus,thenon-interferenceassumptionisviolated

• Thenon-interferenceassumptionstatesthatasubjectcanreceivethetreatmentonlyif heorsheisassignedtotreatmentstatus– e.g.neighborsdiscusswhattheyhavelearnedinaneducationprogramwiththeirneighbors

• Treatingatthecommunitylevelcansometimesmitigatethis

• RandomizationforSpillovers

• Twoleveldesigns• Control• Spillovercontrol• Treatment

Ichino,Nahomi,andMatthiasSchündeln.2012.Deterringordisplacingelectoralirregularities?SpillovereffectsofobserversinarandomizedfieldexperimentinGhana.JournalofPolitics74(1):292-307.

Spillovers

• Whenonesubjectrespondstoanothersubject'streatmentstatus,thenon-interferenceassumptionisviolated

• EXAMPLE:vaccinationsandherdimmunity• EXAMPLE:peoplediscusswhattheyhavelearnedinaneducationprogramwiththeirneighborswhodidnotgototheprogram

• Whatistheproblemifwerandomizetreatmentassignmenttounits,andweestimatetheAverageTreatmentEffectas:

• Mean(Ys forthetreatedunits)– Mean(Ys forthecontrolunits)?

Spillovers

• OneproblemisthatYi(1)andYi(0)canbedifferent(notstable)dependingonwhichotherunitsaretreated,soYi(1)-Yi(0)isnotwell-defined.

• Oneexample:Y=daysIamsickthisyear(myvaccinationstatus,myroommate’svaccinationstatus)

Y(0,0) 30

Y(0,1) 20

Y(1,0) 10

Y(1,1) 5

Spillovers

• Youmightbeabletomitigatetheproblem:

• Ifyou’reworriedaboutspilloverswithinacommunitybecausepeopletalktoeachotherwithinacommunity,randomizeatthecommunitylevel(clusterrandomize).

• Ifyou’reworriedaboutspilloversacrosscommunitiesbecausepeoplevisiteachotheracrosscommunities,sampleyourcommunities(beforetreatmentassignment)inawaythatguaranteesthatthecommunitiesarefarapart.

• Butsometimes,youwanttostudythespillovers.•

DesignV– Two-Level

• CODEO’sobserversforvoterregistrationin2008

• Two-leveldesign• Constituency• RegistrationCenters

• Y(0,0),Y(1,0),Y(1,1)

Ichino,Nahomi,andMatthiasSchündeln.2012.Deterringordisplacingelectoralirregularities?SpillovereffectsofobserversinarandomizedfieldexperimentinGhana.JournalofPolitics74(1):292-307.

NineLimitationsofRandomization

1.Ethics – isthissortofmanipulationethical?Sometimesnot.2.Therealtime constraint.Sometimestoslow.Notmuchgoodtohelpunderstandhistory

3.Theproblemofcost (sometimes;butpossibleverylow)4.Thepowerconstraint.Youneedalotofunits(actually:aproblemforanystatisticalapproaches)

5.Externalvalidity(problemforanyevaluation)6.Theproblemofspillovers(problemforanyevaluation)7.Thevariablesasattributes constraint(problemforanyevaluation)8.Theassignmenttotreatmentconstraint.9.Reducedflexibility fororganization(problemforanyprospectiveevaluation)

Disclaimer

• AfewslidesarefromDonGreenfromhisbookwithAlanGerber,FieldExperiments

• ManyslidesareadaptedfrompreviouslecturesandEGAPworkshops

Lecture 3: Randomization

Documents

Transcript of Lecture 3: Randomization