Design of Randomized Experiments on Networks When Treatment …samsi... · 2014-08-13 · Jake...

Background Estimating Causal E↵ects Testing Causal Hypotheses Summary/Next Steps

Design of Randomized Experiments on NetworksWhen Treatment Propagates: An Exploration

Bruce Desmarais

University of Massachusetts Amherst

Collaborators:Jake Bowers, Wayne Lee, Simi Wang, Mark Frederickson, Nahomi Ichino

Prepared for the 2014 Political Networks Conference, McGill University


How should experiments be designed to learn aboutinterference in networks?

Example: Ghana 2008 Voter Registration Fraud Experiment (Ichino)

868 Electoral RegistrationStations (vertices)connected by roads (edges)

Density of 2.2%

Graph Transitivity of70.3%

2 / 20


Notation for Causal Questions

Treatment assigned to i : Zi 2 {0, 1}

Vector of all treatment assignments: Z = {Z1, . . . ,Zn}

Sharp Null of No Treatment E↵ect: all potential outcomes are equal

Yi (Z) = Yi (Z0) = Yi (Zi = 1,Z�i ) = Yi (Zi = 0,Z0

�i ) 8Z 6= Z

0

Graph-conditioned exposure: Yit(Dit), Dit records exposure condition

6 / 20


Example of treatment propagation: The Ising Model

Initial treatment status drawn from Zi ⇠ Bernoulli(↵). “Infection”/“Exposure”

probability at each iteration is

pr(Ei = 1) =1

1 + exp( 2F (ki � 2mi ))

k is number of (directly adjacent) neighbors (0 ki � K )

m is number of already infected neighbors (0 mi � M)

F is “temperature” or “propensity to be infected”

This example: Only two time periods (t 2 {0, 1}). The Ising model controlsactual infection after an experimenter assigns Zi at t = 0. Record infection afterthe first period.

7 / 20


Example of exposure types on the Ghana nodesOne draw from the Ising propagations with pr(Zi = 1) ⇠ Bernoulli(.15) fort 2 {0, 1} and Temperature= 10.

d1 ⌘ Di (Zi = 1, 0 mit � M)

d(0,0) ⌘ Di (Zi = 0,mit = 0)

d(0,1) ⌘ Di (Zi = 0,mit � 1)

d01

d01

d01

d1

d1

d01

d01

d01d01

d01

d01d1

d01

d01d01d01

d01

d01

d01

d00

d00

d00

d00

d01

d00

d00

d01

d00

d00

d01d1

d01

d1

d00 d00

d01d01

d1

d00

d1

d01 d01

d1d01

d1

d01

d01

d01d00

d01d1

d1

d01 d01

d01

d01

d01

d01d01

d01d01

8 / 20


ATE: Aronow and Samii Estimator

Definitions:

Di indexes the exposure condition of observation i

⇡i (dk) is the probability i ends up in condition dkWe consider

d(0,1), untreated with at least one treated neighbor

d(0,0), untreated with no treated neighbor

Estimand:

µ̂(dk) =1

n

nX

i=1

I(Di = dk)Yi (dk)

⇡i (dk)

⌧̂(d(0,1), d(0,0)) = µ̂(d(0,1))� µ̂(d(0,0))

Testing:

Following Horvitz and Thompson (1952) and CLT:Var(⌧̂(d(0,1), d(0,0))) 1/N2

�Var(µ̂(d(0,1))) + Var(µ̂(d(0,0)))

�

9 / 20


Simulations to assess power and guide design

Baseline (pre-treatment) response

Y (0, 0) ⇠ U(0, 1)

Treatment allowed to propagate for one period and true (constant,multiplicative) model of causal e↵ects computed:

Y (1, 0) = Y (0, 1) = �Y (0, 0)

Consider parametersTreatment assignment probability ↵ 2 {0.05, 0.10, . . . , 0.50}‘Temperature’ T 2 {0, 10, . . . , 100}� 2 {0.26, 0.63}

Simulate 1,000 propagations at each combination of T and ↵.

For each realized propagation, assess power to reject H0 : ⌧ = 0 atsignificance level .05.

10 / 20


Power as a Function of N Indirectly Treated

� = 0.26 � = 0.63

100 200 300 400

0.0

0.2

0.4

0.6

0.8

Prop

ortio

n p=

=0.0

5

0.05

0.05

0.05

0.05

0.050.050.050.050.050.05

0.1

0.1

0.10.1

0.10.10.10.10.10.1

0.15

0.15

0.150.150.150.150.15

0.150.150.15

0.2

0.2

0.20.20.20.20.20.20.20.2

0.25

0.250.250.250.250.250.250.250.250.25

0.3

0.30.30.30.30.30.30.30.30.3

0.350.350.350.350.350.350.350.350.350.35

0.40.40.40.40.40.40.40.40.40.40.450.450.450.450.450.450.450.450.450.450.50.50.50.50.50.50.50.50.50.5

N Exposed to Treatment by Propagation in 1 Period100 200 300 400

0.00

0.05

0.10

0.15

Prop

ortio

n p=

=0.0

5

0.05

0.05

0.05

0.050.05

0.05

0.05

0.050.05

0.05

0.1 0.10.1

0.10.10.10.1

0.10.10.1

0.15 0.15 0.150.150.150.150.150.150.150.150.2 0.2 0.20.20.20.20.20.20.20.20.25 0.250.250.250.250.250.250.250.250.250.3 0.30.30.30.30.30.30.30.30.30.350.350.350.350.350.350.350.350.350.350.40.40.40.40.40.40.40.40.40.40.450.450.450.450.450.450.450.450.450.450.50.50.50.50.50.50.50.50.50.5

N Exposed to Treatment by Propagation in 1 Period

11 / 20


Power With Additive E↵ects Model

Y (0, 0) = 1 + U(0, 1)

Y (0, 1) = �+ U(0, 1)

� = 0.26 � = 0.63

100 200 300 400

0.00

0.05

0.10

0.15

0.20

0.25

0.30

Prop

ortio

n p=

=0.0

5

0.05

0.05

0.05

0.05

0.05

0.050.05

0.050.050.05

0.10.1

0.1

0.10.10.10.10.1

0.10.1

0.15 0.150.150.150.150.150.150.150.150.15

0.2 0.2 0.20.20.20.20.20.20.20.20.25 0.250.250.250.250.250.250.250.250.250.3 0.30.30.30.30.30.30.30.30.30.350.350.350.350.350.350.350.350.350.350.40.40.40.40.40.40.40.40.40.40.450.450.450.450.450.450.450.450.450.450.50.50.50.50.50.50.50.50.50.5


0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

Prop

ortio

n p=

=0.0

50.05

0.05

0.05

0.050.050.05

0.05

0.050.05

0.05

0.1 0.10.1

0.10.10.10.1

0.10.10.1

0.15 0.15 0.150.150.150.150.150.150.150.150.2 0.2 0.20.20.20.20.20.20.20.20.25 0.250.250.250.250.250.250.250.250.250.3 0.30.30.30.30.30.30.30.30.30.350.350.350.350.350.350.350.350.350.350.40.40.40.40.40.40.40.40.40.40.450.450.450.450.450.450.450.450.450.450.50.50.50.50.50.50.50.50.50.5


12 / 20


Power With Certain Propagation

Y (d00) = 1 + U(0, 1)

Y (d01) = �+ U(0, 1)

� = 0.26 � = 0.63

100 200 300 400

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Prop

ortio

n p=

=0.0

5

0.05 0.050.05

0.050.05

0.050.050.050.050.05

0.1 0.10.1 0.10.1

0.10.10.1

0.1

0.1

0.15 0.15 0.150.150.150.150.150.150.150.15

0.2 0.2 0.20.20.20.20.20.20.20.20.25 0.250.250.250.250.250.250.250.250.250.3 0.30.30.30.30.30.30.30.30.30.350.350.350.350.350.350.350.350.350.350.40.40.40.40.40.40.40.40.40.40.450.450.450.450.450.450.450.450.450.450.50.50.50.50.50.50.50.50.50.5


0.00

0.02

0.04

0.06

0.08

0.10

0.12

Prop

ortio

n p=

=0.0

5

0.05

0.050.05 0.050.05

0.05

0.05

0.05

0.05

0.05

0.1

0.10.1

0.10.10.10.10.10.10.1

0.15 0.15 0.150.150.150.150.150.150.150.150.2 0.2 0.20.20.20.20.20.20.20.20.25 0.250.250.250.250.250.250.250.250.250.3 0.30.30.30.30.30.30.30.30.30.350.350.350.350.350.350.350.350.350.350.40.40.40.40.40.40.40.40.40.40.450.450.450.450.450.450.450.450.450.450.50.50.50.50.50.50.50.50.50.5


13 / 20

Correlating Treatment with Degree

●

●

●●●●●●

●●

●●●

●●●

●

●●●●●

● ●

●

−0.10 −0.05 0.00 0.05 0.10

0.45

0.50

0.55

0.60

0.65

0.70

0.75

Prop

ortio

n Si

g. (p

=0.0

5)

Degree−Treatment Corr.

α = 0.05 ●

●

●●

●

●●●●●

●

●

●

●

●●

●

●

●●

●

●

● ●

●

−0.10 −0.05 0.00 0.05 0.10

0.35

0.40

0.45

0.50

0.55

0.60

0.65

Prop

ortio

n Si

g. (p

=0.0

5)


α = 0.1 ●

● ●●●

●●●●

●●●

●

●●●

●●●●

●

●

●

●

●

−0.10 −0.05 0.00 0.05 0.10

0.2

0.3

0.4

0.5

Prop

ortio

n Si

g. (p

=0.0

5)


α = 0.15

●

●

●

●

●

●●

●

●

●

●●

●

●

●●

●●●●●

●

●

●

●

−0.10 −0.05 0.00 0.05 0.10

0.100.150.200.250.300.350.400.45

Prop

ortio

n Si

g. (p

=0.0

5)


α = 0.2 ●

●

●

●●

●

●

●

●

●

●

●●●

●

●●

●

●

●

●

●

●

●

●

−0.10 −0.05 0.00 0.05 0.10

0.05

0.10

0.15

0.20

0.25

Prop

ortio

n Si

g. (p

=0.0

5)


α = 0.25 ●

●

●

●

●

●●●

●●

●●

●

●●

●

●

●●●●

●

●●

●

−0.10 −0.05 0.00 0.05 0.10

0.05

0.10

0.15

0.20

0.25

Prop

ortio

n Si

g. (p

=0.0

5)


α = 0.3

●

●

●

●●

●

●

●

●

●

●●

●●

●

●●●

●●

●●

● ●

●

−0.10 −0.05 0.00 0.05 0.10

0.02

0.04

0.06

0.08

0.10

0.12

0.14

Prop

ortio

n Si

g. (p

=0.0

5)


α = 0.35 ●

● ●●●●●●

●

●●●

●●

●

●

●

●

●

●

●

●

●

●

●

−0.10 −0.05 0.00 0.05 0.10

0.00

0.02

0.04

0.06

0.08

Prop

ortio

n Si

g. (p

=0.0

5)


α = 0.4 ●

●

●●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

−0.10 −0.05 0.00 0.05 0.10

0.01

0.02

0.03

0.04

0.05

Prop

ortio

n Si

g. (p

=0.0

5)


α = 0.45


Tests of a Sharp Null: Distributional Di↵erences

Bowers, Jake, Mark M. Fredrickson and Costas Panagopoulos. 2013.“Reasoning about Interference Between Units: A General Framework.” PoliticalAnalysis 21(1):97–124.

Details: Anderson-Darling k-sample test statistic (Scholz and Stephens, 1987);randomization distributions via RItools (Fredrickson, Bowers, Hansen 2014).

15 / 20


Power as a Function of N Indirectly Treated

100 200 300 400

0.5

0.6

0.7

0.8

0.9

1.0

Power to Reject the Sharp Null of No Effects


Prop

ortio

n p=

0.05

0.05

0.1

0.15

0.2

0.25 0.30.350.40.450.5

0.050.10.150.20.250.30.350.40.450.5

100 200 300 400

0.5

0.6

0.7

0.8

0.9

1.0

Power to Reject the Sharp Null of No Difference D01 vs D00


Prop

ortio

n p=

0.05

0.05

0.1

0.15

0.2 0.25 0.3 0.350.40.450.5

0.050.10.150.20.250.30.350.40.450.5

16 / 20


Power With Additive E↵ects Model

100 200 300 400

0.70

0.75

0.80

0.85

0.90

0.95

1.00



Prop

ortio

n p

<= 0

.05

0.05

0.1

0.15

0.2 0.25 0.30.350.40.450.5

0.050.10.150.20.250.30.350.40.450.5

100 200 300 400

0.70

0.75

0.80

0.85

0.90

0.95

1.00



Prop

ortio

n p

<= 0

.05

0.05 0.1

0.150.2 0.25

0.30.350.40.45

0.5

17 / 20


Power With Certain Propagation

100 200 300 400

0.70

0.75

0.80

0.85

0.90

0.95

1.00



Prop

ortio

n p

<= 0

.05

0.05

0.1 0.15 0.2 0.25 0.30.350.40.450.5

0.050.10.150.20.250.30.350.40.450.5

100 200 300 400

0.70

0.75

0.80

0.85

0.90

0.95

1.00



Prop

ortio

n p

<= 0

.05

0.05 0.1 0.15 0.2 0.25 0.30.350.40.450.5

18 / 20


Summary

Under propagation, power optimizing proportion assigned to treatment maybe much less than 0.5.

Higher order parameters of treatment assignment distribution candramatically a↵ect statistical power

Trade-o↵ between power to detect network-moderated andnon-network-moderated e↵ects (see also Bowers, Fredrickson,Panagopoulos 2013).

Test statistics matter: even if interest focuses on ATE, the ATE mayprovide little power when the true model is not a simple distributional shift.

Limitation on power driven by the need for strong (i.e., isolated) controls.

19 / 20


Next Steps

More measures of design adequacy: RMSE, Type I error, Power.

Make pr(Zi = 1) depend on topology if not also covariates.degree-correlated assignment

community-wise assignment schemes

Consider higher-order spreading

Re-parameterize Ising for substantive interpretation.

Provide more general simulation system that accommodates di↵erent:models of propagation

models of e↵ects

statistical/causal inferential focus

20 / 20

Design of Randomized Experiments on Networks When Treatment …samsi... · 2014-08-13 · Jake...

Documents

Transcript of Design of Randomized Experiments on Networks When Treatment …samsi... · 2014-08-13 · Jake...