Nigeria Impact EvaluationCommunity of Practice

Abuja, Nigeria, April 2, 2014

Measuring Program Impacts ThroughRandomization

David Evans (World Bank)

2

Objective

Evaluate the causal impact of a program or an intervention on some outcome

Examples How much did free distribution of

bednets decrease malaria incidence?

Which of two supply chain models was most effective at eliminating drug shortages?

3

Counterfactual Criteria

Treated & comparison groups…Have identical average

characteristics (observed & unobserved)

The only difference is the treatment

Therefore the only reason for any difference in outcomes is the treatment

Key question: What would participant look like if she hadn’t received the program?

4

Perfect Experiment

1. Identify target beneficiaries

2. Clone them!

Identical on the outside (observable)

Identical on the inside (unobservable)

Chief Ahun

We’re both middle-aged

chiefs

We both love to take up new

health interventions!

Chief Batun

5

Perfect Experiment

Give the intervention to one set of clones

Ahun Batun

6

Perfect Experiment

Observe some time later

Because the groups are identical (inside & out), the difference is due to the bednets!

Ahun

Batun

7

Back to Reality

What would Batun look like if he didn’t receive the bednet?

Room For Improvement Control Groups

Before – After Participants – Non Participants

???

RFI: Before-After

BEFORE BEDNETS

6 malaria episodes in 6 months

AFTER BEDNETS

2 malaria episodes in 6 months

What else might be going on besides the bednets?

• Seasonal differences• Rising incomes: Households invest in other measures

Too many other factors!

Impact of bednets = ???

RFI: Before-After

Important to monitor before-after Insufficient to show impact of

programToo many factors changing over time

Example of cash transfers in Nicaragua!

Counterfactual: What would have happened in the absence of the project, with everything else the same

10

RFI: Participants vs Non-Participants

Compare recipients of a program to People who were not eligible for the

program People who chose not to enroll in the

program

Home births Clinic births

Example: Complications in childbirth

Impact of clinic births?

What else might explain the difference?

11

RFI: Participants vs Non-Participants

Observable differences Income Education

Unobservable differences Heard rumor about hospitals Neighbor available to care for other

children

Mercy Patience

12

RFI: Participants vs Non-Participants

How much of difference is because of clinic?

Impact of clinic births = ???

Home births Clinic births

Example: Complications in childbirth

Impact of clinic births

Other factors!

13

Selection bias

People who choose to join the program are different!

If we cannot account completely for those differences in our data…

We never can How do you capture attitudes toward

health systems? Initiative? …then our comparison will not show

the true impact of the program

What should we do?

Gold standard:Randomized experimental design

15

Randomized Experimental Design

Randomly assign potential beneficiaries to be in the treatment or comparison group

Treatment and comparison have the same characteristics (observed and unobserved), on average, so…

Any difference in outcomes is due to treatment

16

Why Randomization Works

Randomization with two doesn’t work!

But differences average out in a big sample

On average, same number of Ahuns and Patiences Observable AND unobservable

Result: Measure true impact of program

ComparisonTreatment

Comparison

Treatment

17

Random Sample or Assignment?

RANDOM SAMPLE

Select randomly who to gather data on

Gives unbiased average of the group NOT of impact

If take random sample of group: Half women, half men – Sample should be about ½ women, ½ men

RANDOM ASSIGNMENT

Randomly assign who gets the program

Gives unbiased estimate of program impact

Why? Treatment & comparison

are IDENTICAL (on average)

T

T

C

C

18

LET’S RANDOMIZE!

1. Identify the eligible participants

What is the impact of

receiving a new car on body-mass

index?

19

LET’S RANDOMIZE!

2. Generate a random number for each one

20

LET’S RANDOMIZE!

2. Generate a random number for each one

21

LET’S RANDOMIZE!

3. Re-order based on the random number

22

LET’S RANDOMIZE!

4. Assign the first ten to receive cars

I really wanted a car!

23

LET’S RANDOMIZE!

5. Check for balance across treatment & control

Treatment# Drs: 4

Control# Drs: 6

Total observations21

Not a very big sample!

How close? 2/3

24

LET’S RANDOMIZE!

What if we had 500 observations?

Treatment# Drs: 114

Control# Drs: 116

How close? 98/100

25

Is there more?

That’s a simple way to randomize Works with BIG samples

You can help randomization by stratifying Randomize within each sub-group Ensure that each group is equally

represented

26

Stratifying

1. Identify the characteristic(s) to stratify on

27

Stratifying

2. Sort on those characteristics

28

Stratifying

3. Generate those random numbers

29

Stratifying

4. Sort on those characteristics

30

Stratifying

5. Assign treatment within each sub-group

Result: Equal doctors in each group

31

Can we really randomize?

Randomization does not mean denying people the benefits of the project

Usually existing constraints in project roll-out allow randomization

Randomization often the fairest way to allocate treatment

32

Use Staggered Roll-out

Roll-out to 200 clinicsRoll-out to 400 more

clinics

Jan 2014

July 2014

Jan 2015

• Randomize the order in which clinics receive program

• Compare Jan 2014 group to Jan 2015 group at end of first year

33

Some groups must get the program

34

Vary treatment

INTENSITY OF TREATMENT

Malaria Information Campaign•100 villagesMalaria Information Campaign + SMS Reminders•100 villages

NATURE OF TREATMENT

Radio campaign•100 villages

Newspaper campaign•100 villages

35

Randomization is often the fairest

Watch the movie![link]

Randomization of an early child development program in Côte d’Ivoire

36

What if randomization is impossible?

Think again: It often is possible on some level, and it’s the best way to get a clear measure of impact Always begin the IE with imagining what the

ideal would look like

With a national policy Use randomization to test implementation

37

Key takeaway #1

The single best way to evaluate the unbiased average impact of an activity is by randomizing treatment.

38

Key takeaway #2

It is more ethical to test programs rigorously before universally implementing them than it is to use scarce public resources to implement a universal program with uncertain benefits.

39

Key takeaway #3

Randomization is more flexible than you think:

It does not require withholding of benefits.

It can take advantage of necessary staggered roll-out.

It can test different reforms or packages of services across groups at the same time (so all receive at least some package).

40

Let’s randomize

41

Thank you!

BONUS SLIDES

43

44

LET’S RANDOMIZE!

What if we had 100 observations?

Treatment# Drs: 26

Control# Drs: 22

Total observations21

Not a very big sample!