The Role of Experimentation in Policy Making

The Role of Experimentation in Policy Making

Abhijit Vinayak BanerjeeIDB, September 2009

Theory is a very limited guide to policy

• It is very good at alerting us to various possible mechanisms

• It just does not tell us which of those mechanisms actually work and which don’t

• Because theory is always necessarily incomplete. – It is based on concepts, which come out of our

need to limit the complexity of reality– It is based on behavioral assumptions and

assumptions about the environment that are untested and often untestable

An example of a challenging concept• Education

• Let us accept the evidence that education promotes growth

• How to promote education?

• Concepts often used in the growth literature are things like “fraction of population completing primary school”– These are not policy levers

• Policy levers are things like:– Teacher-student ratio, textbooks, school uniforms,

school meals, etc. • Theory tells us nothing about which of works best

• Can we assume that the policy makers know?

An example of unexpected behavior• The demand for health goods

– Like immunization, insecticide treated bed-nets, chlorine for purifying drinking water

• Standard economic theory– These goods are valuable to some people (people who

live in malarial zones, say) less to others

– Pricing at marginal social cost correctly discriminates between those who want them more than the social costs and those who don’t

– However there are externalities: infectious diseases, intra-family issues

– So some subsidy may be justified.

– But no argument for giving them away

– Sunk cost fallacy argues for charging a positive price.

The pricing of health goods• Several studies are exploring

the impact of charging or subsidizing people for health behavior

• First example: Lentils for vaccine. – Immunization is really low in

Rajasthan (less than 5%)

– What is the price elasticity of vaccine demand?

Abdul Latif Jameel Poverty Action Lab

Testing Demand and Supply• One first possibility is that the supply channel is the

problem:– Conducted monthly immunization camps in 60 villages:

regular camps held rain or shine

• One second possibility is that there is a problem of demand: – People not interested in immunization, scared, etc. – Can demand be affected?– Extra incentive: in 30 of the camps, provided a kg of

lentils for each immunization

• 60 camps’ villages remained the control group. Immunization rates were followed in treatment villages, control villages, and one neighboring village of each of the treatment villages


Results•Immunization coverage

increased dramatically

At endline:

Impact of Immunization ProgramPercentage of children age 1-2 years fully

immunized

5.3%

36.9%

17.5%

0.0%

10.0%

20.0%

30.0%

40.0%

Control Villages Camp Villages Camp &Encouragement

Villages

Geographic Impact of Immunization ProgramsPercentage of children age 1-2 years outside of

treatment villages fully immunized

5.3%8.4%

27.2%

0.0%

10.0%

20.0%

30.0%

Control Villages Camp Villages Camp &Encouragement

VillagesAbdul Latif Jameel Poverty Action Lab

High price elasticity• Surprisingly high price elasticity, in light of the

expected benefits. Why might that be so: – Information?

– Discount rate?

– Something else?

• Smokeless stoves• Chlorine for water: same effect• Bed-Nets:

– Dupas and Cohen : randomize the price at which bednets are offered in pre-natal clinic

– Dupas: similar experiment among households.

• Deworming drugs: take up collapses with (small) positive price

Results: Demand Monthly Net Sales by ITN Price

0

20

40

60

80

100

120

140

160

180

200

0 10 20 40

ITN Price (Ksh)

Av

era

ge

Nu

mb

er

of

Ne

ts

So

ld/D

istr

ibu

ted

pe

r M

on

th


Should People Pay for Health Goods? • Conventional economic theory:

– If the health good has a high externality and is expensive, you want to subsidize its use (possibly even subsidize people to do it, like in the case of the immunization program), in particular when the price elasticity is high

• “New” conventional wisdom (social marketing)– Health goods you give away may not end up in

the hands of those who need it the most– The fact of paying may encourage people to use

the good (sunk cost fallacy)


Evidence from Bed nets

• Do people who get a free bed net use it less than those who have to pay?

• Do people who get a free bed net need it less than those who have to pay?


Results: Usage Share Observed Using ITN at follow-up

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 10 20 40

ITN Price (Ksh)

All

First Visits Only

First Pregnancy Only


Effective Coverage: Share of Prenatal Clients Sleeping Under ITN, by Price

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0 10 20 40 50ITN Price (in Ksh)

Nu

mb

er

Sle

ep

ing

un

de

r IT

N


Selection Effects of Price on Health

• Do higher insecticide treated net prices induce selection of more vulnerable women (i.e. sickest)?

• Result: No. People who buy nets at current cost-sharing price are healthier than the average prenatal client in the region – Ability to pay seems to be

the binding factor


Lessons• Find no evidence that cost-sharing reduces wastage on

those who will not use the product• Also find no evidence that cost-sharing induces selection

of those who need net more• Cost-sharing does considerably dampen demand

– As the price goes from zero to the prevailing cost-share price, uptake drops by 75%

• Another example where the cost benefit analysis is not what you thought!– Free distribution seems at least as cost-effective as

cost-sharing (because of externalities)• The debate generated by the paper: does this generalize?

– Proponents of cost sharing proposed various theories to explain why the results may not generalize.

– But they had to formulate new arguments, which can then be tested: this moved the debate beyond where it was… “cost sharing is good”.


Policy has to be based on learning from experience• History is a set of experiments…

• Why do we need Experiments?

Why Experiment: First answer Evaluating impact of alternative policies is not easy• What effect (if any) did the policy have?

– How would individuals who experienced the program have fared in the absence of the program?

– How would those who did not experience the program have fared if they had been exposed to the program?

• To know the impact of a program one must be able to answer the counterfactual:– How would an individual have fared without the program?– But one can’t observe the same individual with and without

the program• Need an adequate comparison group

– Individuals who, except for the fact that they were not beneficiaries of the program, are similar to those who received the program


Randomization • Gives us a control group that is ex ante

identical to the treatment group.

• Does not rely on theory to give us exclusion restrictions that then allow us to make valid comparisons– Very convenient when the theory is suspect

• May not always be useful without theory– You need a priori hypotheses

• May not always be useful without some econometrics.– You need some distributional assumptions

Why Experiment: Second answer• You typically know exactly who was treated or

something close

• And you control data collection

• Therefore you can target your data collection.

• As against in an observational study where the quasi-randomization emerges opportunistically. – You study a policy and typically decide ex post who was treated

and who was not. Unless you were directly involved in the policy making—in which case why not try to randomize?

• And especially with Instrumental Variables methods, you don’t always know who you are comparing

• And even when you do, the actual event may be many years past

• Therefore you cannot rely on data collected for you

• Hence detail/creativity in data collection is lower

Example: Decoy visits in the Evaluation of Police Reform in Rajasthan • Can the police be taught to investigate better, treat

people better and not to try to fob off complainants?

• In collaboration with Rajasthan Police Department

• How do you measure the willingess to register crimes

• Decoy visits: Surveyors act as real complainants. – Revealed themselves if the case was about to be registered

– Only 54% of complaints were registered

• Neither training nor more vacations for the police nor community observers in the police station had any effect on registration

• However the (random) number of previous revealed decoy visits matters: each visit increased the probability of FIR registration by 4%

Why experiment: Third answer• Experiments can be designed to test very

specific hypotheses

• Whereas observational studies, because they rely on policy variation that exists in the world, can rarely be mapped to something quite so narrow.

• Therefore we can test individual elements of a policy rather than the policy as a whole

Technology adoption: fertilizer in Kenya (Duflo-Kremer-Robinson)

• Current hike in food price underscore the needs to improve the productivity of agriculture in developing countries.

• The Green revolution has changed the face of Asian Agriculture since the 1970s. Need a green revolution for Africa

• Part of the solution is to develop new technologies, and part is to make sure the ones we have get used.

• Big puzzle: why do so few farmers adopt fertilizer in Western Kenya– The technology is well known– It has high returns (which we confirmed in experiments)– It can be done in small quantity

Possible reasons• Development economists have worked on

this topic for decades. Hypotheses:– Low actual returns– Lack of information/knowledge– Financing difficulties

• Duflo-Kremer-Robinson set up a series of field experiments over several years in Western Kenya to investigate these issues. – Controlled trial on (randomly selected) farmers’

plot: fertilizer seem profitable

Information• Several experiments to understand the role and

the diffusion of information:– Experiments on farmers plot were done with randomly

selected farmers: we follow their adoption and that of the control farmers

– Other ways to disseminate information: • Starter kits (Malawi)• Demonstration farms• Neighbors

– Invited– Non invited (natural diffusion)

• Bottom line: – information plays a role (10% increase in adoption

among pilot farmer)– However it does not diffuse naturally (no effect of

friends network)– Financial barriers remain a concern for farmers

Present-bias• Farmers seem to make plans to use fertilizer at harvest

time, but not carry them out• By the time of planting they have no money left. • Possibly it is because they postpone the purchase until

they need the fertilizer, but then they consume too much in the intervening period (may be because of family demands).

• If this is the case, a program that encourages them to purchase fertilizer right after harvest, rather than later, should be effective.

• With an NGO (ICS), we offer free delivery of fertilizer right after harvest (the “SAFI” program)

Results• SAFI is taken up by 40% of farmer and

led to a 14 percentage point increase in fertilizer use (a 60% increase).

• As large as the impact of a 50% subsidy• Offering free delivery at the time they

need it is not sufficient (it led to a 7 percentage point increase in use).

• When farmer are asked to chose the timing, they chose the early delivery.

What can’t experiments do?• Tell you what exchange rate to set

• But it is not clear that regressions of any kind are the right methodology for answering such questions

• May better to be build simulation models based on reliable parameter estimates.

• From micro regressions.

• Experiments may play an important role here

The question of external validity• How do we know that the result from an experiment

will generalize to other places

• Problem for all forms of empirical research.

• However it might seem that it is less of a problem for cross-country comparisons– Because the result is an average covering a wide range of

environments

• Not exactly right– Different results when you include different controls

– Could be because the set of countries being compared are different

• One advantage of experiments is you are much more likely to know the exact population that is being compared.

However• There is no dismissing this issue.

• Replications help build some confidence in the results– Theory helps give us identify the kind of

replications that make sense.

• The problem remains that as of now organizations that participate in randomized trials tend to be “strong” organizations

• We need more randomized evaluations in weak organizations

Most importantly• There is an infinity of policy questions out

there. – Textbooks, uniforms, meals, teachers,

blackboards, toilets, scholarships, computers..

• Theory is the only reliable guide to what are the right questions to answer

• Unfortunately in many cases we do not have a useable theory: that tells us, for example, textbooks are more like computers or more like teachers.

On balance however• We have learnt things from experiments we

could have not learnt otherwise• The benefits from the experimental approach

go beyond the specific learnings (with their limitations).– Have helped popularize a culture of emphasizing

experimentation and learning from experiments

– And have imposed some discipline on what can and cannot be claimed as known

• Does not mean that we learn only from Randomized Experiments

• But experiments set a standard of evidence and a culture of being willing to embrace failures.

• THANK YOU!

The Role of Experimentation in Policy Making

Documents

Transcript of The Role of Experimentation in Policy Making