Randomized Experiments

11-07-15 2:02 PMMore Than 1 Billion People Are Hungry in the World - By Abhijit Banerjee and Esther Duflo | Foreign Policy

Page 1 of 15http://www.foreignpolicy.com/articles/2011/04/25/more_than_1_billion_people_are_hungry_in_the_world?print=yes&hidecomments=yes&page=full

More Than 1 Billion People AreHungry in the WorldBut what if the experts are wrong?

BY ABHIJIT BANERJEE, ESTHER DUFLO | MAY/JUNE 2011

For many in the West, poverty is almost synonymous with hunger. Indeed, the

announcement by the United Nations Food and Agriculture Organization in

11-07-15 2:02 PMMore Than 1 Billion People Are Hungry in the World - By Abhijit Banerjee and Esther Duflo | Foreign Policy

Page 2 of 15http://www.foreignpolicy.com/articles/2011/04/25/more_than_1_billion_people_are_hungry_in_the_world?print=yes&hidecomments=yes&page=full

2009 that more than 1 billion people are suffering from hunger grabbed

headlines in a way that any number of World Bank estimates of how many poor

people live on less than a dollar a day never did.

But is it really true? Are there really more than a billion people going to bed

hungry each night? Our research on this question has taken us to rural villages

and teeming urban slums around the world, collecting data and speaking with

poor people about what they eat and what else they buy, from Morocco to

Kenya, Indonesia to India. We've also tapped into a wealth of insights from our

academic colleagues. What we've found is that the story of hunger, and of

poverty more broadly, is far more complex than any one statistic or grand

theory; it is a world where those without enough to eat may save up to buy a TV

instead, where more money doesn't necessarily translate into more food, and

where making rice cheaper can sometimes even lead people to buy less rice.

But unfortunately, this is not always the world as the experts view it. All too

many of them still promote sweeping, ideological solutions to problems that defy

one-size-fits-all answers, arguing over foreign aid, for example, while the facts

on the ground bear little resemblance to the fierce policy battles they wage.

Jeffrey Sachs, an advisor to the United Nations and director of Columbia

University's Earth Institute, is one such expert. In books and countless speeches

and television appearances, he has argued that poor countries are poor because

they are hot, infertile, malaria-infested, and often landlocked; these factors,

however, make it hard for them to be productive without an initial large

investment to help them deal with such endemic problems. But they cannot pay

for the investments precisely because they are poor -- they are in what

economists call a "poverty trap." Until something is done about these problems,

neither free markets nor democracy will do very much for them.

But then there are others, equally vocal, who believe that all of Sachs's answers

are wrong. William Easterly, who battles Sachs from New York University at the

other end of Manhattan, has become one of the most influential aid critics in his

11-07-15 2:02 PMMore Than 1 Billion People Are Hungry in the World - By Abhijit Banerjee and Esther Duflo | Foreign Policy

Page 2 of 15http://www.foreignpolicy.com/articles/2011/04/25/more_than_1_billion_people_are_hungry_in_the_world?print=yes&hidecomments=yes&page=full

2009 that more than 1 billion people are suffering from hunger grabbed

headlines in a way that any number of World Bank estimates of how many poor

people live on less than a dollar a day never did.

But is it really true? Are there really more than a billion people going to bed

hungry each night? Our research on this question has taken us to rural villages

and teeming urban slums around the world, collecting data and speaking with

poor people about what they eat and what else they buy, from Morocco to

Kenya, Indonesia to India. We've also tapped into a wealth of insights from our

academic colleagues. What we've found is that the story of hunger, and of

poverty more broadly, is far more complex than any one statistic or grand

theory; it is a world where those without enough to eat may save up to buy a TV

instead, where more money doesn't necessarily translate into more food, and

where making rice cheaper can sometimes even lead people to buy less rice.

But unfortunately, this is not always the world as the experts view it. All too

many of them still promote sweeping, ideological solutions to problems that defy

one-size-fits-all answers, arguing over foreign aid, for example, while the facts

on the ground bear little resemblance to the fierce policy battles they wage.

Jeffrey Sachs, an advisor to the United Nations and director of Columbia

University's Earth Institute, is one such expert. In books and countless speeches

and television appearances, he has argued that poor countries are poor because

they are hot, infertile, malaria-infested, and often landlocked; these factors,

however, make it hard for them to be productive without an initial large

investment to help them deal with such endemic problems. But they cannot pay

for the investments precisely because they are poor -- they are in what

economists call a "poverty trap." Until something is done about these problems,

neither free markets nor democracy will do very much for them.

But then there are others, equally vocal, who believe that all of Sachs's answers

are wrong. William Easterly, who battles Sachs from New York University at the

other end of Manhattan, has become one of the most influential aid critics in his

11-07-15 2:02 PMMore Than 1 Billion People Are Hungry in the World - By Abhijit Banerjee and Esther Duflo | Foreign Policy

Page 3 of 15http://www.foreignpolicy.com/articles/2011/04/25/more_than_1_billion_people_are_hungry_in_the_world?print=yes&hidecomments=yes&page=full

books, The Elusive Quest for Growth and The White Man's Burden.

Dambisa Moyo, an economist who worked at Goldman Sachs and the World

Bank, has joined her voice to Easterly's with her recent book, Dead Aid. Both

argue that aid does more bad than good. It prevents people from searching for

their own solutions, while corrupting and undermining local institutions and

creating a self-perpetuating lobby of aid agencies. The best bet for poor

countries, they argue, is to rely on one simple idea: When markets are free and

the incentives are right, people can find ways to solve their problems. They do

not need handouts from foreigners or their own governments. In this sense, the

aid pessimists are actually quite optimistic about the way the world works.

According to Easterly, there is no such thing as a poverty trap.

This debate cannot be solved in the abstract. To find out whether there are in

fact poverty traps, and, if so, where they are and how to help the poor get out of

them, we need to better understand the concrete problems they face. Some aid

programs help more than others, but which ones? Finding out required us to

step out of the office and look more carefully at the world. In 2003, we founded

what became the Abdul Latif Jameel Poverty Action Lab, or J-PAL. A key part of

our mission is to research by using randomized control trials -- similar to

experiments used in medicine to test the effectiveness of a drug -- to understand

what works and what doesn't in the real-world fight against poverty. In practical

terms, that meant we'd have to start understanding how the poor really live

their lives.

Take, for example, Pak Solhin, who lives in a small village in West Java,

Indonesia. He once explained to us exactly how a poverty trap worked. His

parents used to have a bit of land, but they also had 13 children and had to build

so many houses for each of them and their families that there was no land left

for cultivation. Pak Solhin had been working as a casual agricultural worker,

which paid up to 10,000 rupiah per day (about $2) for work in the fields. A

recent hike in fertilizer and fuel prices, however, had forced farmers to

economize. The local farmers decided not to cut wages, Pak Solhin told us, but to

11-07-15 2:02 PMMore Than 1 Billion People Are Hungry in the World - By Abhijit Banerjee and Esther Duflo | Foreign Policy

Page 3 of 15http://www.foreignpolicy.com/articles/2011/04/25/more_than_1_billion_people_are_hungry_in_the_world?print=yes&hidecomments=yes&page=full

books, The Elusive Quest for Growth and The White Man's Burden.

Dambisa Moyo, an economist who worked at Goldman Sachs and the World

Bank, has joined her voice to Easterly's with her recent book, Dead Aid. Both

argue that aid does more bad than good. It prevents people from searching for

their own solutions, while corrupting and undermining local institutions and

creating a self-perpetuating lobby of aid agencies. The best bet for poor

countries, they argue, is to rely on one simple idea: When markets are free and

the incentives are right, people can find ways to solve their problems. They do

not need handouts from foreigners or their own governments. In this sense, the

aid pessimists are actually quite optimistic about the way the world works.

According to Easterly, there is no such thing as a poverty trap.

This debate cannot be solved in the abstract. To find out whether there are in

fact poverty traps, and, if so, where they are and how to help the poor get out of

them, we need to better understand the concrete problems they face. Some aid

programs help more than others, but which ones? Finding out required us to

step out of the office and look more carefully at the world. In 2003, we founded

what became the Abdul Latif Jameel Poverty Action Lab, or J-PAL. A key part of

our mission is to research by using randomized control trials -- similar to

experiments used in medicine to test the effectiveness of a drug -- to understand

what works and what doesn't in the real-world fight against poverty. In practical

terms, that meant we'd have to start understanding how the poor really live

their lives.

Take, for example, Pak Solhin, who lives in a small village in West Java,

Indonesia. He once explained to us exactly how a poverty trap worked. His

parents used to have a bit of land, but they also had 13 children and had to build

so many houses for each of them and their families that there was no land left

for cultivation. Pak Solhin had been working as a casual agricultural worker,

which paid up to 10,000 rupiah per day (about $2) for work in the fields. A

recent hike in fertilizer and fuel prices, however, had forced farmers to

economize. The local farmers decided not to cut wages, Pak Solhin told us, but to

11-07-20 1:00 AMSearch | The Abdul Latif Jameel Poverty Action Lab

Page 1 of 1http://www.povertyactionlab.org/search/apachesolr_search?view=map&filters=type:evaluation

The Abdul Latif Jameel Poverty Action Lab Contact J-PAL | Subscribe

TRANSLATING RESEARCH INTO ACTION

Legend Education Finance & Microfinance Environment & Energy Health Political Economy &Governance

Labor Markets Agriculture

Standard Search Region-Theme Grid

Search

To refine displayed results, select one ormore of the categories below:

Keyword:

Themes all...

Policy Goals all...

Region all...

Country all...

Researchers all...

Status all...

Data all...

11-07-15 2:02 PMMore Than 1 Billion People Are Hungry in the World - By Abhijit Banerjee and Esther Duflo | Foreign Policy

Page 6 of 15http://www.foreignpolicy.com/articles/2011/04/25/more_than_1_billion_people_are_hungry_in_the_world?print=yes&hidecomments=yes&page=full

But what if the poor are not, in general, eating too little food? What if, instead,

they are eating the wrong kinds of food, depriving them of nutrients needed to

be successful, healthy adults? What if the poor aren't starving, but choosing to

spend their money on other priorities? Development experts and policymakers

would have to completely reimagine the way they think about hunger. And

governments and aid agencies would need to stop pouring money into failed

programs and focus instead on finding new ways to truly improve the lives of

the world's poorest.

Consider India, one of the great puzzles in this age of food crises. The standard

media story about the country, at least when it comes to food, is about the rapid

rise of obesity and diabetes as the urban upper-middle class gets richer. Yet the

real story of nutrition in India over the last quarter-century, as Princeton

professor Angus Deaton and Jean Drèze, a professor at Allahabad University and

a special advisor to the Indian government, have shown, is not that Indians are

becoming fatter: It is that they are in fact eating less and less. Despite the

country's rapid economic growth, per capita calorie consumption in India has

declined; moreover, the consumption of all other nutrients except fat also

appears to have gone down among all groups, even the poorest. Today, more

than three-quarters of the population live in households whose per capita

calorie consumption is less than 2,100 calories in urban areas and 2,400 in rural

areas -- numbers that are often cited as "minimum requirements" in India for

those engaged in manual labor. Richer people still eat more than poorer people.

But at all levels of income, the share of the budget devoted to food has declined

and people consume fewer calories.

What is going on? The change is not driven by declining incomes; by all

accounts, Indians are making more money than ever before. Nor is it because of

rising food prices -- between the early 1980s and 2005, food prices declined

relative to the prices of other things, both in rural and urban India. Although

food prices have increased again since 2005, Indians began eating less precisely

Notes from: “Using Randomization in Development Economics Research: A Toolkit” (Duflo, Glennerster, Kremer 2008)

Randomization is now integral part of development economics

Over last ten years has grown to dominate the field

Diverse topics: education, health, technology adoption, micro-credit…

Randomized evaluations in developing countries – relatively cheap

Working with local partners allows for flexibility to researchers

Why Randomize?

Problem of causal inference

What is causal impact of specific program requires a counterfactual

At a given point in time, an individual is either exposed to the program or not

Comparing same individual over time does not work because other factors which affect outcomes have changed --- cannot get reliable estimate of program’s impact

Cannot obtain impact of program on a given individual – can obtain average impact of program on a group of individuals by comparing them to similar group of individuals who were not exposed to program

Need a comparison group – group of people who in absence of treatment would have had similar outcomes to those who received treatment

Reality- individuals exposed to treatments differ from those not exposed

programs placed in specific areas (poorer)

individuals screened for participation

decision to participate is voluntary, creating self-selection

Any difference between groups can be attributed to both impact of program and pre-existing differences (selection bias) – cannot decompose

Randomized evaluation removes selection bias

N individuals selected from population (not a random sample)

Divide N individuals into treatment and control randomly

Average treatment effect is difference in empirical means across two groups

Had the treatment not occurred – their outcomes would have been in expectation the same (selection bias equal to zero)

Provides and unbiased estimate of the impact of the program in the sample under study – estimate is internally valid

Many ways in which the assumptions in the simple framework may fail when randomized evaluations are implemented in the field in developing countries

Practicalities

Partners: governments, NGOs, private companies

Implementing real-world programs and interested in finding out whether they work or how to improve them

Partner is always in charge of implementing program

Government programs:

Meant to serve entire eligible population -- pilot programs run before

Most famous example in development: PROGRESA (grants to women) – randomized evaluation by government of Mexico --- due to budgets constraints could not implement in all communities at once

Randomized evaluations in collaboration with governments still relatively rare (require cooperation at high political levels – difficult to generate consensus for implementation)

Recent spread of randomized evaluations owes much to move towards working with non-governmental organizations (NGOs)

NGOs not expected to serve entire population

Even small organizations can substantially affect budgets for households, schools, or health clinics

Often eager to work with researchers to assess effectiveness of operations

For profit firms also started getting interested in randomized evaluations, goal is to understand their business and better serve their clients

Karlan et al. worked with private lender in South Africa – able to test models of lending

Design

Sample size and other design choices will affect the power of the experiment

Tradeoffs between how expensive data collection is compared to treatment – determines size of comparison group (data collection cheap if using administrative data)

Control Variables:

Controlling for covariates affected by the treatment would bias the estimate of the treatment effect by capturing part of the impact

Information on covariates should be collected in the baseline surveys (pre-treatment value of the outcome is of special interest)

Can reduce sample size requirement needed – e.g. controlling for baseline test scores in evaluations of education interventions greatly improves precision of estimates

Practical design choice is whether to randomize the intervention at the level of the individual, family, village, district etc.

Conducting baseline survey – in theory randomization renders baseline surveys unnecessary since it ensures treatment and comparison group are similar in expectation

Still want to collect:

Control variables will reduce sample size requirements

Check randomization was done appropriately

Notes from: “The Experimental Approach to Development Economics” (Banerjee and Duflo 2009)

Veritable explosion of randomized experiments in development economics

Rising tide of criticism

(most found in Heckman 1992 – mainly commenting on field of labour economics)

Promise of experiments:

Surprisingly positive results

o (Miguel/Kremer 2004) showed that deworming treatment (costs 49 cents/child per year) can reduce abesenteeims from by school by one-quarter

o In terms of increasing attendance – deworming is 20 times as effective as hiring an extra teacher, even though both work in the sense of generating statistically significant improvements

o Economic intuition would not have helped us come to this conclusion

o NGOs were equally uniformed about this comparison

Multiple treatment experiments can be informative

o Duflo, Kremer, Robinson (2010) reflects an iterative process

o succession of experiments on fertilizer use were run over a period of several years

o each set of results prompting the need to try out a series of new variation in order to better understand results of previous one

Theoretical Motivation

o Experiments designed to assess whether there is a demand for commitment products (Ashraf, Karlan, and Yin 2006) – came from theoretical motivation

o Karlan and others – experiments emerging as powerful too for testing theories

Biggest Advantage:

Experiments may be that they take us into terrain where observational approaches are not available

Objections raised by critics best viewed as warnings against over-interpreting experimental results

Also concerns about what experiments are doing to development economics as a field

Generalizability

Environmental Dependence - Core element of generalizability – would the same result occur in a different setting?

Effect is not constant across individuals – likely vary systematically with covariates?

Concern of implementer effects and compliance – smaller organization (NGO) – estimated treatment effect reflects unique characteristics of implementer

e.g. some NGOs refuse to randomize

Randomization Issues

Fact that there is an experiment going on might generate selection effects that would not arise in non-experimental setting (being part of an experiment and being monitored influences participants)

Villagers not used to private organization going around offering them things

Necessary that individuals are not aware that they are excluded from program (difficult when randomization is at individual level, easier if randomization is at village level)

Equilibrium Effects

Program effects from small study may not generalize when program is scaled up

e.g. :

Vouchers to go to private school

Students end up with better education and higher incomes

Scale up program to national level

Crowding in private schools (collapse of public schools)

Returns to education fall because of increased supply

Experimental evidence overstates returns to vouchers program

Notes from: “Instruments, Randomization, and Learning about Development” (Deaton 2010)

Effectiveness of development assistance is topic of great public interest

Much public debate among non-economists takes it for granted that, if the funds were made available, poverty would be eliminated -- Amongst economists, it is mixed.

Macro perspective: can foreign assistance raise growth and eliminate poverty?

Micro perspective: what sorts of projects are likely to be effective? Should aid focus on roads, electricity, schools, health clinics?

Answer – we don’t know – how should we go about finding out?

Frustration with Aid organizations

Particularly the World Bank

Allegedly failing to learn from its projects and to build up a systematic catalogue of what works and what does not

Movement toward randomized controlled experiments:

Esther Duflo:

“ randomized trials can revolutionize social policy during 21st century just as they revolutionized medicine during the 20th”

---- Lancet editorial headed “ The World Bank is finally embracing science”

Deaton argues:

under ideal circumstances randomized evaluations of projects are useful for obtaining convincing estimates of the average treatment effect of a program or project

This focus is too narrow and too local to tell us “what works” in development and to design policy or to advance scientific knowledge about development processes

Argues that work needs to be refocused – not answer which projects work but why

Randomization in the Tropics

Under ideal conditions an RCT can estimate certain quantities of interest with minimal assumptions

Originally laid out by Neyman in 1920s:

What we observe in the data:

E(Y1|T=1) – E(Y0|T=0) [*]

What we would like to know is effect of being treated on the treated:

E(Y1-Y0|T=1)

[*] rewritten as:

[E(Y1|T=1) – E(Y0|T=1)] + [E(Y0|T=1) – E(Y0|T=0)]

second term is = 0 by randomization

given expectation is a linear operator we have: E(Y1-Y0|T=1)

Difference in means between treatment and control groups is estimate of the treatment effect on the treated

Since the treatment and control groups are random, is an estimate of the treatment effect for all

Result depends on randomization and linearity of expectations

RCT is only informative of the mean of the treatment effects

Cannot identify anything else in the distribution --- e.g. median which could be of equal interest to policy makers

Also would like to know the fraction of population for which the treatment was positive (cannot know this either)

So if very few are receiving large benefits, everyone else is hurting – could not know this

In practice, researchers who run RCTs present results other than the mean:

Yi = b0 + b1Ti + b2Xi + b3Ti*Xi + ei

Xi are baseline controls

Interaction between Xi and Ti – can see if treatment affected some groups more than others

These estimates depend on more assumptions than trial itself

One immediate charge is data mining

Sufficiently determined examination of any trial will eventually reveal some subgroup for which treatment effect was significant

Such analysis does not share the special features of an RCT – must be assessed same way we assess any other non experimental or econometric study

Also no (econometric) guarantee that another RCT run just on that subgroup will yield same result

Also relevant are assumptions – regarding variances between treatment and control group

Bigger question:

RCTs allow investigator to induce variation that might not arise nonexperimentally – but are these the relevant ones?

RCTs of “what works”

even when done without error of contamination

unlikely to be helpful for policy or move beyond the local

unless they tell us something about why

RCTs are not targeted or suited to these questions

Actual policy will always be different than experiments:

General equilibrium effects that operate on large scale

Outcomes are different when everyone is covered by treatment rather than a few

Experimental subjects are not representative of population

Small development projects at village level do not attract attention of corrupt politicians

Scientists or experimentalists more careful than government implementers

Transporting successful experiments?

Mexico’s PROGRESA program

Conditional cash transfer program paid to parents if children attend schools and clinics

Now in 30 other countries

Is this a good thing?

Cannot simply be exported if countries have

Pre-existing anti-poverty programs with conditional transfers

No capacity to meet increased demands of education and health care

No political support

Combination of mechanism and context that makes for scientific progress

Much interest in RCTs, and instrumental variables, and other econometric techniques that mimic random allocation

comes from skepticism of economic theory

impatience with its ability to deliver structures that seem helpful in interpreting reality

Internal versus external validity:

Contrast between the rigor applied to establish internal validity and the looser analysis to render it policy relevant

To do this typically use some theory or some other information from observables – both go against simplicity of RCTs

Applied and theoretical economists have never been so far apart

Failure to reintegrate is not an option

Otherwise no chance of long term scientific progress extending from the RCTs.

RCTs that are not theoretically guided are unlikely to have more than local validity

11

The millennium development goal calls for a universal primary education by 2015 little consensus on how to achieve this goal or how much it

would cost

12

One view attracting additional children to school will be difficult since

most children not in school in developing countries are earning income their families need

Another view potential contribution of children of primary school age to family

income is very small hence modest incentives could significantly increase enrollment

13

Reducing the Cost of Education Some argue school fees prevent many students from attending school cite dramatic estimates from sub-Saharan Africa

free schooling introduced -- primary school enrollment

reportedly doubled Often data used for these estimates are unclear: free schooling is sometimes announced simultaneously with

other policy initiatives often accompanied by programs that replace school fees with per

pupil grants from the central government which create incentives for schools to over-report enrollment

14

Randomized experiments can isolate the impact of reducing costs on the quantity of schooling Several programs have gone beyond simply reducing school fees by actually paying students to attend school in the form of either cash grants or school meals School health programs can also increase quantity of schooling but this raises the question of how best to implement such programs One view is that the reliance on external financing of medicine is not sustainable and instead advocates health education, water and sanitation improvements and so forth

15

Quality of Education Notes from “Teacher Absence in India” (Kremer et. al.) Study entails a nationally representative survey on 3700 schools in India Three unannounced visits were made to each school

16

Absence data comes from direct physical verification of teacher’s presence not relying on logbooks, interviews, etc.

Teacher is recorded as absent if investigator could not find the teacher in the school during regular working hours

Journal of the European Economic Association (Resubmitted version, 11/27/04)

4

which absence calculations based on a similar methodology are available

(Table 1).3 Only 45 percent of teachers were actively engaged in teaching at

the time of the visit.4

Within India, the absence rate ranged from 15 percent in Maharashtra to 42

percent in Jharkand (Table 2).5 Absence rates are generally higher in low-

income states: doubling per capita income is associated with a 4.7 percentage

3 Most of these estimates come from other countries covered by the same research project on

provider absence in education and health, carried out by the authors of this study and using

standardized methodology (Chaudhury and others 2004).

4 Even with a generous allowance for the possibility that enumerators’ visits diverted some

teachers from teaching, it is unlikely that more than half of the teachers would have been teaching

at the time of the visit. See Kremer and others (2004).

5 Table 2 includes 19 of the 20 states surveyed. Fieldwork in the twentieth state, Delhi, was

delayed for bureaucratic reasons, and the data were received too late to be analyzed here.

Teacher absence (%)

Peru 11Ecuador 14Papua New Guinea 15Bangladesh 16Zambia 17Indonesia 19India 25Uganda 27

TABLE 1: Teacher absence rates by country

Source: Chaudhury, Hammer, Kremer, Muralidharan, and Rogers (2004) for most countries; Habyarimana and others (2004) for Zambia; World Bank (2004) for Papua New Guinea.

Journal of the European Economic Association (Resubmitted version, 11/27/04)

5

point lower predicted absence. The rates of teaching activity among the

teachers who are present are lower in higher-absence states and schools. In

some states, only 20 to 25 percent of teachers were engaged in teaching at the

time of the visit.

Absence rates are considerably higher than could be accounted for by

official non-teaching duties, such as staffing polling stations during elections or

conducting immunization campaigns, which are sometimes cited as important

causes of absence. Based on the responses of each school’s head teacher or

primary respondent, official non-teaching duties account for only about 4

percent of total absences. In other words, on any given day, only about 1

percent of primary teachers are absent because they are carrying out official

non-teaching-related duties.6 Preliminary calculations by the authors suggest

6 While stated reasons for absence should be taken with a grain of salt, there does not appear to

be any reason for head teachers to understate this cause of absence.

State Absence (%) State Absence (%)

Maharashtra 14.6 West Bengal 24.7Gujarat 17.0 Andhra Pradesh 25.3Madhya Pradesh 17.6 Uttar Pradesh 26.3Kerala 21.2 Chhatisgarh 30.6Himachal Pradesh 21.2 Uttaranchal 32.8Tamil Nadu 21.3 Assam 33.8Haryana 21.7 Punjab 34.4Karnataka 21.7 Bihar 37.8Orissa 23.4 Jharkhand 41.9Rajasthan 23.7 Weighted Average 24.8

TABLE 2: Teacher absence in public schools by state

19

One in four teachers are absent in a typical primary school in India Absence rates are generally higher in low-income states Higher teachers’ salaries do not seem to be associated with lower teacher absence Since nominal teachers’ salaries are very similar across states relative teachers’ salaries are higher in poorer states

yet poorer states have higher absence rates

20

Determinants of Teacher Absence

Journal of the European Economic Association (Resubmitted version, 11/27/04)

7

Higher teacher salaries do not seem to be associated with lower teacher

absence. We did not directly collect data on individual teacher salaries, but in

every Indian state, salaries increase with education, experience, and rank.

Teachers with a college degree are 2-2.5 percentage points more likely to be

absent. Being 10 years older increases the probability of absence by around

1.0-1.5 percentage points (though this effect becomes weaker after controlling

for state fixed effects). Head teachers are 4-5 percentage points more likely to

be absent than regular teachers (even after controlling for age and education).

Since absence increases with all three of these variables, it is likely that better-

paid teachers are more absent. Similarly, although regular teachers are typically

Category Yes No

Rich state (96-97 per capita income > $275)? 21.7 28.0

Female? 21.9 27.2

Older than 40 years? 27.1 21.4

Completed bachelors degree? 24.2 21.7

School has a toilet for teachers? 21.6 27.1

School has electricity connection? 19.2 28.2

Commute < 30 minutes? 21.4 25.2

Rural school? 25.2 22.9

Headteacher is absent? 22.2 18.5

School inspected in the past 3 months? 21.0 27.0

Belongs to this town/village? 21.6 23.3

PTA has met in the past 3 months? 21.0 26.5

TABLE 5. Absence rates (in percent) by various individual and school-level characteristics (public schools only)

Journal of the European Economic Association (Resubmitted version, 11/27/04)

8

[1] [2] [3]

No state or village fixed effects

With state fixed effects

With village/town fixed effects

1.34 2.09 2.06(0.54)** (0.53)*** (0.53)***

0.12 0.05 0.08(0.03)*** (0.03) (0.03)**

2.31 1.89 1.86(0.51)*** (0.51)*** (0.51)***

0.26 0.17 -0.58(0.59) (0.60) (0.65)4.19 4.54 4.61

(0.64)*** (0.62)*** (0.64)***-0.6 0.06 1.49

(1.32) (1.25) (1.28)-0.31 0.56 -0.07(0.64) (0.64) (0.73)0.56 1.12 0.94

(0.75) (0.74) (0.76)-1.08 -1.06 -0.84

(0.53)** (0.51)** (0.51)*0.5 0.18 1.86

(0.79) (0.78) (0.96)*-0.87 -0.9 -0.08(0.68) (0.65) (0.85)-1.5 -1.49 -0.93

(0.32)*** (0.32)*** (0.39)**0.77 -0.09

(0.84) (0.79)0.84 0.95 1.08

(0.37)** (0.37)** (0.64)*1.17 0.67 0.9

(0.70)* (0.67) (0.82)-0.05 -0.04 -0.08

(0.02)*** (0.02)** (0.02)***-2.28 -1.94 -2.45

(0.66)*** (0.64)*** (0.85)***-1.42 -1.15 -0.74

(0.36)*** (0.38)*** (0.46)-0.04 0 0.02(0.04) (0.04) (0.04)-0.37 -0.96 -0.86(0.54) (0.53)* (0.56)-0.79 0.44 -1.83(0.96) (0.92) (1.21)-1.7 -1.36 0.16

(0.72)** (0.68)** (0.87)-1.99 -1.36 -2.35

(0.70)*** (0.97) (1.33)*2.34 3.58 7.88

(1.23)* (1.21)*** (1.37)***0.01 1.65 7.54

(1.41) (1.41) (1.64)***-6.73

(2.49)***Constant 51.14 22.81 15.59

(9.80)*** (2.45)*** (2.82)***

Observations 41725 43130 43142R-squared 0.09 0.09 0.18

* significant at 10%; ** significant at 5%; *** significant at 1%

Regressions include a full set of controls for the day of week of the visit, for the round of the visit, and for the time of day of the visit relative to the school day (see full paper for details).

Mid-day meal exists

Government public school

Private aided school

Log of state per-capita income

The binary absence variable has been multiplied by 100 to allow the coefficients to be read as percentage changes.

Notes: Robust standard errors clustered at the school level are given in parentheses.

Duration of current posting (years)

Teacher's place of origin is village/town where school is located

PTA exists

PTA met at least once in last 3 months

Multigrade teaching

Pupil-teacher ratio

School has been inspected in past 3 months

Mean parental education of 4th-grade children (1-7 scale)

Recognition/award scheme exists in district

School infrastructure index (1 to 5)

Rural school

Distance to nearest paved road (1 to 5)

Belongs to a teachers union

Married

Has children of school age (5-14)

Paid regularly (1=Yes)

Has a college degree

Attended training in last 6 months

Head teacher

Contract teacher

TABLE 6: OLS estimates of teacher absence Dependent variable (visit-level observation):

Gender (1 = Male)

Age

Journal of the European Economic Association (Resubmitted version, 11/27/04)

8

[1] [2] [3]

No state or village fixed effects

With state fixed effects

With village/town fixed effects

1.34 2.09 2.06(0.54)** (0.53)*** (0.53)***

0.12 0.05 0.08(0.03)*** (0.03) (0.03)**

2.31 1.89 1.86(0.51)*** (0.51)*** (0.51)***

0.26 0.17 -0.58(0.59) (0.60) (0.65)4.19 4.54 4.61

(0.64)*** (0.62)*** (0.64)***-0.6 0.06 1.49

(1.32) (1.25) (1.28)-0.31 0.56 -0.07(0.64) (0.64) (0.73)0.56 1.12 0.94

(0.75) (0.74) (0.76)-1.08 -1.06 -0.84

(0.53)** (0.51)** (0.51)*0.5 0.18 1.86

(0.79) (0.78) (0.96)*-0.87 -0.9 -0.08(0.68) (0.65) (0.85)-1.5 -1.49 -0.93

(0.32)*** (0.32)*** (0.39)**0.77 -0.09

(0.84) (0.79)0.84 0.95 1.08

(0.37)** (0.37)** (0.64)*1.17 0.67 0.9

(0.70)* (0.67) (0.82)-0.05 -0.04 -0.08

(0.02)*** (0.02)** (0.02)***-2.28 -1.94 -2.45

(0.66)*** (0.64)*** (0.85)***-1.42 -1.15 -0.74

(0.36)*** (0.38)*** (0.46)-0.04 0 0.02(0.04) (0.04) (0.04)-0.37 -0.96 -0.86(0.54) (0.53)* (0.56)-0.79 0.44 -1.83(0.96) (0.92) (1.21)-1.7 -1.36 0.16

(0.72)** (0.68)** (0.87)-1.99 -1.36 -2.35

(0.70)*** (0.97) (1.33)*2.34 3.58 7.88

(1.23)* (1.21)*** (1.37)***0.01 1.65 7.54

(1.41) (1.41) (1.64)***-6.73

(2.49)***Constant 51.14 22.81 15.59

(9.80)*** (2.45)*** (2.82)***

Observations 41725 43130 43142R-squared 0.09 0.09 0.18

* significant at 10%; ** significant at 5%; *** significant at 1%

Regressions include a full set of controls for the day of week of the visit, for the round of the visit, and for the time of day of the visit relative to the school day (see full paper for details).

Mid-day meal exists

Government public school

Private aided school

Log of state per-capita income

The binary absence variable has been multiplied by 100 to allow the coefficients to be read as percentage changes.

Notes: Robust standard errors clustered at the school level are given in parentheses.

Duration of current posting (years)

Teacher's place of origin is village/town where school is located

PTA exists

PTA met at least once in last 3 months

Multigrade teaching

Pupil-teacher ratio

School has been inspected in past 3 months

Mean parental education of 4th-grade children (1-7 scale)

Recognition/award scheme exists in district

School infrastructure index (1 to 5)

Rural school

Distance to nearest paved road (1 to 5)

Belongs to a teachers union

Married

Has children of school age (5-14)

Paid regularly (1=Yes)

Has a college degree

Attended training in last 6 months

Head teacher

Contract teacher

TABLE 6: OLS estimates of teacher absence Dependent variable (visit-level observation):

Gender (1 = Male)

Age

23

Higher teacher salaries do not seem to be associated with lower absence No direct evidence but salaries increase with education, experience, rank Absence increases with: college degree older than 40 head teacher

Teacher absence is lower in schools with better infrastructure Toilets for teachers Electricity connection Library Covered classrooms Non-mud floors

24

Notes from “Addressing Absence” (Banerjee and Duflo) Obvious method to fight teacher absence is to monitor more intensively External control need not always be about monetary incentives Most common type control: someone in the institutional hierarchy (headmaster of a school) is

giventask of keeping an eye on teacher and penalizing absences Alternative method use some impersonal method, such as a camera, for recording absence An NGO in rural India experimented with a camera

25

In this area absence rate was 44% Most schools are one-teacher schools: when the teacher is absent children just go back home and lose entire day of schooling

120 schools were selected to participate in this study 60 randomly selected schools (treatment schools) NGO gave the teacher a camera with instructions to take a picture of himself /herself every day at opening time and at closing time

Figure 1

Figure 1

Figure 2: Impact of the CamerasNumber of Schools Found Open Times in Treatment and

Comparison schools(out of 13 visits)

0

2

4

6

8

10

12

0 1 2 3 4 5 6 7 8 9 10 11 12 13

Attendance Frequency (x)

Num

ber o

f Tea

cher

s pr

esen

t exa

ctly

x ti

mes

Treatment Control

27

Experimental Design

Teachers received a bonus as a function of the number of days they actually attended Teachers received a salary of 1,000 Rs. monthly if they were present at least 21 days in a month Each additional day carried a bonus of 50 Rs. up to a maximum of 1,300 per month. Each day missed carried a penalty of 50 Rs. Therefore the way the bonus was set up the average teacher’s salary remained 1,000 Rs. per month which was what teachers were paid in the remaining 60 schools (the comparison schools).

28

The program resulted in an immediate improvement in teacher attendance The absence rate of teachers was cut by one half Given the structure of the payment, the average salary in the treatment schools ended up matching almost exactly the average salary in the comparison schools The incentives were therefore effective without an increase in teachers’ net pay

Treatment Control Difference(1) (2) (3)

School Open 0.66 0.64 0.02(0.11)

41 39 80

Number of Students Present 17.71 15.92 1.78(2.31)

27 25 52

Teacher Test Scores 34.99 33.62 1.37(2.01)

53 56 109

Teacher Highest Grade Completed 10.21 9.80 0.41(0.46)

57 54 111

0.83 0.84 0.00(0.09)

27 25 52

0.78 0.72 0.06(0.12)

27 25 52

Blackboards Utilized 0.85 0.89 -0.04(0.11)

20 19 39

Infrastructure Index 3.39 3.20 0.19(0.30)

57 55 112

Fstat(1,110) 1.21p-value (0.27)

Table 1: Is School Quality Similar in Treatment and Control Groups Prior to Program?

E. School Infrastructure

Percent of Teachers Interacting with Students

Percentage of Children Sitting Within Classroom

Notes: (1) Teacher Performance Measures from Random Checks only includes schools that were open during the random check. (2) Infrastructure Index: 1-5 points, with one point given if the following school attribute is sufficient: Space for Children to Play, Physical Space for Children in Room, Lighting, Library, Floor Mats

A. Teacher Attendance

B. Student Participation (Random Check)

C. Teacher Qualifications

D. Teacher Performance Measures (Random Check)

Treatment Control Difference Treatment Control Difference(1) (2) (3) (4) (5) (6)

Took Written Exam 0.17 0.19 -0.02(0.04)

1136 1094 2230

Math Score on Oral Exam 7.82 8.12 -0.30 -0.10 0.00 -0.10(0.27) (0.09)

940 888 1828 940 888 1828

Language Score on Oral Exam 3.63 3.74 -0.10 -0.03 0.00 -0.03(0.30) (0.08)

940 888 1828 940 888 1828

Total Score on Oral Exam 11.44 11.95 -0.51 -0.08 0.00 -0.08(0.48) (0.07)

940 888 1828 940 888 1828

Math Score on Written Exam 8.62 7.98 0.64 0.23 0.00 0.23(0.51) (0.18)

196 206 402 196 206 402

Language Score on Written Exam 3.62 3.44 0.18 0.08 0.00 0.08(0.46) (0.20)

196 206 402 196 206 402

Total Score on Written Exam 12.17 11.41 0.76 0.16 0.00 0.16(0.90) (0.19)

196 206 402 196 206 402

Levels Normalized by ControlTable 2: Are Students Similar Prior To Program?

Notes: (1) Children who could write were given a written exam. Children who could not write were given an oral exam. (2) Standard errors are clustered by school.

A. Can the Child Write?

B. Oral Exam

C. Written Exam

Treatment Control Diff Until Mid-Test Mid to Post Test After Post Test(1) (2) (3) (4) (5) (6)

0.79 0.58 0.21 0.20 0.20 0.23(0.03) (0.04) (0.04) (0.04)

1575 1496 3071 882 660 1529

0.78 0.63 0.15 0.15 0.15 0.14(0.04) (0.05) (0.05) (0.06)

843 702 1545 423 327 795

0.78 0.53 0.24 0.21 0.14 0.32(0.04) (0.05) (0.06) (0.06)

625 757 1382 412 300 670

Figure 3: Impact of the Cameras(out of at least 25 visits)

Notes: (1) Child learning levels were assessed in a mid-test (April 2004) and a post-test (November 2004). After the post-test, the "official" evaluation period was ended. Random checks continued in both the treatment and control schools. (2) Standard errors are clustered by school. (3) Panels B and C only include the 109 schools where teacher tests were available.

Table 3: Teacher AttendanceSept 2003-Feb 2006 Difference Between Treatment and Control Schools

A. All Teachers

B. Teachers with Above Median Test Scores

C. Teachers with Below Median Test Scores

0

2

4

6

8

1 4 7 10 13 16 19 22 25Atte ndance Fre que ncy

Num

ber

of T

each

ers p

rese

nt e

xact

ly x

tim

es

Treatment

Control

30

In another experiment: in treatment schools, if the headmasters marked the preschool

teachers present a sufficient number of times for the teacher to receive a prize (a bicycle).

This experiment had no effect Absence rates were not reduced This outcome suggests that when human judgment is involved in a system where rules are often bent incentives may easily be perverted

31

Notes on “Remedying education: Evidence from two randomized experiments in India” (Banerjee, Cole, Duflo, and Linden)

Millennium Development Goals – primary education should be universal Dismal quality of educational services that developing countries offer the poor 2005 – India-wide survey: 44% children aged 7-12 cannot read 50% cannot subtract even though most are enrolled in school

32

In these conditions: policies that promote enrollment may not promote learning recent evidence suggests this to be the case

students do not learn anything in additional days they spend at

school We know very little about how to improve the quality of schools in a cost-effective way

33

Number of randomized experiments have demonstrated: spending more on resources such as textbooks, flip charts, or

additional teachers has had no impact on childrens’ test scores advocated reforms designed to change incentives faced by

teachers, parents and children We do not know enough to give up on inputs need to be more school-specific inputs targeted at weaker children may be effective

34

Consider two randomized experiments in urban India:

(1) remedial education program targeted weakest children

children taken out of regular classroom to work with young

women (from the community who has completed secondary school) for 2 hours per day (school is at least 4 hours per day)

35

(2) computer assisted learning program addressed to all children but adapted to each child’s current level

of achievement

children offered two hours of shared computer time per week play games that involve solving math problems at appropriate

difficulty level

36

Evaluation design Half schools given tutors for grade 3 (group A) other half for grade 4 (group B) in 2001-2002 (year 1). Program continued in 2002-2003 (year 2) – schools in group A were now assigned tutors in grade 4. Schools in group B in year 1 received assistance for grade 3 in year 2. In each year children in grade three in schools that received the program for

grade four = comparison group children that receive program for grade 3 = treatment group

Computer assisted learning: ½ schools given computers in first year (treatment), other half next year (comparison).

TABLE ISAMPLE DESIGN AND TIME LINE

Year 1 (2001–2002) Year 2 (2002–2003) Year 3 (2003–2004)

Grade 3 Grade 4 Grade 3 Grade 4 Grade 3 Grade 4

(1) (2) (3) (4) (5) (6)

Panel A: VadodaraBalsakhi

Group A (5,264 students in 49 schools in year 1;6,071 students in 61 schools in year 2) Balsakhi No balsakhi No balsakhi Balsakhi No balsakhi No balsakhi

Group B (4,934 students in 49 schools in year 1;6,344 students in 61 schools in year 2) No balsakhi Balsakhi Balsakhi No balsakhi No balsakhi No balsakhi

Computer-Assisted Learning (CAL)Group A1B1 (2,850 students in 55 schools in year

2; 2,814 students in 55 schools in year 3) No CAL No CAL No CAL CAL No CAL No CALGroup A2B2 (3,095 students in 56 schools in year

2; 3,131 students in 56 schools in year 3) No CAL No CAL No CAL No Cal No CAL CALPanel B: Mumbai

BalsakhiGroup C (2,592 students in 32 schools in year 1;

5,755 students in 38 schools in year 2) Balsakhi No balsakhi No balsakhi Balsakhi No balsakhi No balsakhiGroup D (2,182 students in 35 schools year 1;

4,990 students in 39 schools in year 2) No balsakhi No balsakhi Balsakhi No balsakhi No balsakhi No balsakhi

Notes: This table displays the assignment to schools in various treatment groups in the three years of the evaluation.Group A1B1 and A2B2 were constituted by randomly assigning half the schools in Group A and half the schools in Group B to the Group A1B1 and the remaining schools to

the Group A2B2.Schools assigned to Group A (resp. B) in 2001–2002 remained in Group A (resp. B) in 2002–2003. Twelve new schools were brought in the study and assigned randomly to Groups

A and B.Schools assigned to Group C (resp. D) in 2001–2002 remained in Group C (resp. D) in 2002–2003. Ten new schools were brought in the study and assigned randomly to Groups

C and D.

1242Q

UA

RT

ER

LY

JO

UR

NA

LO

FE

CO

NO

MIC

S

38

Key Outcomes: Learning measured using annual pre-tests given during first few weeks of school and post-tests given at end of term

39

Short-term effects Simple differences between post-test scores tutor program successful

TABLE IITEST SCORE SUMMARY STATISTICS FOR BALSAKHI AND CAL PROGRAMS

Pretest Posttest

Treatment Comparison Difference Treatment Comparison Difference

(1) (2) (3) (4) (5) (6)

A. Balsakhi: VadodaraYear 1 (grades 3 and 4)

Math �0.007 0.000 �0.007 0.348 0.171 0.177(0.059) (0.070)

Language 0.025 0.000 0.025 0.794 0.667 0.127(0.061) (0.076)

Year 2 (grades 3 and 4)Math 0.046 0.000 0.046 1.447 1.046 0.401

(0.053) (0.078)Language 0.055 0.000 0.055 1.081 0.797 0.285

(0.058) (0.071)B. Balsakhi: Mumbai

Year 1 (grade 3)Math 0.002 0.000 0.002 0.383 0.227 0.156

(0.108) (0.126)Language 0.100 0.000 0.100 0.359 0.210 0.149

(0.108) (0.102)Year 2 (grades 3 and 4)

Math �0.005 0.000 �0.005 1.237 1.034 0.203(0.058) (0.107)

Language 0.056 0.000 0.056 0.761 0.686 0.075(0.054) (0.061)

1246Q

UA

RT

ER

LY

JO

UR

NA

LO

FE

CO

NO

MIC

S

41

Math competencies: Vadodara: 19.5% third grade children pass grade one test in math

competencies Mumbai: 33.7%

Verbal competencies:

Vadodara: 20.9%

Mumbai: 83.7%

41

Because test scores have a strong persistent component precision of the estimated program effect can be increased

substantially by controlling for child’s pre-test score Table III – controls for pre-test score of child, grade and school –

find substantial treatment effect

42

Longer run impacts Must determine whether changes generated by interventions persist over time and last beyond period in which intervention is administered Compare effect of being exposed one or two years if effects are durable should be cumulative

Last two rows of Table III: present estimate of the impact of two years of exposure to the program These estimates are the difference between the year 1 (2001-2002) pre-test score and year 2 (2002-2003) post-test score for students

TABLE IIIESTIMATES OF THE IMPACT OF THE BALSAKHI PROGRAM, BY CITY AND SAMPLE

Number ofobservations

Dependent variable: test scoreimprovement

(posttest � pretest)

Math Language Total

(1) (2) (3) (4)

A: Pooling grades andlocations

Mumbai and Vadodaratogether year 1 12,855 0.182 0.076 0.138

(0.046) (0.056) (0.047)Mumbai and Vadodara

together year 2 21,936 0.353 0.187 0.284(0.069) (0.050) (0.060)

B: Pooling both gradesVadodara year 1 8,426 0.189 0.109 0.161

(0.057) (0.057) (0.057)Vadodara year 2 11,950 0.371 0.246 0.331

(0.073) (0.061) (0.070)Mumbai year 1

(grade 3 only) 4,429 0.161 0.086 0.127(0.075) (0.066) (0.067)

Mumbai year 2 9,986 0.324 0.069 0.188(0.145) (0.081) (0.112)

C: Grade 3Vadodara year 1 4,230 0.179 0.102 0.152

(0.086) (0.085) (0.085)Vadodara year 2 5,819 0.418 0.233 0.354

(0.107) (0.089) (0.100)D: Grade 4

Vadodara year 1 4,196 0.190 0.114 0.166(0.072) (0.076) (0.073)

Vadodara year 2 6,131 0.307 0.240 0.289(0.078) (0.068) (0.074)

E: Two year (2001–2003)Mumbai pretest year 1 to

posttest year 2 3,188 0.612 0.185 0.407(0.141) (0.094) (0.106)

Vadodara pretest year 1 toposttest year 2 3,425 0.282 0.181 0.250

(0.094) (0.079) (0.088)

Notes: This table reports the impact of the Balsakhi Program, for different groups and years. Each cellrepresents a separate regression of test score improvement on a dummy for treatment school, controlling for initialpretest score. Standard errors, clustered at the school-grade level, are given in parentheses. Estimates, whichinclude Mumbai year 2, use intention to treat as an instrument for treatment. Normalized test score gain is thedifference between posttest and pretest for Panels A–D and the difference between posttest in year 2 and pretestin year 1 for panel E. The total score is the sum of the normalized math and language scores.

1250 QUARTERLY JOURNAL OF ECONOMICS

45

Investigate whether the program had effects which lasted beyond the years during which the children were exposed Tested children in grade 4 and 5 at the end of year 3 (2003-2004) when the program had ended

TABLE ISAMPLE DESIGN AND TIME LINE

Year 1 (2001–2002) Year 2 (2002–2003) Year 3 (2003–2004)

Grade 3 Grade 4 Grade 3 Grade 4 Grade 3 Grade 4

(1) (2) (3) (4) (5) (6)

Panel A: VadodaraBalsakhi

Group A (5,264 students in 49 schools in year 1;6,071 students in 61 schools in year 2) Balsakhi No balsakhi No balsakhi Balsakhi No balsakhi No balsakhi

Group B (4,934 students in 49 schools in year 1;6,344 students in 61 schools in year 2) No balsakhi Balsakhi Balsakhi No balsakhi No balsakhi No balsakhi

Computer-Assisted Learning (CAL)Group A1B1 (2,850 students in 55 schools in year

2; 2,814 students in 55 schools in year 3) No CAL No CAL No CAL CAL No CAL No CALGroup A2B2 (3,095 students in 56 schools in year

2; 3,131 students in 56 schools in year 3) No CAL No CAL No CAL No Cal No CAL CALPanel B: Mumbai

BalsakhiGroup C (2,592 students in 32 schools in year 1;

5,755 students in 38 schools in year 2) Balsakhi No balsakhi No balsakhi Balsakhi No balsakhi No balsakhiGroup D (2,182 students in 35 schools year 1;

4,990 students in 39 schools in year 2) No balsakhi No balsakhi Balsakhi No balsakhi No balsakhi No balsakhi

Notes: This table displays the assignment to schools in various treatment groups in the three years of the evaluation.Group A1B1 and A2B2 were constituted by randomly assigning half the schools in Group A and half the schools in Group B to the Group A1B1 and the remaining schools to

the Group A2B2.Schools assigned to Group A (resp. B) in 2001–2002 remained in Group A (resp. B) in 2002–2003. Twelve new schools were brought in the study and assigned randomly to Groups

A and B.Schools assigned to Group C (resp. D) in 2001–2002 remained in Group C (resp. D) in 2002–2003. Ten new schools were brought in the study and assigned randomly to Groups

C and D.

1242Q

UA

RT

ER

LY

JO

UR

NA

LO

FE

CO

NO

MIC

S

already 0.13 higher in the treatment group in year 3 (as shown incolumn (3)).

Table IV corrects for this initial difference by estimating (1),where the treatment is the participation of the school in the CALprogram. The CAL program has a strong effect on math scores(0.35 standard deviations in the first year (year 2) and 0.47standard deviations in the second year (year 3)). It has no dis-cernible impact on language scores (the point estimates are al-ways very close to zero). This is not surprising, since the softwaretargeted exclusively math skills, although some spillover effectson language skills could have occurred (for example, because theprogram increased attendance, or because the children got prac-tice in reading instructions, or because the teachers had reallo-cated time away from math to reading). The effect on the sum oflanguage and math test scores is 0.21 standard deviations in year

TABLE IVIMPACT OF THE CAL PROGRAM, BY YEAR

Number ofobservations

Dependent variable:Test score improvement

(posttest � pretest)

Math Language Total

(1) (2) (3) (4)

A: Effect of the CAL programVadodara both years 11,255 0.394 �0.025 0.191

(0.074) (0.082) (0.083)Vadodara Year 2 5,732 0.347 0.013 0.208

(0.076) (0.069) (0.074)Vadodara Year 3 5,523 0.475 �0.005 0.225

(0.068) (0.042) (0.051)B: Balsakhi and CAL program: Main effects and interactions (Vadodara, Year 2)

CAL 5,732 0.408 0.017 0.242(0.087) (0.084) (0.087)

Balsakhi — 0.371 0.229 0.315(0.112) (0.104) (0.112)

CAL � balsakhi — �0.144 �0.020 �0.086(0.141) (0.134) (0.141)

This table reports the impact of the CAL program. In Panel A, each cell represents a separate regression,of test score gain on a dummy for treatment school, controlling for initial pretest score. In Panel B, eachcolumn represents a regression, of test score improvement on a dummy for the CAL program, a dummy forthe Balsakhi program, and an interaction term, as well as a control for initial pretest score. Standard errors,clustered at the school-grade level, are given in parentheses. Normalized test score improvement is thedifference between posttest and pretest. The total score is the sum of the normalized math and languagescores.

1252 QUARTERLY JOURNAL OF ECONOMICS

TABLE VSHORT- AND LONGER-RUN IMPACTS OF PROGRAMS, BY INITIAL PRETEST SCORE

Sample

Probability ofassignmentto balsakhi

Program effect in year 2 Persistence of program effect

Math Language TotalNumber of

observations Math Language TotalNumber of

observations

(1) (2) (3) (4) (5) (6) (7) (8) (9)

Panel A: Balsakhi, 2002–2003All children 0.313 0.371 0.246 0.331 11,950 0.053 0.033 0.040 9,925

(0.073) (0.061) (0.070) (0.047) (0.041) (0.041)Bottom third 0.446 0.469 0.317 0.425 4,053 0.096 0.097 0.103 3,356

(0.088) (0.074) (0.084) (0.045) (0.038) (0.040)Middle third 0.341 0.374 0.240 0.339 3,874 0.021 �0.024 0.001 3,226

(0.082) (0.069) (0.080) (0.056) (0.054) (0.052)Top third 0.162 0.229 0.174 0.216 4,023 0.015 0.006 0.009 3,343

(0.076) (0.076) (0.077) (0.069) (0.062) (0.061)Panel B: CAL, 2002–2003

All children — 0.347 0.013 0.208 5,732 0.092 �0.072 0.008 4,688(0.076) (0.069) (0.074) (0.045) (0.048) (0.045)

Bottom third — 0.425 0.086 0.278 1,962 0.107 0.004 0.046 1,586(0.106) (0.089) (0.102) (0.046) (0.047) (0.046)

Middle third — 0.316 0.005 0.183 1,844 0.085 �0.105 �0.015 1,511(0.081) (0.081) (0.082) (0.055) (0.069) (0.058)

Top third — 0.266 �0.033 0.146 1,926 0.073 �0.105 �0.013 1,591(0.073) (0.081) (0.078) (0.072) (0.064) (0.068)

This table reports the effects of the Balsakhi and CAL Programs over the short- and medium-term, according to the child’s position in the initial pretest score distribution. Column(1) reports the probability of actually being taught by the balsakhi, conditional on being in a treatment school. Each cell in columns (2)–(8) represents a separate regression of testscore gain on a dummy for treatment, controlling for initial pretest score. In Panel A, intention to treat is used as an instrument for treatment. Columns (2)–(4) give the one-yearprogram effect, estimated as the difference in normalized test score between the posttest and pretest in year 2 (2002–2003). Columns (6)–(8) give the cumulative effect of each programone year after both interventions had stopped. The dependent variable for these regressions is the difference between an end of year test in year 3 (2003–2004), and the pretest inyear 2 (2002–2003). Standard errors, clustered at the school-grade level, are given in parentheses.

1254Q

UA

RT

ER

LY

JO

UR

NA

LO

FE

CO

NO

MIC

S

48

Using difference between 2004 and 2002 Average effect becomes insignificant Not case for weaker students – effect remains significant

49

Find that both programs has a substantial positive effect on children’s academic achievement at least in the short run

Find remedial program impacts weaker students no average effects (consistent with previous literature)

50

Remedial program is very low cost and easily replicated. Relies on local personnel who are trained for a short period earn 10-15 dollars/month.

Curriculum and pedagogy are simple and standardized rapid turnover among tutors.

Use whatever space available (free classroom, playground, hallways)

51

Computer assisted learning Government delivered 4 computers to each government run primary school in the city. Idea of using computers to remedy shortage of qualified teachers is very popular in Indian policy circles. Evidence on impact of computers in developed countries is not encouraging