Randomized Controlled Trials, Development Economics and...
Transcript of Randomized Controlled Trials, Development Economics and...
-
Randomized Controlled Trials, Development Economics and Policy Making in Developing Countries
Esther Duflo Department of Economics, MITCo-Director J-PAL [Joint work with Abhijit Banerjee and Michael Kremer]
-
Randomized controlled trials have greatly expanded in the last two decades
• Randomized controlled Trials were progressively accepted as a tool for policy evaluation in the US through many battles from the 1970s to the 1990s.
• In development, the rapid growth starts after the mid 1990s– Kremer et al, studies on Kenya (1994)– PROGRESA experiment (1997)
• Since 2000, the growth have been very rapid.
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 2
-
Cameron et al (2016): RCT in development
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 3
0
50
100
150
200
250
300
1975 1980 1985 1990 1995 2000 2005 2010 2015Publication Year
Figure 1: Number of Published RCTs
-
BREAD Affiliates doing RCT
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 4
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1980 or earlier 1981-1990 1991-2000 2001-2005 2006-todayPhD Year
Figure 4. Fraction of BREAD Affiliates & Fellows with 1 or more RCTs
* Total Number of Fellows and Affiliates is 166.
-
Top Journals
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 5
3ie_time
Sector2000200120022003200420052006200720082009201020112012% of IER pre-2013
Agriculture and rural development126681251522291548489.70%
Economic policy01000001046220.70%
Education911121028223425356667908223.10%
Energy00001102021610.60%
Environment and disaster management00110138781118173.40%
Finance0121364915141728175.50%
Health, Nutrition, and Population 2133315450787410110017219621623364.90%
Information and communications technology0000119336814182.80%
Private sector development012335585122226215.10%
Public sector management02032435614613133.30%
Social protection1268217141820294751525715.10%
Transportation00001111002210.40%
Urban development00101201244130.80%
Water and sanitation012547558121312134.20%
Total studies3045466782114111143173274296370377.
YearNumber of Published RCTsNumber of Published Impact EvalsFraction of Published Impact Evaluations that are RCTYearNumber of Published RCTsNumber of Published Impact EvalsFraction of Published Impact Evaluations that are RCT
198111100%1981-1990111385%
198211100%1991-200011715874%
1983000%2001-200523636664%
1984000%2006-today1405238659%
1985010%
198622100%
198711100%
19882367%
198922100%
199022100%
199133100%
19921250%
19935683%
199481080%
199591182%
1996192095%
199791753%
1998192383%
1999213560%
2000233174%
2001304764%
2002274560%
2003516974%
2004548762%
20057411863%
20067812761%
20079315460%
200811719261%
200916629057%
201019732161%
201125339564%
201227243063%YearNumber of Published RCTsNumber of Published Impact EvalsFraction of Published Impact Evaluations that are RCT
201313423457%1981-2000ERROR:#REF!ERROR:#REF!ERROR:#REF!
20147615748%2001-2005ERROR:#REF!ERROR:#REF!ERROR:#REF!
2015198622%2006-todayERROR:#REF!ERROR:#REF!ERROR:#REF!
Figure 1: Fraction of Published Impact Evaluations that are RCTs
Fraction of Published Impact Evaluations that are RCT1981198219831984198519861987198819891990199119921993199419951996199719981999200020012002200320042005200620072008200920102011201220132014201511000110.666666666666666631110.50.833333333333333370.80.818181818181818230.950.529411764705882360.826086956521739140.60.741935483870967750.638297872340425560.60.739130434782608650.620689655172413810.62711864406779660.614173228346456710.603896103896103930.6093750.572413793103448310.613707165109034230.640506329113924020.632558139534883710.572649572649572610.484076433121019110.22093023255813954
Publication Year
Figure 1: Fraction of Published Impact Evaluations that are RCT
Fraction of Published Impact Evaluations that are RCT1981-19901991-20002001-20052006-today0.846153846153846150.7405063291139240.644808743169398960.58885163453478628
Publication Date
Number of Published RCTs1981-19901991-20002001-20052006-today111172361405
Figure 1: Number of Published RCTs
Number of Published RCTs1981198219831984198519861987198819891990199119921993199419951996199719981999200020012002200320042005200620072008200920102011201211000212223158919919212330275154747893117166197253272
Publication Year
3iePaper
SectorDifference-in-DifferenceInstrumental VariablesRCTRegression Discontinuity DesignPSM or OMMNotesDifferences in DifferencesInstrumental VariablesRCTRegression DiscontinuityPropensity Score Matching, or Other Matching Method
Agriculture and rural development32.00%21.00%20.50%0.00%43.80%16.70%8.30%66.40%2.40%16.10%
Economic policy43.80%31.30%0.00%6.30%37.50%
Education20.90%10.90%58.30%6.30%17.70%
Energy35.70%35.70%0.00%0.00%57.10%
Environment and disaster management27.60%22.40%30.30%2.60%34.20%
Finance29.80%17.70%41.10%0.00%24.20%
Health, Nutrition and Population9.50%4.40%83.20%0.70%6.50%
Information and communications technology25.40%3.20%66.70%3.20%20.60%
Private sector development36.50%19.10%26.10%7.00%33.90%
Public sector management31.10%23.00%44.60%5.40%16.20%
Social protection31.40%12.30%40.80%6.20%34.30%
Transportation55.60%11.10%22.20%11.10%22.20%
Urban development36.80%15.80%15.80%5.30%36.80%
Water and sanitation13.80%5.30%72.30%0.00%17.00%s
Total16.70%8.30%66.40%Source: Drew B. Cameron , Anjini Mishra , Annette N. Brown "The growth of impact evaluation for international development: how much have we learned?" Journal of Development Effectiveness Vol. 8, Iss. 1, 20162.40%16.10%Source: Drew B. Cameron , Anjini Mishra , Annette N. Brown "The growth of impact evaluation for international development: how much have we learned?" Journal of Development Effectiveness Vol. 8, Iss. 1, 2016
Source: Drew B. Cameron , Anjini Mishra , Annette N. Brown "The growth of impact evaluation for international development: how much have we learned?" Journal of Development Effectiveness Vol. 8, Iss. 1, 2016
Figure 2: Evaluations by Type
Differences in DifferencesInstrumental VariablesRCTRegression DiscontinuityPropensity Score Matching, or Other Matching Method0.167000000000000018.3000000000000004E-20.664000000000000032.4E-20.161Differences in DifferencesInstrumental VariablesRCTRegression DiscontinuityPropensity Score Matching, or Other Matching Method0
Aidgrade
YearPercent RCTs out of TotalPercent RCTs in EconPercent RCTs in Other FieldsTotal Number of EvaluationsTotal # of Econ EvaluationsTotal # of Other EvaluationsTotal # of RCTsTotal # of RCTs (Econ)Total # of RCTs (other)YearPercent RCTs in EconPercent RCTS in Other FieldsNon-RCTs
19821000100101101198201000
19891000100101101198901000
19901000100202202199001000
19911000100101101199101000
199250050202101199205050
19931000100101101199301000
19951000100101101199501000
19961000100404404199601000
19971000100202202199701000
19981000100707707199801000
1999100100100321321199966.666666666733.33333333330
20001000100303303200001000
200188.8888950100927817200111.111111111177.777777777811.1111111111
200266.666665075624413200216.66666666675033.3333333333
20039066.6666610010379272003207010
200458.3333344.444441001293743200433.33333333332541.6666666667
20056033.333331005323122005204040
200658.3333328.571431001275725200616.666666666741.666666666741.6666666667
200762.525100844514200712.55037.5
200856.2536.363641001611594520082531.2543.75
200947.3684237.510019163963200931.578947368415.789473684252.6315789474
20104018.1818210015114624201013.333333333326.666666666760
20115038.88889100221841174201131.818181818218.181818181850
201272.7272766.666661001192862201254.545454545518.181818181827.2727272727
201386.6666684.615391001513113111201373.33333333336.666666666720
2014100100440440201410000
Figure 3A. Aidgrade.org Evaluations
Total Number of Evaluations1982198919901991199219931995199619971998199920002001200220032004200520062007200820092010201120122013201411212114273396101251281619152211154Total # of Econ Evaluations198219891990199119921993199519961997199819992000200120022003200420052006200720082009201020112012201320140000000000202239374111611189134Total # of Other Evaluations1982198919901991199219931995199619971998199920002001200220032004200520062007200820092010201120122013201411212114271374732545344210
Figure 3B: Aidgrade.org Evaluations By Type
Percent RCTs out of Total198219891990199119921993199519961997199819992000200120022003200420052006200720082009201020112012201320141001001001005010010010010010010010088.88889000000000466.6666599999999939058.3333299999999976058.33332999999999762.556.2547.36842405072.72727000000000486.666659999999993100Percent RCTs in Econ1982198919901991199219931995199619971998199920002001200220032004200520062007200820092010201120122013201400000000001000505066.66665999999999344.4444433.33332999999999728.5714299999999992536.36363999999999737.518.18181999999999838.88889000000000466.66665999999999384.615390000000005100Percent RCTs in Other Fields198219891990199119921993199519961997199819992000200120022003200420052006200720082009201020112012201320141001001001005010010010010010010010010075100100100100100100100100100100100
Figure 3. Aidgrade.org Evaluations by Type
Percent RCTs in Econ19821989199019911992199319951996199719981999200020012002200320042005200620072008200920102011201220132014000000000066.666666666666657011.11111111111111116.6666666666666642033.3333333333333292016.66666666666666412.52531.57894736842105113.33333333333333431.81818181818181754.5454545454545473.333333333333329100Percent RCTS in Other Fields198219891990199119921993199519961997199819992000200120022003200420052006200720082009201020112012201320141001001001005010010010010010033.33333333333332910077.7777777777777865070254041.6666666666666715031.2515.78947368421052626.66666666666666818.18181818181818318.1818181818181836.6666666666666670Non-RCTs19821989199019911992199319951996199719981999200020012002200320042005200620072008200920102011201220132014000050000000011.111111111111133.3333333333333431041.6666666666666714041.66666666666666437.543.7552.631578947368425605027.27272727272728200
BREAD Affiliates
Total Number of Affiliates & Fellows# with some RCTs# mostly RCTs
1658662Year of PhD# with some RCTs# mostly RCTs# of Affiliates in yearPercent of Affiliates doing some RCT
1980 or earlier321225%
1981-1990962733%
Number of BREAD Affiliates who do RCTs1991-200015133444%
Year of PhD# with some RCTs# mostly RCTs# of Affiliates in yearPercent of Affiliates doing some RCT2001-200521154151%
19620010%2006-today38265767%
19660030%
19720010%
1973111100%
1974101100%
19750020%
19760010%
19770010%
1979111100%
198110333%
198221367%
19830020%
19840010%
198511333%
198610250%
19870020%
198822633%
198922450%
19900010%
19910010%
199222633%
19930010%
1994111100%
19950030%
199610333%
19970010%
199855771%
199932650%
200033560%
200142757%
200222450%
2003757100%
200444850%
2005421040%
2006861080%
200765875%
200832475%
200954956%
201032560%
2011521050%
201264875%
201321367%
Total8662
*Total Number of Fellows and Affiliates is 165
Total Number of Some RCTTotal Number of Mainly RCT
8662
Figure 4. Fraction of BREAD Affiliates & Fellows with 1 or more RCTs
Percent of Affiliates doing some RCT1980 or earlier1981-19901991-20002001-20052006-today0.250.333333333333333310.441176470588235280.512195121951219520.66666666666666663
PhD Year
BREAD Conf Graph
Row LabelsSum of Total # of PapersSum of Total # of RCTs
2003144
2004122
2005121
2006132
2007143
2008205
2009154
2010138
2011145
20122211
2013209
2014156
2015146
201671
Grand Total20567
YearTotal # of PapersTotal # of RCTs% of Papers using a RCT
200314429%
200412217%
20051218%
200613215%
200714321%
200820525%
200915427%
201013862%
201114536%
2012221150%
201320945%
201415640%
201514643%
20167229%
Figure 5. Percent of BREAD Concerence Papers using a RCT
% of Papers using a RCT
200320042005200620072008200920102011201220132014201520160.28571428571428570.166666666666666668.3333333333333329E-20.153846153846153850.214285714285714270.250.266666666666666660.615384615384615420.357142857142857150.50.450.40.428571428571428550.2857142857142857
Year
Top Journals
Info Needed:How many development papers, how many RCTs
MIT Lib?199020002015
AERYesYesYesYes
QJE1 year lagYesYesYes
JPEYesYes
Restud
Econometrica
Number of Development Papers
Year
Journal Title201520001990
AER
QJE
JPE
Restud
Econometrica
Number of RCTs
Year
Journal Title201520001990
AER
QJE
JPE
Restud
Econometrica
Table 6: Papers in Top Journals
JournalYearTotal # of Papers# of Development Papers# of which are RCTs
AER2015101154
20004860
19905720
QJE20154011
20004350
19905230
JPE20153643
20005170
19906590
Restud20154872
20003630
19904010
Econometrica20154650
20003700
19906420
* These include papers in the following journals: AER, QJE, JPE, Restud, Econometrica
Table 2: Papers in Top 5 Journals
YearTotal # of Papers# of Development Papers# of which are RCTs
20152713210
2000215210
1990278170
* These include papers in the following journals: AER, QJE, JPE, Restud, Econometrica
https://www.aeaweb.org/journals/aer/issueshttp://qje.oxfordjournals.org/content/by/yearhttp://www.journals.uchicago.edu/toc/jpe/currenthttp://www.restud.com/accepted-papers/http://www.econometricsociety.org/publications/econometrica/browse
NEUDC Conference
Info Needed:How many RCTs by year
Host UniversityYearTotal # of PapersTotal # of RCTsPercent RCTRounded
Brown20152204018.181818181818.2%
BU20142013617.910447761217.9%
Harvard 20132024924.257425742624.3%
Dartmouth20121692715.976331360916.0%
Table 3: NEUDC Conference Papers
YearTotal # of RCTsPercent RCTs
20154018.2%
20143617.9%
20134924.3%
20122716.0%
-
Many sectors, many countries
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 6
-
Why have RCT had so much impact?
• Focus on identification of causal effects (across the board)
• Assessing External Validity
• Observing Unobservables
• Data collection
• Iterative Experimentation
• Unpack impacts
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 7
-
Focus on Identification… across the board!• The key advantage of RCT was perceived to be a clear
identification advantage
• With RCT, since those who received a treatment are randomly selected in a relevant sample, any difference between treatment and control must be due to the treatment
• Most criticisms of experiment also focus on limits to identification (imperfect randomization, attrition, etc. ) or things that are not identified even by randomized trials (distribution of treatment effects, effects elsewhere).
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 8
-
Focus on Identification… across the board!• Before the explosion of RCT in development, a literature
on RCT in labor and public finance has thought of other ways to identify causal effects
• In development economics, there was a joint development of the two literatures (natural experiment and RCT), which has made both literatures stronger, and perhaps less different than we initially thought they would be: – Natural experiments think of RCT as a natural benchmark (not just
an hypothetical gold standard). – Development of methods to go beyond simple comparison of
treatment and control in experiments, and richer designs
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 9
-
Encouragement design
Difference in take up caused by encouragement
People who take up program
-
Focus on Identification… across the board!• Before the explosion of RCT in development, a literature on
RCT in labor and public finance has thought of other ways to identify causal effects
• In development economics, there was a joint development of the two literatures (natural experiment and RCT), which has made both literatures stronger, and perhaps less different than we initially thought they would be: – Natural experiments think of RCT as a natural benchmark (not just an
hypothetical gold standard). Extremely well identified non randomized studies.
– Development of methods to go beyond simple comparison of treatment and control in experiments, and richer designs
• Ultimately, the advantage of RCT in terms of identification is a matter of degree, rather than a fundamental difference.
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 11
-
Why have RCT had so much impact?
• Focus on identification of causal effects (across the board)
• Assessing External Validity
• Observing Unobservables
• Data collection
• Iterative Experimentation
• Unpack impacts
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 12
-
External Validity
• Will results obtained somewhere generalize elsewhere?
• A frequent criticism of RCT is that they don’t guarantee external validity
• Which is quite right, but it is not like they are less externally valid…
• And because they are internally valid, and because you can control where they will take place: – compared across contexts. – they can be purposefully run in different contexts – Prediction can be made of what the effects of related programs
could be.
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 13
-
Bayesian Hierarchical Modelling of all the MF results : Profits Meager (2015)
-
Bayesian Hierarchical Modeling-- Meta analysis (consumption)
15
-
Example 2: Targeting the Ultra Poor Program: Coordinated evaluation in several countries
Beneficiary
Productive asset transfer
Health
Consumption support Technical skills training
Home visits
Savings
Banerjee et al, 2015
16
http://www.sciencemag.org/content/348/6236/1260799.full
-
Country by country results: Assets
Banerjee et al, 2015
-0.1
0.2
0.5
0.8Endline 1 Endline 2
Ass
et c
hang
e (s
tand
ard
dev
iatio
ns)
17
http://www.sciencemag.org/content/348/6236/1260799.full
-
Country by country results: Consumption
-5%
0%
5%
10%
15%
20%
Endline 1 Endline 2
% C
hang
e in
per
ca
pita
con
sum
ptio
n
18
-
Structured Speculation
• Ultimately, if the results are similar it is nice, but if they are different the ex-post analysis is speculative.
• Banerjee, Chassang, Snowberg (2016) propose to be explicit about such speculation, and that researchers should predict what the effect may be for other interventions, or in other contexts.
• This can then motivate running such experiments, and guesses can be falsified.
• Example: Dupas (2014)—Effect of short run subsidies on long run adoption depend on the timing of costs and benefits, and how quickly uncertainty about them is resolved: this allows her to classify the goods.
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 19
-
Why have RCT had so much impact?
• Focus on identification of causal effects (across the board)
• Assessing External Validity
• Observing Unobservables
• Data collection
• Iterative Experimentation
• Unpack impacts
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 20
-
Observing unobservables
• Some things simply cannot be observed in the wild, with naturally occurring variation
• Negative income tax experiment was designed as an experiment to separate income and substitution effects
• Many experiments in development are designed likewise to capture such effects: – Karlan Zinman Observing Unobservables– Cohen Dupas and Ashraf Dupas Shapiro: selection and treatment
effect of prices. – Bertrand et al. Corruption in driving licences in Delhi.
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 21
-
Why have RCT had so much impact?
• Focus on identification of causal effects (across the board)
• Assessing External Validity
• Observing Unobservables
• Data collection
• Iterative Experimentation
• Unpack impacts
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 22
-
Innovative data collection
• Innovative data collection does not require an experiment.
• But experiments have two features which have motivated creativity in measurement– We know precisely what we are trying to measure: payoff to the
person who is designing the questionnaire– We know that there will likely be enough power to measure such
effects
• As a result, lots of innovation in measurement:– Borrowing from other fields : psychology, political science,
agriculture, web scraping, wearable techology, – Inventing new methods: e.g. Olken 2007
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 23
-
Why have RCT had so much impact?
• Focus on identification of causal effects (across the board)
• Assessing External Validity
• Observing Unobservables
• Data collection
• Iterative Experimentation
• Unpack impacts
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 24
-
Iterative experimentations
• Some great natural experiment leave us with some unanswered questions: – Why are elite school not working for the marginal child? – Why are (some) charter school working so well?
• One other advantage of experiments is that one is never stuck with one particular surprising answer: you can continue to experiment in the same setting till you have some clarity.
• Example: Duflo, Kremer, Robison multi-year work on fertilizer. – People don’t use fertilizer, even though it is profitable– One set of experiment on financing– One set on learning and social learning.
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 25
-
Why have RCT had so much impact?
• Focus on identification of causal effects (across the board)
• Assessing External Validity
• Observing Unobservables
• Data collection
• Iterative Experimentation
• Unpack impacts
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 26
-
Unpack impacts
• This is a related point, but more narrowly focused on policy design.
• There are many many possible ways to design a particular programs
• Usually, one version is tried out
• But if it works what was essential? – Effort to unpack Conditional Cash Transfer– Example of doing everything at once: Raskin program, Indonesia
• Many people do not receive the rice they are eligible for, or over pay• Would transparency help?
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 27
-
They distribute 4 version of a cards to eligible villagers in 378 villages, randomly chosen out of 572
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 28
-
Other sources of variation and results
• They also varied:– Public (common) knowledge of the program– Fraction of people who get the phyisical card
• Results:– Making the card distribution public knowledge makes it more
effective– The physical card matter: information (in the form of list) alone is not
sufficient– (Perception of) accountability does not seem to make much of a
difference.
• The government decided to scale the version of the card with most info and the list to 65 million beneficiaries!
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 29
-
What has been the policy impacts of RCTs? • Is the Raskin Case unique or unusual?
– A study designed by researchers with several treatments and an underlying economic model, destined to be published in a to academic journal
– but that still had large policy impact
• Some have argued that the research impact of RCT has potentially come at the expense of real-world impact:
– Researchers’ and policy makers interests may diverge– Research slow down the process of iteration
• Evidence – Out of 700 projects on going or completed on the J-PAL site, there are
only 9 story of scale up or policy impact.– However, this is not a census of J-PAL study (or of RCT). Story selected for
high impacts : the sum of people reached is about 200 million.
30
-
Over 200 million people reached through scale-ups of programs evaluated by J-PAL researchers
ProgramPeople
Reached (mn)
School-based Deworming 95
Raskin: Subsidized Rice (Indonesia) 66
Teaching at the Right Level (India) 34
Generasi: Conditional Community Block Grants (Indonesia)
6
Chlorine Dispensers for Safe Water (East Africa) 0.5
Free Insecticidal Bednets Policy influence
Police Skills Training Policy influence
TOTAL 202 mn31
-
Getting a sense of overall influence is difficult• Returns to R&D highly skewed.
- Most scholarly articles are never cited- Most start-ups fail- Venture capitalists get most of their revenue from a small number of
investments
• Still huge payoffs to R&D
• Ideas take a long time to percolate through the system, and many RCT are fairly recent
• Many RCT find that things DO NOT work as well as hoped (microcredit, smokeless stoves)
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 32
-
RCTs and real world impact: The case study of DIV
•To solve the “census” problem, we focus on one case study: USAID DIV.
•USAID’s Development Innovation Ventures (DIV) offers an opportunity to compare outcomes in selected sample of award winners:
• DIV has open approach: no top-down restriction on sector, strategy • Grantees include social entrepreneurs, NGOs and development
researchers
• Staged financing (Pilot ≤ $150,000; Testing ≤ $1,500,000; Transition to Scale ≤ $15,000,000)
• Openness on scaling strategy (Commercial, public-sector or hybrid funding
• Emphasis on cost effectiveness; • Attention to management team, external commitments, but no
rigid litmus test
33
-
Methods
• Coverage: 43 DIV awards made from 2010-2012; total value $17.3m
• Here just examine reach, the estimated number of people exposed to the original and adapted versions of the innovation, after the DIV funding.
• Do not compare measures of the size of impact per beneficiary
• Do not estimate the likelihood that reach will be sustained or increased in the future
• Does not assume the credit to further expansion all goes to DIV
• One (of several) components of social return calculation
• Specifically, we focus on number of awards reaching more than 100k or more than 1 M people.
34
-
DIV-Supported several Innovations Reaching > 100K People5 INNOVATIONS REACHED MORE THAN 1 MILLION PEOPLE• Voter Report Cards (2
awards)
• Election Monitoring Technology
• Consumer Action and Matatu Safety
• Digital Attendance Monitoring
• Dispensers for Safe Water
6 INNOVATIONS REACHED MORE THAN 100K AND LESS THAN ONE MILLION PEOPLE• Scaling CommCare for
Community Health Workers (2 awards)
• d.light Innovative Financing for Solar Systems
• Sustainable Distribution for Improved Cookstoves
• Recruiting and Compensating Community Health Workers
• VisionSpring BoPtical Care
• Renewable Powered Microgrids for Rural Lighting
35
-
High reach of innovations with RCTs/involvement of development economics researchers • 24 awards had RCT/researcher involvement, of which:
• 42% (10 awards) reached more than 100,000 people • 25% (6 awards) reached more than one million people
• 19 awards did not have RCT, of which:• 16% (3 awards) reached more than 100,000 people• No awards have yet reached more than one million
people
• Overall DIV numbers favorable relative to many impact investors• Arbitrage opportunity from openness to multiple type s of
innovation? • Discipline of evidence useful?
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 36
-
While early stage awards have low probability of attaining reach, they have high expected reach per dollar spent
Award StageNumber
of Awards
Award Value
Fraction Reaching More than
100,000 people
Fraction Reaching More than 1,000,000 people
People Reached
Expenditure per Person Reached
Stage 1 (< $100,000) 24 $2,353,136 17% (4/24) 8% (2/24) 6,723,733 $0.35
Stage 2 (
-
Pathways to reach
• DIV awards for innovations with RCTs reached > 100,000 people through a variety of partnerships:– Country governments (e.g. Zambia CHW recruiting, India
biometric monitoring)– Donors (e.g. cookstoves in Ethiopia and Sudan)– NGOs/Social Enterprises (e.g. Dispensers for Safe Water)– Private sector firms (e.g. newspapers, banks, insurance
companies, Qualcomm, Safaricom)• Three of five innovations reaching more than one million people
had earlier RCTs demonstrating impact, potential for cost effectiveness: researcher/project selection?
38
-
Why might projects involving RCT be more likely to have future reach?• Convincing force of evidence [most projects that do not
involve RCT try to scale through retails sale, which is harder]
• Nothing to do with the RCT per se:– Close involvement of researchers help ideas grounded in basic
science percolate research (like in biotech). – In particular: Influence of behavioral economics/information:
focus on low cost interventions , which are more likely to scale
• Selection of good projects [willing to do an RCT]
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 39
-
Conclusion
• The projects evaluated by RCT that then have reached many people tend to be low-cost, well defined, simple.
• So what has made RCT useful as a research tool (ability to iterate, zero-down to component, test a theory) is exactly what has turned out to make them policy relevant: details matter tremendously, and RCT tend to get the details right.
• An alternative pathway: BRAC, PROGRESA. Complexedinterventions replicated in many contexts.
• And a third one: innovation within existing governments and institutions.
– Tamil Nadu innovation Fund, Nudge Unit– Gujarat Pollution Control Board
J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 40
-
Conclusion
• For RCT to move from the large research impacts to large policy impact, we need a range of complementary institutions:– Meta-analysis– Review article– Review panels– Registry of Experiments has started and is successful (706 studies as of
June 8),• Appropriate support and experiment to support the learning
needed to move from successful pilot to policy at scale– Iterations to design scalable (robust) versions and measure their
effects– Equilibrium effects– Political economy /industrial organization of implementation
J-PAL | THE IMPACT OF RCT ON RESEARCH AND POLICY 41
Randomized Controlled Trials, Development Economics and Policy Making in Developing CountriesRandomized controlled trials have greatly expanded in the last two decades Cameron et al (2016): RCT in developmentBREAD Affiliates doing RCT Top Journals Many sectors, many countriesWhy have RCT had so much impact? Focus on Identification… across the board!Focus on Identification… across the board!Encouragement designFocus on Identification… across the board!Why have RCT had so much impact? External ValidityBayesian Hierarchical Modelling of all the MF results : Profits Meager (2015)Bayesian Hierarchical Modeling-- Meta analysis (consumption)Example 2: Targeting the Ultra Poor Program: Coordinated evaluation in several countries Country by country results: �Assets Country by country results: ConsumptionStructured Speculation �Why have RCT had so much impact? Observing unobservables Why have RCT had so much impact? Innovative data collection Why have RCT had so much impact? Iterative experimentationsWhy have RCT had so much impact? Unpack impactsThey distribute 4 version of a cards to eligible villagers in 378 villages, randomly chosen out of 572 Other sources of variation and resultsWhat has been the policy impacts of RCTs? Over 200 million people reached through scale-ups of programs evaluated by J-PAL researchersGetting a sense of overall influence is difficultRCTs and real world impact: �The case study of DIVMethodsDIV-Supported several Innovations Reaching > 100K PeopleHigh reach of innovations with RCTs/involvement of development economics researchers �While early stage awards have low probability of attaining reach, they have high expected reach per dollar spent�Pathways to reach Why might projects involving RCT be more likely to have future reach?ConclusionConclusion