Cost-Effectiveness Analysis and Cost-Benefit Analysis

Dagmara Celik Katreniak HSE

27.10.2014

Proposal Presentations

• Work in a pair or alone?

• Pick a date: – November 17th, 2014 – November 24th, 2014 – December 1st, 2014 – December 8th, 2014

Overview

• So far: – Randomization – Experimental design – Sample Size and Power Calculations

• Next: – Cost-Effective Analysis, Cost-Benefit Analysis – Data Analysis – Topics

Next class

• Data analysis

• Read: – Pitfalls of Participatory Programs (Banerjee et al) – Worms: Identifying impacts on education and

health in the present of treatment externalities (Miguel and Kremer)

Terminology

• Minimum Detectable Effect • Type I error, Type II error • Connection between Minimum Detectable

Effect, sample size and the level of power? • Statistic versus Scientific significance • Level of randomization • Clustering versus stratification • Imperfect Compliance versus Sample size/power

Today

• Cost-Effectiveness Analysis and Cost-Benefit Analysis

• Data collection • Outcome Variables • Data analysis *** • Questions • Case study

Notes to covariates

• Adding sum_r1 into regression of sum_r5 on treatment – Adding covariates (baseline survey!) decreases

variance of estimated coefficient → does not influence size of effect but its significance

– Be careful! – controlling for covariate influenced by treatment biases the treatment effect cause it captures part of its impact → use covariates from baseline, esp. pre-treatment value of outcome variable

• Covariates should be chosen in advance to avoid data mining

Notes to stratification

• After randomization – treatment and control are similar in expectations

• After stratification – treatment and control groups within stratas are similar in practice along the stratified variables

• Pairwise matched design • When stratified – what is the average treatment effect?

– Weighted average of difference between |T-C| across all stratas

• Blocking versus controlling for observables – Blocking more efficient, reduces variance

Notes to design

• Cross-cutting design – Solo A, Solo B, A+B, control – A+B and control – Solo A and control / solo B and control – Policy relevant research vs. academic research

• Important due to cost-reduction

Budget • How to plan your budget? • Main sections

– Main travel cost (visa, flight, vaccination, per-diem) – Technical equipment – Salary for you and your team – Accommodation, office – Rewards for subjects – Office support – Transportation in the field – Overhead – Other expenses (research permit, …) – Ideally 10-15% as safety fund

• Exchange rate differences, Inflation

Where to get funding?

• International funding – USAID, UNHCR, UNICEF, .. – 3IE, GDN, …

• National funding – University grants – Governmental grants

• Firms • Individuals / Foundations

– Bill and Melinda Gates, Karel Janecek, …

Cost Effectiveness

• Ratio of costs to effects or effects to costs – Costs for a given level of effectiveness

• How costly is additional year of schooling? – Effectiveness level for a given cost

• Given extra $100, how will attendance change?

• Ratio can be compared to other studies

– Conditional on similar computation • Helpful mainly for policymakers to be able to

choose between multiple programs • “What are the most cost-effective ways to increase school

attendance?” J-PAL training course, 2011

CEA versus CBA

• Cost-Effectiveness Analysis (CEA) – Outputs of the project that are non-monetary

• E.g. sick days, stress level, happiness level, etc. • Cost-Benefit Analysis (CBA)

– Monetary comparison of the benefits and costs of a project

• E.g. savings in USD in reduced sick days, etc. – Measure of economic worth of a project – Question is how to assign value to different benefits

(e.g. saved life, lowered stress level, etc.) • CEA versus CBA

– Key difference is the unit of measurement • Dollars (CBA) versus reduction in stress (CEA)

J-PAL training course, 2011

CEA versus CBA • CBA allows for the judgment whether a program

is worth implementing or not • Hard to measure

• CEA does not require measurement assumptions • Relatively simple, easy to compare to other

studies of the same outcome of interest • Does not offer monetary value of a project • Despite multiple impacts, CEA only compares

costs to effect on one outcome J-PAL training course, 2011

Comparative Cost-Effectiveness in the field

• Rare to see within a country • Even more rare across countries • Why?

– Lack of data – Different time of program implementation

• Different exchange rates, inflation, discount rates, etc. – Hard to replicate studies (limited funding) – No incentives to do CEA

• Examples: – Kremer, Miguel and Thornton (2005) – Banerjee, Cole, Duflo and Linden (2007)

J-PAL training course, 2011

Comparative Cost-Effectiveness in the field

Reasons: • Time and budget limited

• Learning from past mistakes

• Better implementation of programs

C-E comparison of Deworming Program to other programs

• Source: J-PAL

Quantification of Impacts (1)

• Good measures – the choice of outcome variables

• Multiple Impacts – We can only compare costs of an intervention to

effects on one outcome

• Imprecision in Estimating Impacts – Imprecision coming from data collection – Non-significant results are also results

• There is no effect or there is no power to confirm the effect • Include even insignificant results in the analysis

J-PAL training course, 2011

Quantification of Impacts (2)

• Spillover Effects – Not easy to capture

• Deworming program – Spillover effect type 1: effect on untreated – Spillover effect type 2: effect on control group – Which spillover effect to count?

J-PAL training course, 2011

Partial versus General Equilibrium

• The impact of an intervention tested on a sub-sample is positive

• What would be the effect if the intervention was scaled up to the national level? – Both higher and lower effect possible – Declining marginal benefit of the intervention

• Increase attendance from 40-50% is easier compared to attendance from 85-90%

• Do not estimate general equilibrium effect

J-PAL training course, 2011

Aggregate Impact

Aggregate Impact = Impact per unit * Sample size * duration • Time = to make sure it comes from the same

time interval • Proximal effect may need to be recalculated to

the final effect (e.g. increase in water chlorination → incidence of diarrhea)

J-PAL training course, 2011

Quantification of Costs (1)

• Keep your budget up-to-date and detailed

• Check J-PAL for general worksheet

• Talk to researchers, people with local experience

J-PAL training course, 2011

Quantification of Costs (2) • What was in the field already and what is new?

– Computer learning program • Usage of already existing computers • Purchase of new computers

– Additional teacher (balshaki program) • Impact of the additional teacher + impact of having

smaller classes

• Goods and services provided for free – is it cost? – Think about larger scale; – Always free, subsidized or just now? What is the

market situation? J-PAL training course, 2011

Quantification of Costs (3) • Costs of user time, cost to beneficiaries

– If time not voluntarily used – In terms of local wages (usually low)

• Overlapping goods and services – Hard to disentangle sometimes – Anything paid before the implementation of the project

should not be included • Transfers

– E.g. cash transfer programs – Administrative costs to be included, cash transfers

themselves? • Overhead (usually given by organization)

J-PAL training course, 2011

Quantification of Costs (5) • Inflation

– Inflate forward to today as a base – Inflate backward to some base year

• Exchange rates – Exchange all costs to common unit – Disadvantage – exchange rate reflect prices of

tradables; Not real purchasing power differences documented

• E.g. “skilled labor is much cheaper in India than in Mexico, so a highly labor-intensive program piloted in Mexico will appear less cost-effective when adjusted by PPP”

– Standard versus PPP Exchange rates J-PAL training course, 2011

CEA/CBA – to conclude

• Not popular and not widely implemented • Hard to do follow up studies and compare

CEA/CBA – Funding – Low academic and publication motivation

• Clear benefits of CEA/CBA

J-PAL training course, 2011

CEA/CBA Literature

• Recommended source for cost-effectiveness analysis – Henry Levin and Patrick McEwan (2001): Cost-

Effectiveness Analysis, book, SAGE Publications, Inc;

• J-PAL website

• J-PAL training course, 2011