Water, Sanitation & Hygiene (WASH) Sector Strategic Framework
A Guide to Water and Sanitation Sector Impact...
Transcript of A Guide to Water and Sanitation Sector Impact...
Guidelines for Impact Evaluations in the Water and Sanitation Sector
Christine Poulos, Ph.D.Sr. Economist, RTI International
Subhrendu Pattanayak, Ph.D.Sr. Economist and Research Fellow, RTI International
Associate Professor, NC State University
RTI International
June 12, 2007
Motivation
Numerous texts on impact evaluation (IE), in general
Few rigorous IEs in the WSS sector
The Guide:
focuses on specific considerations in evaluating WSH programs and projects; and
supplements existing IE guidelines.See Baker [2000], Ravallion[2001,2005] and others for a comprehensive overview of IE, evaluation designs, or statistical methods
Roadmap
1. IE and IE in WSSInterpret/apply concepts and practices in context of WSS
2. Illustrative Examples of IEs in WSS
3. WSS IEs in practice
What is Impact Evaluation?
IE measures impacts on beneficiaries that are caused by the intervention (programs and policies)
Need to construct counterfactual What would have happened to the beneficiaries in the absence of the intervention?
Identify control or comparison group similar to beneficiaries but for the intervention
Compare comparison group to beneficiaries to measure impact
IE Steps
1. Decide to do an IE
2. Describe the interventionneed/motivation, context, stage of implementation, inputs, and results
3. Design and implement evaluationThe inner workings of a project project-specific issues (Bamberger 2006)
Why decide to do an IE?
Use findings to:
Add to global evidence base
Identify favorable conditions
Identify effective components
Build support
Ensure accountability
Why decide to do an IE?
Select candidates that provide opportunities for feasible, informative studies:
IE has political and financial support, and/or intervention is well-defined
Results generalizable –intervention is scalable, replicable
Interventions that are innovative, controversial, resource intensive
WSS interventions areMulti-dimensional and multi-sectoralRural Development, Private Sector Development, Urban and Local Government, and Environment
Goals guiding most WSS projects:Based on World Bank Water and Sanitation Sector Board guidelines 1. efficient access to safe drinking water and/or basic sanitation
services;2. sustainable access to safe drinking water and/or basic sanitation
services; and3. equitable access to safe drinking water and/or basic sanitation
services.
Describe the Intervention: Objectives of Intervention
WSS
Rural
Env.
BusinessUrban
WSS Interventions
Three types of reform measures:
1. improving operator performance,
2. service provision by the private sector or small-scale independent providers, and
3. decentralized delivery, typically relying on community demand, participation and management.
Outputs of sector reform initiatives:
Hardware: new or improved WSH infrastructure and services
Software: training, education, better provider performance
Describe the Intervention: Features and Linkages
Inputs:
Resources: financial, institutional, legal and regulatory
Activities: what the intervention does
Describe the Intervention: Features and Linkages
Inputs:
Resources: financial, institutional, legal and regulatory
Activities: interventions
Results:
Outputs: direct product of program activities
Outcomes: short-term changes in beneficiaries’ behaviors & knowledge
Impacts: long-term changes in beneficiaries’ wellbeing
Indicators: direct measure of progress toward goals
Also need to understand external influences
Describe the Intervention: Rehabilitating UWSS
Resources Activities Outputs Outcomes Impacts
Funding
Staff
Technical Assistance
Laboratory services
Replacement of pipes, pumps, meters
Trainings for staff
Water quality testing
No. of rehab WSS systems
No. of connections to rehab WSS systems
Operating cost of systems
Water rehab (hrs)
WQ tests
% of pop. with access to WSS
% of pop. served by rehab WSS
Coping costs
Lpcdconsumed
Use of rehab WSS
Individual and household incomes
School enrollment and attendance
Prev. of diarrhealdisease
Under 5 mortality
Design and Implement Evaluation, I
Question: is the intervention effective in increasing efficient, sustainable, and equitable access to improved WSS?
Robust IE design:
Time Pre-intervention
(baseline)Intervention Post-
interventionBeneficiaries (treatment group)
T1 X T2
Control/comparison group
C1 C2
Question: is the intervention effective in increasing efficient,sustainable, and equitable access to improved WSS IEs use baselines, controls/comparisons, and covariates to ensure causal effects can be identified
No baseline, no control – how measure change? what to compare with (i.e., counterfactual)?
Baseline, no control – what to compare with (i.e., counterfactual that captures trends and history)? sufficiently account for selection bias?
No baseline, control – how to sweep out pre-existing differences (behaviors, rates, trends)? sufficiently account for selection bias?
All the above, but no covariates – are you sure nothing else matters? No other factors effect program selection and or modify or mediate treatment?
Design and Implement Evaluation, II
Design and Implement Evaluation, III
Design determines where controls/comparisons come from
Randomized trials
Quasi-experiments – longitudinal or cross-sectional
natural experiments matching (propensity score, covariate, pipeline)
Analysis
Brief Randomized Trial Example
Design – treatment assigned randomly (not purposive, strategic or selective) so that confounders (alternative causes) are balanced across treatment and control group
Example: Do information treatments change hygiene behavior in Delhi, India? [Jalan & Somanathan, 2004]
Tell 500 households (out of 1000) about quality of their drinking water, and check after 1 year if they change hygiene behaviorSimilar in education, health literacy and hygiene behaviorsInformed household 11% more likely to purify water
Challenges: difficult to control, ethical concerns, political issues, and limited external validity
Brief Quasi-Experimental Example
Design: match treatments to controls based on observable factors
Example: Does Jalswarajya – a public, community-driven rural WSH program in Maharashtra, India – improve access to improved WSH and improve children’s health outcomes?
~250 villages (2 propensity score matched controls for each treatment)Baseline and post-intervention data collection from ~10,000 householdsHousehold and community surveys
Challenges: need lots of data, assumes unobservables are uncorrelated with exposure to intervention
OR IS S A
GA N J AM
KO R A PU T
PH U LB A N I
AN U G U L
KE O N JH A R
M A YU R B H A NJ
RA Y G AD A
PU R IKA L AH A N D I
SU N D AR G A R H
BO L AN G I R
SA M B A LP U R
BO U D
BA R A GA R H
M A L K A N G IR I
GA J AP A TI
NU A P AD ACU T TA C K
J A J P URDH E N K AN A L
BA LA S O RE
NA Y A G AR H
NA B A RA N G A PU R
DE O G AR H
KH U R D H A
SO N P UR BH A D RA K
KE N D RA P A DA
JH A RS U G U DA
JA G AT SI N G H PU R
#Y
#Y
#Y#Y
#Y#Y
#Y#Y#Y
#Y
#Y#Y
#Y #Y#Y
#Y
#Y#Y
#Y
#Y
#Y
#Y#Y#Y #Y
#Y#Y #Y
#Y#Y#Y#Y
#Y
#Y
#Y
#Y
#Y#Y#Y
#Y
#Y
#Y #Y
#Y #Y#Y
#Y#Y
#Y
#Y#Y
#Y
#Y
#Y #Y
#Y
#Y
#Y#Y
#Y
#Y
#Y#Y#Y
#Y#Y
#Y
#Y#Y
#Y
#Y#Y
#Y #Y
#Y
#Y#Y
#Y#Y
#Y#Y
#Y
#Y
#Y
#Y
%[
%[%[ %[
%[
%[
%[
%[
%[
%[
%[
%[
%[%[
%[%[
%[%[
%[
%[
%[
%[%[
%[
%[
%[
%[
%[
%[ %[
%[%[%[
%[%[
%[
%[
%[
%[
%[%[
%[
%[%[
%[
%[
%[%[
%[
%[
%[
%[
%[
%[
%[
%[
%[
B O N TH
B H A D R A K
B H A N D A R IP O K H A R I
D H A M N A GA R
TIH I D I
B A S U D E V P U R
C H A N D A B A L I
O R IS S A
B H A D R A K
M a p S h o w in g B h a d ra k D is tr ic tN
EW
S1 0 0 1 0 K m .
B L O C K B O U N D A R YRiv er
RO A DNA TI O N A L H IG H W A YM A JO R R O A DO T H E R R O A D
Dr a in a g e
L EG E N D :
Detailed Example: Randomized Trial of TSC’s IEC in Orissa, India
Randomized Trial of TSC’s IEC in Orissa, India: Describe Intervention, I
Government of India’s Total Sanitation Campaign
Goal: increase use of IHLsBackdrop: inadequate services, high child mortality, and MDGs
“Community-Led Total Sanitation”
Intensive IECChange knowledge Change attitudesCommunity demand and plan
Small subsidies to the poor; know-how and material to allImplement via local NGOs
Randomized Trial of TSC’s IEC in Orissa, India: Describe Intervention, II
Resources Activities Outputs Outcomes ImpactsFunding: Bank,
GoI, GoO, community contributions
Staff: GoO, NGOs, community monitors
Supply Chain
Technical assistance
Enabling programming -TSC
IEC: walk of shame, fecal calculation, defecation mapping
Distribute subsidies
Training in IHL construction
Establishment and/or stocking of rural sanitation marts
No. of focus groups and completed IECs
No. of community agreements
Total subsidies
No. of households trained in construction
No. of rural sanitation marts
% of households owing IHL
% of pop. using IHL, by gender and age
Coping costs-time
Awareness of environment-health link
Prevalence of diarrhealdisease, by age and gender
Norms regarding OD
Individual and household incomes
School enrollment and attendance
Randomized Trial of TSC’s IEC in Orissa, India: Design and Implement IE, I
Research Questions: Does the TSC’s IEC cause
increases access to and use of IHL? decreases in child morbidity due to diarrheal disease?
Features:Controls: Leverage phasing to randomly assign IEC to 20 out of 40 communities in 1st phaseBaseline: pre-intervention surveys Covariates: household and village surveys Indicators: selection based on literature, pretestsSample: ~1000 households
Randomized Trial of TSC’s IEC in Orissa, India: Design and Implement IE, II
0
10
20
30
40
%
Treatment Control
IHL Ownership and Use
20052006
05
101520253035
%
Treatment Control
U5 Diarrha Rate
20052006
Randomized Trial of TSC’s IEC in Orissa, India: Design and Implement IE, III
0
100
200
300
400
Rs.
Treatment Control
Medical Costs (Rs)
20052006
0
10
20
30
Min.
Treatment Control
Time Spent Walking (min)
20052006
Galiani, Gertler, and Schargrodsky(2005): Describe Intervention, I
Intervention: Privatization of local water companies in Argentina
In the 1990s, about 30% of the country’s municipalities covering almost 60% of the country’s population were privatized. The remaining municipalities continued receiving water services from either public companies or nonprofit cooperatives.
Research question:
While efficiency gains have been demonstrated, does privatization of water supply improve health outcomes and alleviate poverty?
Galiani, Gertler, and Schargrodsky(2005): Describe Intervention, II
Resources Activities Outputs Outcomes ImpactsEnabling
regulation
Private sector resources
Access to credit
New incentives
PSP process and contracts
New systems for billing, accounting, maintenance
Installation or replacement of pipes, pumps, meters
Training
Efficiency in billing and collection
Improved service
System performance (e.g., hours of service)
Water and sewerage connections , by municipality-level poverty rate†
Under five mortality rate, by municipality-level poverty rate
Galiani, Gertler, and Schargrodsky(2005): Design IE, I
Research Questions:Does privatization of water supply improve health outcomes?
Features:Comparisons: Propensity score matching of municipalitiesBaselines: secondary data Covariates: secondary data on socioeconomic and political characteristics of municipalitiesIndicators: U5 mortality rate, water and sewerage connection rate, private ownership of water company serving majority of municipalities' populationSample: ~4000 municipalities
Galiani, Gertler, and Schargrodsky(2005): Key Findings
Analysis:Difference-in-difference estimation with PSM
Findings:Child mortality fell by 8%
Effect was largest in poorest areas (26%)
Connections to the water network increased by 4.2%
Challenges
Short project cycles1. Measuring sustainability within the project cycle
“no less than 3-5 years are required for an intervention to show an impact.”(Habicht et al. 1999)Use program theory (White 2005)
2. Turnover, impatience, fatigue3. Devarajan and Kanbur (2004) “In short, we probably need to scale up
something that is in short supply, namely, patience.”
Constraints – Bamberger (2006)BudgetTimeDataBamberger discusses the acceptability of compromises
Strengthen Overall Quality
Allow sufficient time to meet with clients and key stakeholdersBuild political and institutional support early and often – counteract turnover, fatigueConvince them to stay the course
Get off to a quick start
If possible, change program design to allow for less expensive IE methods
If possible, make mid-course corrections
Develop program theory/logic modelConsensus on how it work, identify where it breaks down, determine what to measure
Alternative data collection methods
Conclusions
Call for increase in use of rigorous IE methods Address project-specific issues in selection of study design and features
Baseline, controls/comparison, and covariates are key
Endline is equally important! Engage and encourage stakeholders to stay the course
The End
Contact information: