Post-Project Sustainability Study:

Health, Wat/San and Agricultural Interventions in Bolivia...6 Years Later

Janine SchooleyCORE Spring Meeting 2013

Relevant BackgroundPCI has prioritized moving beyonddonor-driven M&E

Recognized need tomeasure what we saywe are all about….Real and lastingchange!

Identified private funding and decided to pilot a postproject sustainability studymethodology in Bolivia

Sustainable impact and measurement of same is now one of 7 Strategic Directions in PCI’s 2013-2016 Strategic Plan

Description of Subject ProjectPCI implemented the Bolivia DAP/ MYAP (USAID Title II Development Assistance Program) from 1997 to 2002. Overall project goal was to reduce high levels of food insecurity and poverty in select rural municipalities of Cochabamba and northern Potosi.

Maternal and child health (MCH)

Water and sanitation (W&S)

Agriculture

Project budget: $ 25,600,000

240,525 beneficiaries (annual investment of US$ 21 per beneficiary)

Purpose of the StudyThe primary purpose of the study was to assess the sustainability and impact of behavioral, livelihood, health and infrastructure interventions among participating communities. The key study questions were the following:

What is the real and lasting impact of the Bolivia DAP?

To what extent did the Bolivia DAP change behavior?

What are perceptions about the impact of PCI’s DAP interventions?

What can we learn from this study about how best to achieve sustainable impact?

MethodologySimilar methodologies to those used in the project’s baseline assessment, mid-term and final evaluations

Same 15 communities as for final project evaluation conducted in 2002

Also collected information on non-PCI projects implemented in the same communities

2 control communities also studied

Bolivia Post-Project Evaluation at a Glance

16 PCI staff persons participated in the evaluation

66 mothers interviewed

15 community leaders interviewed

11 water committee leaders interviewed

19 agricultural producers interviewed

10 silo owners interviewed

8 health promoters interviewed

2 government center staff interviewed

10 mothers’ focus groups conducted

5 Irrigation committees’ focus groups conducted

12 water systems inspected

5 irrigation systems inspected

75 home water systems and latrines inspected

10 silos inspected

300+ people contributed to data collection

90 households were selected for participation from 517 total households in the catchment area

90% confidence interval used for sample size calculation

1,149 miles were covered by evaluation team

Key Results

DAP communities were universally still better off than the control communities

100% of non-PCI respondents attributed positive health and development changes in DAP communities to PCI’s earlier presence

Strong evidence of sustainability of results across all 3 intervention areas

Communities which participated in interventions in the three key intervention areas had stronger indicators of success than those communities that participated in only two intervention areas

Sustainability of community interventions, such as water and irrigation systems, was stronger than household interventions such as latrines and individual health knowledge and practices

Key Results cont’d

Maternal and Child Health

Positive Results Areas for Improvement

86% of PCI ‘infants’ still have health cards

Only 69% of younger infants had health cards

82% of families still receive visits from community health promoters trained by PCI

Fewer community health promoters active at the local level over time

84% of households still seek care at local health centers

Water and Sanitation

Positive Results Areas for Improvement

98% of community W&S committees still functioning

72% of homes had latrines but only 51% used these latrines

Community water systems universally still in good working condition

Household water infrastructure sustainability 28% lower than community

80% of all households still using one or more treatment methods for domestic water use

Coordination with municipal water committees weak

Agriculture

Positive Results Areas for Improvement

Irrigation systems still in good condition

Silos still working well and excellently maintained

Lack of transparency and equitable distribution of silos

Household incomes continued to increase from the end of the DAP

Other Important Results

Impact on PCI staff: Pride and commitment to our mission; flood of positive emotions

Current and future project improvements: Useful lessons learned for future program design; what tends to “stick” and what needs more attention if we want it to “stick”

From M&E to Strategic Information for Impact: Expanded organizational commitment to M&E beyond donor requirements and proved the feasibility and value of conducting post-project sustainability assessments

ChallengesInaccurate/incomplete data (eg, census)

Logistics and coordination

Lack of detailed recollections

Locating government health service personnel

Attribution

Lack of $

Lack of complete documentation

Insufficient time allocated for planning

Lessons LearnedTiming of post-project reviews is crucial and should be carefully assessed, weighing the need to allow enough time after the end of a project to accurately assess sustainability against the possibility that waiting too long will limit the ability to collect adequate information about the project.

The involvement of PCI staff who participate in project design and implementation is absolutely essential.

While post-project evaluations build the morale of staff, independent consultants should be engaged as part of the evaluation team to minimize risks of bias and ensure a higher level of independence and validity.

Adequate time should be budgeted for study planning and data collection, taking into consideration logistical and documentation challenges.

Evaluators need to communicate purpose clearly to community leaders and participants in order to keep expectations in line with reality.

Lessons Learned cont’d

If you are interested in more information…

jschooley@pciglobal.org

Thank You!

1) You’re a consulting team visiting a new project (eg, 3 -6 months from startup or at mid-term). Propose 5-10 questions for testing of potential for sustainability during the life of project.

2) List Top 10 challenges or constraints (eg, conflict between achieving results and sustainability) which you encounter in your programs related specifically to assessing progress toward sustainability.

3) Outline various possible strategies for CORE and Member Organizations for obtaining resources for post-project sustainability studies