Outcomes Working GroupWebinar 3: Indicators to measure outcomes

Presenters: Bobbi Gray, Research and Evaluation Specialist, Freedom from Hunger

Anne Hastings, Executive Director,Microfinance CEO Working Group

Working Group Facilitator: Frances Sinha, Director EDA Rural Systems (India) and board member of SPTF

4 February 2015

Agenda • Introduction (5mins)

• Bobbi (FFH): criteria to develop indicators related to health outcomes; application by 4 MFIs, lessons (15-20mins)

• Anne (MCWG): developing a system for standardized measurement, review of outcomes studies (69), selecting themes and indicators for microfinance (15-20mins)

DISCUSSION

Recap

• Previous webinar : Theory of Change framework: ▫ Defining short-term/intermediary and long term changes▫ Assumptions – links from different services/uses to changes ▫ Attrition factor - not all participants will stay the course

• Next logical step to identify relevant practical indicators to measure changes - and the steps that lead to changes:• Criteria for identifying practical, relevant indicators• Selection relevant to microfinance • What is the experience? • Can we build consensus on a ‘standard menu’ ?

Indicators The basics• Specific, relevant - define objectives• Practical to measure - credible• Unambiguous - clear• Can compare, benchmark • Limited number – necessarily selectiveTrade-offs in e.g. what is practical to measure

(household income or expenditure) and/or unambiguous (women’s role in decision making)

Health Outcome Performance Indicators Project

6

Theories of Change: Improved Health

Access to and use of financial services: loans;

insurance; savings;

payments; health loans and savings

Increased income

Consumption smoothing

Coping with shocks

Seek prompt medical

treatmentSeek

preventive health care Access to and

use of health services: education,

provision of curative and

preventive health services

Improved health knowledge and seeking prompt medical treatment

and preventive health care

Integrated health and financial

services: direct provision and

linkages between sectors

Cross-sectoral efficiency gains in

provision of financial and

health services to poor populations

Seek prompt medical

treatmentSeek

preventive health care

Improved health

outcomes

7

Choosing Health IndicatorsCriteria

Feasibility Usability Usability/ Reliability

Likelihood of inclusion

Measurable by an FSP

Can be reported in client survey

Can change in short-term

Addresses relevant measures for FSPs

Cannot rely on specific interventions to change outcomes

Be applicable for both genders

Can be benchmarked to other data (MDGs, regional data, etc.)

Reliability

PPI/PAT Yes Yes Maybe Yes Yes Yes Yes Yes High

Food security index

Yes Yes Yes Yes Yes Yes Yes Maybe High

Use of preventive health services

Yes Yes Yes Maybe Maybe Yes Maybe Maybe High/ Moderate

Access to safe drinking water (MDG 7)

Yes Yes Maybe Yes Maybe Yes Yes Maybe High/ Moderate

8

Current Pilot PartnersFinancial

Service ProviderCountry No of Clients

being served by FSP

No. of clients participating in

health indicators

surveyADRA Peru 17,039 95

CARD Philippines 1,828,052 472

ESAF India 322,590 1,000

Equitas India 1,344,361 250

9

Survey Adaptations Poverty measurement – Use of country-specific PPIs Food security and nutrition –added a focus on food items in

India to reflect stronger focus on nutrition Preventive health care – focuses on institutional births in India;

annual exams and Pap tests in Peru; various annual exams in the Philippines and use of health insurance (PhilHealth)

Curative Health Care: Questions same in all three countries – forgoing medical treatment and purchase of medicines due to cost

Water and Sanitation– Focuses on defecating in the open and treating water to make it safer to drink in India; Open defecation, water sources, and water treatment in Peru; Water sources and treatment of water in the Philippines

Attitudes: Only measured in Peru and Philippines, accesses levels of confidence related to ability to cover future medical costs and seek adequate medical care

10

Standardization of indicators may be difficult. Proceed with caution in the interpretation of results. Baseline values will be important to establish. Baseline values with high levels of performance may not be

useful to track. The value of statistical analysis – ex. correlations between

indicators of interest with poverty – can help refine our “theory of change” as well as determine which indicators may be the most useful to help us understand changes in client outcomes.

Who to track and for how long? This is a very important question to answer as it influences which indicators will be the most useful.

This process requires patience.

Lessons Learned

Collaborating on Outcomes

AccionBRAC & BRAC InternationalCARE Access AfricaFINCA InternationalFreedom from Hunger

Grameen FoundationOpportunity InternationalPro MujerVisionFund InternationalWomen’s World Banking

The Goal of Our Work on OutcomesTo develop, test and disseminate a common core measurement and monitoring system designed to:

Be cost-effectively embedded within an MFI’s operations Use industry-standard indicators and metrics (PPI) Provide affiliated MFIs with actionable client outcome data

that can be tracked over time Allow MFIs the flexibility to measure those changes in

client outcomes they wish to affect using the same indicators for the same outcomes

Enhance the Working Group’s ability to demonstrate and improve the benefits of Responsible Microfinance for the clients being served

Contribute to the sector’s understanding of client outcomes

The Proposed Method1. Select the indicators to test 2. Target countries and the MFIs3. Pilot the indicators

Social performance (SP) leads will work with their partners to collect, analyze and report on the data

SP leads and MFI partners would convene to discuss experiences, identify lessons learned and develop recommendations for future use

SP leads will develop and publish a series of briefs by outcome area with theory of change and recom-mended indicators

Package and disseminate the final indicators along with supporting documentation

Lessons from the Working Group’s Review of Outcomes•Phase 1: Catalogued 69 different

outcomes research from all 8 members and their affiliates▫Wide variation in product studied,

methodology, indicators, quality of research, etc.

▫Difficult to draw definitive conclusions when comparing outcomes across programmes

Different methods

Baseline

Midline

Endline

Other data

FGD

Bank records

KII

Lit Review

Control group

Qual. survey

Quant. survey

0 5 10 15 20 25 30 35 40 45 5024

425

1151616

2730

1544

RESEARCH COM-PONENTS

Different themes

Child growthSocial capital

NutritionShocks / coping mechs

Personal view of ec'c standingFarming (business)

Food securitySavings

EmpowermentPoverty status

Poverty outreachFin literacy

Experience with MF programsBusiness (non-farming)

HealthWell-being

IncomeEducation

Assets

0 5 10 15 20 25 30 353

1010

1410

1525

2317

144

716

2930

1128

316

INDICATORS

A Closer Look at Education Studies1. There was a big range of indicators

being used across and within the networks.

2. There was little consistency across institutions.

3. Most studies only looked at one dimension of education.

Phase 2: Identifying Potential Indicators for 7 Types of Client Outcomes1. Food security2. Coping strategies/shocks3. Economic poverty (income or financial

status)4. Health5. Assets, housing and business6. Social capital and empowerment7. Child and youth education

Phase 2: Summary of IndicatorsFood security

% of households that are food insecure (levels measured vary by tool)Coping strategies/shocks% of households that had to [make a specific adjustment or level of change] to cope with the effects of [household or community shock]Economic poverty (income or financial status)% of households living above/below a given poverty lineHealth% of households with [level of access] to health care [services/supplies]

Phase 2: Summary of IndicatorsAssets, housing and business% of households that were able to [purchase assets/improve their homes] in the past yearSocial capital and empowerment% of [women] expressing confidence in making decisions regarding [specific category]% of clients who perceive [specify benefit or value] in group participationChild and youth education% of children in the household who are regularly attending school% of households with increased ability to pay for school fees

Discussion

1. Any questions/clarifications? 2. What is your experience in indicator selection?3. Can we aim at standardisation - synchronisation?

4. Our working group: contributing to guidelines for selection (and a menu) of indicators to measure client outcomes?

Thank you•For follow up, please contact:

info@sptf.info, francessinha@edarural.com

•Please note: presentations and recordings from all Outcomes Working Group Meetings are being posted to the SPTF website, working groups page: http://sptf.info/sp-task-force/working-groups