VALUE CHAIN MONITORING AND EVALUATION GUIDE

MODULE 4

Select Performance Indicators

April 10, 2023

2

GETTING STARTEDOBJECTIVE OF MODULE 4

•Analyze your causal model to identify key linkages and the results you want to monitor and measure.

•Select the indicators you will track through your M&E system.

•Identify which indicators you will track through traditional, routine measurement and which indicators you will track through routine observation.

April 10, 2023

3

GETTING STARTEDRoutine Measurement vs. Routine Observation

Routine Measurement•Indicators tracked using traditional M&E tools – surveys, focus group discussions, etc.•Traditional M&E timelines – monthly, quarterly, beginning/middle/end of project, etc.

Traditional, Rigid

Routine Observation•Indicators tracked using non-traditional M&E tools – staff observation, team reflections, meetings and after action reviews•Traditional M&E timelines or whenever staff learn something important about that indicator.

Flexible, Reactive

April 10, 2023

4

GETTING STARTED

Including both types of indicators sets VC M&E apart. But why both?

• Project teams are working in a constantly changing environment

• VC M&E requires a delicate balance between rigorously collected and standardized data and observations made by knowledgeable team members.

April 10, 2023

5

GETTING STARTED

Identifying Good IndicatorsRoutinely Measured

Indicators

• Specific

• Measurable

• Achievable

• Relevant

• Time-bound

Routinely Observed Indicators

•Open-ended, indentified through participatory exercises•Focused on anticipated changes such as impact group attitude and behavior•Weighted against validity of ‘killer’ assumptions•Revised as needed to suit implementation needs

CONSULT CARE’S UNIVERSAL INDICATORS FOR MARKET ENGAGEMENT

April 10, 2023

6

Causal model

M&E client map

Donor indicator list

CARE CO program and global indicator lists

Key team members and/or partner

representatives

MATERIALS / INPUTS RECOMMENDED

April 10, 2023

7

STEP-BY-STEP GUIDE

1

•DEFINE YOUR IMPACT INDICATORS

2

•IDENTIFY AND DEFINE THE CRITICAL ASSUMPTIONS IN YOUR CAUSAL MODEL

3

•DEVELOP INDICATORS FOR YOUR OUTPUTS AND LEADING AND LAGGING OUTCOMES

4

•REVIEW THE INDICATORS WITH KEY M&E CLIENTS

April 10, 2023

8

STEP-BY-STEP GUIDE: STEP 1

Defining Indicators•These will ultimately measure program’s success•Household, Enterprise and Sectoral Levels•Should align with CO, Donor, National Government and CARE Universal Indicators•Again, DO NOT SELECT MORE THAN YOU CAN MEASURE!!!

April 10, 2023

9

STEP-BY-STEP GUIDE

1

•DEFINE YOUR IMPACT INDICATORS

2

•IDENTIFY AND DEFINE THE CRITICAL ASSUMPTIONS IN YOUR CAUSAL MODEL

3

•DEVELOP INDICATORS FOR YOUR OUTPUTS AND LEADING AND LAGGING OUTCOMES

4

•REVIEW THE INDICATORS WITH KEY M&E CLIENTS

April 10, 2023

10

STEP-BY-STEP GUIDE: STEP 2WORK THROUGH THE

CAUSAL MODEL•Develop indicators at other levels:

• Outputs• Leading outcomes• Lagging outcomes

•To avoid piling up more indicators than we can measure:

• Focus on priority clients of M&E system

• Prioritize most critical assumptions from causal model

• Use this participatory activity……………..

April 10, 2023

11

STEP-BY-STEP GUIDE: STEP 2Tool: Identifying & Prioritizing

AssumptionsObjective: This activity will help key system clients articulate and identify the most critical assumptions.Format: A group of 10-15 M&E clients (or more) broken into groups of 4-5. Ensure plenty of open wall space and a work table for each group.Materials: A projector, laptop, flip chart paper, markers and plenty of small stickers of different colors (for voting).Time: Will vary depending on the number of M&E clients and project complexity but should be no less than 1 hour for a simple exercise and can be ¾ of a day or more for a large-scale program.

April 10, 2023

12

STEP-BY-STEP GUIDE

1 •DEFINE YOUR IMPACT INDICATORS

2 •IDENTIFY AND DEFINE THE CRITICAL ASSUMPTIONS IN YOUR CAUSAL MODEL

3 •DEVELOP INDICATORS FOR YOUR OUTPUTS AND LEADING AND LAGGING OUTCOMES

4 •REVIEW THE INDICATORS WITH KEY M&E CLIENTS

April 10, 2023

13

Resources for ID’ing Indicators for Outputs and Outcomes

•Donor requirements

•Country Office Program Impact Measurement

Frameworks

•CARE’s Universal Indicators for Market

Engagement

•Other relevant projects operating within the

country

•CARE’s global impact measurement system

•Government measurement systems.

STEP-BY-STEP GUIDE: STEP 3

April 10, 2023

14

STEP-BY-STEP GUIDE: STEP 3COMPLETING STEP 3

•Start with a particular intervention and work through the causal pathway until you link to your impact level indicators•Trace the causal pathway that links indicator to outputs leading outcomes lagging outcomes•Choose a second and repeat the process until you have identified indicators for each causal pathway

April 10, 2023

15

STEP-BY-STEP GUIDE

1 •DEFINE YOUR IMPACT INDICATORS

2 •IDENTIFY AND DEFINE THE CRITICAL ASSUMPTIONS IN YOUR CAUSAL MODEL

3 •DEVELOP INDICATORS FOR YOUR OUTPUTS AND LEADING AND LAGGING OUTCOMES

4 •REVIEW THE INDICATORS WITH KEY M&E CLIENTS

April 10, 2023

16

STEP-BY-STEP GUIDE: STEP 4

Validate Indicators•Review the selected indicators with donor, project staff, managers, partner organizations and target participants•Group meetings, email reviews and comments, etc.•Whatever process you use should be transparent and M&E clients’ comments should be fully considered before finalizing your indicator set

April 10, 2023

17

CASE EXAMPLE

SDVC

Insert SDVC pic

April 10, 2023

18

COMMON PITFALLS•TOO MANY INDICATORS……..Don’t be afraid to negotiate.

•Contradictions between donor requests and the good of the project (what to measure, approaches, etc.) . Use informed rationale to discuss these disparities with the donor!!!!!!

•Projects are unwilling or unprepared to make difficult choices to weed out ‘nice but not necessary’ indicators.

•Projects fail to engage the donor in dialogue regarding expectations of the M&E system (high or low).

•Projects focus too much on output indicators that will not provide insight into what the project is accomplishing.

•Projects do not use the SMART test on the indicators they plan to track through routine measurement, leading to indicator lists that are poorly aligned with killer assumptions and/or difficult or impossible to measure reliably.

•Projects fail to develop a set of indicators that will be tracked through routine observation at the outset, leading to a lot of information being collected, very little of which can be aggregated and/or applied to inform project decision making.

April 10, 2023

19

TEMPLATES AND SUPPORTING MATERIALS

Result Level Domain (Women's Emp, Enterprise, Sector)

Key Change (Y/N)

Indicator - Measurement

Indicator - Observation

Target Data Users (Clients)

Data Source Collection Method / Tool

Collection Frequency

Responsible for Collection

Impact

Lagging outcome

Leading outcome

Output

CARE Market Engagement Indicator Information Table

Note: At this stage, you will not fill in the information in these three columns. You will do so after the next Chapters in this guide.

April 10, 2023

20

TEMPLATES AND SUPPORTING MATERIALS

Narrative Summary

Results Indicators Assumptions

Impacts

Lagging Outcomes

Leading Outcomes

Outputs

Interventions

LOGICAL FRAMEWORKS

April 10, 2023

21

TEMPLATES AND SUPPORTING MATERIALS

Country PPI PAT (USAID)

AngolaBangladesh X X+BeninBolivia X XCambodia X XCote d’IvoireEcuador X X+El Salvador X X+Ethiopia X XGhana X X+Guatemala X XHaiti X XHonduras XIndia X XKenya X X+Liberia X X+Malawi X XMali XMozambiqueNepal X XNicaragua XNigerPakistan XPeru X XRwanda XSierra LeoneSri Lanka XTanzaniaUganda X+Vietnam X X

Poverty Measurement Tools Available by Country

April 10, 2023

22

QUESTIONS?

COMMENTS?

April 10, 2023

23

Multiple ways to continue the discussion and continue learning:• Initiate a monthly session on the M&E guide

and case studies from across CARE. Contact cpennotti@care.org

• Join the Market Engagement Community of Practice on LinkedIn.

• Join a task force to review and refine the universal indicators. Contact nardi@careinternational.org

Want to Learn More?