How to evaluate ultimate impact of value chain interventions? Mixed methods design for attributing indirect interventions to farmers’ income.

The case of maize in Bangladesh

Gideon Kruseman and Shovan Chakraborty25 March 2013

Klik op het pictogram als u een afbeelding wilt toevoegen

Overview

Case

Evaluation questions

Design (conference theme)

Impact logics

Mixed methods design

Results

Conclusions

Communication and results (conference theme)

Lessons learned

Case description

2004 Katalyst introduced maize through

Retailer training

Extension officers training

Contract farming

2011 impact evaluation indicated strong income impact but method lacked before-after and with-without

2012 new impact evaluation based on rigorous scientific methods

Questions

1. What ultimate impact can be expected 8 years later?

2. How can that be measured?

No baseline possible in VC (uncertain who will benefit ex-ante)

Difficult to determine comparison group with notorious spill-over effects

3. What is the impact?

Outreach

Impact

Attribution

Q3 challenging

-> Mixed

methods

Q1&2 uncommon

-> Multi-

disciplinary team with

knowledge of value chains

Design of IE

What expectations underpinned choices for design?

1. There is impact that is attributable to the intervention

2. Measurement of impact is limited by lack of baseline, no clean control groups and many external factors

3. Attribution is strongest low in impact chain (with retailers, officials and companies, not with farmers)

4. If intervention is unique, attribution is easier to establish

Impact logics and the IE (Q1)

Level of attribution of Katalyst interventions to

outcomes

Demonstration Effect: other farmers grow maize

Ultimate outcomesKatalyst’s impact)

Intermediate outcomes

Immediate outcomes

InterventionPartner with company to

promote quality seeds and info (2008-2010)

Client farmers get quality seeds and info on cultivation

techniques and improve practices

167 Partners (120 retailers, rest dealers, agrovets, MSVs,

farmers) promote quality seeds & cultivation techniques

Client farmers get high yields and income

Partner with company to establish contract farming

(2008 - 2011)

Contracted farmers get info on cultivation techniques, buy-

back, finance and inputs (seeds) and improve practices

50 contractors (retailers/traders) start contract farming

Contracted farmers get high yields and income

Partner with government to improve extension services

(2009 – 2011)

Trained farmers get info on cultivation techniques and

improve practices

75 extention workers start promoting maize and cultivation techniques

Trained farmers get high yields and income

Other sources of quality seeds

Other sources of quality seeds

Macro-economy and climateMacro-economy and climate

Other information sourcesOther information sources

Other information sourcesOther information sourcescopy other contractorscopy other contractors

Retailer training Contract farming Maize-based cropping

Focu

s as

sess

men

t

Design (Q2)

Approach

For each question test assumptions of the impact logic at every step

Core methods

Large n farmer survey data

Small n in-depth interviews with farmers, contractors and retailers, treated and non-treated

Maize sector study

Analysis

Production cost-revenue analysis

Factor analysis

Qualitative analysis of in-depth interviews

Assumptions in impact logic (Q1&2)

Mixed methods (Q3)

What outreach?

In-depth contractors interviews

Validation with in-depth farmers interviews

What income effects?

Large n farmer survey for production cost analysis and quantitative cohort comparison

Validation with in-depth farmers interviews

To what extent can impact be attributed?

Qualitative analysis of all in-depth interviews

Factor analysis of large n survey

Results: outreach

Farmers affected by contract farming

 char new char old

mainland new

mainland old Total

Direct outreach 867 2203 966 1753 5.789

Indirect outreach 1185 17675 510 4058 23.428

Results: income effect

Relevant assumption to be tested: farmers have higher yields than the benchmark

  char new char oldmainland new

mainland old copy

yield (maund/dec) 35 87 5 9 18

Total Rev. Impact not considering Land size Change (BDT) 17,410 42,980 1,938 3,879 7,947

Total Rev. Impact including Land size Change (BDT) 38,523 80,757 4,742 10,399 16,902

Total income increase (1 yr after intervention) 33,399,450 177,908,515 4,580,293 18,230,162 395,988,876

Results: contribution of Katalyst

Contract farming has started as a result of Katalyst interventions.

●Only contractors are those involved in the intervention

●These contractors have growing number of contract farmers

Conclusion: True

Knowledge passed on through Katalyst training of contractors is crucial for contractors

●Knowledge comes from many sources including Katalyst

Conclusion: True / False

Results: contribution of Katalyst

Knowledge passed on from contractors to farmers is crucial for farmers

●Knowledge comes from many sources including Contractors

●Contractors are main source of information

Conclusion: partly true

Service provision by contractors is important to contract famers

Conclusion: true

There is a spill over effect of maize cultivation from contract farmers to neighbouring farmers

Conclusion: true

Results: contribution of Katalyst

Knowledge passed on through Katalyst training of Retailers is crucial for retailers

●Knowledge comes from many sources including Katalyst

Conclusion: True / False

Knowledge passed on from retailers to farmers is crucial for farmers

●Knowledge comes from many sources including retailers

Conclusion: True / False

Conclusions

There is impact of contract farming especially because of service provision specifically related to this production form

The impact of contract farming is 100% attributable to Katalyst

Knowledge on maize cultivation comes from many sources, knowledge is vitally important but relative importance of information sources cannot be attributed to any single source.

There is a contribution to the knowledge base through Katalyst interventions

Communication and use of IE

Communication is focussed on:

Underpinning and justifying estimated impact

To which interventions could this be attributed

Some interventions (contract farming) have attributable impact

Some interventions (retailer training) do not

Results are used for:

Fine-tuning current interventions

Design of improved monitoring of interventions and immediate and intermediate results to ensure more robust future impact evaluations (counterfactual analysis)

Lessons learned

Importance of impact logic framework

Importance of defining hypothesis based on impact logic

Importance of design based on hypotheses and application of mixed methods to overcome threats to validity of conclusions

Separation of impact and attribution/contribution

Need to inform donors on costs benefits of IE

END

Thank you for your attention.