Designing Rigorous Impact Evaluations of Agricultural ... · Designing Rigorous Impact Evaluations...

Abt Associates | pg 1

Designing Rigorous Impact Evaluations of Agricultural “Pull” Mechanisms Stephen Bell, Tulika Narayan, & Judy Geyer Abt Associates 4 September, 2014


“Pull” Mechanims: – Why? – What?

Missing markets for socially-essential activities/ products in the agricultural value chain, such as -- healthy strains of grain -- effective harvest management

Traditional “push” mechanisms have failed to provide sustainable solutions to agricultural market failures

Growing appreciation of the potential role of the private sector

Create “prizes”/economic rewards to private sector actors when they deliver desired results

With time, phase out the prizes


3

Outline of Talk

Introducing the AgResults initiative

Research questions and intervention model

Nigeria Aflasafe Pilot

Kenya On-Farm Storage Pilot

General lessons for designing agricultural IEs


Testing Agricultural “Pull” Mechanisms: AgResults


AgResults: A Program for Trying

and Learning Many “pull” pilots:

Nigeria (Aflasafe)

Kenya (On-Farm Storage)

Zambia (Provitamin-A Maize)

Myanmar and/or India (Newcastle Disease Innoculation)

[others in conception stage]

-----------------------------------------------------------------

Many donors:

Australia (DFAT), Canada (DFA), U.K. (DFID), U.S. (USAID), Bill & Melinda Gates Foundation, World Bank


The AgResults Evaluation

Goal = answer two policy questions:

1. Do “pull” mechanisms increase farmers’ adoption of

improved agricultural technologies?

2. Are sustainable markets created that outlive “pulls”?

Basic tasks for Abt Associates:

Define evaluation framework (integrative across pilots)

Specify data and methodologies for 5 “pull” pilot studies

Carry out 3 of those studies

Provide input to donors’ learning agenda (reports, data)


Research

Questions and

Intervention

Model


Detailed Research Questions

ions What is the impact of the [name] pilot on… … private sector involvement in markets for improved technologies? … farmers’ uptake of improved technology? … smallholder incomes? … consumers’ use of derivative products?

Are effects sustainable once the “pull” incentives end?

What are the lessons for future “pull” initiatives?


Evaluation Framework: Economic

Model of the Agriculture Value Chain

Farmer Distributor Consumer Social Planner

Goal Maximize Profit

Maximize Profit

Maximize Utility

Maximize Social Welfare

Choices Technology, capital/labor

Technology, capital/labor

Consumption Interference Mechanisms

Constraints.. Capital/labor costs

Capital/labor costs

Wealth, information

Funding Knowledge

Price, Quantity

Onfarm Product

Price, Quantity

Market Product


10

The Social Planner’s Problem and

Tools

Market failures:

– Insufficient consumer / farmer (i) knowledge of risks, and

(ii) demand for solutions: aflatoxins, Newcastle disease

– Consumer reluctance to engage in behavioral change:

bio-fortified “orange” maize

– Initial technology too expensive until production and

demand increase: grain storage solutions

“Push” mechanisms lower costs

“Pull” mechanisms provide extra revenue conditional on

desired change in quantity/kind of products sold in market


Market without Intervention

Farmer Distributor Consumer

Maximize Profit Maximize Profit Maximize Utility

Social Planner


costs costs

Price, Quantity

Price, Quantity

Onfarm Product

Market Product

costs


Add “Push” Mechanisms



Social Planner


costs costs

New Price Quantity

New Price Quantity

Onfarm Product

Market Product

costs


Add “Pull” Mechanism Instead



Social Planner


costs costs

New Price Quantity

New Price Quantity

Onfarm Product

Market Product

costs


Sustain the Change, Long-Run

Farmer Technology

Distributor Technology

Consumer Information


Social Planner


costs costs

New Price Quantity

New Price Quantity

Onfarm Product

Market Product

costs


Aflasafe Pilot - Nigeria


Theory of Change - Nigeria

Incentives for sale of aflatoxin-free maize

• Increase in treatment of crops with Aflasafe

• Increase in availability of aflatoxin-free maize

Increased Awareness

•Adoption of Aflasafe by farmers

• Increase in awareness of health benefits of aflatoxin-free maize

Smallholder benefits

• Increase in smallholder income (incentive for greater yields)

•Consumption of aflatoxin-free maize

Technical Assistance


Impact Inference Strategy

Random assignment of 72 villages

3 years of phased implementation (2014-16)

Baseline survey of farmers from cohorts B & C

(2014)

End-line survey of farmers from cohorts A, B, & C

(2015)

1-year impact (B vs. C)

2-year impact (A vs. C)


Challenges and Responses

Implementer buy-in

-- two Evaluation Design Workshops with

implementers (welcomed a “lottery”!)

Sampling farmers to maximize detection of impacts -- implementers provided lists of likely participating

farmers in each village before randomizing

can measure impacts on projected participants

-- lists proved uneven for cohort A

-- obtained post-randomization updates for cohorts

B & C use “ASPES” to measure impact on a

symmetrically-predicted endogenous subgroup

(Bell & Peck, 2013)


On-Farm Storage Pilot - Kenya


Theory of Change - Kenya

Incentives for sale of on-farm storage

•Sales and marketing by private sector to promote on-farm storage

Increased Awareness

•Adoption of on-farm storage by farmers

•Awareness of proper use of on-farm storage

•Awareness of benefits of storing maize

Smallholder benefits

• Increase in smallholder income (better timing of sales to open market)

• Increased food security


Impact Inference Strategy

Implementation plan rules out contemporaneous

comparison group

All-at-once rollout no comparable maize-growing

regions for comparison sites

Community-wide marketing no “untreated”

farmers in intervention sites

No choice but to use pre-post comparison

-- do it with strongest possible data

-- apply robustness check


Challenges and Responses

The future, absent the intervention, is not static -- use trend line in outcomes before the pilot to project “untreated” trajectory past that point -- prioritize addition of more pre-program time points (over post-program time points) when collecting data interrupted time-series

Trend may shift on its own -- track an outcome not affected by the intervention but affected by the same confounding factors (rainfall, pest burden) -- test for “false effects” on yield per acre -- include as a control variable


General

Lessons


Broader Lessons for Agricultural IE

Be flexible and creative with impact inference

strategies

Build a constituency “on the ground” to support key

evaluation functions

Build theory into research design and data priorities

Worry about statistical precision to detect impacts –

and be cautious in interpreting null findings

Preserve resources to allow long-run (e.g., post-

”pull”) follow-up


Questions? Discussion

[email protected]

Designing Rigorous Impact Evaluations of Agricultural ... · Designing Rigorous Impact Evaluations...

Documents

Transcript of Designing Rigorous Impact Evaluations of Agricultural ... · Designing Rigorous Impact Evaluations...