Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012
Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012
description
Transcript of Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012
![Page 1: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/1.jpg)
Africa RISINGM&E Expert Meeting
Addis Ababa, 5-7 September 2012
![Page 2: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/2.jpg)
Outline
• Timeline
• Objectives of M&E
• Targeting vs Research
• Monitoring and Evaluation
• Theory of Change
• Evaluation designs
• Open issues
![Page 3: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/3.jpg)
3-9 Months
9-12 Months
Timeline
✔1-3
Months
Site selection/Characterization
ComponentInventory
Activity -> Indicator List
Survey Design
Baseline Survey
Component DB
✔
Evaluation Design
![Page 4: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/4.jpg)
Objectives of M&E
• Keep track of the AR’s outputs (M)
• Assess the impact of AR (E)
• Create knowledge of what works (M&E):
-assess effectiveness
-rank policy/project alternatives
-appraise scale-up and external validity
-identify problems in the project
-inform project management
![Page 5: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/5.jpg)
What is AR M&E after?
• M could be an easy(ier) task but:
-we need constant and timely information
-we need close collaboration among CG, implementers, and IFPRI
->Solution: Outcome mapping? Cost-benefit analysis? Surveys?
• E must be well thought in advance…
![Page 6: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/6.jpg)
Targeting vs Research
1. Do we need to select the poorest/hungriest farmers to benefit from AR?
2. Or, being AR a research project, is it possible to use a stratified sample (based on different criteria)?
2a. common E approach?
2b. project-specific E approach? It depends on the question(s)!
![Page 7: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/7.jpg)
Evaluation
• Determine the causal effect of AR on outcomes (not only on outputs):– Farmers’ wellbeing?– Land productivity?– Input supply, labor productivity, environment, women’s
conditions, health and nutrition,…?– …all of the above plus-> for whom? For which development
domain? For which type of households? For which livelihood?
• What would be the impact with a different technology package?
![Page 8: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/8.jpg)
Monitoring Evaluation
‘Traditional’ M&E: monitoring to track implementation efficiency
(input - output)
Impact Evaluation: estimate causal effectiveness on outcomes (output
- outcome)
INPUTS OUTCOMES/IMPACTSOUTPUTS
MONITOR EFFICIENCY
EVALUATE EFFECTIVENESS
$$$
BEHAVIOUR
Note: Diagram from WorldBank training material produced by Arianna Legovini, Lead Economist - AIEI
![Page 9: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/9.jpg)
Theory of Change
• Impact evaluation must be based on a set of hypotheses on the change that can be achieved as a consequence of AR
• How would you think that AR can affect the life of (beneficiary) farmers?
![Page 10: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/10.jpg)
Livelihood Strategies /Coping Strategies
/Vulnerability to Shocks
AR implementation
Productivity increase
Income and Expenditure Saving/
Investment
General Household Expenditure
School Enrolment and
Attendance
Expenditure on Health and Education
for children
School ProgressionHealth Status
Food IntakeDietary
DiversityFood Security
Psychological well being *
Other Expenditures for children: Food, Clothing,
Recreation
Targeting ACTIVITIES
OUTPUTS
FIRST ORDER OUTCOMES
SECOND ORDER OUTCOMES
Asset Building
THIRD ORDER OUTCOMES
Labour Participation
Child Labour Remittances
Access to Credit
Time Allocation of
Children
Time Allocation/ Caring arrangements/
Migration of Adults and Caregivers *
Intra-household decision making *
Utilization of Health Services
Time and risk preferences
![Page 11: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/11.jpg)
Impact Evaluation
• How would you go about measuring the causal impact of AR on …
-productivity?
![Page 12: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/12.jpg)
Y
t
Impact?
Impact Evaluation - Method
AfterBefore
Bens
![Page 13: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/13.jpg)
Y
t
Impact
Impact Evaluation - Method
AfterBefore
Bens
Non Bens
RCTs
![Page 14: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/14.jpg)
Y
t
Pre-existing Difference
Impact
Impact Evaluation - Method
AfterBefore
Bens
Non Bens
Diff in Diff
![Page 15: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/15.jpg)
Y
t
Propensity Score Matching
Impact
Impact Evaluation - Method
AfterBefore
Bens
Non Bens
Diff in Diff
with
![Page 16: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/16.jpg)
Impact Evaluation
• How would you go about measuring the causal impact of AR on …
-Gross margin per unit of land? [assume we are not interested in farmers with <.3 ha]
![Page 17: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/17.jpg)
RDD0
500
01
0000
150
002
0000
250
00G
ross
ag
r. r
eve
nue
per
hec
tare
in P
PP
0 .2 .4 .6 .8 1(sum) landcult
![Page 18: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/18.jpg)
Impact Evaluation - Method
• Causal effect: change that is due to AR and not to other actors or factors (confounders)– … taking into account any other factors also
changing during the program period– … taking into account any systematic
differences between beneficiaries and non-beneficiaries of AR intervention
It is very important that the “control group” is comparable to the “treatment group”
![Page 19: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/19.jpg)
Evaluation DesignTreatment Sites Control Sites
• How can we ensure that treatment and control sites are comparable?
![Page 20: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/20.jpg)
Evaluation Design
• How can we ensure that treatment and control farmers are comparable?
![Page 21: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/21.jpg)
Evaluation Design
Random Treatment Assignment
![Page 22: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/22.jpg)
Evaluation Design
Random Treatment Assignment
![Page 23: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/23.jpg)
Evaluation DesignTreatment sites Control Sites
BENEFICIARY
FARMERS
WOULD BE BENEFICIARY
FARMERS
A B
![Page 24: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/24.jpg)
Y
t
Pre-existing Difference
Impact
Where do we stand?
AfterBefore
BASELINE
A
B
![Page 25: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/25.jpg)
BENEFICIARY
FARMERS
Evaluation DesignTreatment sites Control sites
WOULD BE BENEFICIARY
FARMERS
A BWOULD BE NON
BENEFICIARY FARMERS
NON BENEFICIARY
FARMERS
C D
![Page 26: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/26.jpg)
Open issues/1• What questions we would like AR to
answer?
• What R4D lessons can we learn from jumpstart projects? What would you carry over to longer-term AR activities?
• Do we need to target AR to specific farmers?
• Role of FtF indicators -which data to collect-?
![Page 27: Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012](https://reader035.fdocuments.us/reader035/viewer/2022062807/568150a1550346895dbea088/html5/thumbnails/27.jpg)
Open issues/2• Choice of outcome indicators/variables
• Ethics for control (same as placebo effect in medicine)
• Sampling frame for randomization
• Sample design
• Statistical power for causal impact
Site 1 Site 2