How an impact evaluation will look like - World...
Transcript of How an impact evaluation will look like - World...
![Page 1: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/1.jpg)
How an impact evaluation will
look like
Impact Evaluation Workshop
Belgrade, Serbia
Robin Audy
June 4, 2015
![Page 2: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/2.jpg)
Key Messages
• Good Impact evaluations require careful planning from the start of a project
The earlier the better
• Collecting data electronically is now accessible to all
It allows new types of indicators
![Page 3: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/3.jpg)
1. The steps of an impact evaluation
1. Identify the problem
3. Design
an
evaluation
4. Do Initial
Survey
5. Randomize
the program
6. Conduct
follow-up
survey
7. Crunch the
numbers
8.
Announce
the result
2. Create a new or
target an ongoing
program
![Page 4: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/4.jpg)
1. The steps of a randomized impact evaluation
4. Do Initial
Survey
5.
Randomize
the program
6. Conduct
follow-up
survey
![Page 5: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/5.jpg)
2. Who do we interview?
![Page 6: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/6.jpg)
Who are the final beneficiaries
Who should be interviewed, and when
How to identify them?
2. Who do we interview?
![Page 7: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/7.jpg)
2. Who do we interview, and when?Imagine a simple conditional cash transfer for the poor
![Page 8: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/8.jpg)
2. Who do we interview, and when?You will target among your population. A census can help determine who is eligible.
![Page 9: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/9.jpg)
2. Who do we interview, and when?From the list of targeted eligible individuals, do your baseline survey
Do the baseline survey before assigning treatment status!
![Page 10: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/10.jpg)
2. Who do we interview, and when?Only then should you assign treatment and control statuses
![Page 11: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/11.jpg)
2. Who do we interview, and when?Perhaps take-up will not be complete among all those in the treatment group.
![Page 12: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/12.jpg)
2. Who do we interview, and when?Perhaps take-up will not be complete among all those in the treatment group.
You could also have individuals from the control who manage to get treatment.
![Page 13: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/13.jpg)
2. Who do we interview, and when?Perhaps take-up will not be complete among all those in the treatment group.
You could also have individuals from the control who manage to get treatment.
At endline, follow all the individuals from the baseline
![Page 14: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/14.jpg)
2. Note about sample representativenessInternal validity vs External validity
![Page 15: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/15.jpg)
2. Baseline surveyAnother type: clustered randomization by village
![Page 16: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/16.jpg)
2. Baseline survey
![Page 17: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/17.jpg)
2. Baseline survey
![Page 18: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/18.jpg)
2. Baseline survey
![Page 19: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/19.jpg)
3. What does a baseline interview look like?
![Page 20: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/20.jpg)
3. What about electronic data collection?Preparation is key!
• Design questions and prepare deployment
• Test, pilot and refine design
• Train and run interviews
![Page 21: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/21.jpg)
4. Resources for electronic data collection
• SurveyCTO www.surveycto.com
• Kobo Collect (www.kobotoolbox.org)
![Page 22: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/22.jpg)
5. Resources for electronic data collection
![Page 23: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/23.jpg)
6. Types of indicators to consider
• Contextual setting variables
• Age, education, poverty, community characteristics
• Intermediate outcomes
• Enrolment in school, frequency of health care visits, sanitation behavior
• Final outcomes
• Measures of learning, health outcomes, measures of deforestation
• Contact information for follow-up
• Map, name, phone number, etc…
![Page 24: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/24.jpg)
6. Types of indicators to consider
![Page 25: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/25.jpg)
24
7. Technology allows for new types of indicators
![Page 26: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/26.jpg)
25
7. Technology allows for new types of indicators
![Page 27: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/27.jpg)
26
7. Technology allows for new types of indicators
![Page 28: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/28.jpg)
8. Randomly assign treatment and comparison
![Page 29: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/29.jpg)
8. Randomly assign treatment and comparison
• The lottery assigns who gets what
• Perception of fairness
• Transparency:
-> Big public event with dignitaries and all stakeholders
• Communications plan is important
• Explain equal chances to all
![Page 30: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/30.jpg)
8. Randomly assign treatment and comparison
• Imagine we put in place right now a program that tests the attention level of
workshop attendees here today. We administer a test about the knowledge
retained from the presentation.
• For this, we divide all attendees into 3 groups:
• Stretch
• Caffeine
• Control group
• We have ~18 attendees, we want to divide them into 3 equal groups:
Example:
Stretch Caffeine Control Total
6 6 6 18
![Page 31: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/31.jpg)
8. Randomly assign treatment and comparison
All 18 participants are assigned to a ticket
A, B or C
It is possible to stratify. For example,
from each row
These tickets can be shown to
stakeholders in advance of
randomization
During the lottery, we randomly assign a
treatment to each letter…
Example:
Person Ticket
12 A
8 A
6 A
1 A
17 A
11 A
13 B
15 B
9 B
3 B
18 B
4 B
7 C
2 C
5 C
16 C
10 C
14 C
![Page 32: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/32.jpg)
8. Randomly assign treatment and comparisonExample:
Treatment Ticket
Stretch ?
Caffeine ?
Control group ?
Person Ticket
12 A
8 A
6 A
1 A
17 A
11 A
13 B
15 B
9 B
3 B
18 B
4 B
7 C
2 C
5 C
16 C
10 C
14 C
![Page 33: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/33.jpg)
8. Randomly assign treatment and comparison
• An early childhood development program has 3 components:
• Free access
• Free access + small incentive
• Free access + large incentive
• Control group
• Also, orthogonally Info and No info
• We have 240 participating villages
• We want to divide the villages into 3 equal groups:
Example of a larger program:
Free
access
Free
access +
small
incentive
Free
access +
large
incentive
Control Total
villages
No Info 30 30 30 30 120
Info 30 30 30 30 120
![Page 34: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/34.jpg)
8. Randomly assign treatment and comparison
All 240 villages are assigned to a ticket
A, B or C and X or Y
The tickets can be stratified, i.e. same
number of letters by region
These tickets can be shown to
stakeholders in advance of
randomization
In the lottery, we randomly assign a
treatment to each letter
Example:Village Ticket
Bansko AX
Belitza BX
Blagoevgrad CY
Gotse Delchev AX
Garmen CX
Krupnik BY
Mikrevo AY
Petrich BX
Sandanski CY
Satovcha BY
Simitly AY
Yakorouda CX
Ajtos AY
Burgas BY
Karnobat BX
Pomorie AX
Vresovo BY
Troianovo CY
Ahtopol AX
...
![Page 35: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/35.jpg)
8. Randomly assign treatment and comparisonLottery results
![Page 36: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/36.jpg)
8. Verify Baseline balance
Variable NControl
No info
Control
Info
Free
No info
Free
Info
Twenty lev
No info
Twenty lev
Info
Seven lev
No info
Seven lev
Info
Age 57723.90
{0.043}
3.85
[0.557]
3.93
[0.675]
3.80
[0.172]
3.86
[0.607]
3.92
[0.786]
3.89
[0.967]
3.91
[0.860]
Female 57720.460
{0.020}
0.505
[0.131]
0.491
[0.276]
0.479
[0.520]
0.502
[0.184]
0.502
[0.181]
0.454
[0.828]
0.477
[0.535]
Ethnicity: Bulgarian 57720.241
{0.040}
0.220
[0.707]
0.219
[0.709]
0.193
[0.424]
0.154
[0.126]
0.165
[0.181]
0.182
[0.279]
0.202
[0.501]
Ethnicity: Turkish 57720.150
{0.048}
0.131
[0.754]
0.143
[0.908]
0.139
[0.858]
0.227
[0.331]
0.212
[0.408]
0.331**
[0.021]
0.147
[0.959]
Ethnicity: Roma 57720.602
{0.059}
0.629
[0.741]
0.614
[0.882]
0.631
[0.724]
0.614
[0.895]
0.614
[0.881]
0.475
[0.151]
0.611
[0.916]
Ethnicity: Other 57720.001
{0.001}
0.016
[0.206]
0.021
[0.158]
0.015
[0.131]
0.001
[0.979]
0.004
[0.399]
0.008
[0.108]
0.031
[0.206]
Attended KG before 57720.711
{0.043}
0.669
[0.506]
0.666
[0.528]
0.661
[0.439]
0.725
[0.809]
0.706
[0.941]
0.638
[0.256]
0.695
[0.808]
Past 3 days child got read
books5683
0.530
{0.043}
0.439
[0.169]
0.499
[0.653]
0.441
[0.191]
0.451
[0.237]
0.499
[0.631]
0.552
[0.734]
0.482
[0.455]
Past 3 days child got told
story5687
0.723
{0.048}
0.605
[0.104]
0.635
[0.224]
0.685
[0.570]
0.622
[0.120]
0.635
[0.197]
0.738
[0.813]
0.661
[0.347]
Past 3 days child sang 56870.760
{0.049}
0.690
[0.314]
0.710
[0.471]
0.740
[0.753]
0.732
[0.664]
0.743
[0.783]
0.776
[0.807]
0.749
[0.855]
![Page 37: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/37.jpg)
8. Verify Baseline balance
Variable NControl
No info
Control
Info
Free
No info
Age 57723.90
{0.043}
3.85[0.557]
3.93[0.675]
Female 57720.460{0.020}
0.505[0.131]
0.491[0.276]
Ethnicity: Bulgarian 57720.241{0.040}
0.220[0.707]
0.219[0.709]
Ethnicity: Turkish 57720.150{0.048}
0.131[0.754]
0.143[0.908]
![Page 38: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/38.jpg)
9. Follow-up surveys
• Initiate intervention
• Option: collect additional intermediate outcomes
• Option: collect program implementation information
• Option: collect costing information for CBA / CEA analysis (opportunity costs
need to be collected
• Follow up beneficiary survey: back in the fields!
![Page 39: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/39.jpg)
Key Messages
• Good Impact evaluations require careful planning from the start of a project
The earlier the better
Know who, when and how
Plan a transparent, public lottery (especially for large programs)
• Collecting data electronically is now accessible to all
It allows new types of indicators
![Page 40: How an impact evaluation will look like - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2015/11/... · •Kobo Collect () 5. Resources for electronic data collection. 6. Types](https://reader034.fdocuments.us/reader034/viewer/2022042418/5f33e53c112e34613d601fe1/html5/thumbnails/40.jpg)
Questions?