The counterfactual logic for public policy evaluation Alberto Martini
-
Upload
alec-nieves -
Category
Documents
-
view
17 -
download
0
description
Transcript of The counterfactual logic for public policy evaluation Alberto Martini
![Page 1: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/1.jpg)
The counterfactual logic for public policy evaluation
Alberto Martini
hard at first, natural later
1
![Page 2: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/2.jpg)
Everybody likes “impacts” (politicians, funders, managing
authorities, eurocrates)
Impact is the most used and misusedmisused term in evaluation
![Page 3: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/3.jpg)
Impacts differ in afundamental way from
outputs and resultsOutputs and results are
observable quantities
![Page 4: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/4.jpg)
Can we observe an impact?
No, we can’tThis is a major point of departure
between this and other paradigms
![Page 5: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/5.jpg)
As output indicators measure outputs, as result indicators
measure results, so supposedly
impact indicators measure impacts
Sorry, they don’t
![Page 6: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/6.jpg)
Almost everything about programmes can be observed (at least in principle):
outputs (beneficiaries served, activities done, training courses offered,
KM of roads built, sewages cleaned)
outcomes/results (income levels, inequality, well-being of the population,
pollution, congestion, inflation, unemployment, birth rate)
![Page 7: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/7.jpg)
Unlike outputs and results, to measure impacts
one needs to deal with
unobservables
![Page 8: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/8.jpg)
To measure impacts, it is not enough to “count” something,
or compare results with targets, or to check progress from baseline
It is necessary to deal with
causality
![Page 9: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/9.jpg)
“Causality is in
the mind” J.J. Heckman
Nobel Prize Economics 2000
![Page 10: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/10.jpg)
How would you define impact/effect?
“the difference between a situation observed after an intervention
has been implemented
and the situation that………………………………………………………………….. ?
would have occurred without the intervention 10
![Page 11: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/11.jpg)
There is just one tiny problem with this definition
the situation that would have occurred without the
intervention cannot be observed
11
![Page 12: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/12.jpg)
The social science scientific community has developed the notion of potential outcomes
“given a treatment, the potential outcomes is what we would observe for the same individual for different
values of the treatment” 12
![Page 13: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/13.jpg)
13
Hollywood’s version of potential
outcomes
![Page 14: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/14.jpg)
A priori there are only potential outcomes of the
intervention, but later one becomes an observed outcome, while the other
becomes the counterfactual outcome
![Page 15: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/15.jpg)
A very intuitive example of the role of counterfactual
analysis in producing credible evidence for policy decisions
![Page 16: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/16.jpg)
Does learning and playing chess have a positive impact on achievement in
mathematics?
![Page 17: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/17.jpg)
Policy-relevant question:
Should we make chess part of the regular curriculum in elementary schools, to improve mathematics
achievement? Or would it be a waste of time?
Which kind of evidence do we need to make this decision in an informed way?
17
![Page 18: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/18.jpg)
Let us assume we have a crystal ball
and we know “truth”:
for all pupils we know both potential outcomes—the math score they would obtain if they practiced chess or the score they would
obtain if they did not practice chess
![Page 19: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/19.jpg)
General rule:
what we observe can be very different
than what is true
![Page 20: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/20.jpg)
Mid ability1/3
Mid ability1/3
Types of pupilsTypes of pupils
Low ability1/3
Low ability1/3
Practice chess at home and do not gain much if taught in schoolPractice chess at home and do
not gain much if taught in school
Practice chess only if taught in school, otherwise they do not
learn chess
Practice chess only if taught in school, otherwise they do not
learn chess
Unable to play chess effectively, even if taught in school
Unable to play chess effectively, even if taught in school
20
High ability1/3
High ability1/3
What happens to themWhat happens to them
![Page 21: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/21.jpg)
Mid abilityMid ability
High abilityHigh ability
Low abilityLow ability
If they do play chess at school
If they do play chess at school
If they do NOT play at school
If they do NOT play at school differencedifference
7070 70 70 00
5050 40 40 1010
20 20 20 20 0 0
Potential outcomes
21
math test scores
Try to memorize these numbers: 70 50 40 20 10Try to memorize these numbers: 70 50 40 20 10
![Page 22: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/22.jpg)
SO WE KNOW THAT
1. there is a true impact but it is small
2. the only ones to benefit are mid ability students, for them
the impact is 10 points
22
![Page 23: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/23.jpg)
The naive evidence:observe the differences between chess
players and non players and infer something about the impact of chess
The difference between players and non players measures the effect of playing chess.
DO YOU AGREE?23
![Page 24: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/24.jpg)
The usefulness of the potential outcome way of reasoning is to
make clear what we observe and we do not observe,
and what we can learn and cannot learn from the data, and how
mistakes are made
24
![Page 25: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/25.jpg)
Mid abilityMid ability
High abilityHigh ability
Low abilityLow ability
7070
20 20
25
What we observe
DO YOU SEE THE POINT?DO YOU SEE THE POINT?
average=30average=30
![Page 26: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/26.jpg)
Results of the direct comparison
Pupils who play chess
Average score = 70 points
Pupils who do not play chess
Average score = 30 points
Difference = 40 pointsis this the impact of playing chess?
26
![Page 27: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/27.jpg)
Can we attribute the difference of 40 points to playing chess alone?
There are many more factors at play that influence math scores
27
OBVIOUSLY NOT
![Page 28: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/28.jpg)
Play chessPlay
chess
Math abilityMath ability
Math test
scores
Math test
scoresCS
SELECTION PROCESS DIRDIREDIRECT INFLUENCE
Ignoring math ability could severly mislead us, if we intend to interpret the difference in test
scores as a causal effect of chess
Does it have an impact on?
28
![Page 29: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/29.jpg)
First (obvious) lesson we learn
Most observed differences tell us nothing about causality
We should be careful in general to make causal claims based on the
data we observe29
![Page 30: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/30.jpg)
is pretty silly, isn’t it?30
However, comparing math test scores for kids who have learned chess by themselves and kids
who have not
![Page 31: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/31.jpg)
Comparing enterprises applying for subsidies with those not applying and call the
difference in subsequent investment “the impact of the subsidy”
31
Almost as silly as:Comparing participants of training courses with non participants and
calling the difference in subsequent earnings “the impact of training”
![Page 32: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/32.jpg)
The raw difference between self-selected participants and non-
participants is a silly way to apply the counterfactual approach
the problem is selection bias (pre-existing differences)
32
![Page 33: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/33.jpg)
Now we decide to teach pupils how to play
chess in school
Schools can participate
or not
Now we decide to teach pupils how to play
chess in school
Schools can participate
or not
33
![Page 34: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/34.jpg)
We compare pupils in schools that participated in the program and pupils in schools which did not
in order to get an estimate of the impact of teaching chess in school
34
![Page 35: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/35.jpg)
Pupils in the treated schools
Average score = 53 points
Pupils in the non treated schools
Average score = 29 points
Difference = 24 pointsis this the TRUE impact?
35
We get the following results
![Page 36: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/36.jpg)
Mid abilityMid ability
High abilityHigh ability
Low abilityLow ability
Schools that participatedSchools that participated
Schools that did NOT participateSchools that did NOT participate
30% 30%
60% 60%
70%
36
There is an evident difference in composition between the two
types of schools
There is an evident difference in composition between the two
types of schools
20%
10%
10% 10%
![Page 37: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/37.jpg)
Mid abilityMid ability
High abilityHigh ability
Low abilityLow ability
Schools that participatedSchools that participated
Schools that did NOT
30% 30%
60% 60%
70 %
WEIGHTED Average of 70, 50 and 20 = 53
WEIGHTED Average of 70, 50 and 20 = 53
37
WEIGHTED Average of 70, 40 and 20 = 29
20 %
10 %
10% 10%
Average impact = 53 – 29 = 24Average impact = 53 – 29 = 24
![Page 38: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/38.jpg)
The difference of 24 points is a combination of the true impact and
of the difference in composition
If we did not know the truth, we might take 24 as the true impact on math
score, and being a large impact,make the wrong decision
38
![Page 39: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/39.jpg)
We have two alternatives:
statistically adjusting the data or
conducting an experiment The mostt
39
![Page 40: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/40.jpg)
Any form of adjustment assumes we have a model in mind,
we know that ability influences math scores and we know how to
measure ability
40
![Page 41: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/41.jpg)
But even if we do not have all this information we can conduct a
randomized experiment
The schools who participate get free instructors to teach chess , provided
they agree to exclude
one classroom at random
41
![Page 42: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/42.jpg)
Results of the randomized experiment
Pupils in the treated classes in the volunteer schools
Average score = 53 points
Pupils in the excluded classes in the volunteer schools Average score
= 47 points
Difference = 6 pointsthis is very close the TRUE impact
42
![Page 43: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/43.jpg)
Mid abilityMid ability
High abilityHigh ability
Low abilityLow ability
Schools that volunteeredSchools that volunteered
Schools that did NOT volunteer
Schools that did NOT volunteer
30%30%
60% 60%
EXPERIMENTALSaverage of 70, 50 & 20 = 53
EXPERIMENTALSaverage of 70, 50 & 20 = 53
Impact = 53 – 47 = 6 Impact = 53 – 47 = 6 43
CONTROLS mean of 70, 40 & 20 = 47
CONTROLS mean of 70, 40 & 20 = 47
random assignment: flip a coin random assignment: flip a coin
10% 10%
![Page 44: The counterfactual logic for public policy evaluation Alberto Martini](https://reader034.fdocuments.us/reader034/viewer/2022051620/56813324550346895d9a0402/html5/thumbnails/44.jpg)
Experiments are not they only way to identify impacts
However, it is very unlikely that an experiment will generate grossly mistaken estimates
If anything, they tend to be biased toward zero
On the other hand, some wrong comparisons can produce wildly mistaken
estimates44