Econometric Evaluation of Social Programs Part I: Causal Models...

Introduction Policy Roy Treatment Counter Problems ID Summary

Econometric Evaluation of Social ProgramsPart I: Causal Models, Structural Models,

and Econometric Policy Evaluation

James J. Heckman and Edward J. Vytlacil

The University of Chicago and Columbia University

ECON 312 Spring 2019

1 / 281


Introduction

Evaluating policy is a central problem in economics.

This requires the economist to construct counterfactuals.

The existing literature on “causal inference” in statistics isthe source of inspiration for the recent econometrictreatment effect literature and we examine it in detail.

2 / 281


The literature in statistics on causal inference confusesthree distinct problems that are carefully distinguished inthis chapter and in the literature in economics:

(1) Definitions of counterfactuals.

(2) Identification of causal models from idealized data ofpopulation distributions (infinite samples without anysampling variation). The hypothetical populations may besubject to selection bias, attrition and the like. However, allissues of sampling variability are irrelevant for this problem.

3 / 281


(3) Identification of causal models from actual data, wheresampling variability is an issue. This analysis recognizesthe difference between empirical distributions based onsampled data and population distributions generating thedata.

Table 1 delineates the three distinct problems.

4 / 281


Table 1: Three distinct tasks arising in the analysis of causal models

Task Description Requirements

1 Defining the Set of Hypotheticals A Scientific Theoryor Counterfactuals

2 Identifying Parameters Mathematical Analysis of(Causal or Otherwise) from Point or Set IdentificationHypothetical Population Data

3 Identifying Parameters from Data Estimation andTesting Theory

5 / 281


A model of counterfactuals is more widely accepted themore widely accepted are its ingredients:

(1) the rules used to derive a model, including whether or notthe rules of logic and mathematics are followed;

(2) its agreement with other theories; and

(3) its agreement with the evidence.

Models are of hypothetical worlds obtained byvarying — hypothetically — the factors determiningoutcomes.

6 / 281


The second problem is one of inference in very largesamples.

Can one recover counterfactuals (or means or distributionsof counterfactuals) from data that are free of any samplingvariation problems?

This is the identification problem.

The third problem is one of inference in practice.

Can one recover a given model or the desiredcounterfactual from a given set of data?

7 / 281


Some of the controversy surrounding construction ofcounterfactuals and causal models is partly aconsequence of analysts being unclear about these threedistinct problems and often confusing them.

Particular methods of estimation (e.g., matching orinstrumental variable estimation) have become associatedwith “causal inference” and even the definition of certain“causal parameters” because issues of definition,identification, and estimation have been confused in therecent literature.

The econometric approach to policy evaluation separatesthese problems and emphasizes the conditional nature ofcausal knowledge.

8 / 281


Human knowledge advances by developingcounterfactuals and theoretical models and testing themagainst data.

The models used are inevitably provisional and conditionalon a priori assumptions.

Blind empiricism leads nowhere.

Economists have economic theory to draw on but recentdevelopments in the econometric treatment effect literatureoften ignore it.

9 / 281


Current widely used “causal models” in epidemiology andstatistics are incomplete guides to interpreting data or forsuggesting estimators for particular problems.

Rooted in biostatistics, they are motivated by theexperiment as an ideal.

They do not clearly specify the mechanisms determininghow hypothetical counterfactuals are realized or howhypothetical interventions are implemented except tocompare “randomized” with “nonrandomized” interventions.

10 / 281


Because the mechanisms determining outcome selectionare not modeled in the statistical approach, the metaphorof “random selection” is often adopted.

Since randomization is used to define the parameters ofinterest, this practice sometimes leads to the confusionthat randomization is the only way — or at least the bestway — to identify causal parameters from real data.

In truth, this is not always so, as we demonstrate in thispresentation.

11 / 281


One reason why epidemiological and statistical models areincomplete is that they do not specify the sources ofrandomness generating variability among agents.

I.e., they do not specify why observationally identicalpeople make different choices and have different outcomesgiven the same choice.

They do not distinguish what is in the agent’s informationset from what is in the observing statistician’s informationset, although the distinction is fundamental in justifying theproperties of any estimator for solving selection andevaluation problems.

12 / 281


They are also incomplete because they are recursive.

They do not allow for simultaneity in choices of outcomesof treatment that are at the heart of game theory andmodels of social interactions.

13 / 281


The goal of the econometric literature, like the goal of allscience, is to model phenomena at a deeper level, tounderstand the causes producing the effects so that onecan use empirical versions of the models to forecast theeffects of interventions never previously experienced, tocalculate a variety of policy counterfactuals, and to useeconomic theory to guide the choices of estimators and theinterpretation of the evidence.

These activities require development of a more elaboratetheory than is envisioned in the current literature on causalinference in epidemiology and statistics.

14 / 281


The recent literature sometimes contrasts structural andcausal models.

The contrast is not sharp because the term “structuralmodel” is often not precisely defined.

There are multiple meanings for this term, which areclarified in this presentation.

15 / 281


The essential contrast between causal models and expliciteconomic models as currently formulated is in the range ofquestions that they are designed to answer.

Causal models as formulated in statistics and in theeconometric treatment effect literature are typicallyblack-box devices designed to investigate the impact of“treatment” — which are often complex packages ofinterventions — on some observed set of outcomes in agiven environment.

16 / 281


Explicit economic models go into the black box to explorethe mechanism(s) producing the effects.

In the terminology of Holland (1986), the distinction isbetween understanding the “effects of causes” (the goal ofthe treatment effect literature) versus understanding the“causes of effects” (the goal of the literature buildingexplicit economic models).

By focusing on one narrow black-box question, thetreatment effect and natural experiment literatures canavoid many of the problems confronted in the econometricsliterature that builds explicit economic models.

This is its great virtue.

17 / 281


At the same time, it produces parameters that are morelimited in application.

The parameters defined by instruments or “naturalexperiments” are often hard to interpret within anyeconomic model.

Without further assumptions, these parameters do not lendthemselves to extrapolation out of sample or to accurateforecasts of impacts of policies besides the ones beingempirically investigated.

18 / 281


By not being explicit about the contents of the black-box(understanding the causes of effects), it ties its hands inusing information about basic behavioral parametersobtained from other studies as well as economic intuitionto supplement available information in the data in hand.

It lacks the ability to provide explanations for estimated“effects” grounded in economics or to conduct welfareeconomics.

When the components of treatments vary across studies,knowledge does not accumulate across treatment effectstudies, whereas it does accumulate across studiesestimating common behavioral or technologicalparameters.

19 / 281


Economic Policy Evaluation Questions and Criteria ofInterest

Three broad classes of policy evaluation questions areconsidered in this presentation.

Policy evaluation question one is:

P1 Evaluating the impact of historical interventions onoutcomes, including their impact in terms of welfare.

By historical, we mean interventions actually experiencedand documented.

20 / 281


It is useful to distinguish objective or public outcomes from“subjective” outcomes.

Objective outcomes are intrinsically ex post in nature.

Subjective outcomes can be ex ante or ex post.

Thus the outcome of a medical trial produces both a curerate and the pain and suffering of the patient.

Ex ante expected pain and suffering may be different fromex post pain and suffering.

Agents may also have ex ante evaluations of the objectiveoutcomes that may differ from their ex post evaluations.

21 / 281


P1 is the problem of internal validity.

It is the problem of identifying a given treatment parameteror a set of treatment parameters in a given environment.

The econometric approach emphasizes valuation of theobjective outcome of the trial (e.g., health status) as wellas subjective evaluation of outcomes (patient’s welfare),and the latter may be ex post or ex ante.

22 / 281


Most policy evaluation is designed with an eye toward thefuture and towards informing decisions about new policiesand application of old policies to new environments:

P2 Forecasting the impacts (constructing counterfactual states)of interventions implemented in one environment in otherenvironments, including their impacts in terms of welfare.

23 / 281


Included in these interventions are policies described bygeneric characteristics (e.g., tax or benefit rates) that areapplied to different groups of people or in different timeperiods from those studied in implementations of thepolicies on which data are available.

This is the problem of external validity.

24 / 281


Finally, the most ambitious problem is forecasting the effectof a new policy, never previously experienced:

P3 Forecasting the impacts of interventions (constructingcounterfactual states associated with interventions) neverhistorically experienced to various environments, includingtheir impacts in terms of welfare.

This problem requires that we use past history to forecastthe consequences of new policies.

It is a fundamental problem in knowledge.

25 / 281


Knight (1921, p. 313) succinctly states the problem:

The existence of a problem in knowledge depends on thefuture being different from the past, while the possibility ofa solution of the problem depends on the future being likethe past.

26 / 281


Notation

Economic Policy Evaluation Questions and Criteria ofInterest: Notation and Definition of Individual Level

Treatment Effects

To evaluate is to value and to compare values amongpossible outcomes.

These are two distinct tasks, which we distinguish in thispresentation.

We define outcomes corresponding to state (policy,treatment) s for an agent characterized by ω as Y (s, ω),ω ∈ Ω.

The agent can be a household, a firm, or a country.

27 / 281


Notation

One can think of Ω as a universe of agents eachcharacterized by an element ω.

The ω encompasses all features of agents that affect Youtcomes.

Y (·, ·) may be generated from a scientific or economictheory.

It may be vector valued.

The Y (s, ω) are outcomes realized after treatments arechosen.

In advance of treatment, agents will not know the Y (s, ω)but may make forecasts about them.

28 / 281


Notation

Let S be the set of possible treatments with elementsdenoted by s.

For simplicity of exposition, we assume that this set is thesame for all ω.

For each ω, we obtain a collection of possible outcomesgiven by

{Y (s, ω)

}s∈S.

The set S may be finite (e.g., there may be J states),countable, or may be defined on the continuum (e.g.,S = [0,1]), so that there are an uncountable number ofstates.

29 / 281


Notation

For example, if S = {0,1}, there are two treatments, one ofwhich may be a no-treatment state — e.g., Y (0, ω).

This is the outcome for an agent ω not getting a treatmentlike a drug, schooling, or access to a new technology, whileY (1, ω) is the outcome in treatment state 1 for agent ωgetting the drug, schooling, or access.

30 / 281


Notation

Begin material from previous slide presentations (1).

31 / 281


Notation

To focus ideas, analyze a prototypical policy evaluationproblem.

Country can adopt a policy (e.g., democracy).

Choice Indicator:

D = 1 if it adopts.D = 0 if not.

32 / 281


Notation



Choice Indicator:D = 1 if it adopts.

D = 0 if not.

32 / 281


Notation



Choice Indicator:D = 1 if it adopts.D = 0 if not.

32 / 281


Notation

Two outcomes (Y0(ω),Y1(ω)), ω ∈ Ω

Y0(ω) if country does not adoptY1(ω) if country adopts

Causal effect on observed outcomes

Marshallian ceteris paribus causal effect:

Y1(ω) − Y0(ω)

33 / 281


Notation

Two outcomes (Y0(ω),Y1(ω)), ω ∈ ΩY0(ω) if country does not adopt

Y1(ω) if country adopts



Y1(ω) − Y0(ω)

33 / 281


Notation

Two outcomes (Y0(ω),Y1(ω)), ω ∈ ΩY0(ω) if country does not adoptY1(ω) if country adopts



Y1(ω) − Y0(ω)

33 / 281


Notation

Figure 1: Extended Roy economy for policy adoptionDistribution of gains and treatment parameters

-5 -4 -3 -2 -1 0 1 2 3 4 50

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

Gain

TT=2.517

AM C=1.5* ATE=0.2

TUT=-0.595

Figure 1. Extended Roy Economy for Policy Adoption Distribution of Gains and Treatment Parameters

*C = Marginal Return

Suppose that a country has to choose whether to implement a policy. Under the policy, the GDP would be Y1.Without the policy,the GDP of the country would be Y0. For sake of simplicity, suppose that

Y1 = μ1 + U1

Y0 = μ0 + U0

where U0 and U1 are unobserved components of the aggregate output. The error terms (U0, U1) are dependent in a general way. Letδ denote the additional GDP due to the policy, i.e. δ = μ1−μ0. We assume δ > 0. Let C denote the cost of implementing the policy.We assume that the cost is a fixed parameter C. We relax this assumption below. The country’s decision can be represented as:

D =

½1 if Y1 − Y0 − C > 00 if Y1 − Y0 − C ≤ 0,

so the country decides to implement the policy (D = 1) if the net gains coming from it are positive. Therefore, we can define theprobabily of adopting the policy in terms of the propensity score

Pr(D = 1) = P (Y1 − Y0 − C > 0)

We assume that (U1, U0) ∼ N (0,Σ), Σ =∙1 −0.5−0.5 1

¸, μ0 = 0.67, δ = 0.2 and C = 1.5.

Gain=Y1 − Y0

34 / 281


Notation

Figure 1 Legend

Suppose that a country has to choose whether to implement a policy. Underthe policy, the GDP would be Y1. Without the policy, the GDP of the countrywould be Y0. For the sake of simplicity, suppose that

Y1 = µ1 + U1Y0 = µ0 + U0

where U0 and U1 are unobserved components of the aggregate output. Theerror terms (U0,U1) are dependent in a general way. Let δ denote theadditional GDP due to the policy, i.e. δ = µ1 − µ0. We assume δ > 0. Let Cdenote the cost of implementing the policy. We assume that the cost is afixed parameter C.

35 / 281


Notation

Figure 1 Legend

We relax this assumption below. The country’s decision can berepresented as:

D ={

1 if Y1 − Y0 − C > 00 if Y1 − Y0 − C ≤ 0,

so the country decides to implement the policy (D = 1) if thenet gains coming from it are positive. Therefore, we can definethe probability of adopting the policy in terms of the propensityscore

Pr(D = 1) = P(Y1 − Y0 − C > 0).

We assume that (U1,U0) ∼ N (0,Σ), Σ =[

1 −0.5−0.5 1

],

µ0 = 0.67, δ = 0.2, and C = 1.5.

36 / 281


Notation

More generally, define outcomes corresponding to state(policy, treatment) s for an “agent” characterized by ω asY (s, ω), ω ∈ Ω = [0,1], s ∈ S, set of possible treatments.

The agent can be any economic agent such as ahousehold, a firm, or a country.

The Y (s, ω) are ex post outcomes realized aftertreatments are chosen.

Consider uncertainty and related ex ante and ex postevaluations later on.

37 / 281


Notation

The individual treatment effect for agent ω.

Y (s, ω) − Y (s′, ω) , s , s′, s, s′ ∈ S , (2.1)

Individual level causal effect.

Comparisons can also be made in terms of utilitiesR (Y (s, ω)).

R (Y (s, ω) , ω) > R (Y (s′, ω) , ω) if s is preferred to s′.

The difference in subjective outcomes is[R (Y (s, ω) , ω) − R (Y (s′, ω) , ω)], and is another possibledefinition of a treatment effect. Holding ω fixed holds allfeatures of the person fixed except the treatment assigned,s.

38 / 281


Notation

The question,“What question is the analysis supposed to answer?”

is the big unanswered question in the recent policyevaluation literature.

The question is usually unanswered because it is unaskedin much of the modern treatment effect literature whichseeks to estimate “an effect” without telling you whicheffect or why it is interesting to know it.

The answer to the question shapes the way we go aboutpolicy evaluation analysis.

A central point in the Cowles research program (Marschak,1949, 1953).

39 / 281


Notation

To evaluate is to value and to compare values amongpossible outcomes.

These are two distinct tasks, which we distinguish.

We define outcomes corresponding to state (policy,treatment) s for an agent characterized by ω as Y (s, ω),ω ∈ Ω.

One can think of Ω as a universe of agents eachcharacterized by an element ω.

The ω encompasses all features of agents that affect Youtcomes.

Y (·, ·) may be generated from a scientific or economictheory.

40 / 281


Notation

The Y (s, ω) are outcomes realized after treatments arechosen.

In advance of treatment, agents may not know the Y (s, ω)but may make forecasts about them.

These forecasts may influence their decisions toparticipate in the program or may influence the agents whomake decisions about whether or not an individualparticipates in the program.

41 / 281


Notation

Let S be the set of possible treatments with elementsdenoted by s.

For simplicity of exposition, we assume that this set is thesame for all ω.

For each ω, we obtain a collection of possible outcomesgiven by

{Y (s, ω)

}s∈S.

The set S may be finite (e.g., there may be J states),countable, or may be defined on the continuum (e.g.,S = [0,1]) so there are an uncountable number of states.

42 / 281


Notation

For example, if S = {0,1}, there are two treatments, one ofwhich may be a no-treatment state (e.g., Y (0, ω) is theoutcome for an agent ω not getting a treatment like a drug,schooling or access to a new technology, while Y (1, ω) isthe outcome in treatment state 1 for agent ω getting thedrug, schooling or access).

43 / 281


Notation

Each “state” (treatment) may consist of a compound ofsubcomponent states.

In this case, one can define s itself as a vector (e.g.,s = (s1, s2, . . . , sK ) for K components) corresponding to thedifferent components that comprise treatment.

Thus a job training program typically consists of a packageof treatments.

We might be interested in the package of one (or more) ofits components.

44 / 281


Notation

The outcomes may be time subscripted as well, Yt (s, ω)corresponding to outcomes of treatment measured atdifferent times.

The index set for t may be the integers, corresponding todiscrete time, or an interval, corresponding to continuoustime.

The Yt (s, ω) are realized or ex post (after treatment)outcomes.

45 / 281


Notation

Under this assumption, the individual treatment effect foragent ω comparing objective outcomes of treatment s withobjective outcomes of treatment s′ is

Y (s, ω) − Y (s′, ω) , s , s′, (2.2)

where we pick two elements s, s′ ∈ S.

This is also called an individual level causal effect.

This may be a nondegenerate random variable or adegenerate random variable.

The causal effect is the Marshallian (1890) ceteris paribuschange of outcomes for an agent across states s and s′.

46 / 281


Notation

Economists are interested in the welfare of participants aswell as the objective outcomes (see Heckman and Smith,1998).

Although statisticians reason in terms of assignmentmechanisms, economists recognize that agent preferencesoften govern actual choices.

Comparisons across outcomes can be made in terms ofutilities (personal, R (Y (s, ω) , ω), or in terms of plannerpreferences, RG, or both types of comparisons might bemade for the same outcome and their agreement orconflict evaluated).

Write R (Y (s, ω) , ω) as R(s, ω), suppressing the explicitdependence of R on Y (s, ω).

47 / 281


Notation

One can ask if R(s, ω) > R(s′, ω) or not (is the agent betteroff as a result of treatment s compared to treatment s′?).

The difference in subjective outcomes is[R(s, ω) − R(s′, ω)], and is another possible definition of atreatment effect.

Since the units of R(s, ω) are arbitrary, one could insteadrecord for each s and ω an indicator if the outcome in s isgreater or less than the outcome in s′, i.e.R(s, ω) > R(s′, ω) or not.

This is also a type of treatment effect.

48 / 281


Notation

These definitions of treatment effects embody Marshall’s1890 notion of ceteris paribus comparisons but now inutility space.

A central feature of the econometric approach to programevaluation is the evaluation of subjective evaluations asperceived by decision makers and not just the objectiveevaluations focused on by statisticians.

49 / 281


Notation

The term “treatment” is used in multiple ways in thisliterature and this ambiguity is sometimes a source ofconfusion.

In its most common usage, a treatment assignmentmechanism is a rule τ : Ω→ S which assigns treatment toeach ω.

The consequences of the assignment are the outcomesY(s, ω), s ∈ S, ω ∈ Ω.

The collection of these possible assignment rules is Twhere τ ∈ T .

There are two aspects of a policy under this definition.

The policy selects who gets what.

50 / 281


Notation

More precisely, it selects individuals ω and specifies thetreatment s ∈ S received.

51 / 281


Notation

We offer a more nuanced definition of treatmentassignment that explicitly recognizes the element of choiceby agent ω in producing the treatment assignment rule.

Treatment can include participation in activities such asschooling, training, adoption of a particular technology, andthe like.

Participation in treatment is usually a choice made byagents.

Under a more comprehensive definition of treatment,agents are assigned incentives like taxes, subsidies,endowments and eligibility that affect their choices, but theagent chooses the treatment selected.

52 / 281


Notation

Agent preferences, program delivery systems, aggregateproduction technologies, market structures, and the likemight all affect the choice of treatment.

The treatment choice mechanism may involve multipleactors and multiple decisions that result in an assignmentof ω to s.

For example, s can be schooling while Y(s, ω) is earningsgiven schooling for agent ω.

A policy may be a set of payments that encourageschooling, as in the Progressa program in Mexico, and thetreatment in that case is choice of schooling with itsconsequences for earnings.

53 / 281


Notation

We specify assignment rules a ∈ A which map individualsω into constraints (benefits) b ∈ B under differentmechanisms.

In this notation, a constraint assignment mechanism a is amap

a : Ω→ Bdefined over the space of agents.

The constraints may include endowments, eligibility, taxes,subsidies and the like that affect agent choices oftreatment.

The map a defines the rule used to assign b ∈ B.

54 / 281


Notation

Formally, the probability system for the model withoutrandomization is (Ω, σ (Ω) ,F ) where Ω is the probabilityspace, σ (Ω) is the σ-algebra associated with Ω and F isthe measure on the space.

When we account for randomization we need to extend Ωto Ω′ = Ω ×Ψ, where Ψ is the new probability spaceinduced by the randomization, and we define a system(Ω′, σ (Ω′) ,F ′).

55 / 281


Notation

Some policies may have the same overall effect on theaggregate distribution of b, but may treat given individualsdifferently.

Under an anonymity postulate, some would judge suchpolicies as equivalent in terms of the constraints (benefits)offered, even though associated outcomes for individualsand aggregates may be different.

Another definition of equivalent policies is in terms of thedistribution of aggregate outcomes associated with thetreatments.

In this chapter, we characterize policies at the individuallevel recognizing that sets of A that are characterized bysome aggregate distribution over elements of b ∈ B maybe what others mean by a policy.

56 / 281


Notation

57 / 281


Notation

Given b ∈ B allocated by constraint assignmentmechanism a ∈ A, agents pick treatments.

We define treatment assignment mechanismτ : Ω ×A ×B → S as a map taking agent ω ∈ Ω facingconstraints b ∈ B assigned by mechanism a ∈ A into atreatment s ∈ S.

Note that including B in the domain of definition of τ isredundant since the map a : Ω→ B selects an elementb ∈ B.

We make b explicit to remind the reader that agents aremaking choices under constraints.

In settings with choice, τ is the choice rule used by agentswhere τ ∈ T , a set of possible choice rules.

58 / 281


Notation

It is conventional to assume a unique τ ∈ T is selected bythe relevant decision makers, although that is not requiredin our definition.

A policy regime p ∈ P is a pair (a, τ) ∈ A × T that mapsagents denoted by ω into elements of s.

In this notation, P = A×T .

59 / 281


Notation

Incorporating choice into the analysis of treatment effectsis an essential and distinctive ingredient of the econometricapproach to the evaluation of social programs.

The traditional treatment-control analysis in statisticsequates mechanisms a and τ.

An assignment in that literature is an assignment totreatment, not an assignment of incentives and eligibilityfor treatment with the agent making treatment choices.

In this notation, the traditional approach has only oneassignment mechanism and treats noncompliance with itas a problem rather than as a source of information onagent preferences, as in the econometric approach.

60 / 281


Notation

Thus, under full compliance, a : Ω→ S and a = τ, whereB = S.

61 / 281


Notation

Policy invariance is a key assumption for the study of policyevaluation.

It allows analysts to characterize outcomes withoutspecifying how those outcomes are obtained.

In our notation, policy invariance has two aspects.

The first aspect is that, for a given b ∈ B (incentiveschedule), the mechanism a ∈ A by which ω is assigned ab (e.g. random assignment, coercion at the point of a gun,etc.) and the incentive b ∈ B are assumed to be irrelevantfor the values of realized outcomes for each s that isselected.

62 / 281


Notation

Second, for a given s for agent ω, the mechanism τ bywhich s is assigned to the agent under assignmentmechanism a ∈ A is irrelevant for the values assumed byrealized outcomes.

Both assumptions define what we mean by policyinvariance.

63 / 281


Notation

Policy invariance allows us to describe outcomes byY(s, ω) and ignore features of the policy and choiceenvironment in defining outcomes.

If we have to account for the effects of incentives andassignment mechanisms on outcomes, we must work withY (s, ω,a,b , τ) instead of Y (s, ω).

The following policy invariance assumptions justifycollapsing these arguments of Y (·) down: Y (s, ω).

64 / 281


Notation

Policy invariance for objective outcomes:

PI-1

For any two constraint assignment mechanisms a,a′ ∈ A andincentives b ,b ′ ∈ B, with a(ω) = b and a′(ω) = b ′, and for allω ∈ Ω, Y(s, ω,a,b , τ) = Y(s, ω,a′,b ′, τ), for alls ∈ Sτ(a,b)(ω) ∩ Sτ(a′,b′)(ω) for assignment rule τ whereSτ(a,b)(ω) is the image set for τ (a,b). For simplicity weassume Sτ(a,b)(ω) = Sτ(a,b) for all ω ∈ Ω.

65 / 281


Notation

This assumption says that for the same treatment s andagent ω, different constraint assignment mechanisms aand a′ and associated constraint assignments b and b ′

produce the same outcome.

66 / 281


Notation

A second invariance assumption invoked in the literature isthat for a fixed a and b, the outcomes are the sameindependent of the treatment assignment mechanism:

PI-2

For each constraint assignment a ∈ A, b ∈ B and all ω ∈ Ω,Y(s, ω,a,b , τ) = Y(s, ω,a,b , τ′) for all τ and τ′ ∈ T withs ∈ Sτ(a,b) ∩ Sτ′(a,b), where Sτ(a,b) is the image set of τ for agiven pair (a,b).

Again, we exclude the possibility of ω-specific image setsSτ(a,b) and Sτ′(a,b).

67 / 281


Notation

These invariance postulates are best discussed in thecontext of specific economic models.

These conditions are closely related to the invarianceconditions of Hurwicz (1962).

68 / 281


Notation

If treatment effects based on subjective evaluations arealso considered, we need to broaden invarianceassumptions (PI-1) and (PI-2) to produce invariance inrewards for certain policies and assignment mechanisms.

It would be unreasonable to claim that utilities R (·) do notrespond to incentives.

Suppose, instead, that we examine subsets of constraintassignment mechanisms a ∈ A that give the sameincentives (elements b ∈ B) to agents, but are conferred bydifferent delivery systems, a.

69 / 281


Notation

For each ω ∈ Ω, define the set of mechanisms deliveringthe same incentive or constraint b as Ab (ω):

Ab (ω) ={a | a ∈ A,a(ω) = b} , ω ∈ Ω.

The set of delivery mechanisms that deliver b may varyamong the ω.

Let R (s, ω,a,b , τ) represent the reward to agent ω from atreatment s with incentive b allocated by mechanism a withan assignment to treatment mechanism τ.

70 / 281


Notation

PI-3

For any two constraint assignment mechanisms a,a′ ∈ A andincentives b ,b ′ ∈ B with a(ω) = b and a′(ω) = b ′, and for allω ∈ Ω, Y (s, ω,a,b , τ) = Y (s, ω,a′,b ′, τ) for alls ∈ Sτ(a,b)(ω) ∩ Sτ(a′,b′)(ω) for assignment rule τ, whereSτ(a,b)(ω) is the image set of τ (a,b) and for simplicity weassume that Sτ(a,b)(ω) = Sτ(a,b) for all ω ∈ Ω. In addition, forany mechanisms a,a′ ∈ Ab (ω), producing the same b ∈ Bunder the same conditions postulated in the precedingsentence, and for all ω, R (s, ω,a,b , τ) = R (s, ω,a′,b , τ) .

71 / 281


Notation

This assumption says, for example, that utilities are notaffected by randomization or the mechanism of assignmentof constraints.

Corresponding to (PI-2) we have a policy invarianceassumption for the utilities with respect to the mechanismof assignment:

72 / 281


Notation

PI-4

For each pair (a,b) and all ω ∈ Ω,

Y (s, ω,a,b , τ) = Y (s, ω,a,b , τ′)R (s, ω,a,b , τ) = R (s, ω,a,b , τ′)

for all τ, τ′ ∈ T and s ∈ Sτ(a,b) ∩ Sτ′(a,b).

73 / 281


Notation

This assumption rules out general equilibrium effects,social externalities in consumption, etc. in both subjectiveand objective outcomes.

74 / 281


Notation

End material from previous slide presentations (1).

75 / 281


Notation

Notation and Definition of Individual Level TreatmentEffects: More General Criteria

One might compare outcomes in different sets that areordered.

Thus if Y (s, ω) is scalar income and we compareoutcomes for s ∈ SA with outcomes for s′ ∈ SB , whereSA ∩ SB = ∅, then one might compare YsA to YsB , where

sA = argmaxs∈SA

(Y (s, ω)) ,

sB = argmaxs∈SB

(Y (s, ω)) ,

and we suppress the dependence of sA and sB on ω.76 / 281


Notation

This compares the best in one choice set with the best inthe other.

Another contrast compares the best choice with the nextbest choice.

To do so, define s′ = argmaxs∈S (Y (s, ω)) andSB = S r {s′}, and define the treatment effect as Ys′ − YsB .

77 / 281


Notation

Social welfare theory constructs aggregates over Ω ornonempty, nonsingleton subsets of Ω (see Sen, 1999).

Let sp (ω) denote the s ∈ Sp that ω receives under policy p.

This is a shorthand notation for the element in Sτdetermined by the map p = (a, τ) assigned to agent ωunder policy p.

78 / 281


Notation

A comparison of two policy outcomes{sp(ω)

}ω∈Ω and{

sp′(ω)}ω∈Ω, where p , p

′ for some ω ∈ Ω, can beexpressed as

RG({

Y(sp (ω) , ω

)}ω∈Ω

)− RG

({Y(sp′(ω), ω)

}ω∈Ω

)using the social welfare function defined over outcomesRG (

{Y (s, ω) , ω

}ω∈Ω).

79 / 281


Notation

The cost-benefit comparison of two policies p and p′ is

CBp,p′ =∫

Ω

W(Y

(sp(ω), ω

))dµ(ω)−

∫Ω

W(Y

(sp′(ω), ω

))dµ(ω),

where p,p′ are two different policies and p′ maycorrespond to a benchmark of no policy and µ(ω) is thedistribution of ω.

The Benthamite criterion replaces W (Y (s (ω) , ω)) withR(Y(s(ω), ω)) in the preceding expressions and integratesutilities across agents:

Bp,p′ =∫

Ω

R(Y(sp(ω), ω))dµ(ω) −∫

Ω

R(Y(sp′(ω), ω))dµ(ω).

80 / 281


Problem

Economic Policy Evaluation Questions and Criteria ofInterest: The Evaluation Problem

Assume a well-defined set of individuals ω ∈ Ω and auniverse of counterfactuals or hypotheticals for each agentY (s, ω), s ∈ S.

Different policies p ∈ P give different incentives byassignment mechanism a to agents who are allocated totreatment by a rule τ ∈ T .

81 / 281


Problem


82 / 281


Problem

How To Construct Counterfactuals?

Central problem in the evaluation literature is the absenceof information on outcomes for person ω other than theoutcome that is observed.

Even a perfectly implemented social experiment does notsolve this problem.

Randomization with full compliance identifies only onecomponent of

{Y(s, ω)

}s∈S for any person.

In addition, some of the s ∈ S may never be observed.

83 / 281


Problem

For each policy regime, at any point in time we observeperson ω in some state but not in any of the other states.

Do not observe Y (s′, ω) for person ω if we observeY (s, ω), s , s′.

Let D (s, ω) = 1 if we observe person ω in state s underpolicy regime p.

Observed objective outcome

Y(ω) =∑s∈S

D (s, ω) Y (s, ω) . (2.3)

84 / 281


Problem

The evaluation problem in this model is that we onlyobserve each individual in one of S̄ possible states.

We do not know the outcome of the individual in otherstates and hence cannot directly form individual leveltreatment effects.

The selection problem arises because we only observecertain persons in any state.

We observe Y (s, ω) only for persons for whomD (s, ω) = 1.

In general, the outcomes of persons found in S = s are notrepresentative of what the outcomes of people would be ifthey were randomly assigned to s.

85 / 281


Problem

The Roy model (1951): Two possible treatment outcomes(S = {0,1}) and a scalar outcome measure and a particularassignment mechanism D (1, ω) = 1 [Y (1, ω) > Y (0, ω)](reveals R(1, ω) − R(0, ω) ≥ 0).

The economist’s use of choice data distinguishes theeconometric approach from the statistical approach.

86 / 281


Problem


Two main avenues of escape from this problem.

The first avenue, featured in explicitly formulatedeconometric models and often called “structuraleconometric analysis ”, derives from the Cowles tradition.

Models Y(s, ω) explicitly in terms of its determinants asspecified by theory.

This entails describing the random variables characterizingω and carefully distinguishing what agents know and whatthe analyst knows.

87 / 281


Problem


This approach also models D(s, ω) and the dependencebetween Y(s, ω) and D(s, ω) produced from variablescommon to Y (s, ω) and D (s, ω).

Specifies a full model and attempts to addressproblems (P1)–(P3).

88 / 281


Problem


A second avenue, pursued in the recent treatment effectliterature, redirects attention away from estimating thedeterminants of Y(s, ω) toward estimating some populationversion of individual “causal effects,” without modeling whatfactors give rise to the outcome or the relationship betweenthe outcomes and the mechanism selecting outcomes.

Agent valuations of outcomes are typically ignored.

The treatment effect literature focuses largely on policyproblem (P-1) for the subset of outcomes that is observed.

Seeks to answer a narrower problem.

89 / 281


Population Level Treatment Parameters

For program (state, treatment) j compared to program(state, treatment) k ,

ATE(j, k ) = E (Y(j, ω) − Y(k , ω)) .

TT(j, k ) =E (Y(j, ω) − Y(k , ω) | D(j, ω) = 1) . (2.4)

These are the traditional parameters for average returns.

But for economic analysis, marginal returns are moreimportant.

90 / 281



The distinction between the marginal and average return isa central concept in economics.

The Effect Of Treatment for People at the Margin ofIndifference (EOTM) between j and k , given that theseare the best two choices available is, with respect topersonal preferences, and with respect to choice-specificcosts C (j, ω).

91 / 281



EOTMR(j, k ) = (2.5)

E

Y(j, ω)−Y(k , ω)∣∣∣∣∣∣∣∣

R (Y (j, ω) ,C (j, ω) , ω) = R (Y (k , ω) ,C (k , ω) , ω) ;R (Y (j, ω) ,C (j, ω) , ω)R (Y (k , ω) ,C (k , ω) , ω)

}≥ R (Y (`, ω) ,C (`, ω) , ω)

,` , j, k .

92 / 281



A generalization of this parameter called the MarginalTreatment Effect, introduced into the evaluation literatureby Björklund and Moffitt (1987).

Return to people at the margin of choice.

Will discuss methods for identifying this return.

93 / 281



Policy relevant treatment effect

Effect on aggregate outcomes of one policy regime p ∈ Pcompared to the effect of another policy regime p′ ∈ P:

PRTE: E(Y(sp(ω), ω) − Y(sp′(ω), ω)),where p,p′ ∈ P.

sp(ω) is treatment allocated under policy p.

Corresponding to this objective outcome is the subjectivecounterpart:

Subjective PRTE: E(R(sp(ω), ω)) − E(R(sp′(ω), ω)),where p,p′ ∈ P.

94 / 281


Criteria of interest besides the mean

Modern political economy seeks to know the proportion ofpeople who benefit from policy regime p compared with p′.Voting Criterion:

Pr(Y

(sp(ω), ω

)> Y

(sp′(ω), ω

)).

For particular treatments within a policy regime p, it is alsoof interest to determine the proportion who benefit from jcompared to k as

Pr (Y (j, ω) > Y (k , ω)) .

Option values also interesting: option of having access to aprogram.

Uncertainty and regret.

95 / 281


A generalized Roy model under perfect certainty

Given an economic model, we can trivially derive thetreatment effects.

S̄ states associated with different levels of schooling, orsome other outcome such as residence in a region, orchoice of technology.

Associated with each choice s is a valuation of theoutcome of the choice R (s) where R is the valuationfunction and s is the state. (We drop the ω argument hereto simplify notation.)

Z : observed individual variables that affect choices.

Each state may be characterized by a bundle of attributes,characteristics or qualities Q (s) that fully characterize thestate. If Q (s) fully describes the state, R (s) = R (Q (s)) .

R (s) = µR (s,Z) + υ (s,Z , ν)

96 / 281



Associated with each choice is outcome Y (s) which maybe vector valued.

The set of possible treatments S is{1, . . . , S̄

}, the set of

state labels.

The assignment mechanism is specified by utilitymaximization:

D (j) = 1 if argmaxs∈S

{R (s)

}= j, (2.6)

where in the event of ties, choices are made by a flip of acoin.

People self-select into treatment.

97 / 281



End material from previous slide presentations (2).

98 / 281


Accounting for Private and Social Uncertainty

Economic Policy Evaluation Questions and Criteria ofInterest: Accounting for Private and Social Uncertainty

Systematically accounting for uncertainty introducesadditional considerations that are central to economicanalysis but that are ignored in the treatment effectliterature as currently formulated.

Even if, ex post, agents know their outcome in abenchmark state, they may not know it ex ante, and theymay always be uncertain about what they would haveexperienced in an alternative state.

99 / 281



This creates a further distinction: that between ex post andex ante evaluations of both subjective and objectiveoutcomes.

An alternative criterion is required if agents act in their ownself-interest.

Because agents typically do not possess perfectinformation, the simple voting criterion that assumesperfect foresight over policy outcomes that is discussedabove may not accurately predict choices.

100 / 281



Let Iω denote the information set available to agent ω.

He or she evaluates policy j against k using thatinformation.

Under an expected utility criterion, agent ω prefers policy jover policy k if

E(R(Y(j, ω), ω) | Iω) > E(R(Y(k , ω), ω) | Iω).

The proportion of people who prefer j is

PB (j | j, k) =∫

1 [E[R (Y(j, ω), ω) | Iω] ≥ E[R (Y(k , ω), ω) | Iω] dµ (Iω) ,(2.7)

where µ(ω) is the distribution of ω in the population whosepreferences over outcomes are being studied.

101 / 281



The voting criterion presented previously is the specialcase where the information set Iω contains(Y (j, ω) ,Y (k , ω)), so there is no uncertainty about Y(j)and Y(k ).

Accounting for uncertainty in the analysis makes itessential to distinguish between ex ante and ex postevaluations.

Ex post, part of the uncertainty about policy outcomes isresolved although agents do not, in general, have fullinformation about what their potential outcomes wouldhave been in policy regimes they have not experiencedand may have only incomplete information about the policythey have experienced.

102 / 281



Ex post assessments of a program through surveysadministered to agents who have completed it maydisagree with ex ante assessments of the program.

An agent who has completed program j may know Y(j, ω)but can only guess at the alternative outcome Y(k , ω)which they have not experienced.

103 / 281



In this case, ex post “satisfaction” with j relative to k foragent ω who only participates in k is synonymous with theinequality

R(Y (j, ω) , ω) > E(R(Y (k , ω) , ω) | Iω), (2.8)

where the information is post-treatment.

Survey questionnaires about “client” satisfaction with aprogram may capture subjective elements of programexperience not captured by “objective” measures ofoutcomes that usually exclude psychic costs and benefits.

104 / 281


The Data Needed to Construct the Criteria

Economic Policy Evaluation Questions and Criteria ofInterest: The Data Needed to Construct the Criteria

Four ingredients are required to implement the criteriadiscussed in this section:

(1) private preferences, including preferences over outcomesby the decision maker;

(2) social preferences, as exemplified by the social welfarefunction;

(3) distributions of outcomes in alternative states, and for somecriteria, such as the voting criterion, joint distributions ofoutcomes across policy states; and

(4) ex ante and ex post information about outcomes.

105 / 281


A Generalized Roy Model Under Perfect Certainty

Suppose that there are S̄ states associated with differentlevels of schooling, or some other outcome such asresidence in a region, or choice of technology.

Associated with each choice s is a valuation of theoutcome of the choice R (s) where R is the valuationfunction and s is the state.

106 / 281



107 / 281


The Roy model (1951) and its extensions (Gronau, 1974;Heckman, 1974; Willis and Rosen, 1979; Heckman, 1990;Carneiro, Hansen, and Heckman, 2003) are at the core ofmicroeconometrics.

Y1 = Xβ1 + U1 (3.1a)Y0 = Xβ0 + U0, (3.1b)

and associated costs (prices) as a function of W

C = WβC + UC . (3.1c)

Can embed into general equilibrium models (Heckman,Lochner and Taber, 1998; Wolpin and Lee, 2006)

108 / 281


The valuation of “1” relative to “0” is R = Y1 − Y0 − C.Substituting from (3.1a)–(3.1c) into the expression for R:

R = X(β1 − β0) −WβC + U1 − U0 − UC ,

and sectoral choice is indicated by D where D = 1 if the agentselects 1; = 0 otherwise:

D = 1 [R > 0] .

109 / 281


υ = (U1 − U0 − UC), Z = (X ,W).

γ = (β1 − β0,−βC).

Thus R = Zγ+ υ.

Generalized Roy model:

Z ⊥⊥ (U0,U1,UC) (independence),(U0,U1,UC) ∼ N(0,Σ) (normality).

110 / 281


υ = (U1 − U0 − UC), Z = (X ,W).

γ = (β1 − β0,−βC).

Thus R = Zγ+ υ.


Z ⊥⊥ (U0,U1,UC) (independence),

(U0,U1,UC) ∼ N(0,Σ) (normality).

110 / 281


υ = (U1 − U0 − UC), Z = (X ,W).

γ = (β1 − β0,−βC).

Thus R = Zγ+ υ.


Z ⊥⊥ (U0,U1,UC) (independence),(U0,U1,UC) ∼ N(0,Σ) (normality).

110 / 281


For the Generalized Roy Model, the probability of selectingtreatment 1 or “propensity score” is

Pr (R > 0 | Z = z) = Pr (υ > −zγ)

= Pr(υσυ>−zγσυ

)= Φ

(zγσυ

),

where Φ is the cumulative distribution function of the standardnormal distribution.

111 / 281


The Average Treatment Effect given X = x is

ATE(x) = E (Y1 − Y0 | X = x)= x (β1 − β0) .

Treatment on the treated is

TT (x , z) = E (Y1 − Y0 | Z = z,D = 1)= x (β1 − β0) + E (U1 − U0 | υ > −Zγ,Z = z)= x (β1 − β0) + E (U1 − U0 | υ > −zγ) .

112 / 281


The local average treatment effect (LATE) of Imbens andAngrist (1994) is the average gain to program participationfor those induced to receive treatment through a change inZ [= (X ,W)] by a component of W not in X .

The change affects choices but not potential outcomesY(s).

Let D (z) be the random variable D when we fix W = wand let D (z′) be the random variable when we fix W = w′.

This definition is instrument dependent.

Econometric Evaluation of Social Programs Part I: Causal Models...

Documents

Transcript of Econometric Evaluation of Social Programs Part I: Causal Models...