Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish...

34
Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December 6, 2013 Dr. Laura Stapleton Mr. Dan McNeish, Ms. Xiulin Mao Measurement, Statistics and Evaluation Program College of Education University of Maryland, College Park

Transcript of Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish...

Page 1: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

1

Causal Inference in Education Policy Research

MLDS Research SeriesDecember 6, 2013

Dr. Laura Stapleton

Mr. Dan McNeish, Ms. Xiulin MaoMeasurement, Statistics and Evaluation Program

College of Education

University of Maryland, College Park

Page 2: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

2

Outline

Introduction and ContextCausal Inference, the Counterfactual, and the “Fundamental Problem of Causal Inference”Three Approaches to Evaluate Cause in Observational Studieso Condition on Covariateso Instrumental Variable Estimationo Regression Discontinuity Analysis

RecommendationsReferences for More Information

Page 3: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

3

Introduction

Data, too often, are turned into “information” that can be misleading

“…many college students who take developmental education classes… fail to graduate. Only 28% of two-year college students who took at least one developmental course earned a degree or certificate within 8.5 years, compared to 43% of non-remedial students…” (USA Today, July 25, 2013)

“Financial aid can make or break a college education…any student headed to study at a private school with financial gaps over $3,000 would be at significant risk of not graduating.” (www.bottomline.org)

Page 4: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

4

Introduction

“employment and finances [are seen] as the main reasons for departure…nearly 40 percent of students who worked full time while enrolled dropped out within three years, compared to 19 percent of students who worked part time and 13 percent who did not work.” (Demos, 2011)

“The evidence is clear: Undergraduates enrolled full-time —specifically, 30 or more credits completed in their first year —are more likely to graduate on time than students who complete fewer credits per year.” (Complete College America, 2013)

Page 5: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

5

Introduction

These examples focus on bivariate relations:

Taking a remedial course and earning a degree/certificate

Having unmet financial need and graduating

Working and dropping out of college

Enrollment in 30 credits and time-to-degree

“Correlation does not imply causation”…what can we do instead?

First, we need an understanding of the concept of causal

inference and the counterfactual

Page 6: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

6

Causal Inference

In order to attribute “cause” -- or measure the Treatment Effect of a given “treatment” -- we need to see the person under both conditions

“Fundamental Problem of Causal Inference” – we never obtain the counterfactual in social scienceIf we can assume that treatment assignment is unrelated to potential outcomes, then the Average Treatment Effect gives us an estimate of the Treatment Effect

no college college Treatment effect = $ - $

of college on earnings??

??

Page 7: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

7

Causal Inference

We can obtain an unbiased estimate of the Average Treatment Effect (ATE) with a well-designed random assignment studyY(C) Y(T)

Pote

ntia

l Out

com

es

high

low

Observed Control Observed Treatment

Y Y

Page 8: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

8

Causal Inference

However, with observational studies, Treatment Assignment tends to be related to potential outcomes (it is endogenous) Y(C) Y(T)

Pote

ntia

l Out

com

es

high

low

Observed Control Observed Treatment

Y

Y

Page 9: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

9

Aug 27, 2013, there was a great piece by Dylan Matthews in the Washington Post: The Tuition is Too Damn High: Part II Why College is Still Worth It

Causal Inference

College IncomeWe see a correlation…

College Income

ResourcesA different reason for the correlation…

College IncomeWe may be tempted to conclude…

Page 10: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

10

Causal Inference in Observational Studies

College Income

We would like to estimate the treatment effect (TE)… TE

But with observational studies, there is also a relation (r) between the treatment assignment and the potential outcome…

If we naively estimate the effect of T on Y, then our estimate is biased… it will be TE + r

r

College IncomeTE + r

Page 11: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

11

Approaches to Evaluate the Causal Effect

Three approaches we will introduce today:conditioning on covariatesinstrumental variable estimatorregression discontinuity estimator

Each of these approaches differ on:“who” is included in the analysisthe required type of auxiliary information / datastatistical (and conceptual) assumptions

Page 12: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

12

Conditioning on Covariates

Page 13: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

13

Conditioning on Covariates

We can address selection bias in our observed data by using other data that we know about the participantsUsing the other data (which we call “covariates”), we can:

adjust for differences in a multiple regression frameworkstratify people by levels of the covariates and examine the effect of treatment within stratamatch on the covariates to create groups that only differ on the treatment variable

Page 14: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

14

Controlling for Covariates Using Multiple Regression

College IncomeTE

HS GPA

Parent SES

Assumptions:1) Covariates have eliminated the

relation between College and Income residual (ignorability)

2) Correct functional form of relation of covariates and outcome

3) Relation of College and Income same at all levels of covariates

4) Homoscedastic spread of observations over the outcome/covariate spaceProvides us with the “ATE” –

the average treatment effect if all people were exposed to the treatment

r

Page 15: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

15

Controlling for Covariates Using Multiple Regression

What if all high GPA and high SES students went to college and low GPA and low SES students did not go to college?We would be comparing completely different groups – no overlap (referred to as “common support”) – but we would not “see” it when we run this regression analysisAdditionally, we never know whether we have met assumptions about functional form (we cannot separate the evaluation of it from possible treatment effects given that both are included in the model simultaneously)

Page 16: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

16

Controlling for Covariates Using Stratification

High GPA, High SES

High GPA, Medium SES

High GPA, Low SES

Low GPA, Low SES

Page 17: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

17

Controlling for Covariates Using Stratification

High GPA, High SES

High GPA, Medium SES

High GPA, Low SES

Low GPA, Low SES

College IncomeTE

College IncomeTE

College IncomeTE

College IncomeTE

The treatment effect estimates are combined across the strata, weighted by number in treatment in each stratum

Provides us with the ATT –the average treatment effect on the treated

This approach removes the requirement that we know the functional form of the relation between covariates and outcome

Page 18: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

18

Propensity Scores

What if we have more than two variables that theoretically determine membership into treatment groups?Estimate a propensity to be in the treatment group:

Sibling College

CollegeHS GPA

Parent SES

Motiv-ation

Using the observed membership, we create a prediction equation to determine a person’s probability of being in treatment given covariates

Each person then receives a propensity score (ranging from 0 to 1)

This propensity score estimation process is done without knowledge of the value on the outcome

Page 19: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

19

Controlling for Covariates Using Propensity Score Stratification

Highest Propensity

Next Highest Propensity

Middle Propensity

Lowest Propensity

College IncomeTE

College IncomeTE

College TE

The treatment effect estimates are averaged across the strata

Provides us with the ATT –the average treatment effect on the treated

College IncomeTE

Income

Page 20: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

20

Controlling for Covariates Using Propensity Score Matching

Low

Pro

pens

ity

Hig

h Pr

open

sity For each treatment person, a match

is found in the control group based on propensity score

Control Treatment

Page 21: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

21

The mean on the outcome for the treated group is then compared to the mean of the matched control group

Unmatched control group members are not included in any analysis

Controlling for Covariates Using Propensity Score Matching

Provides us with the ATT – the average treatment effect on the treatedLo

w P

rope

nsity

H

igh

Prop

ensit

y For each treatment person, a match is found in the control group based on propensity score

Control Treatment

Page 22: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

22

Controlling for Covariates Using Propensity Score Matching

Low

Pro

pens

ity

Hig

h Pr

open

sity What if there is minimal

overlap?Control Treatment

Any comparison between the groups may be problematic

May need to compare only those persons with reasonable matches and confine generalization to just that population

Page 23: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

23

Instrumental Variable Estimator

Page 24: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

24

Instrumental VariableEstimator

We may not have access to the many covariates needed to “equate” our treatment and control groupsIt may be possible to locate and “carve out” an “exogenous” part of the variability in treatment assignmentWe then use this “exogenous” part only to estimate the causal effectThe variable that is used to carve out exogeneity is referred to as an “instrumental” variable

Page 25: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

25

College Income

We would like to find the unbiased treatment effect estimate…

TE

We can locate an instrumental variable to estimate…

proximityβ

Instrumental Variable Estimator

College

Assumption:There is no direct relation between the instrumental variable and the outcome

True instrumental variables are difficult to find

This analysis provides the local average treatment effect (LATE), localized just to that part of the treatment variable that is “sensitive to” differences in the instrumental variable (that part of college that is a function of differences in geographic location)

Page 26: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

26

Regression Discontinuity Estimator

Page 27: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

27

Regression Discontinuity Estimator

Sometimes in education, there is a “natural experiment” with what is referred to as a “discontinuity design”

Underlying continuum along which people are arrayed (called the “forcing” variable)An exogenously determined cut-point on this continuum divides participants into levels of a treatment

Goal is to limit the analysis to just those observations “at the cut-off” of this forcing variable

Compare the outcome for those just below the cut-off to those just above the cut-off

Page 28: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

28

Regression Discontinuity Estimator

Cum

ulat

ive

GPA

Placement Exam

The cut-point is determined and those just above and just below the cut-point are compared

The size of the comparison “bandwidth” is crucial

As the “bandwidth” is increased, the functional form between the forcing variable and outcome should be correctly specified

The treatment effect is the difference in predicted values at the location of the cut-point

This analysis provides us with an estimate of the effect of the POLICY based on the cut-point and not of actually being exposed to the treatment

Page 29: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

29

Recommendations and Issues for MLDS Center to Consider

Use an appropriate analytic approach for the research or policy question

try not to take a quick or easy road inform our various audiences over timebe leaders in analytic rigor

Build models based on appropriate theory need to partner with experts on each issueexamples: early education/childcare, financial aid, labor migration, etc.

Page 30: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

30

Recommendations and Issues for MLDS Center to Consider

Data. These analytic techniques require that we need to capture those data elements that would allow us to:

make “matches” identify instrumental variablesuse a forcing variable in regression discontinuitythese variables may include those not typically considered of interest in analyses

Page 31: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

31

Recommendations and Issues for MLDS Center to Consider

Be responsive and efficient

Policy decisions are often made quickly

We need to be similarly quick (yet rigorous…a hard balancing act)

Learn how to tell a story

These analyses can be very intricate -- our policy briefs need to mask that intricacy

“Any fool can make something complicated. It takes a genius to make it simple.”

(Woody Guthrie)

Page 32: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

32

References for More InformationMethodological resources:

Murnane, R. J., & Willett, J. B. (2010). Methods matter: Improving causal inference in educational and social science research. New York, NY: Oxford University Press.Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25, 1-21.

Empirical examples of methods:Jacob, B. A., & Lefgren, L. (2004). Remedial education and student achievement: A regression-discontinuity analysis. Review of Economics and Statistics, 86, 226-244.

Long, B. T., & Kurlaender, M. (2009). Do community colleges provide a viable pathway to a baccalaureate degree? Educational Evaluation and Policy Analysis, 31, 30-53.Titus, M. A. (2007). Detecting selection bias, using propensity score matching, and estimating treatment effects: An application to the private returns to a Master’s degree. Research in Higher Education, 48, 487-521.

Page 33: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

33

Page 34: Causal Inference in Education Policy Research · 2020-02-28 · Causal Inference Stapleton, McNeish & Mao Ñ 1 Causal Inference in Education Policy Research MLDS Research Series December

Causal Inference Stapleton, McNeish & Mao

Ñ

34

Questions?

([email protected])


Introduction to causal inference

Introduction to causal inference

Causal Inference

Causal Inference

Causal inference in data science

Causal inference in data science

Module . s: Causal Inference - edX€¦ · CAUSAL INFERENCE AND IMPACT EVALUATION Causal inference and impact evaluation is all about attributing a change in an outcome of interest

Module . s: Causal Inference - edX€¦ · CAUSAL INFERENCE AND IMPACT EVALUATION Causal inference and impact evaluation is all about attributing a change in an outcome of interest

Simplifying Causal Inference - Harvard University

Simplifying Causal Inference - Harvard University

Causal Inference in Observational Studies

Causal Inference in Observational Studies

Machine Learning and Causal Inference

Machine Learning and Causal Inference

Causal Inference Overview - Harvard University · Fundamental Problem of Causal Inference (ctd). I The fundamental problem of causal inference is that at most only one of the two

Causal Inference Overview - Harvard University · Fundamental Problem of Causal Inference (ctd). I The fundamental problem of causal inference is that at most only one of the two

Causal Inference MSM

Causal Inference MSM

Causal Inference Woth Observational Data

Causal Inference Woth Observational Data

Applying Causal Inference Techniques to Strengthen Dual ... · Modern Causal Inference Techniques Modern causal inference techniques can be used to account for the absence of random

Applying Causal Inference Techniques to Strengthen Dual ... · Modern Causal Inference Techniques Modern causal inference techniques can be used to account for the absence of random

Causal inference for binary regression

Causal inference for binary regression

Elements of Causal Inference - OAPEN

Elements of Causal Inference - OAPEN

Causal Inference and Ambiguous Manipulations

Causal Inference and Ambiguous Manipulations

Lecture 1: Causal Inference in Social Science€¦ · Outlines 1 Course Overview 2 Causal Inference in Social Science Causal Inference: The Core of Empirical Studies in Economics

Lecture 1: Causal Inference in Social Science€¦ · Outlines 1 Course Overview 2 Causal Inference in Social Science Causal Inference: The Core of Empirical Studies in Economics

Bayesian Causal Inference - uni-muenchen.de...from causal inference have been attracting much interest recently. [HHH18] propose that causal [HHH18] propose that causal inference stands

Bayesian Causal Inference - uni-muenchen.de...from causal inference have been attracting much interest recently. [HHH18] propose that causal [HHH18] propose that causal inference stands

Causal Inference and Data-Fusion in Econometrics · 2019-03-29 · Causal Inference in Econometrics I Despite a strong interest in causal inference in general, graphical models of

Causal Inference and Data-Fusion in Econometrics · 2019-03-29 · Causal Inference in Econometrics I Despite a strong interest in causal inference in general, graphical models of

Introducing Counterfactual Causal Inference

Introducing Counterfactual Causal Inference

CompSci 295, Causal Inference

CompSci 295, Causal Inference

Causal inference - pubdocs.worldbank.orgpubdocs.worldbank.org/.../13-b-Technical-Causal-Inference-Leite-ENG… · Fundamental problem of causal inference o For a given unit u, we

Causal inference - pubdocs.worldbank.orgpubdocs.worldbank.org/.../13-b-Technical-Causal-Inference-Leite-ENG… · Fundamental problem of causal inference o For a given unit u, we