Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and...
-
Upload
magdalene-robertson -
Category
Documents
-
view
221 -
download
2
Transcript of Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and...
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
What would it take to Change your Inference? Quantifying the Discourse about Causal Inferences in the Social Sciences
Kenneth A. Frank Help from Yun-jia Lo and Mike Seltzer
AERA workshop April 19, 2015 (AERA on-line video – cost is $95)Motivation Statistical inferences are often challenged because of uncontrolled bias. There may be bias due to uncontrolled confounding variables non-random selection into a sample. We will answer the question about what it would take to change an inference by formalizing the sources of bias and quantifying the discourse about causal inferences in terms of those sources. For example, we will transform challenges such as “But the inference of a treatment effect might not be valid because of pre-existing differences between the treatment groups” to questions such as “How much bias must there have been due to uncontrolled pre-existing differences to make the inference invalid?”
ApproachesIn part I we will use Rubin’s causal model to interpret how much bias there must be to invalidate an inference in terms of replacing observed cases with counterfactual cases or cases from an unsampled population. In part II, we will quantify the robustness of causal inferences in terms of correlations associated with unobserved variables or in unsampled populations. Calculations for bivariate and multivariate analysis will be presented in the spreadsheet for calculating indices [KonFound-it!] with some links to SPSS, SAS, and Stata. FormatThe format will be a mixture of presentation, individual exploration, and group work. Participants may include graduate students and professors, although all must be comfortable with basic regression and multiple regression. Participants should bring their own laptop, or be willing to work with another student who has a laptop. Participants may choose to bring to the course an example of an inference from a published study or their own work, as well as data analyses they are currently conducting.
Materials to download: spreadsheet for calculating indices [KonFound-it!] powerpoint with examples and calculations
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
overviewReplacement Cases Framework (40 minutes to reflection)
Thresholds for inference and % bias to invalidate and inference
The counterfactual paradigm
Application to concerns about non-random assignment to treatments
Application to concerns about non-random sample
Reflection (10 minutes)
Examples of replacement framework
Internal validity example: Effect of kindergarten retention on achievement
(40 minutes to break)
External validity example: effect of Open Court curriculum on achievement
Review and Reflection
Extensions
Extensions of the framework
Exercise and break (20 minutes: Yun-jia Lo, Mike Seltzer support)
Correlational Framework (25 minutes to exercise)
How regression works
Impact of a Confounding variable
Internal validity: Impact necessary to invalidate an inference
Example: Effect of kindergarten retention on achievement
Exercise (25 minutes: Mike Seltzer supports on-line)
External validity (30 minutes)
combining estimates from different populations
example: effect of Open Court curriculum on achievement
Conclusion (10 minutes)
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
https://www.msu.edu/~kenfrank/research.htm#causal
Materials for course
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Quick Survey
Can you make a causal inference from an observational study?
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Answer: Quantifying the Discourse
Can you make a causal inference from an observational study?
Of course you can. You just might be wrong. It’s causal inference, not determinism.
But what would it take for the inference to be wrong?
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
I: Replacement of Cases Framework
How much bias must there be to invalidate an inference?
Concerns about Internal Validity• What percentage of cases would you have
to replace with counterfactual cases (with zero effect) to invalidate the inference?
Concerns about External Validity• What percentage of cases would you have
to replace with cases from an unsampled population (with zero effect) to invalidate the inference?
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
What Would It Take to Change an Inference? Using Rubin’s Causal Model to
Interpret the Robustness of Causal Inferences
Abstract
We contribute to debate about causal inferences in educational research in two ways. First, we quantify how much bias there must be in an estimate to invalidate an inference. Second, we utilize Rubin’s causal model (RCM) to interpret the bias necessary to invalidate an inference in terms of sample replacement. We apply our analysis to an inference of a positive effect of Open Court Curriculum on reading achievement from a randomized experiment, and an inference of a negative effect of kindergarten retention on reading achievement from an observational study. We consider details of our framework, and then discuss how our approach informs judgment of inference relative to study design. We conclude with implications for scientific discourse.
Keywords: causal inference; Rubin’s causal model; sensitivity analysis; observational studies Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. 2013. What would it take to Change an Inference?: Using Rubin’s Causal Model to Interpret the Robustness of Causal Inferences. Education, Evaluation and Policy Analysis. Vol 35: 437-460.
http://epa.sagepub.com/content/early/recent
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
0
1
2
3
4
5
6
7
8
9
A B
Study
Es
tim
ate
d E
ffe
ct
above threshold
below threshold
Figure 1 Estimated Treatment Effects in Hypothetical Studies A and
B Relative to a Threshold for Inference
Threshold
{}
% bias necessary
to invalidate
the inference
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Quantifying the Discourse: Formalizing
Bias Necessary to Invalidate an Inference δ =a population effect, =the estimated effect, andδ# =the threshold for making an inference
An inference is invalid if: (1)
An inference is invalid if the estimate is greater than the threshold while the population value is less than the threshold.
Defining bias as -δ, (1) implies an estimate is invalid if and only if:
Expressed as a proportion of the estimate, inference invalid if:
# #ˆ( )ˆ% ( ) 1ˆ ˆ
bias
#ˆ ˆ( ) (2)bias
#
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Algebra for % Bias to Invalidate#
#
#
#
#
#
inference invalid if
ˆ ,
ˆsubtract from all parts
ˆ ˆ ˆ ˆ
ˆ ˆ0
multiply by -1
ˆ ˆ0
ˆsubstitute bias=
ˆ0 bias
ˆbias > > 0
ˆas % of estimate, divide by
ˆbias >
ˆ
#
#
> 0ˆ
%bias >1 > 0ˆ
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
0
1
2
3
4
5
6
7
8
9
A B
Study
Es
tim
ate
d E
ffe
ct
above threshold
below threshold
Figure 1 Estimated Treatment Effects in Hypothetical Studies A and
B Relative to a Threshold for Inference
Threshold
δ#
{}
% bias necessary
to invalidate
the inference
# #ˆ( ) 4 1ˆ% ( ) to invalidate= 1 1 33%ˆ ˆ 6 3
bias
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Interpretation of % Bias to Invalidate an Inference
% Bias is intuitive
Relates to how we think about statistical significance
Better than “highly significant” or “barely significant”
But need a framework for interpreting
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Framework for Interpreting % Bias to Invalidate an Inference: Rubin’s Causal
Model and the Counterfactual
1) I have a headache
2) I take an aspirin (treatment)
3) My headache goes away (outcome)
Q) Is it because I took the aspirin?
A) We’ll never know – it is counterfactual – for the individual
This is the Fundamental Problem of Causal Inference
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Definition of Replacement Cases as Counterfactual: Potential Outcomes
t ci i iY Y Definition of treatment effect for individual i:
value on outcome if unit received treatment
value on outcome if unit received control
ti
ci
Y
Y
Fundamental problem of causal inference is that we cannot simultaneously observe
and t ci iY Y
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Fundamental Problem of Inference and Approximating the Counterfactual with
Observed Data (Internal Validity)
345
6?6?6?
But how well does the observed data approximate the counterfactual?
91011
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Symbolic: Fundamental Problem of Inference and Approximating the
Counterfactual with Observed Data (Internal Validity)
6?6?6?
But how well does the observed data approximate the counterfactual?
Yt|X=t Yc|X=t
Yc|X=cYt|X=c
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Approximating the Counterfactual with Observed Data
345
But how well does the observed data approximate the counterfactual?Difference between counterfactual values and observed values for the control implies the treatment effect of 1
8910
111
6
is overestimated as 6 using observed control cases with mean of 4
9
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Using the Counterfactual to Interpret % Bias to Invalidate the Inference
How many cases would you have to replace with zero effect counterfactuals to change the inference?Assume threshold is 4 (δ# =4):1- δ# /
=1-4/6=.33 =(1/3)
666
6.00
The inference would be invalid if you replaced 33% (or 1 case) with counterfactuals for which there was no treatment effect. New estimate=(1-% replaced) +%replaced(no effect)=(1-%replaced) =(1-.33)6=.66(6)=4
000
64
345
1011
9
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Which Cases to Replace
• Think of it as an expectation: if you randomly replaced 1 case, and repeated 1,000 times, on average the new estimate would be 4
• Assumes constant treatment effect• conditioning on covariates and
interactions already in the model
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
0
1
2
3
4
5
6
7
8
9
A B
Study
Es
tim
ate
d E
ffe
ct
above threshold
below threshold
Figure 1 Estimated Treatment Effects in Hypothetical Studies A and
B Relative to a Threshold for Inference
Threshold
δ#
{}
% bias necessary
to invalidate
the inference
# #ˆ( ) 4 1ˆ% ( ) to invalidate= 1 1 33%ˆ ˆ 6 3
bias
To invalidate the inference, replace 33% of cases with counterfactual data with zero effect
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Fundamental Problem of Inference and Approximating an Unsampled Population
with Observed Data (External Validity)
910 Yt|Z=p1134 Yc|Z=p5
64
How many cases from p would you have to replace with cases with zero effect from p` to change the inference?Assume threshold is: δ# =4:1- δ# /
=1-4/6=.33 =(1/3)
6 Yt|Z=p´
6 Yc|Z=p´
0
66 6 6
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
0
1
2
3
4
5
6
7
8
9
A B
Study
Es
tim
ate
d E
ffe
ct
above threshold
below threshold
Figure 1 Estimated Treatment Effects in Hypothetical Studies A and
B Relative to a Threshold for Inference
Threshold
δ#
{}
% bias necessary
to invalidate
the inference
# #ˆ( ) 4 1ˆ% ( ) to invalidate= 1 1 33%ˆ ˆ 6 3
bias
To invalidate the inference, replace 33% of cases with cases from unsampled population data with zero effect
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Review & Reflection
Review of Framework
Pragmatism thresholds
How much does an estimate exceed the threshold % bias to invalidate the inference
Interpretation: Rubin’s causal model• internal validity: % bias to invalidate
number of cases that must be replaced with counterfactual cases (for which there is no effect)
• external validity: % bias to invalidate number of cases in sample population p that must be replaced with unobserved population p` (for which there is no effect)
Reflect
Which part is most confusing to you?
Is there more than one interpretation?
Discuss with a partner or two
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Example of Internal Validity from Observational Study : The Effect of Kindergarten Retention on Reading and Math Achievement
(Hong and Raudenbush 2005)
1. What is the average effect of kindergarten retention policy? (Example used here)
Should we expect to see a change in children’s average learning outcomes if a school changes its retention policy?
Propensity based questions (not explored here)
2. What is the average impact of a school’s retention policy on children who would be promoted if the policy were adopted?
Use principal stratification.
Hong, G. and Raudenbush, S. (2005). Effects of Kindergarten Retention Policy on Children’s Cognitive Growth in Reading and Mathematics.
Educational Evaluation and Policy Analysis. Vol. 27, No. 3, pp. 205–224
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Data
• Early Childhood Longitudinal Study Kindergarten cohort (ECLSK)– US National Center for Education Statistics (NCES).
• Nationally representative• Kindergarten and 1st grade
– observed Fall 1998, Spring 1998, Spring 1999 • Student
– background and educational experiences– Math and reading achievement (dependent variable)– experience in class
• Parenting information and style• Teacher assessment of student• School conditions• Analytic sample (1,080 schools that do retain some children)
– 471 kindergarten retainees – 10,255 promoted students
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Effect of Retention on Reading Scores(Hong and Raudenbush)
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Possible Confounding Variables(note they controlled for these)
• Gender• Two Parent Household• Poverty• Mother’s level of Education (especially relevant for reading
achievement)• Extensive pretests
– measured in the Spring of 1999 (at the beginning of the second year of school)
– standardized measures of reading ability, math ability, and general knowledge;
– indirect assessments of literature, math and general knowledge that include aspects of a child’s process as well as product;
– teacher’s rating of the child’s skills in language, math, and science
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Calculating the % Bias to Invalidate the Inference:Obtain spreadsheet
From https://www.msu.edu/~kenfrank/research.htm#causalChoose spreadsheet for calculating indices
Access spreadsheet
spreadsheet for calculating indices [KonFound-it!]
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Kon-Found-it: Basics
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Obtain t critical, estimated effect and standard error
Estimated effect( ) = -9.01 Standard
error=.68n=7168+471=7639;df > 500,
t critical=-1.96
From: Hong, G. and Raudenbush, S. (2005). Effects of Kindergarten Retention Policy on Children’s Cognitive Growth in Reading and Mathematics. Educational Evaluation and Policy Analysis. Vol. 27, No. 3, pp. 205–224
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
covariates
Page 215
Df=207+14+2=223
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Kon-Found-it: Basics% Bias to Invalidate
Estimated effect( ) = -9.01
Standard error=.68
n=7168+471=7639 Covariates=223
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Calculating the % Bias to Invalidate the Inference:Inside the Calculations
δ# =the threshold for making an inference =
se x tcritical, df>230 =.68 x -1.96=-1.33
[user can specify alternative threshold]
% Bias necessary to invalidate inference = 1-δ#/ =1-1.33/-9.01=85%
85% of the estimate must be due to bias to invalidate the inference.
}
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Using the Counterfactual to Interpret % Bias to Invalidate the Inference
How many cases would you have to replace with zero effect counterfactuals to change the inference?Assume threshold is 4 (δ# =4):1- δ# /
=1-4/6=.33 =(1/3)
666
6.00
The inference would be invalid if you replaced 33% (or 1 case) with counterfactuals for which there was no treatment effect. New estimate=(1-% replaced) +%replaced(no effect)=(1-%replaced) =(1-.33)6=.66(6)=4
000
64
345
1011
9
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Original cases that were not replacedReplacement counterfactual cases with zero effect
Original distribution
Retained Promoted
Example Replacement of Cases with Counterfactual Data to Invalidate Inference of an Effect of Kindergarten Retention
Counterfactual:No effect
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Interpretation1) Consider test scores of a set of children who were retained that are considerably lower (9 points) than others who were candidates for retention but who were in fact promoted. No doubt some of the difference is due to advantages the comparable others had before being promoted. But now to believe that retention did not have an effect one must believe that 85% of those comparable others would have enjoyed most (7.2) of their advantages whether or not they had been retained.
This is even after controlling for differences on pretests, mother’s education, etc.
2) The replacement cases would come from the counterfactual condition for the observed outcomes. That is, 85% of the observed potential outcomes must be unexchangeable with the unobserved counterfactual potential outcomes such that it is necessary to replace those 85% with the counterfactual potential outcomes to make an inference in this sample. Note that this replacement must occur even after observed cases have been conditioned on background characteristics, school membership, and pretests used to define comparable groups.
3) Could 85% of the children have manifest an effect because of unadjusted differences (even after controlling for prior achievement, motivation and background) rather than retention itself?
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Evaluation of % Bias Necessary to Invalidate Inference
Compare Bias Necessary to Invalidate Inference with Bias Accounted for by Background Characteristics
1% of estimated effect accounted for by background characteristics (including mother’s education), once controlling for pretests
More than 85 times more unmeasured bias necessary to invalidate the inference
Compare with % Bias necessary to invalidate inference in other studies
Use correlation metric• Adjusts for differences in scale
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
% Bias Necessary to Invalidate Inference based on Correlationto Compare across Studies
2 2
t 13.25r .152
(n q 1) t (7639 223 1) (13.25)
t taken from HLM: =-9.01/.68=-13.25n is the sample size q is the number of parameters estimated
# critical
2 2critical
t 1.96threshold= r .023
(n q 1) t (7639 223 1) 1.96
Where t is critical value for df>200
% bias to invalidate inference=1-.023/.152=85%
Accounts for changes in regression coefficient and standard errorBecause t(r)=t(β)
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
% Bias Necessary to Invalidate Inference based on Correlationto Compare across Studies
2 2
t 13.25r .152
(n q 1) t (7639 238 1) (13.25)
# critical
2 2critical
t 1.96threshold= r .023
(n q 1) t (7639 238 1) 1.96
% bias to invalidate inference=1-.023/.155=85%
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Compare with Bias other Observational Studies
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
% Bias to Invalidate Inference for observational studieson-line EEPA July 24-Nov 15 2012
Kindergarten retention effect
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Exercise 1 : % Bias necessary to Invalidate an Inference
• Take an example from an observational study in your own data or an article
• Calculate the % bias necessary to invalidate the inference– Interpret the % bias in terms of sample
replacement– What are the possible sources of bias?– Would they all work in the same
direction?• Debate your inference with a partner
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum 44
Application to Randomized Experiment: Effect of
Open Court Curriculum on Reading Achievement
• Open Court “scripted” curriculum versus business as usual
• 917 elementary students in 49 classrooms• Comparisons within grade and school• Outcome Measure: Terra Nova
comprehensive reading score Borman, G. D., Dowling, N. M., and Schneck
, C. (2008). A multi-site cluster randomized field trial of Open Court Reading. Educational Evaluation and Policy Analysis, 30(4), 389-407.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Value of Randomization
Few differences between groups
But done at classroom levelTeachers might talk to each other
School level is expensive (Slavin, 2008)Slavin, R. E. (2008). Perspectives on evidence-based research in education-what works? Issues in synthesizing educational program evaluations. Educational Researcher, 37, 5-14.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum n=27+22=49
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum Anadjusted difference of 10
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Obtaining # parameters estimated, t critical, estimated effect and standard error
Estimated effect( ) = 7.95
Standard error=1.83
3 parameters estimated,Df=n of classrooms-# of parameters estimated=49-3=46.t critical = t.05, df=46=2.014
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Quantifying the Discourse for Borman et al:What would it take to change the inference?
δ = population effect, =the estimated effect = 7.95, andδ # =the threshold for making an inference = se x tcritical, df=46 =1.83 x 2.014=3.69
% Bias necessary to invalidate inference = 1- δ #/ =1-3.69/7.95=54%
54% of the estimate must be due to bias to invalidate the inference
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Calculating the % Bias to Invalidate the Inference:Inside the Calculations
=the estimated effect = 7.95standard error =1.83t critical= 2.014
% Bias necessary to invalidate inference = 1-d#/d =1-3.68/7.95=54%
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Calculating the % Bias to Invalidate the Inference:Inside the Calculations
δ# =the threshold for making an inference = se x tcritical,
df=46 =1.83 x 2.014=3.686[user can override to specify hreshold]
% Bias necessary to invalidate inference = 1-d#/d =1-3.686/7.95=54%
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
OCR0
1
2
3
4
5
6
7
8
9
below threshold above threshold
Est
imat
ed E
ffec
t% Exceeding Threshold for Open Court
Estimated Effect
ˆ 7.95
δ# =3.68
54 % above threshold=1-3.68/7.95=.54}54% of the estimate must be due to bias to invalidate the inference
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Fundamental Problem of Inference and Approximating the Counterfactual with
Observed Data (External Validity)
91011345
66 6 6
64
How many cases would you have to replace with cases with zero effect to change the inference?Assume threshold is: δ# =4:1- δ# /
=1-4/6=.33 =(1/3)
6
6
0
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Interpretation of Amount of Bias Necessary to Invalidate the Inference: Sample
RepresentativenessTo invalidate the inference:
54% of the estimate must be due to sampling bias to invalidate Borman et al.’s inference
You would have to replace 54% of Borman’s cases (about 30 classes) with classes in which Open Court had no effect to invalidate the inference
Are 54% of Borman et al.’s cases irrelevant for non-volunteer schools?We have quantified the discourse about the concern of validity
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Example Replacement of Cases from Non-Volunteer Schools to Invalidate Inference of an Effect of the Open Court Curriculum
Open Court
Original volunteer cases that were not replacedReplacement cases from non-volunteer schools with no treatment effectOriginal distribution for all volunteer cases
Business as Usual Unsampled Population:No effect
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
The Fundamental Problem of External Validity
Before a randomized experiment:
People believe they do not “know” what generally works
People choose treatments based on idiosyncratic conditions -- what they believe will work for them (Heckman, Urzua and Vytlacil, 2006)
After a randomized experiment:
People believe they know what generally works
People are more inclined to choose a treatment shown to generally work in a study because they believe “it works”
The population is fundamentally changed by the experimenter (Ben-David; Kuhn)
The fundamental problem of external validity
the more influential a study the more different the pre and post populations, the less the results apply to the post experimental population
All the more so if it is due to the design (Burtless, 1995)
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Restatement of the Fundamental Problem of
External Validity
A researcher cannot simultaneously change the behavior in a population and claim that the pre-study population fully represents the post-study population.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Sampson, R. J. (2010). Gold standard myths: Observations on the experimental turn in quantitative criminology. Journal of Quantitative
Criminology, 26(4), 489-500. page 495
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Sampson, R. J. (2010). Gold standard myths: Observations on the experimental turn in quantitative criminology. Journal of Quantitative
Criminology, 26(4), 489-500. page 496
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Comparisons across Randomized Experiments (correlation metric)
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Distribution of % Bias to Invalidate Inference for Randomized Studies EEPA: On-line Jul 24-Nov 5 2012
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Review & Reflection
Review of applications
Concern about internal validity: Kindergarten retention (Hong and Raudenbush)
• 85% of cases must be replaced counterfactual data (with no effect) to invalidate the inference of a negative effect of retention on reading achievement
– Comparison with other observational studies
Concern about external validity: Open Court Curriculum
• 54% of cases must be replaced with data from unobserved population to invalidate the inference of a positive effect of Open Court on reading achievement in non-volunteer schools
– Comparison with other randomized experiments
Reflect
Which part is most confusing to you?
Is there more than one interpretation?
Discuss with a partner or two
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Exercise 2 : % Bias necessary to Invalidate an Inference
Take an example of a randomized experiment in your own data or an article
Calculate the % bias necessary to invalidate the inference
Interpret the % bias in terms of sample replacement
What are the possible sources of bias?
Would they all work in the same direction?
Debate your inference with a new partner
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Extensions of the Framework
Ordered thresholds for decision-making
Alternative hypotheses and scenarios
Relationship to confidence intervals
Related techniques
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Ordered Thresholds Relative to Transaction Costs
1. Changing beliefs, without a corresponding change in action.
2. Changing action for an individual (or family)
3. Increasing investments in an existing program.
4. Initial investment in a pilot program where none exists.
5. Dismantling an existing program and replacing it with a new program.
Definition of threshold: the point at which evidence from a study would make one indifferent to policy choices
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
How District Leaders Really Make Decisions, and About Whatfrom Design-Based Implementation Research: A Strategic Approach to Scaling and Sustaining Educational Innovation: William R. Penuel
University of ColoradoPhilip Bell
University of Washington
Justify an already-made decision
Choose between programs
Mobilize support for adopted program
Challenge an adopted program
Improve district policy implementation
Find concepts to inform improvement efforts
To select standards focus
Improve existing programs
Improve PD
1 2 3 4 5
3.62
3.72
3.76
3.83
3.90
3.90
3.93
3.97
4.17
Use of Research Evidence
Sampson, R. J. (2010). Gold standard myths: Observations on the experimental turn in quantitative criminology. Journal of Quantitative
Criminology, 26(4), 489-500.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Alternative Hypotheses: Non Zero Null Hypothesis
Non-zero null hypotheses (for kindergarten retention)
H0:δ> −6.
se x tcritical, df=7639=.68 x (−1.645)= −1.12 (one tailed test).
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Alternative Hypotheses: Non Zero Null Hypothesis
δ# = −6−1.12=−7.12
1− δ #/ =1− (−7.12/−9)=.21.
21% of estimated effect would have to be due to bias to invalidate inference for H0:δ> −6.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Alternative Scenario: Failure to Reject Null when null is False
Assume the data are comprised of two subsamples, one with an effect at the
threshold (d#) and the other (d ) with 0 effect, and define as the proportion of the sample that has 0 effect.
dxy =(1-)d# + dxy .
*Frank, K. A. and Min, K. 2007. Indices of Robustness for Sample Representation. Sociological Methodology. Vol 37, 349-392. * co first authors.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Sample Replacement: Estimate Does not Exceed the Threshold
Set dxy=0 and solve for : =1- dxy /d# =1-.5/1.183=.577=58%If of the cases have 0 effect then if you replace
them (with cases at the threshold) then the overall estimate will be at the threshold.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Dehejia, Rajeev H., and Sadek Wahba. "Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs." Journal of the American statistical Association 94, no. 448 (1999): 1053-1062.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Alternative Scenario: Alternative Thresholds for Inference
Use δ# = −4
To Invalidate Hong and Raudenbush’s inference, 56% of the estimate would have to be due to bias if the threshold for inference is -4.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Alternative Scenario: Non-zero Effect in the Replacement (e.g., non-volunteer) Population
Inference invalid if 1-πp<(δp − δ#)/(δ p − δ p´ ).
If δ p´ = −2, and δ#=3.68 and δ p =7.95 (both as in the initial example for Open Court). Inference is invalid if 1-πp<(7.95 – 3.68)/(7.95
− −2 ) =.43
inference invalid if more than 43% of the sample were replaced with cases for which the effect of OCR was −2 .
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
}0 1 2 3 4 5 6 7 8 9 1 1 1 1 0 1 2 3
Confidence Interval
Relationship between the Confidence Interval and % Bias Necessary to Invalidate the Inference of an Effect of Open Court on Comprehensive Reading Score
δ #
Lower bound of confidence interval “far from 0” estimate exceeds threshold by large amount
0 1 2 3 4 5 6 7 8 9 1 1 1 1
0 1 2 3} }
δ #
}}
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Related Techniques
Bounding (e.g., Altonji et, Elder & Tabor, 2005; Imbens 2003; Manski)
lower bound: “if unobserved factors are as strong as observed factors, how small could the estimate be?”
• Focus on estimate
% robustness: “how strong would unobserved factors have to be to invalidate inference?”
• Focus on inference, policy & behavior
External validity based on propensity to be in a study (Hedges and O’Muircheartaigh )
They focus on estimate
We focus on comparison with a threshold
Other sensitivity (e.g., Rosenbaum or Robins)
Characteristics of variables needed to change inference
We focus on how sample must change.• Can be applied to observational study or RCT
Other Sources of Bias
Violations of SUTVA• Agent based models?
Measurement error• Just another source of bias (minor concern for examples here)
Differential treatment effects• Use propensity scores to differentiate, then apply indices
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
II: Correlation FrameworkConcerns about internal validity
At what levels must an omitted variable be correlated with a predictor and outcome to change an inference about the predictor?
Frank, K. 2000. "Impact of a Confounding Variable on the Inference of a Regression Coefficient." Sociological Methods and Research, 29(2), 147-194
Concerns about external validity
What must be the correlation between predictor and outcome in the unsampled population to invalidate an inference about an effect in a population that includes the unsampled population?
*Frank, K. A. and Min, K. 2007. Indices of Robustness for Sample Representation. Sociological Methodology. Vol 37, 349-392. * co first authors.
Combined application (with propensity scores)
Frank, K.A., Gary Sykes, Dorothea Anagnostopoulos, Marisa Cannata, Linda Chard, Ann Krause, Raven McCrory. 2008. Extended Influence: National Board Certified Teachers as Help Providers. Education, Evaluation, and Policy Analysis. Vol 30(1): 3-30.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Making an Inference from an Observational Study
83
•Gather data on people as they naturally choose “treatments”
•Measure covariates, especially pretests•Randomly sample from a larger population – can
do because not assigning to treatments•Examples
•Coleman et al., Catholic schools effect•Hong and Raudenbush: effects of kindergarten
retention
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Linear Adjustment to Invalidate Inference
How many cases would you have to replace with zero effect counterfactuals to change the inference?Assume threshold is 4 (δ# =4):1- δ# /
=1-4/6=.33 =(1/3)
666
6.00
The inference would be invalid if you replaced 33% (or 1 case) with counterfactuals for which there was no treatment effect. New estimate=(1-% replaced) +%replaced(no effect)=(1-%replaced) =(1-.33)6=.66(6)=4
000
64
345
91011
9
+3=6+2=6+1=6
+6=9
Linear model distributeschange over all cases
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Adjusting for Confound to get Correct Effect
666
To replace observed control cases to get estimated effect of 4instead of uncontrolled mean difference of 6, use confound: control group disadvantaged from the start
0 1 2
6 4 x 1 x
y s confound
y s confound
Replacing observed cases with linear counterfactuals
But how do we obtain these estimates?
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Control for Confound ↔Adjusting the Outcome
666
0 1 2
6 4 x 1 x
example: 3=6 4 x 0 1 x -3
to get to 6 on the left, add 3 to each side
3+3=6 4 x 0
control for conound adjust the outcome
y s confound
y s confound
Replacing observed cases with linear counterfactuals
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Regression!SAS Syntax for reading in toy counterfactual data
DATA one;input y s confound;cards;9 1 010 1 011 1 03 0 -34 0 -25 0 -1run;
proc reg data=one;model y=s ;run;
proc reg data=one;model y=s confound; run;
0 1 2
6 4 x 1 x
y s confound
y s confound
Biased estimated
Unbiased estimate
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Regression!: SPSS Syntax for reading in toy counterfactual dataDATA LIST FREE / y s confound.
Begin DATA .9 1 010 1 011 1 03 0 -34 0 -25 0 -1End DATA .
REGRESSION /MISSING LISTWISE /STATISTICS COEFF OUTS R ANOVA /CRITERIA=PIN(.05) POUT(.10) /NOORIGIN /DEPENDENT y /METHOD=ENTER s.
REGRESSION /MISSING LISTWISE /STATISTICS COEFF OUTS R ANOVA /CRITERIA=PIN(.05) POUT(.10) /NOORIGIN /DEPENDENT y /METHOD=ENTER s confound.
0 1 2
6 4 x 1 x
y s confound
y s confound
Biased estimated
Unbiased estimate
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
How Regression Works: Partial Correlation
s· s· ·s· | 2 2 2 2
· s·
.965 .866 .928.866
1 1 1 .866 1 .928
y cv y cvy cv
y cv cv
r r rr
r r
Partial Correlation: correlation between s and y, where s and y have been controlled for the confounding variable
2 2
2 2
( | ) ( ) 1 3.406 1 .928 1.265
(s | ) (s) 1 .547 1 .866 .274
y c
s c
sd y cv sd y r
sd cv sd r
1 s· |
( | ) 1.265ˆ .866 4( | ) .274y cv
sd y cvr
sd x cv
Spss syntaxCORRELATIONS /VARIABLES=y s confound /PRINT=TWOTAIL NOSIG
Sas syntaxtproc corr data=one;var y s confound;run;
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Derivation of Partial from Matrix Algebra (Normal Equations)
1 11
2 2
1 1
( ) ( ) ( ) ( ) /covariance of x and yˆ
variance of x( ) ( ) /
n n
i i i ii i
n n
i ii i
x x y y x x y y n
x x x x n
1 2 1 2
1 2
1 2
1 22 2
11 2 1
1 2 1 2 22
1 2
1 2 1 2 s· s·11 2 2
1
1 11 ' , ' , X'Y=
1 1 1 1
for , ' X'Y1 1
x x x x
x x
x x s cv
x x
x yx x
x x x x x y
x x
x y x y x x s y cv y s c y cv y
r
r r rrX X X X
r r r
r r
r r r r r r v r r rX X
r r
2·
2 2 2· s·
( ) 1
1 1 (s) 1
y cvcv
y cv cv s cv
sd y r
r r sd r
Multivariate: Β=[X’X]-1X’Y For 2 variables that are standardized:
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
s· s· ·1 2
s·
( )
1 (s)y cv y cv
cv
r r r sd y
r sd
Defining the Impact of a Confounding Variable
s· s· ·s· | 2 2
· s·1 1
y cv y cvy cv
y cv cv
r r rr
r r
s· s· ·
The "impact" is the product:
impact (of cv on ) y cv y cvr r r
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
rsy=.18
rscv=.17
rycv=.07rscv×rycv
CVEnhancement ofteaching through
leadership
SBoard
Certification YHelp
Provided
The Impact of a Enhancement of Teaching through leadership on Correlation Between
Board Certification and Help Provided
rsy|cv=.866
How Regression Works:Impact of a Confounding Variable on a Regression Coefficient
rs∙y=.964
rcv∙y=.928.866x.928
s y
rcv∙s=.866
rc∙sxrc∙y
cv
Impact weights the relationship between cv and y by the relationship between c and s: the stronger the relationship between cv and y, the more important the relationship between cv and s.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Alternative Conceptualization (overlapping variance)
(rx2∙y|x1)2
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Everything old is new again
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Regression works…
As long as you have the right confounding variables
Linear relationships assumed
Does this work in practice?
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Regression in Practice: Observational studies with controls for pretests work better than you think
Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the
American Statistical Association, 103(484), 1334-1344.
Quantify how much biased removed by statistical
control using pretests in a given setting
Sample: Volunteer undergraduates
Outcome: Math and vocabulary tests
Treatment: • basic didactic, • showing transparencies• defining math concepts
OLS Regression with pretests removes 84% to 94% of bias relative to RCT!! Propensity by strata not quite as good
See also Concato et al., 2000 for a comparable example in medical research
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Supporting findingsBerk R (2005) Randomized experiments as the bronze standard. J Exp Criminol 1:417–433
Berk R, Barnes G, Ahlman L, Kurtz E (2010) When second best is good enough: a comparison between a
true experiment and a regression discontinuity quasi-experiment. J Exp Criminol 6:191–208. ‘‘the results
from the two approaches are effectively identical’’ page 191.
Pohl, S., Steiner, P. M., Eisermann, J., Soellner, R., & Cook, T. D. (2009). Unbiased causal inference from an observational study: Results of a within-study comparison. Educational Evaluation and Policy Analysis, 31(4), 463-479.
Concato, J., Shah, N., & Horwitz, R. I. (2000). Randomized, controlled trials, observational studies, and the hierarchy of research designs. New England Journal of Medicine, 342(25), 1887-1889.
Cook, T. D., Shadish, S., & Wong, V. A. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy and Management. 27 (4), 724–750.
Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association, 103(484), 1334-1344.
Steiner, Peter M., Thomas D. Cook & William R. Shadish (in press). On the importance of reliable covariate measurement in selection bias adjustments using propensity scores. Journal of Educational and Behavioral Statistics.
Steiner, Peter M., Thomas D. Cook, William R. Shadish & M.H. Clark (2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods. Volume 15, Issue 3. Pages 250-267. more than just pretest.
Kane, T., & Staiger, D. (2008). Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation. NBER working paper 14607.
35. Kane, T., & Staiger, D. (2008).
Bifulco, Robert . "Can Nonexperimental Estimates Replicate Estimates Based on Random Assignment in Evaluations of School Choice? A Within‐Study Comparison." Journal of Policy Analysis and Management 31, no. 3 (2012): 729-751.
Reports 64% to 96% reduced with pre-test
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
New paper on Multiple Approaches (from Tom Dietz)
http://arxiv.org/pdf/1412.3773v1.pdf
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Reflection
What part if most confusing to you?Why?
More than one interpretation?
Talk with one other, share
Find new partner and problems and solutions
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
When Regression Doesn’t Work
Instrumental variables?• Alternative assumptions, may be not better
– Exclusion restriction– Strong instrument– Large n
Propensity scores?No better than the covariates that go into it
• Heckman, 2005; Morgan & Harding, 2006, page 40; Rosenbaum, 2002, page 297; Shadish et al., 2002, page 164);
• How could they be better than covariates?– Propensity=f(covariates).
Paul Holland says:
PAUL ROSENBAUM
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Sensitivity Analysis:What Must be the Impact of an Unmeasured
Confounding variable invalidate the Inference?
Step 1: Establish Correlation Between predictor of interest and outcomeStep 2: Define a Threshold for InferenceStep 3: Calculate the Threshold for the Impact Necessary to Invalidate the InferenceStep 4: Multivariate Extension, with other Covariates
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Obtain t critical, estimated effect and standard error
Estimated effect( ) = -9.01 Standard
error=.68n=7168+471=7639;df > 500,
t critical=-1.96
From: Hong, G. and Raudenbush, S. (2005). Effects of Kindergarten Retention Policy on Children’s Cognitive Growth in Reading and Mathematics. Educational Evaluation and Policy Analysis. Vol. 27, No. 3, pp. 205–224
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Establish Correlation and Threshold for Kindergarten Retention on Achievement
2 2
t 13.25r .150
(n q 1) t (7639 2 1) (13.25)
t taken from HLM: =-9.01/.68=-13.25n is the sample size =7639q is the number of parameters estimated ( predictor+omitted variable=2)
# critical
2 2critical
t 1.96threshold= r .022
(n q 1) t (7639 2 1) 1.96
Where t is critical value for df>200
Step 1: calculate treatment effect as correlation
Step 2: calculate threshold for inference (e.g., statistical significance)
r# can also be defined in terms of effect sizes
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Step 3a: Calculate the Threshold for the Impact Necessary to Invalidate the Inference
.150 .022.130
1 | 022 |TICV
#·
#
r
1 | r |x yrTICV
Set rx∙y|cv =r# and solve for k to find the threshold for the impact of a confounding variable (TICV). Sometimes I say ITCV.
· · · ·· | 2 2
· ·11 1
x y x cv y cv x yx ycv
y cv x cv
r r r r impactr
impactr r
Assume rx∙cv =ry∙cv (which maximizes the impact of the confounding variable – Frank, 2000). Then impact= rx∙cv x ry∙cv = rx∙cv x rx∙cv = ry∙cv x ry∙cv , and
Magnitude of impact of an unmeasured confound > .130 → inference invalidMagnitude of impact of an unmeasured confound < .130 → inference valid.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Maxim
izing impact
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Step 3b: Component Correlations
2r .130 r= .130 .361
The magnitude of each correlation (rx∙cv , ry∙cv ) must be greater than .36 to change inference. The correlations must have opposite signs.
If rx∙cv = ry∙cv =r, then impact= rx∙cv x ry∙cv =r2 .
Component correlations for threshold
.150 .022.130
1 .022TICV
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Calculating the % Bias to Invalidate the Inference:Obtain spreadsheet
From https://www.msu.edu/~kenfrank/research.htm#causalChoose spreadsheet for calculating indices
Access spreadsheet
spreadsheet for calculating indices [KonFound-it!]
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
KonFound-it! SpreadsheetUser enters values in yellow
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Inside the Calculations
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Inside the Calculations
2 2
t 13.25r .150
(n q 1) t (7639 3) (13.25)
# critical
2 2critical
t 1.96threshold= r .022
(n q 1) t (7639 3) 1.96
Step 1: calculate treatment effect as correlation
Step 2: calculate threshold for inference (e.g., statistical significance)
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Inside the Calculations: Step 3a
.150 .022.130
1 | 022 |TICV
#·
#
r
1 | r |x yrTICV
2r .130 r= .130 .361 Step 3b
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Reflection
• What part if most confusing to you?– Why?– More than one interpretation?
• Talk with one other, share• Find new partner and problems and
solutions
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Inside the Calculations: Multivariate
2 2
t 13.25r .152
(n q 1) t (7639 3 238) (13.25)
# critical
2 2critical
t 1.96threshold= r .023
(n q 1) t (7639 3 238) 1.96
Step 1M: calculate treatment effect as correlation with DF correction
Step 2M: calculate threshold for inference (e.g., statistical significance)
DF correction
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Step 3M: Multivariate Extension, with Covariates
#· | |
#|
·cv| y·cv|
r|
1 | r |
and = = |
x y z z
z
x z z
rITCV z
r r ITCV z
2 2 2 2· · | · cv· · cv·
2 2 2 2x· x· | x· cv· x· cv·
1 1
1 1
y cv y cv z y z z y z z
cv cv z z z z z
r r R R R R
r r R R R R
k=rx ∙cv|z× ry ∙ cv|z
Following bivariate approach: maximizing the impact with covariates z in the model implies
And components
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Assumption: Omitted Confound Independent of Observed
Covariates
2 2 2 2 2· · | · cv· · cv· · | ·
2 2 2 2 2x· x· | x· cv· x· cv· x· | x·
1 1 1
1 1 1
y cv y cv z y z z y z z y cv z y z
cv cv z z z z z cv z z
r r R R R R r R
r r R R R R r R
2cv· 0 by assumption (can be altered in spreadsheet)zR
Gives maximal credence to skeptic challenging inference
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Step 3M: Multivariate Extension, with Covariates
#· | |
#|
·cv| y·cv|
r . .152 .023| .132,
1 | r | 1 | .023 |
and = = | =. .132=.364
x y z z
z
x z z
rITCV z
r r ITCV z
2· · | ·
2x· x· | x·
1 .364 1 .201 .325
1 .364 1 .275 .310
y cv y cv z y z
cv cv z z
r r R
r r R
Following bivariate approach: maximizing the impact with covariates z in the model implies
And
2cv· 0 by assumption (can be altered in spreadsheet)zR
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Obtaining R2 YZ
2 22 2|Z . ,. ,2 2
|Z 2 2|Z1 1
YX Y X ZY X Z YZYX YZ
YZ YX
r RR Rr R
r r
2 2 2
|Z . ,2Z 2 2
|Z
.152 .22.201
1 .152 1YX Y X Z
YYX
r RR
r
:
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Inside Calculations: Obtaining R2 YZ
2 2 2|Z . ,2
Z 2 2|Z
.152 .22.201
1 .152 1YX Y X Z
YYX
r RR
r
:
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Inside Calculations: Obtaining R2 xz
xZ:
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Extra Sufficient Statistics for Obtaining R2xZ
number of covariates
% retained= 471/7639=.061Var(x)=.061*.94=.057, std(x)=.24Page 208
Page 210
Residual variance=.55+.88=143Initial variance=13.532=183R2 =1-143/183=.22
or
page 216:page 215:
std(y),
page 217
Already have Se(β1)
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Multivariate Calculations in SpreadsheetUser entersStd(y)Std(x)R2
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Obtaining R2 xz
:
2 2 2. ,2
2 2221
ˆ 1 13.53 1 .221 1 .275
ˆ .24 (7639 241) .68ˆ Se( )
y Y X Z
XZ
x
RR
df
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
What must be the Impact of an Unmeasured Confound to Invalidate the Inference?
If |k| > .132 (or .130 without covariates) then the inference is invalid
If |r x cv|z |= |ry cv|z|, then each would have to be greater than k1/2 =.36 to invalidate the inference.
*Because effect is negative, components take opposite signs
*Correlations must be partialled for covariates z.Multivariate adjustment, to invalidate the inference:
|ry cv| > .325 and |r x cv| >.310
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Links to sas and stata code
sas program for calculating indices and related measures
stata code for calculating indices (beta version – thanks to Yun-jia Lo)
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Alternative: Regression Coefficient and Standard Error
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Maximizing Expression
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Comparing Regression vs Correlation ITCV (example from Frank, 2000)
#· | |
#|
r .1809 .0579| .1305
1 | r | 1 | .0579 |x y z z
z
rITCV z
Regression (page 155)
Correlation (pages 160,182; Frank, Sykes et al., 2008)
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Exercise 3: Impact Threshold1)Identify a statistical inference in an article you are
interested in.2) Describe possible confounds/alternative explanations
that could bias the estimate3) Note the sample size and t-ratio
recall t=estimate/[se(estimate)]4) Calculate robustness of inference usinghttp://www.msu.edu/~kenfrank/papers/calculating%20indices%203.xls5) If you have the data calculate the multivariate ITCV6) Discuss with a partner
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Applications of Impact Threshold in Education and Sociology (also some in accounting)Frank, 2000. ITCV=.228. Frank, K.A., Zhao, Y., Penuel, W.R., Ellefson, N.C., and Porter, S. 2011. Focus, Fiddle and Friends: Sources of Knowledge to Perform the Complex Task of Teaching. Sociology of Education, Vol 84(2): 137-156.Frank, K.A., Gary Sykes, Dorothea Anagnostopoulos, Marisa Cannata, Linda Chard, Ann Krause, Raven McCrory. 2008. Extended Influence: National Board Certified Teachers as Help Providers. Education, Evaluation, and Policy Analysis. Vol 30(1): 3-30. Frisco, Michelle, Muller, C. and Frank, K.A. 2007. Using propensity scores to study changing family structure and academic achievement. Journal of Marriage and Family. Vol 69(3): 721–741 *Frank, K. A. and Min, K. 2007. Indices of Robustness for Sample Representation. Sociological Methodology. Vol 37, 349-392. * co first authors.Frank, K. 2000. "Impact of a Confounding Variable on the Inference of a Regression Coefficient." Sociological Methods and Research, 29(2), 147-194
Crosnoe, Robert and Carey E. Cooper. 2010. “Economically Disadvantaged Children’s Transitions into Elementary School: Linking Family Processes, School Contexts, and Educational Policy.” American Educational Research Journal 47: 258-291. ITCV =.32Crosnoe, Robert. 2009. “Low-Income Students and the Socioeconomic Composition of Public High Schools.” American Sociological Review 74: 709-730. ITCV=.03Augustine, Jennifer March, Shannon Cavanagh, and Robert Crosnoe. 2009. “Maternal Education, Early Child Care, and the Reproduction of Advantage.” Social Forces 88: 1-30. ITCV=.4,.23
Maroulis, S. & Gomez, L. (2008). “Does ‘Connectedness’ Matter? Evidence from a Social Network Analysis within a Small School Reform.” Teachers College Record, Vol. 110, Issue 9.
Cheng, Simon, Regina E. Werum, and Leslie Martin. 2007. “Adult Social Capital: How Family and Community Ties Shape Track Placement of Ethnic Groups in Germany.” American Journal of Education 114: 41-74.
William Carbonaro1 Elizabeth Covay1 School Sector and Student Achievement in the Era of Standards Based Reforms. Sociology of eductaion vol. 83 no. 2 160-182 .Chen, Xiaodong, Frank, Kenneth A. Dietz, Thomas, and Jianguo Liu. 2012. Weak Ties, Labor Migration, and Environmental Impacts: Toward a Sociology of Sustainability. Organization & Environment. Vol. 25 no. 1 3-24.
see alsoPan, W., and Frank, K.A. (2004). "An Approximation to the Distribution of the Product of Two Dependent Correlation Coefficients." Journal of Statistical Computation and Simulation, 74, 419-443
Pan, W., and Frank, K.A., 2004. "A probability index of the robustness of a causal inference," Journal of Educational and Behavioral Statistics, 28, 315-337.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Extensions LogisticSimulation:
• Kelcey, B. 2009. Improving and Assessing Propensity Score Based Causal Inferences in Multilevel and Nonlinear Settings. Unpublished doctoral dissertation, University of Michigan
• Ichino, Andrea, Fabrizia Mealli, and Tommaso Nannicini. 2008. “From temporary help jobs to permanent employment: what can we learn from matching estimators and their sensitivity? ” Journal of Applied Econometrics 23:305–327.
• Nannicini, Tommaso. 2007. “Simulation–based sensitivity analysis for matching estimators.” Stata Journal 7:334–350.
Table• David J. Harding. 2003. “Counterfactual Models of Neighborhood
Effects: The Effect of Neighborhood Poverty on High School Dropout and Teenage Pregnancy.” American Journal of Sociology 109(3): 676-719.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Extensions: Multilevel
If omitted confound is at level 1, no concerns. Covariates include level 2 fixed or random effectsIf omitted confound at level 2:Seltzer, M. H., Frank, K. A., & Kim, J. (2007). Studying the Sensitivity of Inferences to Possible Unmeasured Confounding Variables in Multisite Evaluations. Paper presented at invited session at AERA 2007.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Extensions: Impact Not Maximized and Impact Curve
Impacts above the blue line invalidate the inference
Smallest impact to invalidate inference: rxcv=rycv=.364
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Extensions: Impacts Before and After Pretests
If k > .109 (or .130 without covariates) then the inference is invalid
Impact of strongest measured covariate (student approaches to learning) is -.00126 (sign indicates controlling for it reduces estimated effect of retention)
Impact of unmeasured confound would have to be about 100 times greater than the impact of the strongest observed covariate to invalidate the inference. Hmmm….
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Effect of Retention on Achievement After Adding each Covariate
Controls Est Se t
School -21.24 .63 -33.49
School+Pre2 (spring Kindergarten) -12.01 .45 -26.48
School+Pre2+(Pre2-Pre1)+ -12.10 .47 -26.28
School+Pre2+(Pre2-Pre1)+Momed
-12.00 .47 -26.26
School+Pre2+(Pre2-Pre1)+Female
-12.07 .46 -25.18
School+Pre2+(Pre2-Pre1)+2parent
-12.01 .46 -26.27
School+Pre2+(Pre2-Pre1)+poverty
-12.04 .46 -26.16
Hong and Raudenbush (model based) -9.01 .68 -13.27
n=10,065, R2 =.40Note: 1 year’s growth is about 10 points, so retention effect > 1 year growth
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Consider Alternate Sample(External Validity)
Causal Inference concern: We cannot assert cause if the effect of is not constant across contexts.
Statistical Translation:Would the inference be valid if the sample included more of some population (e.g. non-volunteer schools) for which the effect was not as strong?
Rephrased for robustness: what must be the conditions in the alternative sample to invalidate the inference?
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Consider Alternate Sample(External Validity)
Define as the proportion of the sample that is replaced with an alternate sample.
r is correlation in unobserved data
R is combined correlation for observed and unobserved data:
Rxy=(1-)rxy + rxy .
*Frank, K. A. and Min, K. 2007. Indices of Robustness for Sample Representation. Sociological Methodology. Vol 37, 349-392. * co first authors.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Thresholds for Sample Replacement
Set R=r# and solve for rxy:
If half the sample is replaced (=.5), original
inference is invalid if rxy < 2r#-rxy
Therefore, 2r#-rxy defines the threshold for replacement: TR(=.5)
If rxy =0, inference is altered if π> 1-r#/rxy . Therefore 1-r#/rxy defines the threshold for
replacement: TR(rxy=0)Assumes means and variances are constant across samples, alternative calculations available.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Thresholds for Sample Replacement: Toy example
Set R=r# and solve for rxy:
If half the sample is replaced (=.5), original
inference is invalid if rxy < 2r#-rxy
Therefore, 2r#-rxy defines the threshold for replacement: TR(=.5):
If threshold = r#=.4 and rxy =.6 then inference invalid
if rxy = 2r#-rxy =2x.4-.6=.2.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Fundamental Problem of Inference to an Unsampled Population (External Validity)
But how well does the observed data represent both populations?
91011345
888666
64
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Thresholds for Sample Replacement
If rxy =0, inference is altered if π> 1-r#/rxy . Therefore 1-r#/rxy defines the threshold for
replacement: TR(rxy=0)
If threshold = r#=.4 and rxy =.6 then inference invalid if % replaced = π =1-r#/rxy =1-.4/.6=1/3.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Fundamental Problem of Inference and Approximating the Counterfactual with
Observed Data (External Validity)
91011345
66 6 6
64
How many cases would you have to replace with cases with zero effect to change the inference?Assume threshold is: δ# =4:1- δ# /
=1-4/6=.33 =(1/3)
6
6
0
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Empirical example of Thresholds for Replacement: Effect of Open Court (Borman et al)
TR(=.5)= 2r#-rxy|z =2(.29)-.54=.03. Correlation between Open Court and reading achievement
would have to be less than .03 to invalidate inference if half the classrooms in the sample were replaced (e.g., with classrooms with no effect).
TR(rxy =0)= 1-r#/rxy|z =1-(.29/.54)=.47About 47% (abut 23 classrooms) would have to be replaced
with others for which Open Court had no effect (rxy =0) to invalidate the inference in a combined sample.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Review of Correlational Framework
Statistical control impact of a confound
Internal validity: Impact necessary to change an inference
External validity: unobserved correlations necessary to change an inference
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Exercise 4: Robustness for Sample Representativeness (External Validity)
1)Identify a statistical inference in your own work or in the literature for which there is concern about the external validity
2) Identify possible populations for which the effect may not apply
3) Note the t-ratio and sample size4) Calculate robustness of inference usinghttp://www.msu.edu/~kenfrank/papers/calculating%20indices%203.xls
How do the indices change when you change thesample size?t-ratio?
5) Discuss with a new partner your inference and how robust you think it is. Partner can challenge. Then change roles.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Thresholds for Sample Replacement: Toy example
Set R=r# and solve for rxy:
If half the sample is replaced (=.5), original
inference is invalid if rxy < 2r#-rxy
Therefore, 2r#-rxy defines the threshold for replacement: TR(=.5):
If threshold = r#=.4 and rxy =.6 then inference invalid
if rxy = 2r#-rxy =2x.4-.6=.2.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Conclusion
Applications of frameworks
Limitations
There will be debate about inferences: quantify
Assumptions as the bridge between statistics and science
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Applications of Frameworks% bias to invalidate
Any estimate, threshold
CounterfactualGood for experimental settings (treatments)
Think in terms of replacement of cases
CorrelationalGood for continuous predictors
Think in terms of correlations
Both can be applied to Internal and External Validity
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Limitations on Inference
Without random sample and random assignment causal inferences are uncertain. They are inferences.Observation study
Internal validity: omitted variables Randomized Study
External validity: fidelity, attritionParadox of external validity
If results of randomized study influence behavior, then inherent limit on external validity
• Prior to study, subjects sign up because of fit, political pressure, etc
• After study, subjects sign up because they think it generally works
Inferences will be debated based on differences in ExperiencePower Goals
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
It’s all in how you talk about it!
Do best method you canInclude relevant controls!
But science is as much in the nature of the discourse as the method
Virtue epistemology • Greco, 2009; Kvanig, 2003; Sosa, 2007
What would it take to invalidate the inference?• Frank, K.A., Duong, M.Q., Maroulis, S., Kelcey, B.
under review. Quantifying Discourse about Causal Inferences from Randomized Experiments and Observational Studies in Social Science Research
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Quantify what it Would Take to Change the Inference
• Objections to moving from statistical to causal inference – No unobserved confounding variables– Treatment has same effect for all
• Robustness indices quantify how much must assumptions must be violated to invalidate inference.
• No new causal inferences!– robustness indices merely quantify terms of debate regarding
causal inferences.• Can be used with any threshold.• Basically, characterizing size of p-value or t-ratio, instead of “barely
significant” versus “highly significant”• Can be used (theoretically) for any t-ratio
– Discuss: Statistical inference as threshold?• Difficult to defend tenuous effects (e.g., p =.049).
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Assumptions are the bridge between statistical and causal inference
Statistical Inference Causal Inference
Assumptions
Cornfield, J., & Tukey, J. W. (1956, Dec.), Average Values of Mean Squares in Factorials. Annals of Mathematical Statistics, 27(No. 4), 907_949.
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
In Donald Rubin’s words “Nothing is wrong with making
assumptions; on the contrary, such assumptions are the strands that join the field of statistics to scientific disciplines. The quality of these assumptions and their precise explication, not their existence, is the issue”(Rubin, 2004, page 345).
Ken adds: and we should talk about the inferences in terms of the sensitivity to the assumptions
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Paul Holland and Don Rubin
This [the use of randomization to alleviate the problems of untestable assumptions] should not be interpreted as meaning that randomization is necessary for drawing causal inferences. In many cases, appropriate untestable, assumptions will be well supported by intuition, theory or past evidence. In such cases we should not avoid drawing causal inferences and hide behind the cover of uninteresting descriptive statements. Rather we should make causal statements that explicate the underlying assumptions and justify them as well as possible.”
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Reflection
What part if most confusing to you?Why?
More than one interpretation?
Talk with one other, share
Find new partner and problems and solutions
Replacement Cases Framework
Conclusion
Correlational Frameworkoverview
Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework
Correlational Framework
Impact of a Confounding variable
Internal validity
Example: Effect of kindergarten retention
External validity
example: effect of Open Court curriculum
Emails of Users'Crosnoe, Robert' <[email protected]>; [email protected]; [email protected]; [email protected] Muller, Chandra L <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Spiro Maroulis <[email protected]>; 'Ben Kelcey' <[email protected]>; 'Minh Duong' <[email protected]>; [email protected]; [email protected]; 'James Moody' <[email protected]>; 'William Carbonaro' <[email protected]>; 'Mark Berends' <[email protected]>; Min Sun <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; '[email protected]; [email protected]; [email protected]; [email protected]; David Williamson Shaffer <[email protected]>; [email protected]; Eric Camburn <[email protected]>; Geoffrey Borman <[email protected]>; [email protected]; [email protected]; Cavanagh, Shannon E <[email protected]>; Dedrick, Robert <[email protected]>; John Lockwood <[email protected]>; Lou Mariano <[email protected]>; Joshua Cowen <[email protected]>; [email protected]; T Gmail <[email protected]>; Yu Xie <[email protected]>; [email protected]; Konstantopoulos, Spyros <[email protected]>; Kimberly S. Maier <[email protected]>; William Schmidt <[email protected]>; Houang <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Last sent april 6 2015, 9:15 a.m.