Copyright © 2011, SAS Institute Inc. All rights reserved.
Educational Value-Added Analysis of Covariance Models with Error in the CovariatesS. Paul WrightEVAASSAS Institute Inc.
2
Copyright © 2011, SAS Institute Inc. All rights reserved.
Introduction
Educational Value-Added Analysis of Covariance Modelswith Error in the Covariates
3
Copyright © 2011, SAS Institute Inc. All rights reserved.
Background
What is analysis of covariance (ANCOVA)? Mathematically, the model is:
Where: : current-year score for student of teacher : vector of covariates (“predictors”), including
» prior-year test score(s)» student-level characteristics?» teacher/school/community characteristics?
: “effect” of -th teacher
4
Copyright © 2011, SAS Institute Inc. All rights reserved.
Key Question of this Study
Question 1: If analysis of covariance (ANCOVA) is to be used for value-added modeling, and there is measurement error in the covariates, what is the most useful approach for dealing with the bias caused by measurement error?
Three options were considered: Classical errors-in-variables (EIV) regression Instrumental variables (IV) regression ANCOVA with multiple covariates
5
Copyright © 2011, SAS Institute Inc. All rights reserved.
Additional Questions of Interest
Question 2: Should be a fixed or random effect? Question 3: Should include student characteristics? Question 4: Should include teacher characteristics?
6
Copyright © 2011, SAS Institute Inc. All rights reserved.
In Short, the Answers are…
1. What is the most useful approach for dealing with the bias caused by measurement error? Use multiple ANCOVA (at least 3 prior test scores)
2. Should be a fixed or random effect? Use random effects
3. Should include student characteristics? Including student characteristics has little impact
4. Should include teacher/etc. characteristics? Including teacher characteristics can be devastating
7
Copyright © 2011, SAS Institute Inc. All rights reserved.
Methods of Analysis
Educational Value-Added Analysis of Covariance Modelswith Error in the Covariates
8
Copyright © 2011, SAS Institute Inc. All rights reserved.
SimulationSpecifics included the following: 1,000 data sets 8 teachers 20 students per teacher 1 response (y) and 8 prior test scores (x): Mean=50, Std. Dev.=21 (“NCE” scores) True score correlation matrix (see next slide)
Based on empirical data Assuming a reliability of .85
Student poverty status (P) Classroom poverty status ()
9
Copyright © 2011, SAS Institute Inc. All rights reserved.
Simulation
Y X1 X2 X3 X4 X5 X6 X7 X8Y 1.00 0.80 0.70 0.65 0.66 0.74 0.67 0.59 0.65
X1 0.94 1.00 0.74 0.70 0.71 0.76 0.68 0.61 0.68
X2 0.82 0.87 1.00 0.74 0.76 0.69 0.76 0.68 0.75
X3 0.76 0.82 0.87 1.00 0.77 0.65 0.68 0.72 0.71
X4 0.78 0.84 0.89 0.91 1.00 0.64 0.72 0.67 0.74
X5 0.87 0.89 0.81 0.76 0.75 1.00 0.73 0.70 0.76
X6 0.79 0.80 0.89 0.80 0.85 0.86 1.00 0.74 0.75
X7 0.69 0.72 0.80 0.85 0.79 0.82 0.87 1.00 0.75
X8 0.76 0.80 0.88 0.84 0.87 0.89 0.88 0.88 1.00
Correlation Matrix• Upper right: empirical observed score correlations• Lower left: true score correlations (reliability = 0.85)
10
Copyright © 2011, SAS Institute Inc. All rights reserved.
Simulation
Teacher effects: normal, mean=0, std. dev.=12 = {-10.80, -18.67, -1.91, -5.95, +5.95, +1.91, +18.67, +10.80}
Student scores: : true score associated with previous-year score in same subject as response : previous-year scores in other subjects two-years-previous score in same subject as response : two-years-previous scores in other subjects using true-score correlation matrix
11
Copyright © 2011, SAS Institute Inc. All rights reserved.
SimulationAssignment of students to teachers: Sort students by the following value: Scenario #1: , random assignment.
Assign students by increasing value of s,1st 20 students to 1st teacher, 2nd 20 students to 2nd teacher, etc.
Scenario #2: .65,lower-achieving students are assigned to less effective
teachers.Assign students by increasing value of s.
Scenario #3: .65,lower-achieving students are assigned to more effective
teachers.Assign students by decreasing value of s.
12
Copyright © 2011, SAS Institute Inc. All rights reserved.
Simulation
Student assignment to poverty status: 50% of students designated as “in poverty” (), based on
lower achieving-students are more likely to be “in poverty”
Classroom poverty level () = classroom average of
13
Copyright © 2011, SAS Institute Inc. All rights reserved.
Analysis
Abbreviations for the Method of Estimation FE: fixed-effects ANCOVA
Estimated effects constrained to sum to zero
RE: random-effect ANCOVA Estimated effects automatically sum to zero
IV: instrumental variables model with fixed effects Estimated effects constrained to sum to zero
EIV: errors-in-variables model with fixed effects Estimated effects constrained to sum to zero
14
Copyright © 2011, SAS Institute Inc. All rights reserved.
AnalysisAbbreviations for the Covariates _1x: one covariate {} _2x: two covariates {} _3x: three covariates {} _4x: four covariates {} _8x: eight covariates _1i: one covariate {} with as instrument _: one covariate {}, the true score for _ : covariates and (student-level poverty indicator) _: covariates and (teacher-level poverty indicator) _: covariates and (classroom average true score) _: covariates _AOG: analysis of gains, response , no covariates
15
Copyright © 2011, SAS Institute Inc. All rights reserved.
Analysis
Software: SAS/STAT® 9.2
MIXED procedure for random-effects models
SAS/ETS® 9.2 SYSLIN procedure for fixed-effects models including IV and EIV RESTRICT statement enforces sum-to-zero constraint For FE and IV, analyze raw data For EIV, analyze corrected variance-covariance matrix
16
Copyright © 2011, SAS Institute Inc. All rights reserved.
Results
Educational Value-Added Analysis of Covariance Models with Error in the Covariates
17
Copyright © 2011, SAS Institute Inc. All rights reserved.
Results = true effect for j-th teacher = estimated effect for j-th teacher in the k-th simulated dataset = mean square error for j-th teacher
Bias = lack of accuracy (how far off target the estimates are) Variance = lack of precision (how scattered the estimates are)
18
Copyright © 2011, SAS Institute Inc. All rights reserved.
Results
Models with and , and Bias2 + Variance = MSE
Model Scenario 1 Scenario 2 Scenario 30.1+2.9 = 2.9 0.1+2.8 = 2.9 0.1+2.9 = 3.1
3.9+17.4 = 21.3 54.0+18.6 = 72.7 51.7+18.9 = 70.6
3.7+17.1 = 20.9 67.9+11.7 = 79.6 68.2+11.5 = 79.7
12.1+25.8 = 37.9 73.4+14.2 = 87.6 73.1+14.1 = 87.1
19
Copyright © 2011, SAS Institute Inc. All rights reserved.
Mean Square Error = Bias2 (red) + Variance (blue)
Scenario 1: Random Assignment of Students to Teachers
20
Copyright © 2011, SAS Institute Inc. All rights reserved.
Lessons from Scenario 1
Little or no bias in the displayed models RE beats FE (slightly): bias vs. variance trade-off AOG, IV, and EIV much noisier than ANCOVA Use 3 or more prior test scores in ANCOVA Adding student-level poverty indicator has little effect
(RE_T versus RE_TP) Adding teacher-level covariates is devastating
both for bias and variance
21
Copyright © 2011, SAS Institute Inc. All rights reserved.
Mean Square Error = Bias2 (red) + Variance (blue)
Scenario 2: Assign Lower-Achieving Students to Less Effective Teachers
22
Copyright © 2011, SAS Institute Inc. All rights reserved.
Lessons from Scenario 2
Non-random assignment can produce serious bias ANCOVA with 3+ prior scores reduces bias (and
variance) RE beats FE: less variance and less bias! AOG is now biased -- IV and EIV are not MSE for AOG, IV, and EIV are still higher than ANCOVA Adding student-level poverty indicator has little effect
(RE_T versus RE_TP) Adding teacher-level covariates is devastating
23
Copyright © 2011, SAS Institute Inc. All rights reserved.
Mean Square Error = Bias2 (red) + Variance (blue)
Scenario 3: Assign Lower-Achieving Students to More Effective Teachers
24
Copyright © 2011, SAS Institute Inc. All rights reserved.
Lessons from Scenario 3
Non-random assignment can produce serious bias FE now beats RE due to much less bias ANCOVA with 3+ prior scores reduces bias (and
variance) AOG, IV, and EIV have little or no bias MSE for AOG, IV, and EIV comparable to 3+ ANCOVA Adding student-level poverty indicator has little effect
(RE_T versus RE_TP) Adding teacher-level covariates is devastating
25
Copyright © 2011, SAS Institute Inc. All rights reserved.
Summarizing the Bias in Scenarios 2 and 3
In Scenario 2, bias and shrinkage act in opposite directions, giving the RE model an advantage.
In Scenario 3, bias and shrinkage act in the same direction, giving the RE model a disadvantage.
26
Copyright © 2011, SAS Institute Inc. All rights reserved.
Additional Models (not shown)
Fixed-effect models with after-the-fact shrinkage ANCOVA, AOG, IV, and EIV results are intermediate between FE and RE
27
Copyright © 2011, SAS Institute Inc. All rights reserved.
In Conclusion, the Answers are…
1. What is the most useful approach for dealing with the bias caused by measurement error? Use multiple ANCOVA (at least 3 prior test scores)
2. Should be a fixed or random effect? Use random effects
3. Should include student characteristics? Including student characteristics has little impact
4. Should include teacher/etc. characteristics? Including teacher characteristics can be devastating
28
Copyright © 2011, SAS Institute Inc. All rights reserved.
Afterword: The Excluded Contender
All student test scores on “left hand side” Why excluded: no covariates, so no bias due to
measurement error in covariates Examples:o AOGo EVAAS “MRM” including “layered model”o HLM cross-classified modelo RAND complete persistence modelo Some panel data models
Repeated Measures Analysis of Variance (RM-ANOVA)
29
Copyright © 2011, SAS Institute Inc. All rights reserved.
Afterword
Lord’s Paradox in analysis of pre-post measurements Example: Children’s height over timeo Growth = Post minus Preo “Growth”? = Observed-Post minus Predicted-from-Pre
Scale requirements: meaningful differenceso Vertical scaleso Standardized scores (z-scores)o Normalized scores (rank-based z-scores, NCEs)
RM-ANOVA versus ANCOVA
Top Related