Educational Value-Added Analysis of Covariance Models with Error in the Covariates

Copyright © 2011, SAS Institute Inc. All rights reserved.

Educational Value-Added Analysis of Covariance Models with Error in the CovariatesS. Paul WrightEVAASSAS Institute Inc.

2


Introduction

Educational Value-Added Analysis of Covariance Modelswith Error in the Covariates

3


Background

What is analysis of covariance (ANCOVA)? Mathematically, the model is:

Where: : current-year score for student of teacher : vector of covariates (“predictors”), including

» prior-year test score(s)» student-level characteristics?» teacher/school/community characteristics?

: “effect” of -th teacher

4


Key Question of this Study

Question 1: If analysis of covariance (ANCOVA) is to be used for value-added modeling, and there is measurement error in the covariates, what is the most useful approach for dealing with the bias caused by measurement error?

Three options were considered: Classical errors-in-variables (EIV) regression Instrumental variables (IV) regression ANCOVA with multiple covariates

5


Additional Questions of Interest

Question 2: Should be a fixed or random effect? Question 3: Should include student characteristics? Question 4: Should include teacher characteristics?

6


In Short, the Answers are…

1. What is the most useful approach for dealing with the bias caused by measurement error? Use multiple ANCOVA (at least 3 prior test scores)

2. Should be a fixed or random effect? Use random effects

3. Should include student characteristics? Including student characteristics has little impact

4. Should include teacher/etc. characteristics? Including teacher characteristics can be devastating

7


Methods of Analysis

Educational Value-Added Analysis of Covariance Modelswith Error in the Covariates

8


SimulationSpecifics included the following: 1,000 data sets 8 teachers 20 students per teacher 1 response (y) and 8 prior test scores (x): Mean=50, Std. Dev.=21 (“NCE” scores) True score correlation matrix (see next slide)

Based on empirical data Assuming a reliability of .85

Student poverty status (P) Classroom poverty status ()

9


Simulation

Y X1 X2 X3 X4 X5 X6 X7 X8Y 1.00 0.80 0.70 0.65 0.66 0.74 0.67 0.59 0.65

X1 0.94 1.00 0.74 0.70 0.71 0.76 0.68 0.61 0.68

X2 0.82 0.87 1.00 0.74 0.76 0.69 0.76 0.68 0.75

X3 0.76 0.82 0.87 1.00 0.77 0.65 0.68 0.72 0.71

X4 0.78 0.84 0.89 0.91 1.00 0.64 0.72 0.67 0.74

X5 0.87 0.89 0.81 0.76 0.75 1.00 0.73 0.70 0.76

X6 0.79 0.80 0.89 0.80 0.85 0.86 1.00 0.74 0.75

X7 0.69 0.72 0.80 0.85 0.79 0.82 0.87 1.00 0.75

X8 0.76 0.80 0.88 0.84 0.87 0.89 0.88 0.88 1.00

Correlation Matrix• Upper right: empirical observed score correlations• Lower left: true score correlations (reliability = 0.85)

10


Simulation

Teacher effects: normal, mean=0, std. dev.=12 = {-10.80, -18.67, -1.91, -5.95, +5.95, +1.91, +18.67, +10.80}

Student scores: : true score associated with previous-year score in same subject as response : previous-year scores in other subjects two-years-previous score in same subject as response : two-years-previous scores in other subjects using true-score correlation matrix

11


SimulationAssignment of students to teachers: Sort students by the following value: Scenario #1: , random assignment.

Assign students by increasing value of s,1st 20 students to 1st teacher, 2nd 20 students to 2nd teacher, etc.

Scenario #2: .65,lower-achieving students are assigned to less effective

teachers.Assign students by increasing value of s.

Scenario #3: .65,lower-achieving students are assigned to more effective

teachers.Assign students by decreasing value of s.

12


Simulation

Student assignment to poverty status: 50% of students designated as “in poverty” (), based on

lower achieving-students are more likely to be “in poverty”

Classroom poverty level () = classroom average of

13


Analysis

Abbreviations for the Method of Estimation FE: fixed-effects ANCOVA

Estimated effects constrained to sum to zero

RE: random-effect ANCOVA Estimated effects automatically sum to zero

IV: instrumental variables model with fixed effects Estimated effects constrained to sum to zero

EIV: errors-in-variables model with fixed effects Estimated effects constrained to sum to zero

14


AnalysisAbbreviations for the Covariates _1x: one covariate {} _2x: two covariates {} _3x: three covariates {} _4x: four covariates {} _8x: eight covariates _1i: one covariate {} with as instrument _: one covariate {}, the true score for _ : covariates and (student-level poverty indicator) _: covariates and (teacher-level poverty indicator) _: covariates and (classroom average true score) _: covariates _AOG: analysis of gains, response , no covariates

15


Analysis

Software: SAS/STAT® 9.2

MIXED procedure for random-effects models

SAS/ETS® 9.2 SYSLIN procedure for fixed-effects models including IV and EIV RESTRICT statement enforces sum-to-zero constraint For FE and IV, analyze raw data For EIV, analyze corrected variance-covariance matrix

16


Results

Educational Value-Added Analysis of Covariance Models with Error in the Covariates

17


Results = true effect for j-th teacher = estimated effect for j-th teacher in the k-th simulated dataset = mean square error for j-th teacher

Bias = lack of accuracy (how far off target the estimates are) Variance = lack of precision (how scattered the estimates are)

18


Results

Models with and , and Bias2 + Variance = MSE

Model Scenario 1 Scenario 2 Scenario 30.1+2.9 = 2.9 0.1+2.8 = 2.9 0.1+2.9 = 3.1

3.9+17.4 = 21.3 54.0+18.6 = 72.7 51.7+18.9 = 70.6

3.7+17.1 = 20.9 67.9+11.7 = 79.6 68.2+11.5 = 79.7

12.1+25.8 = 37.9 73.4+14.2 = 87.6 73.1+14.1 = 87.1

19


Mean Square Error = Bias2 (red) + Variance (blue)

Scenario 1: Random Assignment of Students to Teachers

20


Lessons from Scenario 1

Little or no bias in the displayed models RE beats FE (slightly): bias vs. variance trade-off AOG, IV, and EIV much noisier than ANCOVA Use 3 or more prior test scores in ANCOVA Adding student-level poverty indicator has little effect

(RE_T versus RE_TP) Adding teacher-level covariates is devastating

both for bias and variance

21



Scenario 2: Assign Lower-Achieving Students to Less Effective Teachers

22



Non-random assignment can produce serious bias ANCOVA with 3+ prior scores reduces bias (and

variance) RE beats FE: less variance and less bias! AOG is now biased -- IV and EIV are not MSE for AOG, IV, and EIV are still higher than ANCOVA Adding student-level poverty indicator has little effect


23



Scenario 3: Assign Lower-Achieving Students to More Effective Teachers

24



Non-random assignment can produce serious bias FE now beats RE due to much less bias ANCOVA with 3+ prior scores reduces bias (and

variance) AOG, IV, and EIV have little or no bias MSE for AOG, IV, and EIV comparable to 3+ ANCOVA Adding student-level poverty indicator has little effect


25


Summarizing the Bias in Scenarios 2 and 3

In Scenario 2, bias and shrinkage act in opposite directions, giving the RE model an advantage.

In Scenario 3, bias and shrinkage act in the same direction, giving the RE model a disadvantage.

26


Additional Models (not shown)

Fixed-effect models with after-the-fact shrinkage ANCOVA, AOG, IV, and EIV results are intermediate between FE and RE

27


In Conclusion, the Answers are…

1. What is the most useful approach for dealing with the bias caused by measurement error? Use multiple ANCOVA (at least 3 prior test scores)

2. Should be a fixed or random effect? Use random effects

3. Should include student characteristics? Including student characteristics has little impact

4. Should include teacher/etc. characteristics? Including teacher characteristics can be devastating

28


Afterword: The Excluded Contender

All student test scores on “left hand side” Why excluded: no covariates, so no bias due to

measurement error in covariates Examples:o AOGo EVAAS “MRM” including “layered model”o HLM cross-classified modelo RAND complete persistence modelo Some panel data models

Repeated Measures Analysis of Variance (RM-ANOVA)

29


Afterword

Lord’s Paradox in analysis of pre-post measurements Example: Children’s height over timeo Growth = Post minus Preo “Growth”? = Observed-Post minus Predicted-from-Pre

Scale requirements: meaningful differenceso Vertical scaleso Standardized scores (z-scores)o Normalized scores (rank-based z-scores, NCEs)

RM-ANOVA versus ANCOVA

Educational Value-Added Analysis of Covariance Models with Error in the Covariates

Documents

Transcript of Educational Value-Added Analysis of Covariance Models with Error in the Covariates