INEE. Ponencia Profesor Rivkin. Universidad Illinois. Estimating the Effect of Leaders on Public...
-
Upload
instituto-nacional-de-evaluacion-educativa -
Category
Education
-
view
1.553 -
download
0
description
Transcript of INEE. Ponencia Profesor Rivkin. Universidad Illinois. Estimating the Effect of Leaders on Public...
Estimating the Effect of Leaders on Public Sector Productivity: The Case
of School Principals
Eric Hanushek, Steven Rivkin, and Greg Branch
January 2013
Questions
• Does the quality of leadership explain a substantial share of the variation in organization outcomes?
• Does the variation in principal effectiveness differ by the share of low income students in a school?
• Is the pattern of teacher turnover consistent with the notion that raising the quality of teachers constitutes an important mechanism through which principals influence school quality?
• Are “effective” principals more likely to leave high poverty schools?
Methodological challenges
• Difficult to separate contributions of leaders from other factors– Observed characteristics explain little of the
variation in student performance– Typically behaviors cannot be observed or
directly related to outcomes– Semi-parametric analyses infer effectiveness
from contributions to student outcomes• Unlike the case for teachers, principal actions often
affect quality of school in future periods
Growing Literature on Principals
• Early work focused on observed characteristics– Clark, Martorell and Rockoff also consider
experience
• Other work builds on the panel data methods used by Bertrand and Schoar– raise methodological concerns
• Grissom and Loeb; Miller (2009)
Direct estimation of Variance in Principal Effectiveness
• Builds on Rivkin, Hanushek and Kain (2005) research on teacher quality– Variation in achievement increases with a
principal change
• Coelli and Green (2012) use this approach for Canada
Our Approaches
• Use value-added model with school fixed effects to estimate principal fixed effects– Compute variance of principal effectiveness
• Infer variance in principal effectiveness from relationship between year-to-year fluctuations in school value-added and principal turnover– Carefully examining sensitivity to timing
• Examine relationship between estimates of principal quality and changes in teacher quality during a principal’s tenure
• Describe principal quality differences by transition status
• Consider differences by poverty share throughout analysis
UTD Texas Schools Project
• Stacked panels of students and staff• Annual testing• Student demographic characteristics
– Divide schools by student poverty rate
• Information on staff– Role– Experience– school
• Can follow principals and students who switch schools and roles within Texas public schools
Estimation of Variation in Principal Quality-Broad Issues
• Non-random selection of principals and students into schools complicates analysis
• Tenure-quality relationship complex– Length of tenure unlikely to be monotonically related
to effectiveness• Principals likely learn from experience
– Skills– Behavior rewarded in school and district
• Principal effects on school quality likely grow in magnitude over time
• May be positive or negative
First empirical approach
• Principal by spell fixed effects based on first three years at a school
– Regress math score on lagged math score, student demographic variables, and grade by year fixed effects using aggregate data
– bias potentially introduced by unobserved school factors
Alternative Approach
• Ignore issue of tenure and use all spells
• Control for school fixed effects
• Potential bias if there are time-varying school factors not accounted for
• Estimate of variance includes differences due to tenure
Test Measurement Issues
• Random error inflates estimates of variation in principal quality
– Use Bayesian shrinkage estimator to mitigate effects of random error
– Unlike the case with the estimation of teacher quality, it is not a serious problem given adequacy of sample sizes even in small schools
• Tests focus on basic skills, so initial achievement differences may influence translation of principal quality into test score growth
• Create Z scores and re-weight observations such that average achievement in all schools aggregates over the same test distribution in terms of the share of students in each of ten deciles of the pre-test distribution
Table 3. Distribution of Principal by Spell Fixed Effects by Low Income Share
Share low inc
quartile
Standard deviation
10th 25th 75th 90th
all 0.21 -0.29 -0.15 0.11 0.22
lowest 0.16 -0.18 -0.06 0.13 0.22
2nd 0.18 -0.24 -0.14 0.09 0.19
3rd 0.21 -0.30 -0.16 0.10 0.21
highest 0.26 -0.38 -0.24 0.11 0.29
Sensitivity checks
• Shrink and reweight
• School fixed effects included in specification estimated over sample of all schools with multiple principals during period
Alternative, Test-Measurement Error Adjusted Estimates of the Variance in Principal
Effectiveness
Adjustment Neither Shrunk nor Reweighted
Shrunk Reweighted Shrunk and Reweighted
Standard deviation
0.207 0.200 0.270 0.241
Deficiencies of fixed effects
• Unobservables, even if orthogonal to principal quality, inflate variance estimate• These include changes over time in student
cohort quality and district curricula
• Not accounted for with shrinkage
Principal turnover based variance estimates
• Derive variance estimates from relationship between year-to-year changes in school average achievement and principal turnover– If principal quality matters, changes should be
larger in years in which there is a change in principal
– Builds on Rivkin et al (2005) estimates of variance in teacher effectiveness
• Variance in principal effectiveness equals additional year-to-year variation in transition years over non-transition years– Fluctuations between non-transition years
provide valid counterfactual for what would have taken place in transition years in the absence of a change in leadership
– Not valid if there is additional turbulence during transition years (e.g. Ashenfelter dip)
• If it is actually caused by principal quality, differences in school quality should be larger in non-adjacent years due to an increase over time in principal effects– Compare non-adjacent years around
transitions in order to investigate source of additional variation
specification
principal quality () in cohort y, quality of other school factors not under the control of the principal (γ)
Taking Expectation
)()(2)(2)( 22222''' s
ys
ys eEAAE y
syss
ys
yss
Assume cov(principal quality) = var(principal quality) if principal same Assume cov(principal quality) = 0 if principal different
• Regress squared year-to-year difference in school average test score gains on indicator for principal change
• Assumptions to identify within school variance in principal quality from turnover coefficient– Principal turnover orthogonal to other
unobserved changes that affect achievement– Schools draw principals from common
distributions during this period
Within School Covariance• Covariance between principal quality in
adjacent years with same principal in both years equals variance in principal quality
• Covariance between principal quality in adjacent years equals zero in schools that change principals
• Coefficient on the principal turnover indicator equals 2 times variance in principal quality
Sensitivity checks
• Add squared differences in demographic characteristics in some specifications
• Use non-adjacent years in some specifications
Results for entire sample
Timing of Comparison Adjacent year one year in between
Student Demographic and Mobility controls no yes no yes
Different Principal 0.0052 0.0048 0.0058 0.0056Coefficient (3.41) (3.16) (4.35) (4.28)
estimated standard deviation 0.051 0.049 0.054 0.053of principal quality(square root of 0.5*coefficient)
Estimated Standard Deviation by Poverty Quartile
Quartile Lowest 2nd 3rd highest
Adjacent years
0.029 0.037 0.049* 0.067
Non-adjacent years
0.027 0.035 0.057* 0.064
Principal quality and teacher turnover
• Principal may have limited control over entrants– Job security an issue, but can still exert
influence over who remains• Desirability of school for high quality teachers• Decision to move out lower performers
• Focus on effectiveness of exiting teachers and rate of turnover
Figure 1. Teacher Transitions by Principal Effectiveness and School Poverty Rate
0 .1 .2 .3 0 .1 .2 .3
Top
3rd
2nd
Bottom
Top
3rd
2nd
Bottom
Top
3rd
2nd
Bottom
Top
3rd
2nd
Bottom
Lowest Quartile Disadvantaged 2nd Quartile Disadvantaged
3rd Quartile Disadvantaged Highest Quartile Disadvantaged
Change School Change DistrictExit Sample
Quartiles Principle Effectiveness
Estimation
• Argument is that better principals are more likely to “dismiss” least effective teachers
• Data does not link students and teachers– Focus on differences in grade average value-
added within schools
• Grade with lower mean value-added is more likely to have a teacher below the “dismissal” threshold
Campus by year fixed effect regressions
• Regress share of teachers that exit grade g in school s in year y on controls and grade average value added interacted with principal quality quartile indicators
• Control for – student demographics– grade by year fixed effects– School by year fixed effects
• Potential problem of a mechanical relationship– In future plan to sever time periods
• Measure quality with data in 2nd year of spell• Examine link with teacher turnover in subsequent years
Poverty Quartiles all highest
Grade average gain*2nd quartile-0.018 -0.065
principal quality (0.89) (1.79)
Grade average gain*3rd quartile-0.029 -0.025
principal quality (1.35) (0.65)
Grade average gain*4th quartile-0.079 -0.102
principal quality (3.68) (3.16)
Table 8. Coefficients on Principal Quality Quartile-Grade Average Value-Added Interactions Using First Three Years Sample by School Poverty
Principal Transitions and Value Added
• Transitions categorized by new role and destination• New role
– Principal– Other position in school– Central office administrator
• Destination– same school– New school-same district– Central office-same district– New school-New district– Central office-new district– Exit Texas public schools
Probability Principal Remains in Same Position following 3rd Year in a School, by Quartile of Estimated Quality and
School Poverty Rate (<25 yrs ex) Principal Quality Quartile
Lowest 2nd 3rd highest
School Poverty Quartile
Lowest 59% 68% 73% 76%
2nd 52% 70% 81% 72%
3rd 44% 55% 64% 58%
Highest 63% 73% 72% 67%
Probability Principal with Fewer than 25 Years of Experience Becomes Principal in Different School following 3rd Year in a School, by Quartile of Estimated Quality and School Poverty
Rate (total probability of changing position)
Principal Quality Quartile lowest 2nd 3rd highest
School PovertyQuartile
lowest 7%(41) 6%(32) 8%(27) 9%(24)
2nd 5%(48) 8%(30) 3%(19) 12%(28)
3rd 12%(56) 11%(45) 11%(36) 15%(42)
highest 12%(37) 12%(27) 10%(28) 9%(33)
Future Work
• Estimate very flexible model with school by year fixed effects
• Use these estimates to examine whether variance in estimates of principal quality rises with tenure in a school
• Account for Ashenfelter dip in direct estimates of variance in principal quality
• Modify teacher turnover analysis
Summary
• Purposeful sorting and unobserved factors complicate estimates of leadership quality distribution
• We find substantial variation in estimates of principal quality– A one standard deviation increase in principal quality would
increase school average achievement by roughly 0.05 standard deviations (roughly half as much as a one std dev increase in teacher quality
• Least effective principals least likely to remain in a school– Often transition to other schools, particularly from a high
poverty school
• Details– Turnover based estimates ignore any between school
variation in principal quality– Find a higher quality variance in high poverty schools– Direct estimates of principal quality appear to
overstate variance, even in specifications that include school fixed effects
• Evidence is consistent with notion that the management of teacher composition is one channel through which principals influence school quality
Principal turnover
• Least effective principals least likely to remain in a school– Often transition to other schools, particularly from a
high poverty school
• Little or no evidence that the most effective principals are disproportionately likely to leave even high poverty schools