Under Pressure: Job Security, Resource Allocation, and Productivity in Schools under NCLB
description
Transcript of Under Pressure: Job Security, Resource Allocation, and Productivity in Schools under NCLB
Under Pressure: Job Security, Resource Allocation, and Productivity in Schools under NCLB
Randall RebackBarnard College, Columbia University
Jonah E. RockoffColumbia Business School
Heather L. SchwartzRAND
Excellent research assistant provided by Tamara Lalovic Cox & Elizabeth Davidson
Funding provided by the Spencer Foundation and the U.S. Institute of Education Sciences
October, 2012
Policy Background: No Child Left Behind
● First took effect during spring of 2003● Requires states to adopt school accountability systems that
determine whether public schools satisfy Adequate Yearly Progress (AYP)
● Schools’ AYP status based on – % of students taking and demonstrating proficiency on
statewide exams in math and reading– both overall and student subgroups pass rates– state-specific exams, standards, and rules
● Consequences of failing AYP– Escalating sanctions, including
• Inter-district public school choice • Funds redirected to after-school tutoring for students from
low income families– States are also required to publish annual school report cards,
so schools’ AYP status can also affect school prestige and local property values.
2
Overview of Our Study
● Assemble NCLB related data and outcomes for all schools nationwide for first 2 years of NCLB
● Find schools that were at substantial risk of failing AYP and therefore faced pressure under NCLB–Considerable variation across states
● Use several external data sets to investigate the impact of NCLB pressure on teachers and students
3
Preview of Main Results
● Teacher-level results: Accountability pressure– increases teachers’ concerns about their job security and
decreases their expected career length– decreases frequency of instruction in low-stakes subjects
(e.g., science) and decreases time on whole-class instruction
● Student-level results In schools facing the strongest short term accountability incentives, students…
• score higher on low-stakes readings exams• perform at least as well on low-stakes math and science
exams• do not experience any negative effects in terms of their
enjoyment of math & reading or their anxiety about testing
4
NCLB Policy Variation
Interaction of four features significantly influences the likelihood that a school fails AYP:
(1) state rules for the numerical significance of student subgroups
(2) within-school heterogeneity, which influences how many student subgroups are numerically significant
(3) the generosity of the state’s confidence intervals(4) the generosity of the state’s safe harbor provisions
Several other idiosyncratic policies also matter
Methodology (Part 1)
● Predicting which schools were on the margin of making AYP– Use 2002 data on test scores and demographics (pre-
policy) to predict probability of making AYP in 2003 and 2004 using separate probit regressions for each state
● Define schools as on the AYP margin if……at least one group moderate probability of passing…no group has a very low probability of passing
● Define schools as below the AYP margin if any group has a very low probability of passing
6
Difference-in-differences, An Example
● Take two pairs of schools, from NJ and PA
● Even though pairs are observably similar, differences in state rules create arguably exogenous variation in NCLB pressure
7
School name StateTitle I
eligiblePercent
poorPercent
whitePercent black
Percent Hispanic Size
Hamilton School PA yes 89% 5% 83% 10% 454
Richland Elementary PA no 18% 94% 1% 3% 472
School name StateTitle I
eligiblePercent
poorPercent
whitePercent black
Percent Hispanic Size
On margin of failing AYP?
Hamilton School PA yes 89% 5% 83% 10% 454 no
Richland Elementary PA no 18% 94% 1% 3% 472 no
Bradley Elementary NJ yes 87% 4% 81% 15% 418 yes
Upper Pittsgrove NJ no 16% 93% 4% 3% 419 no
School name StateTitle I
eligiblePercent
poorPercent
whitePercent black
Percent Hispanic Size
Hamilton School PA yes 89% 5% 83% 10% 454
Richland Elementary PA no 18% 94% 1% 3% 472
Bradley Elementary NJ yes 87% 4% 81% 15% 418
Upper Pittsgrove NJ no 16% 93% 4% 3% 419
Difference-in-differences
Regressions control for – state fixed effects– various school-,child-,household- level
controls• including student-level and school-level
pre-NCLB test performance– simulated % of states where school would be
on AYP margin – simulated % of states would be below margin
8
Evidence from RAND-NCLB Survey of Math Teachers
9
0 10 20 30 40 50 60 70 80 90 100
I focus more effort on students who are close to proficient
I spend more time teaching general test-taking strategies
I focus more on topics emphasized in the state test
I focus more on the state standards
I spend more time teaching content
I search for more effective teaching methods
% Agreement Among Math Teachers
High (N=1074) Marginal (N=224) Low (N=19)
Evidence from RAND-NCLB Survey of Principals
10
0 10 20 30 40 50 60 70 80 90 100
Distributed commercial test preparationmaterials
Distributed released copies of the state testor test items
Encouraged or required teachers to spendmore time on tested subjects and less time
on other subjects
My school can attain the AYP targets for2003-04
My school can attain the AYP targets forthe next five years
My school’s AYP status accurately reflectsthe overall performance of our school
% Agreement Among Principals
Marginal (N=21) High (N=104)
Data Used in Second Stage Analysis
● Early Childhood Longitudinal Survey (ECLS): students in the kindergarten cohort of 1998-99, followed up at 1st, 3rd, 5th (and 8th) grade– Nationally representative, but not from all 50 states– Rich data set with low stakes exams. Used adaptive
testing to capture wide range of abilities.● SASS 2003-04: nationally representative sample of
schools and teachers within schools● Restricted-use data allow use to link ECLS/SASS
observations with variables on NCLB pressure
11
Effects of NCLB on Teachers
12
Job Security Concerns for
Relatively Inexperienced
Teachers
Plan to Teach until
Retirement
Work Hours in a Typical Week for
“Generalist” Teachers
Work Hours in a Typical Week
for “Specialist” Teachers
Main Sample: NCLB Sample WaveOn AYP Margin 0.101* -0.057* -1.84** 4.15**
(0.057) (0.033) (0.84) (1.81)
Below AYP Margin
0.170** -0.154*** -2.65** 4.89**
(0.076) (0.048) (1.13) (2.31)
Falsification Sample: Pre-NCLB Sample WaveOn AYP Margin -.071 0.039 0.28 0.63
(.068) (0.036) (0.66) (2.16)Below AYP
Margin-.025 0.059 0.79 -0.40(.093) (0.049) (0.86) (2.73)
Effects of School being on AYP Margin on Student Learning
13
Reading Score Math Score Science Score Anxiety About
Testing
0.08 ** 0.00 0.04 -.09***
(0.04) (0.04) (0.03) (.04)
Reading Enjoyment Math Enjoyment
0.03 0.13 *(0.07) (0.08)
ECLS: Other Findings
● Effects are not very different for– Students in the subgroups under pressure– “Bubble students”– Students from low-income families
● Stronger positive effects for math score gains in states without strong accountability prior to NCLB (related to Dee and Jacob, 2011)
14
Conclusions
● States vary widely in rates of making AYP– Cross-state variation in student academic
aptitude or in exam difficulty explains relatively little of this variation
● Short term NCLB pressure…– Threatens perceived job security of teachers– Influences teachers’ time use – Has positive net effects on average student test
score growth on low-stakes reading exams– Does not have any short-term negative effects
on outcomes for the average student!
15
No Data Left Behind!
Our NCLB data is publicly available from our “No Data Left Behind” website:
http://www7.gsb.columbia.edu/nclb/
16
17
First stage: Pr(Made AYP in ’03 & ’04)
● 1 in 5 marginal; 1 in 10 low probability● Large differences in actual AYP outcomes
18
● Large variation across groups in contributions to risk of AYP failures
Table 3: First Stage Predictions of AYP OutcomesPanel A: School-wide Outcomes OnPercent of Schools 22.9%
Above Below 10.1% 67.0%
Table 3: First Stage Predictions of AYP OutcomesPanel A: School-wide Outcomes OnPercent of Schools 22.9%Percent Actually Made AYP 2003 and 200436.4%
Below 10.1%6.9% 86.0%
67.0%Above
Panel B: Subgroup Outcomes
Math ReadingOverall School Population 92.5% 7.5% 9.3%
Actually made AYP in subject in '03 and '04 52.0% 52.7%Economically Disadvantaged 61.5% 14.9% 18.5%
Actually made AYP in subject in '03 and '04 54.2% 53.7%Limited English Proficient 24.0% 19.4% 37.6%
Actually made AYP in subject in '03 and '04 58.3% 49.8%Disabled 32.0% 31.0% 33.3%
Actually made AYP in subject in '03 and '04 51.0% 51.7%White 71.8% 1.4% 0.9%
Actually made AYP in subject in '03 and '04 56.7% 61.4%Black 33.9% 27.4% 24.9%
Actually made AYP in subject in '03 and '04 51.6% 53.5%Hispanic 30.6% 11.5% 19.9%
Actually made AYP in subject in '03 and '04 55.8% 54.7%Asian/Pacific Islander/Filipino 11.7% 5.6% 11.2%
Actually made AYP in subject in '03 and '04 39.2% 47.3%Native American 5.2% 11.0% 12.9%
Actually made AYP in subject in '03 and '04 53.23% 44.74%
Predicted Marginal ChanceConditional on Numerical Significance
Numerically Significant Subgroup
Panel B: Subgroup Outcomes
Math Reading Math ReadingOverall School Population 92.5% 7.5% 9.3% 2.2% 2.6%
Actually made AYP in subject in '03 and '04 52.0% 52.7% 11.0% 8.6%Economically Disadvantaged 61.5% 14.9% 18.5% 3.7% 4.6%
Actually made AYP in subject in '03 and '04 54.2% 53.7% 12.6% 13.1%Limited English Proficient 24.0% 19.4% 37.6% 4.9% 10.5%
Actually made AYP in subject in '03 and '04 58.3% 49.8% 14.2% 19.2%Disabled 32.0% 31.0% 33.3% 14.6% 17.6%
Actually made AYP in subject in '03 and '04 51.0% 51.7% 14.5% 12.7%White 71.8% 1.4% 0.9% 0.1% 0.0%
Actually made AYP in subject in '03 and '04 56.7% 61.4% 15.8% 33.3%Black 33.9% 27.4% 24.9% 9.8% 7.8%
Actually made AYP in subject in '03 and '04 51.6% 53.5% 16.5% 14.6%Hispanic 30.6% 11.5% 19.9% 1.3% 2.7%
Actually made AYP in subject in '03 and '04 55.8% 54.7% 12.3% 15.8%Asian/Pacific Islander/Filipino 11.7% 5.6% 11.2% 6.6% 9.1%
Actually made AYP in subject in '03 and '04 39.2% 47.3% 43.6% 11.5%Native American 5.2% 11.0% 12.9% 11.3% 10.4%
Actually made AYP in subject in '03 and '04 53.23% 44.74% 8.06% 14.75%
Predicted Marginal Chance Predicted Low ChanceConditional on Numerical Significance
Numerically Significant Subgroup
Panel B: Subgroup Outcomes
Math Reading Math ReadingOverall School Population 92.5% 7.5% 9.3% 2.2% 2.6%
Actually made AYP in '03 and '04 52.0% 52.7% 11.0% 8.6%Economically Disadvantaged 61.5% 14.9% 18.5% 3.7% 4.6%
Actually made AYP in '03 and '04 54.2% 53.7% 12.6% 13.1%Limited English Proficient 24.0% 19.4% 37.6% 4.9% 10.5%
Actually made AYP in '03 and '04 58.3% 49.8% 14.2% 19.2%Disabled 32.0% 31.0% 33.3% 14.6% 17.6%
Actually made AYP in '03 and '04 51.0% 51.7% 14.5% 12.7%White 71.8% 1.4% 0.9% 0.1% 0.0%
Actually made AYP in '03 and '04 56.7% 61.4% 15.8% 33.3%Black 33.9% 27.4% 24.9% 9.8% 7.8%
Actually made AYP in '03 and '04 51.6% 53.5% 16.5% 14.6%Hispanic 30.6% 11.5% 19.9% 1.3% 2.7%
Actually made AYP in '03 and '04 55.8% 54.7% 12.3% 15.8%Asian/Pacific Islander/Filipino 11.7% 5.6% 11.2% 6.6% 9.1%
Actually made AYP in '03 and '04 39.2% 47.3% 43.6% 11.5%Native American 5.2% 11.0% 12.9% 11.3% 10.4%
Actually made AYP in '03 and '04 53.23% 44.74% 8.06% 14.75%
Predicted Marginal Chance Predicted Low ChanceConditional on Numerical Significance
Numerically Significant Subgroup
Which Children Have Been Left Behind?
19
Failed AYP Made AYP Failed AYP Made AYPTotal Number of Schools 19,483 65,332 19,483 65,332Average Enrollment 727 500 1,216 800Student/Teacher Ratio 17.4 16.1 18.4 17.2Percent of Schools… School Title I Eligible 39.9% 29.4% 34.8% 27.6% Located in City 36.5% 23.1% 39.2% 27.7% Located in Urban Fringe 31.9% 33.3% 37.8% 41.1% Located in Town or Rural Area 31.5% 43.5% 23.0% 31.2% Serving Primary Grades 36.0% 64.1% 27.1% 56.3% Serving Middle Grades 27.4% 15.5% 29.4% 17.0% Serving High School Grades 27.9% 17.1% 39.1% 24.5% Ungraded/Other 8.5% 3.2% 4.4% 2.2%Percent of Students… Eligible for Free/Reduced Lunch 54.2% 39.9% 49.3% 37.0% White 46.5% 67.7% 46.2% 63.6% Black 25.6% 13.2% 24.1% 14.1% Hispanic 20.8% 13.6% 23.7% 16.3% Asian 3.4% 3.4% 4.3% 4.5%Note: Includes data from 50 states. Data on school and student characteristics taken from the Common Core of Data, 2003-04. For schools in Tennessee, data on student ethnicity taken from 1998-99 and data on free/reduced price lunch eligibility is unavailable.
Weighted by EnrollmentUnweighted