Targeted Interventions in High School: Preparing Students for
College
Zeyu Xu1, Ben Backes1, Amanda Oliveira1, Dan Goldhaber1, 2
1 American Institutes for Research
2 University of Washington
February 2020
ii
Abstract
This study adds to the currently limited evidence base on the efficacy of interventions
targeting non-college-ready high school students by examining the impact of Kentucky’s
Targeted Interventions (TI) program. We focus on interventions that students received under TI
in the senior year of high school based on their 11th grade ACT test scores. Using difference-in-
regression discontinuity and difference-in-difference designs with seven cohorts of 11th grade
students, we find that, for an average per-student cost of about $600, TI significantly reduces the
likelihood that students enroll in remedial course in both 2- and 4-year postsecondary institutions
by 5–10 percentage points in math and 3–4 percentage points in English. These effects are
similar among students who are eligible for free-or reduced-price lunch, Black and Hispanic
students, students with remediation needs in multiple subjects, and students in lower-performing
schools. Evidence also shows that TI increases the likelihood that students enroll in and pass
college math before the end of the first year by four percentage points in 4-year universities.
However, little evidence exists for TI affecting credit accumulation or persistence.
iii
Acknowledgments
The research reported here was supported by the Institute of Education Sciences, U.S.
Department of Education, through Grant R305A160188 to the American Institutes for Research
and the Kentucky Department of Education. We thank Karen Dodd, April Piper, Aaron Butler,
and Hannah Poquette from the Kentucky Department of Education and Barrett Ross from the
Kentucky Center for Statistics for their support. The opinions expressed are those of the authors
and do not represent the views of the Institute, the U.S. Department of Education, or the
Kentucky Department of Education.
iv
1
1. Introduction
Increasing the share of adults with postsecondary credentials is high on the national
education agenda. This is reflected in national initiatives such as those put forth by the Obama
administration,1 as well as in education attainment goals set by many states.2 For example, in
Kentucky, which is the focus of this study, the Kentucky Council for Postsecondary Education
(CPE) has set a goal for 60 percent of working-age Kentuckians to have high-quality
postsecondary degrees or credentials by 2030 (CPE, 2019).
Yet significant barriers to reaching these goals exist. Postsecondary graduation rates are
low, and they have been stagnant for the last decade: Only about 30 percent of full-time
community college students graduate within three years of initial enrollment and less than 60
percent of 4-year university students graduate within six years.3 Furthermore, low graduation
rates disproportionately affect disadvantaged students. For example, among community college
students, Black students are 30 percent less likely to graduate than White and Asian students.
Numerous institutional and student factors that may explain low postsecondary
attainment (Holzer & Baum, 2017), but weak academic preparation before college matriculation
is often cited as a key roadblock to student success (Bettinger, Boatman, & Long, 2013; Scott-
Clayton, 2011). Estimates, for instance, suggest that only a quarter of high school students who
are assessed based on ACT college benchmarks appear fully college ready (ACT, 2014); not
surprisingly then, the need for remediation in public postsecondary institutions is widespread,
with 68 percent of 2-year college students and 40 percent of 4-year university students taking at
least one remedial course (Chen, 2016; Bailey & Jaggars, 2016).
1 https://obamawhitehouse.archives.gov/blog/2009/07/14/investing-education-american-graduation-initiative 2 Foundations, such as the Lumina Foundation and the Bill & Melinda Gates Foundation, have also invested heavily
in programs designed to promote the share of adults with postsecondary credentials in the labor force (Bailey, 2012). 3 https://nces.ed.gov/programs/raceindicators/indicator_RED.asp
2
In this study, we evaluate the effects of a high school college readiness remediation
program in Kentucky on students’ college outcomes using regression discontinuity (RD) and
difference-in-differences (DD) methods. As the first state to adopt the Common Core State
Standards (CCSS), Kentucky is also one of the earliest states to introduce statewide high school
remediation for college. First implemented in the 2010–11 school year for high school math and
reading and one year later for English, Kentucky’s Targeted Interventions (TI) program uses
11th-grade ACT test scores to identify students for remediation. ACT tests have been mandatory
for all 11th-grade students in Kentucky since 2006. Students scoring below 19 in math, 18 in
English, or 20 in reading are considered not on track to be college-ready in those subjects when
graduating high school. These students are required to receive intervention services or remedial
support, with the goal of getting them ready to access “credit-bearing coursework without the
need for developmental education or supplemental courses” in college (CPE, 2010, p.7),
Kentucky’s working definition of college readiness.
The rationale for addressing college-readiness gaps before college matriculation is
straightforward: Traditional developmental education in college, which typically requires
academically underprepared students to take up to three levels of remedial courses before they
can start accumulating college credits, is both costly (Boatman & Long, 2018) and considered by
many to be ineffective (e.g., Valentine, Konstantopoulos, & Goldrick-Rab, 2017). As of 2017, 17
states offer statewide college-readiness remediation in high school (sometimes also called
transition interventions) and 22 states offer local high school remediation programs (Barnett,
Chavarin, & Griffin, 2018). The number of statewide programs has more than doubled since
2013 (Barnett, Fay, Trimble, & Pheatt, 2013). Florida’s College and Career Readiness Initiative
(FCCRI) (Mokher, Leeds, & Harris, 2018) and Tennessee’s Seamless Alignment and Integrated
3
Learning Support (SAILS) program (Kane et al., 2019) are prominent examples of statewide
high school remediation.
Despite the widespread interest in high school remediation for college readiness,
empirical evidence on its impact on college outcomes is relatively limited. Our study is one of
four studies that look at one of the major efforts that states are adopting to help academically
underprepared students succeed in higher education; moreover, the design and implementation of
high school remediation programs vary considerably from state to state, and currently no
evidence is available for the type of program implemented in Kentucky. As we describe in
Section 3, Kentucky’s high school remediation program has a unique combination of
programmatic and contextual features, and its experience will add significantly to the currently
limited evidence base.
Our study finds empirical evidence that Kentucky’s TI program significantly reduces the
remedial course enrollment rate in both 2- and 4-year postsecondary institutions by 5–10
percentage points in math and 3–4 percentage points in English. The TI program also boosts the
enrollment rate in credit-bearing college math during the freshman year in 4-year universities,
resulting in an overall increase of four percentage points in the likelihood of university students
passing introductory college math by the end of the freshman year. The TI program reduces the
likelihood of college remediation by similar magnitude among students who are eligible for free
and reduced-priced lunch (FRL), Black and Hispanic students, students with remediation needs
in multiple subjects, and students in lower-performing high schools. However, the TI program
has no detectable effect on other measures of progress through college such as total credits
earned by the end of the first two years in college or persistence.
4
In what follows, we first describe the theory and empirical literature on high school
remediation. This is followed by a description of Kentucky’s TI program. We next discuss the
data, analytic samples, and methods. Impact estimates are presented in Section 5. We conclude
this study with a discussion on how we may rethink high school remediation.
2. Theory and Empirical Literature
In recent years, efforts to improve the ways in which we help academically
underprepared students acquire the requisite skills and knowledge for college-level instruction
and contents have started to cut across postsecondary institutions and secondary schools.
Traditionally, college remediation was predominantly delivered by postsecondary institutions
through prerequisite developmental education models (Goudas & Boylan, 2012). Under such
models, underprepared college students must take a series of remedial courses in math and
English before proceeding to credit-bearing courses.
By most accounts, prerequisite remediation appears not to be working as conceived.
Despite total annual spending of $4 billion (Scott-Clayton & Rodriguez, 2015), traditional
remediation sequences mostly have null—and in a few cases negative—effects on subsequent
student outcomes such as passing gatekeeper courses, credit accumulation, and credential
attainment (Bailey, Jaggars, & Scott-Clayton, 2013; Melguizo, Bos, Ngo, Mills, & Prather, 2015;
Valentine, Konstantopoulos, & Goldrick-Rab, 2017).4 Critics argue that traditional prerequisite
remediation diverts resources and efforts away from making progress toward postsecondary
credential completion, and that underprepared students are further disadvantaged by the
4 For example, one study finds that, among students who were subject to lengthy remediation sequences, only 11
percent and 29 percent of students referred to math and English remediation eventually passed introductory college
courses in these subjects (Jaggars & Stacey, 2014).
5
additional burden of remediation that has delivered no apparent academic benefits (Scott-Clayton
& Rodriguez, 2015).
The desire to minimize such “diversion effects” has led some reforms to push for
remediation to be completed before college entry.5, 6 High school remediation programs are
thought to benefit students by making clearer the skills students need to succeed in college.7
Among the 17 states that have statewide remediation programs, 13 use college placement
assessments directly to screen high school seniors for intervention (Fay et al., 2017), thereby
aligning expectations across the two education systems. High school remedial interventions are
also frequently designed to target specific student deficiency areas identified by screener tests.
Finally, by helping students become college-ready before college entry, high school remediation
could potentially save students and the public costs in time and resources that college
developmental education would require.
But high school remediation also raises several concerns. The first concern is the
opportunity cost represented by regular high school courses and other learning opportunities that
are crowded out by mandatory remediation. In states and subjects where four years of courses are
required for high school diploma, remediation in the senior year tends to displace more advanced
regular courses such as pre-calculus, trigonometry and English IV (Kane et al., 2019; Mokher et
5 Widespread reforms have been underway at the postsecondary level as well. These include changes to assessment
and placement practices (such as the abolishment of placement assessments in Florida); accelerated remediation by
combining two sequential remedial courses into one semester (e.g., Denver’s FastStart), reducing the number of
remediation credit hours (e.g., Chabot’s accelerated English), or offering corequisites that allow students to enroll
directly in college-level courses with paired support; and the development of distinct math pathways to better target
what students need in their future jobs (Jaggars & Hodara, 2011). 6 Pre-college remediation also includes a few summer bridge programs. Existing empirical evidence is scant:
whereas Bailey & Jaggars (2016) found a positive correlation between CUNY’s Start program and student
outcomes, Chingos, Griffiths, & Mulhem (2017) found null effects of a low-cost online summer math program using
a randomized experiment. 7 Bettinger et al. (2013) point out that the misalignment between high school graduation requirements and the
expectations of postsecondary institutions is a main reason why many students finish high school unprepared for the
next step in their education.
6
al., 2018).8 Requirements for remediation in high school may also crowd out career technical
education courses that may have higher relevance and labor market value to some students.
Second and relatedly, the quality of remedial education that students receive at the expense of
regular high school coursework is unclear. In both West Virginia and Florida (Mokher et al.,
2018; Pheatt, Trimble & Barnett, 2016), for instance, the emphasis of transition courses leaned
toward test preparation for college placement assessments. In the case of Tennessee (Kane et al.,
2019), transition courses were modeled after college remedial courses. But college remedial
courses have been criticized for their lack of relevant content (Bailey & Jaggars, 2016) and an
instructional approach that emphasizes drilling and practice (Boatman, 2012; Grubb, 2013). In
short, whether substituting regular high school coursework with remediation leads to a net gain
in learning is unclear.
Despite the fast growth of high school remediation programs, we are aware of only three
evaluations of statewide programs. These programs differ in how students were selected for
remediation, the objective of the intervention, and the intervention curriculum (see a summary of
key differences and evaluation findings in Exhibit 1). Pheatt, Trimble, and Barnett (2016)
examine the effect of West Virginia’s Transition Mathematics for Seniors in 2011–12 and 2012–
13. Students were screened for remediation using a test administered in the junior year, and the
primary goal of the math remedial course appeared to be preparing students to take and ideally to
pass the COMPASS, the placement test that colleges used to refer students for developmental
education. Using an RD design, the study finds no significant effect on passing the college
remediation placement test and a negative effect on students’ likelihood of passing a college
gatekeeper math course.
8 Taking advanced courses in high school is shown to be associated with positive postsecondary outcomes (Attewell
& Domina, 2008).
7
Mokher et al. (2018) studies the impact of Florida’s high school remediation program,
FCCRI, on student college outcomes. FCCRI identified students for remediation in two steps:
Students who scored in the midrange of a 10th grade state standardized test (FCAT) were tested
again in the 11th grade using college placement tests. The evaluated program was mandatory
starting in 2011–12 but voluntary before then. Targeted students enrolled in transition courses in
math and English that emphasized test preparation for college placement assessments such as the
ACT or PERT. Using RD and a regression analysis, the study finds that FCCRI generally had no
detectable effect on high school graduation, college enrollment, or the likelihood of enrolling in
or passing college-level courses.
Finally, Kane et al. (2019) evaluates Tennessee’s SAILS high school remediation
program in math on student college outcomes. SAILS was launched in 2011–12, and it gradually
spread to more high schools but never reached more than 60 percent of all schools during the
study period. In SAILS schools, ACT scores from the junior year were used to target students for
both high school remediation and college placement. Successful completion of high school
remedial math automatically exempted students from developmental education in college, and so
it is not surprising that high school remediation reduced the enrollment of college remedial math
by around 30 percentage points. Exemption from college remediation translated into small
increases in college math enrollment, but it failed to increase the passing rate of college math or
accelerate college credit accumulation. A post-intervention assessment also shows no significant
improvement in intervention students’ math skills.
It is worth noting that in all three states reviewed above, high school remediation
programs have changed in important ways over time. In West Virginia, the math remediation
curriculum studied by Pheatt et al (2016) was dropped after three years of full implementation. In
8
Florida and Tennessee, the role of high school remediation diminished significantly after
changes to developmental education at the college level. The abolishment of mandatory college
placement testing in Florida in 2014 and the statewide introduction of corequisite developmental
education in Tennessee in 2015 made high school remediation in those states less relevant,
especially when the high school programs’ goals were limited primarily to test preparation or to
fulfill college remediation requirements.
3. Targeted Interventions in Kentucky
In 2009, the Kentucky General Assembly passed Senate Bill (SB) 1 calling for a
complete overhaul of Kentucky’s assessment and accountability system for PreK–12 education.
Since then, the Commonwealth has embarked on a system of education reform—known as
“Unbridled Learning”—designed to better monitor, document, and support progress toward the
goal of college- and career- readiness. In response to SB1 and the strategic priorities
subsequently established by the Kentucky Board of Education (KBE),9 the Kentucky Department
of Education (KDE) and the Kentucky Council on Postsecondary Education (CPE) jointly
developed the Targeted Interventions (TI) program (see Kentucky Statutes KRS158.6453 and
KRS158.6459, and regulation 704 KAR3:305).
The TI program is designed to monitor student progress on college- and career- readiness
proficiencies, screen students who need help, and provide interventions that are targeted to fulfill
those needs. This program has been in effect since the 2010–11 school year. Performance
benchmarks are established for students in Grades 8, 10, and 11 using the EXPLORE, PLAN,
9 In 2011, the KBE established four strategic priorities under Unbridled Learning: Next Generation Learners, Next
Generation Professionals, Next Generation Support Systems, and Next Generation Schools and Districts. Together,
these four priorities form the basis of Kentucky’s new accountability model, with all of its components fully
implemented starting with the 2014–15 school year.
9
and ACT tests, respectively.10 College readiness is benchmarked in three subject areas: math,
English and reading. The program was gradually phased in over three years, starting with high
school math and reading in 2010–11 and followed by high school English in 2011–12 and
middle school math, reading, and English in 2012–13.
Students scoring below the benchmarks are targeted to receive intervention services in
their senior year. School districts determine the details of TI interventions, but each intervention
cycle typically includes a diagnostic pretest, the development of instructional targets and their
associated formative assessments, direct delivery of instruction, and a posttest. Supplemental
instruction is often delivered through transition courses that are tailored toward individual
students’ deficiency areas and aligned with the Kentucky Core Academic Standards (KY’s
Common Core–aligned education standards). Transition courses can be either integrated into an
existing course or a stand-alone course, and interventions can be delivered during regular school
hours or as extended school services (ESS). Based on phone interviews with program
administrators and surveys of teachers involved in supplemental instruction delivery, most
transition courses make use of online curricula such as those developed by Dreambox, ALEX,
and IXL in combination with teacher-developed materials, and test preparation does not appear
to be the primary goal of these remedial interventions. More details about programmatic features
and implementation details can be found in a companion cost study of the TI program conducted
by Levin et al. (2020).
10 All three assessments were developed by ACT, Inc. as an integrated series of assessment called the Educational
Planning and Assessment System (EPAS). EXPLORE was designed for 8th and 9th grade and PLAN was designed
for the 10th grade. Each assessment provides predicted score ranges for subsequent assessments. EXPLORE and
PLAN have been replaced by the ACT ASPIRE system since 2015. See
https://files.eric.ed.gov/fulltext/ED510457.pdf for more details.
10
Although the design of the TI programexpressly states the intent to intervene early, our
current study focuses on 11th-grade students. This is because intervention reporting is mandatory
for targeted 11th-grade students only. Using reported intervention data, we were able to detect—
as one would expect if the program had been implemented as designed—a sharp jump in TI
participation rates among students who score just below the benchmarks relative to students who
score just above (see more details in Section 4). By comparison, intervention reporting is not
mandatory for students in the 8th and 10th grades, and we failed to detect any sharp jumps in
intervention participation rates around corresponding cutoff scores based on voluntarily reported
intervention data.11 It is unclear whether the lack of discontinuity in intervention participation
rates around cutoff scores is due to reporting issues, program implementation noncompliance, or
both, rendering it impossible to interpret any “program impact” we may or may not find among
8th- and 10th-grade students. In addition, our study also focuses on math and English
interventions only because student performance in these two subjects determines whether or not
a student would be placed directly into college math and English courses that are required for
graduation.
The first row of Table 1 presents the number of 11th-grade students who scored at or
above the ACT benchmarks (19 for math and 18 for English) and those who scored below the
benchmarks in spring 2014 and 2015.12 More than 87,000 high school juniors took the ACT for
the first time. Among those, more than 60 percent failed to meet the math benchmark and 40
percent missed the English benchmark.
About 56–57 percent of students who scored below the cutoff in either subject are
documented to have participated in TI. On the other hand, between one-fifth to one-quarter of
11 Data available upon request. 12 These are the first two cohorts of 11th-grade students for whom TI intervention data are available.
11
students who scored at or above the ACT benchmarks also participated in TI (Table 1, Row 2).
Unfortunately, we do not know what may have led to the observed program participation
noncompliance (both “no shows” and crossovers). Communications with TI administrators
suggest that other factors such as teacher input could have been considered in conjunction with
ACT scores to refer students for TI. Students who failed the benchmarks on the first try might
also retake the test to meet the benchmarks. Staff and parental resistance to high school
remediation was also reported in our interviews with school administrators. Other possible
factors like intervention exemptions and under-reporting cannot be ruled out.
The rest of Table 1 and Table 2 present summary statistics on some of the characteristics
of TI in order to present a more concrete picture of what high school remediation looked like in
Kentucky. The bottom half of Table 1 describes the content and type of remediation that TI
participants received by student group. Students could receive remediation in multiple content
areas and forms. Among all 11th-grade students who participated in TI (column 1), 79 percent
received remediation in math and 48 percent received remediation in English. Most students
received high school remediation in the form of transition courses (63 percent), and close to one-
third of students receive services through extended school services (ESS). It is interesting to note
that half of students who failed to meet the math benchmark (column 3) also participated in
English remediation, and that 78 percent of students who failed ACT English (column 5)
received remediation in math.
Some of the characteristics of TI remediation can be quantified using individual student-
level data (Table 2). For example, an average intervention session lasted 55 minutes and was
delivered four times a week. The most frequently-reported areas of deficiency included algebraic
thinking, math reasoning, math computation, writing mechanics, and writing content. Around
12
half (55 percent for math and 48 percent for English) of TI participants in our sample were
judged by schools to have exited remediation successfully.
The context, design, and delivery of high school remediation vary considerably from state
to state (Barnett et al., 2018), and Kentucky’s TI has several unique characteristics. For example,
most (12 out of 17) states that mandate statewide high school remediation automatically exempt
students from developmental education in college after successful completion of high school
remediation (Fay, Barnett, & Chavarin, 2017). Kentucky, by comparison, is one of five states
that do not directly link high school remediation completion with developmental education
exemption in college. Automatic exemption may be more efficient, but it mechanically produces
an “effect” of reduction in college remedial course-taking without necessarily improving
students’ academic proficiency, as Kane and colleagues (2019) found in Tennessee.
In addition, high school remediation is typically designed to provide differentiated, not
additional, learning opportunities for underprepared students. Because most states (e.g., Florida
and Tennessee) require four years of math and English for high school graduation, remedial
activities crowd out regular math and English courses that high school seniors would take. By
comparison, Kentucky required only three years of math for high school graduation during the
study period, and so Kentucky’s math remediation displaced less advanced, elective math
courses (such as algebra II and college algebra). More importantly, one-third of targeted 11th-
grade students in Kentucky received intervention through extended school services that averaged
over 100 minutes per week, representing net gains in learning time outside of regular school
hours.
Finally, even though the ACT is used by both K–12 and postsecondary systems to assess
college readiness and course placement in Kentucky, benchmarks do not align perfectly.
13
Students considered college-ready by high schools may still be required to undertake
developmental education in college. For example, although Kentucky high schools consider
students who score 19 or higher on ACT math college-ready, only students scoring above 21 can
take any college-level math courses. This contrasts with some states where different placement
assessments are used for remediation referral in high school and college, or where assessment
and placement policies perfectly align.
4. Data and Methods
4.1. Data and Sample Statistics
We utilize longitudinal student-level administrative records provided by the Kentucky
Center for Statistics (KYStats) that track student progress from high school through college
between 2009 and 2018. These records include high school test scores, student background
characteristics, and transcript data from both high school and postsecondary institutions in
Kentucky, including both public and private postsecondary institutions. The data also include
student-level records that document the interventions that remedial students received. The
intervention data were first collected for the 2014 cohort of 11th-grade students, and they include
information on intervention type, content area, curriculum and intensity.
The individual student outcomes we focus on include whether students take remedial
courses in college, whether they pass math and English credit-bearing courses in college, and
persistence and credit accumulation by the second year of college. These outcomes are examined
for 2- and 4-year institutions separately.13
13 One limitation of our data is that we do not have information on out-of-state postsecondary enrollment for all
cohorts. To assess the potential impact of missing data on our study, we examine college enrollment behavior among
the 2013 cohort of 11th-grade students for whom we do have complete information on college enrollment through
the National Student Clearinghouse. We find that out-of-state postsecondary enrollment accounts for less than three
14
The analytic sample of this study consists of seven cohorts of 11th grade students who
took the ACT for the first time in the Spring of their junior year of high school between 2009 and
2015. Each cohort consists of around 43,000 students. As depicted in Table 3, the 2010−2015
cohorts of 11th grade students are subject to math TI and the 2011−2015 cohorts are subject to
English TI (“Treated” cohorts that do not have individual student level intervention data are
represented by solid dots, and cohorts that have individual level intervention data are represented
by squares). The 2009 cohort represents the pre-treatment group for math and the 2009 and 2010
cohorts represent the pre-treatment group for English (“Untreated” cohorts are represented by
open dots in Table 3).
Descriptive statistics are presented in Table 4, with each column representing a specific
sample. Among all students, 85 percent are White and 48 percent are eligible for free/reduced
price lunch (FRL) (column 1). The average ACT scores are very close to the college-ready
benchmarks (Math=18.8, Reading=19.3 and English=18.4), with a standard deviation of about 5-
6 points. The analytic samples for the RD analysis (that is, within 3 to 5 points around the ACT
cut scores) have similar characteristics, with ACT scores slightly lower than the population
average by around one point.
4.2. Methods
The TI program relies on ACT test scores to refer students for supplemental and remedial
support. In a world with perfect compliance, all students with scores S below the cutoff c would
be assigned to treatment (T=1), while those scoring at or above would not (T=0 if S≥c). The
treatment status would be a discontinuous function of ACT score: T=1(S<c). Under the
assumptions that population average potential outcomes Y are continuous functions of the
percent of students in the cohort around the ACT cutoffs, and there is no apparent discontinuity at the cut point in
the likelihood of leaving the state.
15
running variable (i.e., the ACT test score) at the cutoff, and that no determinants of the potential
outcomes change abruptly at the cutoff other than treatment assignment, the causal effect of TI
on students within the immediate vicinity of cutoff c could be reliably estimated using RD. This
can be represented by regression equation (1), where k(.) is a function of the ACT score of
student i, 𝑆𝑖, that is centered around the cutoff such that negative values indicate scores below
cutoff.
Not all students scoring below the cutoff participate in TI (no-shows) and some students
scoring above the cutoff receive treatment (cross-overs). Figure 1 depicts these two types of
noncompliance using math intervention data from the 2014 and 2015 cohorts. Although the ACT
score does not predict TI participation deterministically, it still holds that the probability of
assignment to treatment changes discontinuously (by about 40 percentage points) at the cutoff.
When the intended and actual assignment to TI deviate from each other, 𝛼1 estimates the intent-
to-treat (ITT) effect of high school remediation on student outcomes 𝑌𝑖 instead of the treatment
effect of the treated (TOT).
Because ITT reflects the overall effect that a state could realistically expect from similar
programs, and because we have intervention data for only two cohorts of 11th grade students,
our study focuses on the ITT effect instead of the TOT effect that would be estimated using a
fuzzy RD.14 The decision to focus on ITT is also related to two additional challenges in the
current setting that have led us to use difference-in-RD as the preferred method to estimate the TI
impact.
14 The TOT effect of TI on student outcomes is equivalent to the ITT effect scaled by the difference in remediation
participation rates between students scoring just above and below the cutoff.
𝑌𝑖 = 𝛼0 + 𝛼1𝑇𝑖 + 𝑘(𝑆𝑖) + 𝛼2𝑇𝑖 ∗ 𝑘(𝑆𝑖) + 𝜀𝑖. (1)
16
The first challenge is related to the discreteness of the running variable and the resulting
uncertainty associated with potential specification errors in the RD model. ACT scores are
typically reported in whole number scores in a range of 1-36. Even with finer-grained underlying
scores, the distribution of ACT scores is clearly lumped at several discrete score points (see Kane
et al. [2019] which obtained finer-grained ACT scores for one cohort). When the running
variable is discrete, it becomes infeasible to compare averages within arbitrarily small
neighborhoods around the cutoff (Card & Lee, 2008). Impact estimates would instead have to
rely on parametric assumptions about the relationship between the outcome and the running
variable near the cutoff. Following the recommendation of Card and Lee (2008), researchers
typically widen the confidence interval around the impact estimate to account for the additional
uncertainty brought about by parametric assumptions. This is typically achieved by clustering the
standard error at each distinct value of the running variable (CRV).15 With additional, pre-
treatment data, however, we can also estimate a similar RD “effect” in the pre-treatment sample
and take the difference between the pre- and post-RD estimates. To the extent that the underlying
relationship between ACT scores and student outcomes in the absence of TI has not changed
over time, difference-in-RD can produce specification-robust estimate of high school
remediation on student outcomes.16
The second challenge has to do with the alignment between high school college-readiness
benchmarks and course placement policy at the postsecondary level in Kentucky public
15 The clustered variance estimator is known to be biased when the number of clusters is small (McCaffrey & Bell,
2002), and more recent research provides convincing empirical and theoretical evidence that CRV standard errors
generally understate the uncertainty related to specification errors and lead to substantially worse coverage
properties than Eicker-Huber-White (EHW) heteroskedasticity-robust standard errors (Kolesár and Rothe, 2018).
Because of the issues associated with CRV standard errors, we follow Kolesár and Rothe’s advice and use EHW
standard errors when making inferences about estimated TI impacts on student outcomes. 16 Difference-in-RD, sometimes also called difference-in-discontinuity, has been implemented to study a wide range
of public policy topics. See, for example, Aucejo, Romano, & Taylor (2019); Dantas, Duarte, Neto, & Sampaio
(2018); Jackson (2017); Grembi, Nannicini, & Troiano (2016); Xie (2015); Bettendorf, Folmer, & Jongen (2014).
17
institutions. Because students scoring above and below the math and English cut scores on the
ACT also face different college remediation requirements, it is clear that TI participation is not
the only “treatment” that differentiates students around the cutoffs. For example, in math,
students scoring in the 16-18 range must (if they fail to retake and pass college placement
assessments) take lower level remedial math courses in college (e.g., Basic Algebra and
Mathematical Literacy) than students scoring in the 19-21 range, who are able to directly enroll
in certain credit-bearing math courses with paired supplemental support (e.g., College Algebra
Workshop). Any estimated RD effect on student college outcomes by Equation (1) would reflect
the combined effect of both high school and college remediation. This is the type of scenario
(i.e., the presence of confounding treatments around the cutoff) that typically calls for the use of
difference-in-RD (Eggers, Freier, Grembi, & Nannicini, 2018). Because placement policies into
college remedial sequences have remained constant for all 11th grade cohorts during the study
period, we can use difference-in-RD to net out the effect that is due to differential college
remediation requirements under the assumption that such effect is constant across all student
cohorts.
We conduct the usual validity checks on RD setup. The critical identifying assumption in
an RD design is the continuity assumption: that absent the treatment, expected student outcomes
would remain smooth through the cutoff. This assumption implies that determinants of potential
outcomes other than treatment status are also smooth around the cutoff. Given these
assumptions, any discontinuity in student outcomes can only be caused by the treatment.
One way the continuity assumption may be violated is through the manipulation of the
running variable such that students of different characteristics sort themselves into treatment and
control groups. If TI were viewed as an extra burden, for example, more students might be
18
expected to show up just at or above the cutoff than just below to avoid TI. This type of
manipulation is unlikely because ACT tests are assessed without any teacher, student or principal
influence and it is almost impossible for students to precisely target scores in the close vicinity of
the cutoff.17 The distribution of ACT test scores, centered around the cutoff, indeed shows no
sign of score manipulation as it is smooth through the cutoff (Figure 2).18
Because students are unable to precisely achieve a score at or below the cutoff, we should
not expect student attributes to change sharply around the cutoff. While this assumption cannot
be definitively proven, as many individual and family characteristics are unobservable to the
researchers, we can test for continuity of observable characteristics such as student demographic
characteristics and FRL eligibility. As shown Table 5, within a bandwidth of three points around
the cutoff, we do not generally detect significant discontinuity in student attributes around the
cutoff scores with the exception of fewer females among students who scored just at or above the
math benchmark than just below. However, controlling for gender does not meaningfully change
our estimated TI effects.
In addition to difference-in-RD, we also estimate a DD model in order to explore the
potential effect of TI on students further away from the cutoff. The DD model:
compares the differences in student outcomes between students in different ACT score groups
after the implementation of TI with similar differences before TI. 𝛼2 estimates the DD impact on
student outcomes and 𝑇𝑖𝑡 can accommodate various student groups depending on their score
17 Some students took the ACT more than once before the 12th grade. Unobserved factors, such as motivation, that
influence test re-taking behavior may affect both the likelihood of passing the threshold after retesting and future
education outcomes. To avoid this type of endogenous sorting, we use the required sitting for ACT in March of
students’ junior years scores as the running variable when all students take the test at the same time. 18 We rely on graphical inspection because McCrary’s test for distributional discontinuity (2008) relies on local
linear regressions that might lead to incorrect inferences when the running variable is discrete (Card & Lee, 2008).
𝑌𝑖𝑡 = 𝛼0 + 𝛼1𝑇𝑖𝑡 + 𝛼2𝑃𝑜𝑠𝑡𝑖𝑡 + 𝛼2𝑇𝑖𝑡 ∗ 𝑃𝑜𝑠𝑡𝑖𝑡 + 𝜀𝑖𝑡 (2)
19
range, with the reference group defined as students scoring just above the benchmarks to mimic
the RD samples (i.e., ACT math 19−21 and ACT English 18−20).19 The validity of DD estimates
of the TI effect relies on the assumption that, in the absence of TI, pre-treatment differences
across student groups would have remained the same in the post-treatment period. In our
analysis, we include two “treatment group” indicators in the vector Tit: students scoring just
below the benchmarks (ACT math 16–18 or ACT English 15–17) and students scoring further
below the benchmarks (ACT math 13–15 or ACT English 12–14).
5. Findings
Empirical findings are presented in four sets of tables. Tables 6a-6b display group means
of student outcomes that we will refer to from time to time when discussing the estimated TI
effects in order to put those estimates into context. Tables 7−9 present the estimated TI effects on
high school completion and college enrollment rates. Although TI was not designed to affect
high school-to-college transition outcomes, it is important to examine the extent to which the
program may have altered the composition of entering college students, a potential mechanism
through which TI may affect student college outcomes. Next, conditional on college enrollment
Tables 8–9 present the estimated TI effects on various college outcomes. Finally, Table 10–12
present findings for student subgroups as well as an exploratory analysis of the correlation
between intervention type and student outcomes.
19 The “just below” group resembles the RD sample that uses a bandwidth of three, and the 𝛼2 estimate for this
group can be used as a further check on the robustness of difference-in-RD estimates. The main difference between
the DD and difference-in-RD estimates is that our DD model assumes the student outcome to be constant within
each student score group, whereas the RD framework allows for flexible relationship between the student outcome
and the test score within each score group. The DD model is more restrictive, but it may also be less risky in
situations where a limited number of data points are available to estimate a relationship. See a related discussion in
Bloom (2003).
20
5.1. High School-to-College Transition
The high school graduation rate among 11th grade students, at more than 90 percent, is
high for all student groups in the analytic sample, which is broken down by ACT test score
range, subject, and pre- and post-treatment periods in Tables 6a and 6b. The graduation rate in
our samples is higher than the published high school graduation rate for Kentucky,20 likely
because Kentucky’s compulsory school leaving age at the time was 16 and most high school
dropouts would likely have occurred before the spring of the 11th grade. Between one-third to
two-thirds of 11th grade students in our sample enrolled in college immediately after high school
graduation. Unsurprisingly, higher ACT scores are associated with higher high school graduation
rate and college enrollment rate, and the association appears the strongest between ACT scores
and the rate of enrollment in 4-year institutions.
In order to understand whether the implementation of TI induced more marginal students
to graduate high school and enroll in college, we present the estimated TI impact on these
outcomes in Table 7. The estimated effects of math TI are shown in the top panel of the table,
and the estimated effects of English TI are in the bottom panel. Difference-in-RD estimates,
presented in the first three columns of Table 7 with varying bandwidths, show no detectable
effect of high school math remediation on high school completion or college enrollment. High
school English remediation, on the other hand, appears to lower the overall college enrollment
immediately after high school graduation by 3–4 percentage points. However, the estimated
impact becomes statistically insignificant when enrollment is examined for 2- and 4-year
institutions separately. These results are largely consistent with DD estimates (the last two
20 During the study period, Kentucky graduated 88 percent of all its high school students, with nearly 70 percent of
the state’s school districts reporting graduation rates of 90 percent or higher. See
http://www.americaspromise.org/resource/all-kids-how-kentucky-closing-high-school-graduation-gap-low-income-
students
21
columns of Table 7) for students scoring just below the benchmark scores (16–18 in math and
15–17 in English). Among students with much lower scores (13–15 in math and 12–14 in
English), participation in high school remediation in either subject may have lowered their 2-year
college enrollment by 2–3 percentage points, whereas participation in high school math
remediation seems to have improved the chances of enrolling in 4-year universities. Overall, TI
does not appear to have changed the composition of entering college students in significant
ways.
5.2. College Outcomes
Figure 3 depicts the time trends of the rate of remedial course enrollment in college, the
enrollment rate in introductory credit-bearing courses and the passing rate in credit-bearing
courses. The trends are separately presented by ACT score range, subject and institution type (2-
and 4-year). Among college students who scored below the 11th-grade ACT benchmarks,
including those scoring far below the benchmarks, there is a clear decrease in the likelihood of
remedial course enrollment in the first year in college. For example, among 2-year college
students, even though students scoring just above the ACT benchmarks have been taking college
remedial courses at a relatively steady rate over a seven-year period (about 10–14 percent), the
percentage of college students scoring just below the ACT benchmarks (16–18 in math and 15–
17 in English) who enroll in remedial courses has decreased from 50 percent to 30 percent in
math and from 40 percent to 20 percent in English.21 We observe similar time trends in 4-year
universities, where the remedial course enrollment rates of students scoring above and below the
21 As the time between the original policy change and the cohort in question increases, it becomes more difficult to
attribute change in developmental course-taking patterns in college to the introduction of TI as they could reflect
other simultaneous efforts to reduce remediation at the postsecondary level. Alternatively, it is possible that the
effects of TI increased over time as the program matured and schools had more experience with implementation.
Because the biggest declines in in developmental course-taking are in the later years, omitting them from our results
yields attenuated estimates. Nonetheless, our estimates are still statistically significant when restricting the sample to
smaller windows after the policy change.
22
ACT benchmarks have converged as a result of decreasing likelihood of students scoring below
the cutoff enrolling in college remedial courses.
Low-scoring students appear to use some of the extra time gained through reduced
remediation needs to take credit-bearing courses in the freshman year, leading to an increase in
college course enrollment rate in math and, to a lesser extent, in English. However, the passing
rate of college math has not changed much for students scoring below the math benchmark
relative to those scoring above the benchmark, suggesting that not all students who were induced
to take college math were ready for those courses.
We present the causal estimates of the TI effect on student college outcomes in Table 8
for math and in Table 9 for English. The columns are organized by institution type (2-year vs. 4-
year) and model specification (difference-in-RD with three bandwidths, followed by DD
estimates for students just below and far below the ACT benchmarks). The rows are organized
by college outcomes grouped into course-taking outcomes in college (the top two panels) and
more general outcomes that describe student progress through college (the bottom panel).
Difference-in-RD estimates in Table 8 show that, among students scoring in close
vicinity around the ACT math benchmark, being referred to math remediation in high school
reduces the likelihood of college math remedial course enrollment by 5–10 percentage points in
2-year institutions and by seven percentage points in 4-year institutions. High school English
remediation has similar, albeit weaker, effects, reducing the likelihood of college remedial
English enrollment by 3–4 percentage points (Table 9). The difference-in-RD estimates can be
better understood by an inspection of Figure 4, which plots student outcomes on the y-axis
against ACT scores on the x-axis for students around the cutoff and compares their relationship
in the pre-treatment (represented by circles) period with the relationship in the post-treatment
23
(represented by triangles) period. It reveals that TI helps reduce college remediation needs by
significantly narrowing the remediation gap that existed before the introduction of TI between
students just above and below the ACT cutoffs.
The next six rows of Tables 8 and 9 present the estimated TI impact on the enrollment
and passing rate of college courses. Difference-in-RD estimates show that, among 4-year
university students scoring just below the ACT benchmark, TI increases the likelihood of
enrollment in college math courses in the freshman year by four percentage points (Table 8). For
these same students, TI also increases the overall passing rate of college math by a similar, four
percentage points by the end of both the first and the second year in college. By comparison,
there is no consistent evidence that TI has any impact on the enrollment and passing rates of
college math among 2-year college students. Instead, math TI increases the first-year enrollment
rate in college English by 3–7 percentage points, suggesting that 2-year college students may
have opted to spend the extra time saved from math remedial courses to fulfill college English
requirements. In contrast to math TI, there is no evidence that English TI has any detectable
effects on the likelihood of enrolling in or passing college courses in either English or math in
any types of institutions (Table 9).
The bottom panel in Tables 8 and 9 show the estimated TI effects on credit accumulation
by the end of the first two years in college and college persistence. There is no detectable TI
effect on any of these measures of progress through college. The lack of statistical significance is
not entirely surprising because the expected impact of TI on credit accumulation is much smaller
than what statistical power would allow us to detect. Take math TI among 4-year university
students as an example. Difference-in-RD estimates suggest a reduction of math remedial course
enrollment rate of seven percentage points (Table 8) among students scoring just below the ACT
24
threshold (i.e., 16–18 if we assume a bandwidth of three), representing a 18 percent drop in
remedial math enrollment in this student group.22 These students, who would have had to take
remedial math and earned an average of 23 credits by the end of the freshman year before the
introduction of TI (Table 6a), can now earn three additional credits. Even if all of them took and
successfully completed three additional college credits, the average number of earned credits for
this student group (i.e., ACT math 16–18) would now be 0.18*(23+3) + 0.82*23=23.54, a mere
half-credit increase (equivalent to 0.06 standard deviation in earned credits by the end of
freshman year, which is 9.09).
The jump in the TI participation rate around the ACT benchmarks is 40 percentage
points, and so the treatment effects among the treated (TOT) are likely larger than the intent-to-
treat (ITT) effects reported above. A back-of-the-envelope calculation (i.e., dividing ITT
estimates by 0.40) shows that, among students who participated in interventions, TI reduced the
likelihood of remedial course enrollment by 13–25 percentage points in math and 8–13
percentage points in English; it also increased the likelihood that treated students enrolled in and
passed college math in 4-year universities by 10 percentage points.
Tables 8 and 9 also present DD estimates of the TI impact for the same set of student
outcomes as those investigated using difference-in-RD. As discussed in the methods section,
here we compare the pre-post changes in student outcomes across student groups, with students
scoring just above the ACT benchmarks (ACT math 19–21 or English 18–20) serving as the
reference group. Each model includes indicators for two separate treatment groups: students
scoring just below the benchmarks (ACT math 16–18 or English 15–17) and students scoring far
below the benchmarks (ACT math 13–15 or English 12–14). The estimated TI impacts for both
22 Forty percent of students in the ACT math 16–18 group took remedial math in the pretreatment period (Table 6a),
and so a seven percentage points reduction is equivalent to a 100*7/40=18 percent change.
25
groups of low-scoring students are presented next to each other in Tables 8 and 9. The main
purpose of the DD estimates for the “just below” students is to check the robustness of the
difference-in-RD estimates of the TI effect, and these two sets of estimates are mostly consistent:
DD estimates confirm that, among students who score just below benchmarks, being identified
for high school remediation leads to significant decreases in the likelihood of college
remediation in both English and math. It also increases the likelihood of college math course
enrollment in 4-year universities. However, unlike difference-in-RD estimates, DD estimates fail
to detect significant improvement in the passing rate of college math among 4-year university
students.
Results from DD also suggest that the estimated TI benefits are not limited to marginal
students who just miss the ACT benchmarks. We find similar effects among lower-scoring
students as well. For students scoring well below the benchmarks, TI reduces the likelihood of
remediation in college by 10 or more percentage points in both math and English, and it
increases the enrollment rate of credit-bearing math and English by 6–7 percentage points in both
2- and 4-year institutions.23
5.3. Student and School Characteristics as Moderators
In this section, we investigate possible variation in TI impact on student college outcomes
among students who are underprepared in multiple subjects, minorities students, FRL-eligible
students, and students in low-performing schools. There are reasons to believe that high school
remediation may have differential impacts among these student subgroups. For example, the
23 If high school remediation was successful and it reduced the need for remediation in college, as is the case with
TI, it would also change the mix of students enrolled in college remedial courses simply because those who still
need remediation despite having received interventions in high school are likely the lowest-performing.
Consequently, a possible unintended consequence of a successful high school remediation program is a lower
passing rate of college remedial courses. A quick calculation based on group statistics in Tables 6a and 6b supports
this possibility: Before the implementation of TI, 65 percent of remedial math students in two-year colleges scoring
just below the benchmark passed the course compared to 57 percent after the introduction of TI.
26
summary statistics in Table 1 show that many students who fail the ACT benchmark in one
subject also tend to receive interventions in another subject. The demand for time and resources
is likely higher for students who need remediation in multiple subjects than students who miss
the benchmark in only one subject, potentially lowering the effectiveness of intervention in any
single subject. On the other hand, since academic proficiency (and deficiency) is often correlated
across subjects and cross-subject benefits of interventions are widely documented (e.g., Master,
Loeb & Wyckoff, 2017), interventions in multiple subjects may reinforce one another and
strengthen the effects of each intervention.
TI may also have differential impact among minority and FRL-eligible students. This is
because part of the effect of high school remediation is thought to be achieved through labeling
students as not on track to be college-ready (Mokher et al., 2018). However, being labelled as not
ready for college will motivate higher levels of student effort only when success is considered a
goal that is attainable and when resources to help those students to catch up are available (Ou,
2012; Papay et al., 2011). These conditions are often lacking for FRL and minority students. In
addition, research suggests that Black students may be particularly susceptible to discouragement
effect from being labelled as not college-ready (Dougherty, 2015).
Finally, the effectiveness of TI could potentially vary by school performance level. The
percentage of students failing to meet the ACT benchmark at the school level ranges from 0 to
100. It is plausible that schools with large numbers of students failing to meet benchmarks could
become overwhelmed by the need to tailor interventions to each student, so the intervention
would become less targeted. On the other hand, higher demand for remediation may create
economies of scale that allow schools to secure resources that are unaffordable to schools with
few remediation students.
27
In Tables 10 and 11, we present results that investigate the potential heterogeneity of TI
impact on student outcomes by the four student and school characteristics discussed above. To
limit the size of the tables, we choose to focus on three first-year college outcomes that are most
directly targeted by the TI program: the likelihood of students enrolling in college remedial
courses and the likelihood of enrolling in and passing credit-bearing college math and English.
These tables are structured in a way similar to Tables 8 and 9, with columns representing model
specifications. We test for statistical equivalence between subgroup estimates and their
corresponding population estimates, and bold and italicize cells where estimates are significantly
different.
Most subgroup estimates of the TI effects are qualitatively similar to the average effects
reported in Tables 8 and 9. The most consistent finding is that TI significantly reduces the need
for college remediation in nearly all student subgroups (with the exception of remedial English
enrollment in 2-year colleges among FRL-eligible students and students who have failed
benchmarks in multiple subjects). In very few instances, subgroup estimates are statistically
different from population estimates. Specifically, math TI raises the enrollment rate of credit-
bearing math during the freshman year among FRL-eligible students in 4-year universities by
about 10 percentage points (compared to four percentage points on average), and English TI
increases the enrollment rate of credit-bearing English among Black and Hispanic students in 2-
year colleges by at least 12 percentage points (compared to an estimated zero effect on average).
In some other instances, apparent differences in point estimates between student
subgroups and the whole sample are not statistically significant due to large standard errors.
Such differences concentrate mainly among the estimated TI impact on the likelihood of passing
college-level courses. Notably, for students who fail benchmarks in multiple subjects and
28
students from high schools with the highest concentration (top quarter) of remedial students,
being identified for math TI has a null effect on the likelihood of passing college math in 4-year
universities (where the average effect is a four-percentage-point increase) and a negative effect
in 2-year colleges (where the average effect is null). Among FRL-eligible students, TI has a
stronger than average positive effect (nine percentage points) on the passing rate of college math
in 4-year universities, but a statistically significant negative effect (-7 to -8 percentage points) in
2-year colleges. For Black and Hispanic students, math TI fails to produce any detectable effects
on college outcomes in 4-year universities where the population average effects are all
significant. In contrast, in 2-year colleges where TI has null effects in the full sample, TI
increases in the likelihood of college course enrollment of Black and Hispanic students by more
than 10 percentage points in both math and English. Although the increase in the college English
enrollment rate leads to a 6–13 percentage points increase in the passing rate of college English,
a similar increase in college math enrollment fails to translate into discernible gains in the
passing rate of college math.
5.4. Intervention Type
TI interventions were reported under three categories: transition courses, extended school
services (ESS) and “other” interventions. Of particular interest to us is the impact of TI delivered
through ESS and how it compares to the effect of transition courses. This is because ESS
interventions, which average more than 100 minutes per week and affecting one-third of TI
students, provides additional learning time for students who need help. This contrasts with other
high school remediation programs (such as those evaluated in West Virginia, Florida, and
Tennessee), which typically replaces regular high school coursework with remediation without
adding additional learning opportunities. It is hardly conceivable that, without spending more
29
time to study, underprepared students would be able to catch up with their more advanced peers.
Indeed, abundant evidence shows positive causal effects of additional instruction and learning on
student outcomes.24
Unfortunately, the investigation of potential heterogeneous effect by intervention type is
hampered by two factors. First, our reliance on difference-in-RD to identify TI effects prevents
us from estimating separate TI effects for each intervention type. Moreover, a fundamental
concern is that the choice of intervention type is likely endogenous: Extra learning opportunities
may have been provided to students after regular hours for those who needed more help based on
factors that researchers cannot observe.
With these concerns in mind, we conducted an exploratory analysis by estimating an
ordinary least squares regression for each college outcome, TI subject and postsecondary
institution type, with standard errors clustered at the school level. Student outcome is regressed
on indicator variables for ESS and other interventions (with transition courses as the reference
intervention type), ACT test scores, FRL eligibility, gender and race/ethnicity. The analytic
samples include the 2014 and 2015 cohorts of 11th-grade students who participated in the TI
program and have available intervention data. Estimated coefficients on intervention type
indicators are reported in Table 12. Results are mixed. Relative to transition courses, ESS is
associated with larger increases (by four percentage points) in the passing rate of college math in
4-year universities. For English, however, ESS students are more likely than students who take
transition courses to enroll in college remedial English (three percentage points in 4-year
24 For example, Jacob and Lefgren (2004) find that summer school classes targeting low-performing students in
Chicago elementary schools increase academic achievement in reading and mathematics and that these positive
effects remain substantial at least 2 years following the completion of the program. Taylor (2014) and Cortes,
Goodman and Nomi (2015) both find that adding a remedial math course to a regular math course in middle school
has positive effects on test scores, though the immediate effects decayed quickly in a couple of years. And Figlio,
Holden and Özek (2018) find significant benefits of additional reading instruction provided through extended school
days on reading test scores.
30
institutions and five percentage points in 2-year colleges); ESS students are also four percentage
points less likely to take introductory English in 2-year institutions, but the difference is not
statistically significant.
6. Discussion and Conclusion
This study is part of a growing body of evidence that explores linkages from high school
to college. Understanding these linkages is especially important given that high school and
college are increasingly treated as a continuum as opposed to distinct educational processes in
preparing students for college-level education. One the one hand, postsecondary institutions
frequently admit students deemed not college ready, but react to the preparation level of students
with various initiatives, including the developmental course requirements used as a dependent
variable here.25 On the other hand, getting all students ready for college has become the mantra
for K-12 schools so college outcomes are an important indicator for the success of the K-12
system.
This study adds to the limited evidence about the extent to which high school remediation
can help academically underprepared students succeed in college. We find evidence consistent
with the goal set forth by Kentucky for the TI program: Being assigned to TI is estimated to lead
to reductions in the likelihood that students take remedial courses in college and increases in the
likelihood that 4-year university students will take and pass college math.
The TI program findings are sizable. The intent-to-treat estimates suggest that TI reduces
the likelihood of remedial course enrollment by 5–10 percentage points in math and 3–4
25 The co-requisite model is another strategy by which colleges react to students who are deemed not college ready.
Under this strategy, students enroll directly in college-level courses with a paired support course. In 2016, over a-
third of community colleges across the nation were offering co-requisite options in English and 16% in math (Scott-
Clayton, 2017). The adoption of the co-requisite model has continued to grow at a quick pace, and in 2018, 15 states
recommended or mandated co-requisite reforms (Rutschow, 2019). Kentucky introduced the co-requisite model in
its community college system in 2019.
31
percentage points in English, and the program is estimated to increase the likelihood that 4-year
university students take and pass college math during the freshman year by four percentage
points. Importantly, only half of the students who scored below the ACT benchmarks were
treated, suggesting treatment-on-the treated effects that are roughly twice as large. The estimated
effects are similar in terms of reducing the need for college remediation among disadvantaged
students, students in low-performing schools as well as students scoring far below college-ready
benchmarks.
These effect estimates are also sizeable relative to the cost of the program and remedial
courses (as an assumed necessity if the program did not exist). The resources associated with
developing and implementing TI amounted to about $600 per treated student (Levin et al., 2020).
By comparison, each 3-credit remedial course costs between $830 (in community colleges in
Tennessee, Belfield, Jenkins, & Lahr [2016]) and $1,500 (national average across all institution
types, Barry & Dannenberg [2016]).
On the whole, our findings suggest that the TI program helps address college readiness
issues at the front end of the college pipeline. We did not find significant evidence that it had
positive effects on progress through college (credit accumulation and college persistence). But,
these findings should be interpreted with caution given that the estimated impact on early college
outcomes may be too small to produce discernible changes in later college outcomes.
32
References
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using
instrumental variables. Journal of the American statistical Association, 91(434), 444-455.
Attewell, P., & Domina, T. (2008) Raising the bar: Curricular intensity and academic
performance. Educational Evaluation and Policy Analysis, 30(1), 51–71.
Aucejo, E., Romano, T., & Taylor, E. S. (2019). Does Evaluation Distort Teacher Effort and
Decisions? Quasi-Experimental Evidence from a Policy of Retesting Students. CEP
Discussion Paper No. 1612. Centre for Economic Performance.
Bailey, T. (2012). Can community colleges achieve ambitious graduation goals? In A. Kelly &
M. Schneider (Eds.), Getting to graduation: The completion agenda in higher education
(pp. 73–101). Baltimore, MD: The Johns Hopkins University Press.
Bailey, T. & Jaggars, S. (2016). When college students start behind. The Century Foundation.
Bailey, T., Jeong, D. W., & Cho, S. W. (2010). Referral, enrollment, and completion in
developmental education sequences in community colleges. Economics of Education
Review, 29(2), 255-270.
Barnett, E., Fay, M., & Pheatt, L. E. (2016). Implementation of high school-to-college transition
courses in four states. New York, NY: Community College Research Center, Teachers
College, Columbia University.
Barnett, E. A., Fay, M. P., Trimble, M. J., & Pheatt, L. (2013). Reshaping the College
Transition: Early College Readiness Assessments and Transition Curricula in Four
States. A State Policy Report. New York, NY: Community College Research Center,
Teachers College, Columbia University.
33
Barnett, E. A., Chavarín, O., & Griffin, S. (2018). Math Transition Courses in Context: Preparing
Students for College Success. CCRC Research Brief. Community College Research
Center, Teachers College, Columbia University.
Barry, M. N., & Dannenberg, M. (2016). Out of pocket: The high cost of inadequate high schools
and high school student achievement on college affordability. Education Reform Now.
Belfield, C., Jenkins, P. D., & Lahr, H. E. (2016). Is corequisite remediation cost-effective?
Early findings from Tennessee. (CCRC Research Brief 62). New York, NY: Columbia
University, Teachers College, Community College Research Center.
Bettendorf, L. J., Folmer, K., & Jongen, E. L. (2014). The dog that did not bark: The EITC for
single mothers in the Netherlands. Journal of Public Economics, 119, 49-60.
Bettinger, E. P., Boatman, A., & Long, B. T. (2013). Student supports: Developmental education
and other academic programs. The Future of Children, 23(1): 93–115.
Bloom, H. S. (2003). Using “short” interrupted time-series analysis to measure the impacts of
whole-school reforms: With applications to a study of accelerated schools. Evaluation
Review, 27(1), 3-49.
Boatman, A. (2012). Evaluating Institutional Efforts to Streamline Postsecondary Remediation:
The Causal Effects of the Tennessee Developmental Course Redesign Initiative on Early
Student Academic Success. An NCPR Working Paper. National Center for Postsecondary
Research, Columbia University.
Boatman, A., & Long, B. T. (2018). Does Remediation Work for All Students? How the Effects
of Postsecondary Remedial and Developmental Courses Vary by Level of Academic
Preparation. Educational Evaluation and Policy Analysis, 40(1), 29-58.
34
Boylan, H. R., & Trawick, A. R. (2015). Contemporary developmental education: Maybe it's not
as bad as it looks. Research & Teaching in Developmental Education, 31(2), 26.
Card, David and David S. Lee. (2008). Regression Discontinuity Inference with Specification
Error. Journal of Econometrics, 142 (2): 655-674.
Calcagno, J. C., & Long, B. T. (2008). The impact of postsecondary remediation using a
regression discontinuity approach: Addressing endogenous sorting and
noncompliance (No. w14194). National Bureau of Economic Research.
Chao, J. C., & Swanson, N. R. (2005). Consistent estimation with a large number of weak
instruments. Econometrica, 73(5), 1673-1692.
Chen, X. (2016). Remedial Coursetaking at U.S. Public 2- and 4-Year Institutions: Scope,
Experiences, and Outcomes (NCES 2016-405). U.S. Department of Education.
Washington, DC: National Center for Education Statistics. Retrieved [10/11/2019] from
http://nces.ed.gov/pubsearch.
Chingos, M. M., Griffiths, R. J., & Mulhern, C. (2017). Can low-cost online summer math
programs improve student preparation for college-level math? Evidence from randomized
experiments at three universities. Journal of Research on Educational
Effectiveness, 10(4), 794-816.
Cortes, K. E., Goodman, J. S., & Nomi, T. (2015). Intensive math instruction and educational
attainment long-run impacts of double-dose algebra. Journal of Human Resources, 50(1),
108-158.
Dantas, R. N., Duarte, G., Neto, R. D. M. S., & Sampaio, B. (2018). Height restrictions and
housing prices: A difference-in-discontinuity approach. Economics Letters, 164, 58-61.
35
Dougherty, S. M. (2015). Bridging the discontinuity in adolescent literacy? Mixed evidence from
a middle grades intervention. Education Finance and Policy, 10(2), 157–192.
Edgecombe, N. (2011). Accelerating the Academic Achievement of Students Referred to
Developmental Education. CCRC Working Paper No. 30. Community College Research
Center, Columbia University.
Eggers, A. C., Freier, R., Grembi, V., & Nannicini, T. (2018). Regression discontinuity designs
based on population thresholds: Pitfalls and solutions. American Journal of Political
Science, 62(1), 210-229.
Fay, M. P., Barnett, E., & Chavarín, O. (2017). How states are implementing transition
curricula: Results from a national scan. New York, NY: Columbia University, Teachers
College, Community College Research Center.
Figlio, D., Holden, K. L., & Ozek, U. (2018). Do students benefit from longer school days?
Regression discontinuity evidence from Florida's additional hour of literacy
instruction. Economics of Education Review, 67, 171-183.
Flory, M., & Cramer, E. (2017). Participation in Kentucky's College Preparatory Transition
Courses: An Update. REL 2017-211. Regional Educational Laboratory Appalachia.
Goudas, A. M., & Boylan, H. R. (2012). Addressing flawed research in developmental
education. Journal of Developmental Education, 36(1), 2–13.
Grembi, V., Nannicini, T., & Troiano, U. (2016). Do fiscal rules matter?. American Economic
Journal: Applied Economics, 1-30.
Grubb, W. N. (2013). Basic skills education in community colleges: Inside and outside of
classrooms. New York City, NY: Routledge.
36
Hahn, J., Todd, P., & Van der Klaauw, W. (2001). Identification and estimation of treatment
effects with a regression‐discontinuity design. Econometrica, 69(1), 201-209.
Imbens, G., & Kalyanaraman, K. (2012). Optimal bandwidth choice for the regression
discontinuity estimator. The Review of economic studies, 79(3), 933-959.
Jackson, C. K. (2017). The Effect of Single-Sex Education on Test Scores, School Completion,
Arrests, and Teen Motherhood: Evidence from School Transitions. NBER Working Paper
No. 22222. National Bureau of Economic Research.
Jacob, B. A., & Lefgren, L. (2004). Remedial education and student achievement: A regression
discontinuity analysis. Review of economics and statistics, 86(1), 226-244.
Jaggars, S. S., & Hodara, M. (2011). The Opposing Forces that Shape Developmental Education:
Assessment, Placement, and Progression at CUNY Community Colleges. CCRC
Working Paper No. 36. Community College Research Center, Columbia University.
Kentucky Department of Education and Kentucky Council on Postsecondary Education. (2010).
Unified strategy for college and career readiness: Senate Bill 1 (2009). Retrieved from
http://education.ky.gov/educational/CCR/Documents/CCRUnifiedPlan_draft.pdf.
Kolesár, M., & Rothe, C. (2018). Inference in regression discontinuity designs with a discrete
running variable. American Economic Review, 108(8), 2277-2304.
Levin, J., Carbuccia, M., Adelman-Sil, E., & Danks. A. (2020). Kentucky targeted intervention
program cost study. Washington, DC: American Institutes for Research.
Master, B., Loeb, S., & Wyckoff, J. (2017). More Than Content: The Persistent Cross-Subject
Effects of English Language Arts Teachers’ Instruction. Educational Evaluation and
Policy Analysis, 39(3), 429-447.
37
McCaffrey, D. F., & Bell, R. M. (2002). Bias reduction in standard Errors for linear and
generalized linear models with multi-stage samples. In Proceedings of Statistics Canada
Symposium (pp. 1-10).
McCrary, J. (2008). Manipulation of the running variable in the regression discontinuity design:
A density test. Journal of Econometrics, 142(2), 698–714.
Melguizo, T., Bos, J. M., Ngo, F., Mills, N., & Prather, G. (2016). Using a regression
discontinuity design to estimate the impact of placement decisions in developmental
math. Research in Higher Education, 57(2), 123-151.
Mokher, C. (2014). Participation and pass rates for college preparatory transition courses in
Kentucky (REL 2014–009). Washington, DC: U.S. Department of Education, Institute of
Education Sciences, National Center for Education Evaluation and Regional Assistance,
Regional Educational Laboratory Appalachia. Retrieved from
http://ies.ed.gov/ncee/edlabs.
Ou, D. (2010). To leave or not to leave? A regression discontinuity analysis of the impact of
failing the high school exit exam. Economics of Education Review, 29(2), 171–186.
Papay, J. P., Murnane, R. J. & Willett, J. B. (2011). How performance information affects
human-capital investment decisions: The impact of test-score labels on educational
outcomes. NBER Working Paper No. 17120. Cambridge, MA: National Bureau of
Economic Research.
Pheatt, L. E., Trimble, M. J., & Barnett, E. (2016). Improving the transition to college:
Estimating the impact of high school transition courses on short-term college outcomes.
New York, NY: Community College Research Center, Teachers College, Columbia
University.
38
Rutschow, E. Z. (2019). Developmental mathematics reforms. The National Academies of
Sciences, Engineering, and Medicine Workshop on Understanding Success and Failure of
Students in Developmental Mathematics. Washington, DC: Board on Science Education.
Scott-Clayton, J., & Rodriguez, O. (2015). Development, discouragement, or diversion? New
evidence on the effects of college remediation policy. Education Finance and
Policy, 10(1), 4-45.
Scott-Clayton, J. (2017). Evidence-Based Reforms in College Remediation are Gaining Steam—
And So Far Living Up to the Hype. Washington DC: Brookings Institution.
Sparks, D. & Malkus, N. (2013). First-year undergraduate remedial coursetaking: 1999-2000,
2003-04, 2007-08. Statistics in Brief (NCES 2013-013). Washington, DC: National
Center for Education Statistics, Institute of Education Sciences, U.S. Department of
Education.
Taylor, E. (2014). Spending more of the school day in math class: Evidence from a regression
discontinuity in middle school. Journal of Public Economics, 117, 162-181.
Xie, E. (2015). The Influences of New Rural Pension Scheme on Elderly Labor Supply and
Well-being in Rural Areas. Journal of Finance and Economics, 8, 39-49.
39
Figures and Tables
Figure 1. Math Targeted Intervention (TI) Participation by ACT Math Score
40
Figure 2. Distribution of ACT Test Scores in Kentucky, by Subject
41
Figure 3. Trend in College Outcomes, by Institution Type, Subject, and ACT Scores: 2009-2015
42
Figure 4. College Outcomes by ACT Test Scores, by Subject and Institution Type
43
Exhibit 1. Existing Studies of Statewide High School Remediation: Program Features and
Evaluation Findings
State Subject Screener Main objective Findings
WV (Pheatt et al. 2016) Math Not directly
aligned with
college
placement
Test preparation for
college placement
assessments
No effect on passing the placement
test and negative effect on passing
college math.
FL (Mokher et al., 2018) Math
and
English
2-stage
screening, with
2nd stage using
college
placement tests
Test preparation for
college placement
assessments
No effect on the likelihood of
college remediation placement, or
the enrollment and passing rates of
college-level courses.
TN (Kane et al., 2019) Math Same as college
placement tests
Automatic exemption
from college remediation
upon successful
completion of high school
remediation
Lowered college remediation
placement rate by 30 percentage
points, small increase in enrollment
rate in college math, but no impact
on the passing rate of college math
or credit accumulation.
44
Table 1. Targeted Intervention (TI) Participation Rate, by Test Subject,
Intervention Content and Type: 2014 and 2015 11th Grade Students
Math English
All
At or
above
cut
score
Below
cut
score
At or
above
cut
score
Below
cut
score
Total students with valid ACT score 87,811 35,040 52,771 49,373 38,438
Percent with record of TI 41 18 56 28 57
Among TI participants:
CONTENT
Math (%) 79 47 85 80 78
English (%) 48 39 50 18 67
TYPE
Extended school services (ESS) (%) 31 54 26 37 27
Course (%) 63 38 69 55 68
Other (%) 20 14 22 17 23
Note: Less than 1.5 percent of TI participants have missing information on TI type. Percentages add up to
more than 100 percent because students can receive TI in more than one content area and type.
45
Table 2. Characteristics of Targeted Interventions (TI), by Subject: 2014 and
2015 11th Grade Students
Math English
Deficiency area (% of intervention by subject)
Algebraic Thinking 36 —
Math Reasoning 32 —
Math Computation 32 —
Writing Mechanics — 40
Writing Content — 32
Other TI characteristics
Duration (Minutes/Session) 55 55
Frequency (Number of Sessions/Week) 4 4
Teacher-developed materials (%) 79 81
In-person delivery (%) 83 81
Successful exit of TI (%) 55 48
— Not applicable
46
Table 3. Pre- and Post-Treatment 11th Grade Cohorts
2008-09 2009-10 2010-11 2011-12 2012-13 2013-14 2014-15 2015-16
Math ○ ● ● ● ● □ □ □
English ○ ○ ● ● ● □ □ □
Note: Years refer to the time when a student was in the 11th grade.
○ Pre-treatment
● Post-treatment without intervention data
□ Post-treatment with intervention data
47
Table 4. Descriptive Statistics of Study Samples: 2009−2015
Full
Sample
RD Sample Math RD Sample English
BW=3 BW=4 BW=5 BW=3 BW=4 BW=5
ACT-M 16-21 ACT-M 15-22 ACT-M 14-23 ACT-E 15-20 ACT-E 14-21 ACT-E 13-22
Demographics Female (%) 50 52 52 51 51 51 51
White (%) 85 85 84 84 85 85 85
Black (%) 10 9 11 11 10 10 10
Hispanic (%) 5 5 5 5 5 5 5
FRL (%) 48 49 52 52 50 50 49
Test Scores ACT Math 18.8 17.6 17.3 17.3 17.7 17.8 17.9
(4.5) (1.6) (2.1) (2.6) (2.8) (3.0) (3.1)
ACT Reading 19.3 18.7 18.2 18.1 18.0 18.1 18.2
(5.9) (4.6) (4.7) (4.9) (3.6) (3.8) (4.0)
ACT English 18.4 17.9 17.2 17.0 17.4 17.5 17.6
(6.3) (4.5) (4.9) (5.1) (1.8) (2.4) (2.8)
Note: Standard deviation in parentheses.
48
Table 5. Covariate Balance Check
Math English
All 2 year 4 year All 2 year 4 year
Female -0.04** -0.07** -0.05* 0.00 0.03 -0.02
(0.02) (0.03) (0.03) (0.02) (0.03) (0.03)
White 0.01 -0.02 0.03* 0.00 0.01 -0.01
(0.01) (0.02) (0.02) (0.01) (0.02) (0.02)
Black -0.01 0.01 -0.02 -0.00 -0.02 -0.01
(0.01) (0.01) (0.02) (0.01) (0.02) (0.02)
Hispanic 0.00 0.01 -0.01 -0.02** -0.02 0.00
(0.01) (0.01) (0.01) (0.01) (0.01) (0.01)
FRL -0.01 -0.01 0.00 -0.01 -0.05* -0.01
(0.02) (0.03) (0.02) (0.02) (0.03) (0.03)
Note: * and ** denote statistical significance at the 0.10 and 0.05 levels, respectively. Standard errors in
parentheses. RD estimates assume linear association between the running variable and outcomes with a
bandwidth of three. Results are robust to other bandwidth choices.
49
Table 6a. Group Means of Student Outcomes, by Treatment Period and ACT Math
Score Range: 2009−2015
Student Outcomes
Pre-Treatment Period (2009)
Post-Treatment Period (2010−2015)
ACT 19-21
ACT 16-18
ACT 13-15
ACT 19-21
ACT 16-18
ACT 13-15
High school-to-college transition
High school graduate (%) 98 96 92 98 96 92
Enroll in college immediately after high school (%) 71 57 34 67 52 30
2-year institution (%) 22 26 22 23 26 20
4-year institution (%) 49 31 12 44 26 9
Two-year college outcomes
Take remedial math by end of 1st year (%) 12 52 60 12 42 54
Pass remedial math by end of 1st year (%) 9 34 31 7 24 27
Take credit-bearing math by end of 1st year (%) 71 40 15 71 45 21
Pass credit-bearing math by end of 1st year (%) 42 24 7 46 25 9
Take credit-bearing math by end of 2nd year (%) 78 51 24 78 54 30
Pass credit-bearing math by end of 2nd year (%) 51 34 13 53 33 15
Total credits earned by end of 1st year 22 17 12 24 19 13
Total credits earned by end of 2nd year 31 24 15 31 24 16
Return for 2nd year of college (%) 66 61 51 64 57 48
Four-year university outcomes
Take remedial math by end of 1st year (%) 11 40 54 9 29 41
Pass remedial math by end of 1st year (%) 7 25 30 5 18 23
Take credit-bearing math by end of 1st year (%) 67 46 28 68 51 33
Pass credit-bearing math by end of 1st year (%) 42 25 11 43 27 11
Take credit-bearing math by end of 2nd year (%) 76 60 43 77 64 47
Pass credit-bearing math by end of 2nd year (%) 51 36 20 52 37 20
Total credits earned by end of 1st year 27 23 18 28 24 19
Total credits earned by end of 2nd year 43 36 28 43 36 27
Return for 2nd year of college (%) 83 77 69 79 73 65
50
Table 6b. Group Means of Student Outcomes, by Treatment Period and ACT English
Score Range: 2009−2015
Student Outcomes
Pre-Treatment Period (2009−10)
Post-Treatment Period (2011−2015)
ACT 18-20
ACT 15-17
ACT 12-14
ACT 18-20
ACT 15-17
ACT 12-14
High school-to-college transition
High school graduate (%) 97 95 94 97 96 94
Enroll in college immediately after high school (%) 66 55 40 62 49 35
2-year institution (%) 26 28 25 26 26 23
4-year institution (%) 40 27 16 35 23 13
Two-year college outcomes
Take remedial English by end of 1st year (%) 5 30 43 2 18 29
Pass remedial English by end of 1st year (%) 3 19 27 1 11 18
Take credit-bearing English by end of 1st year (%) 82 63 44 85 68 54
Pass credit-bearing English by end of 1st year (%) 59 43 28 62 47 34
Take credit-bearing English by end of 2nd year (%) 86 71 54 88 74 62
Pass credit-bearing English by end of 2nd year (%) 64 51 37 66 53 42
Total credits earned by end of 1st year 21 17 13 22 18 15
Total credits earned by end of 2nd year 28 22 17 28 22 18
Return for 2nd year of college (%) 64 59 53 61 55 50
Four-year university outcomes
Take remedial English by end of 1st year (%) 1 23 44 0 13 29
Pass remedial English by end of 1st year (%) 0 15 28 0 8 18
Take credit-bearing English by end of 1st year (%) 86 80 72 87 84 79
Pass credit-bearing English by end of 1st year (%) 58 51 41 60 52 45
Take credit-bearing English by end of 2nd year (%) 90 87 82 91 89 85
Pass credit-bearing English by end of 2nd year (%) 64 58 50 65 58 51
Total credits earned by end of 1st year 25 22 19 27 23 20
Total credits earned by end of 2nd year 39 34 29 40 34 28
Return for 2nd year of college (%) 79 75 71 75 71 65
51
Table 7. Estimated Effect of Targeted Interventions (TI) on High School to College
Transition Outcomes, By Intervention Subject and Model
Difference-in-RD estimates
DD estimates BW=3 BW=4 BW=5
Based on ACT math cutoff ACT 16-18 ACT 13-15
High school graduate 0.01 0.01 0.01* 0.00 -0.00
(0.01) (0.01) (0.01) (0.00) (0.00)
Enroll in college immediately after high school 0.00 0.00 -0.00 -0.00 0.01
(0.01) (0.01) (0.01) (0.01) (0.01)
Enroll in 2-year institutions 0.00 -0.00 0.00 -0.01* -0.02***
(0.01) (0.01) (0.01) (0.01) (0.01)
Enroll in 4-year institutions 0.00 0.00 -0.00 0.01 0.03***
(0.01) (0.01) (0.01) (0.01) (0.01)
Based on ACT English cutoff ACT 15-17 ACT 12-14
High school graduate 0.00 0.01 0.01 0.01 0.00
(0.01) (0.01) (0.01) (0.00) (0.00)
Enroll in college immediately after high school -0.04** -0.04** -0.03* -0.02*** -0.02***
(0.02) (0.02) (0.01) (0.01) (0.01)
Enroll in 2-year institutions -0.02 -0.02 -0.01 -0.02*** -0.03***
(0.02) (0.02) (0.01) (0.01) (0.01)
Enroll in 4-year institutions -0.02 -0.02 -0.01 -0.00 0.01
(0.02) (0.02) (0.01) (0.01) (0.01)
Note: *, ** and *** denote statistical significance at the 0.10, 0.05 and 0.01 levels, respectively. Standard errors in parentheses. Difference-in-RD estimates assume linear association between the running variable and outcomes. DD estimates use students scoring up to three points above the cutoff as the comparison and control for student ACT scores, FRL-eligibility, gender and race/ethnicity.
52
Table 8. Estimated Effect of Targeted Interventions (TI) in Math on College Outcomes
2 year 4 year
Difference-in-RD DD Difference-in-RD DD
BW=3 BW=4 BW=5 ACT 16-18
ACT 13-15 BW=3 BW=4 BW=5
ACT 16-18
ACT 13-15
College math outcomes
Take remedial math by end of 1st year -0.05* -0.09*** -0.10*** -0.12*** -0.10*** -0.07*** -0.07*** -0.07*** -0.10*** -0.14***
(0.03) (0.02) (0.02) (0.01) (0.01) (0.02) (0.02) (0.02) (0.01) (0.02)
Take credit-bearing math by end of 1st year 0.04 0.03 0.02 0.04** 0.07*** 0.04 0.04* 0.04** 0.04*** 0.06***
(0.03) (0.03) (0.02) (0.02) (0.02) (0.03) (0.02) (0.02) (0.01) (0.02)
Take credit-bearing math by end of 2nd year 0.04 -0.00 -0.00 -0.00 0.05*** 0.03 0.01 0.02 0.03** 0.06***
(0.03) (0.03) (0.03) (0.02) (0.02) (0.02) (0.02) (0.02) (0.01) (0.02)
Pass credit-bearing math by end of 1st year -0.00 -0.02 -0.03 -0.03 -0.01 0.04* 0.04* 0.04* 0.01 0.00
(0.03) (0.03) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.01) (0.01)
Pass credit-bearing math by end of 2nd year 0.01 -0.03 -0.04* -0.04*** -0.00 -0.00 -0.02 -0.02 -0.02 -0.01
(0.03) (0.02) (0.02) (0.01) (0.01) (0.02) (0.02) (0.02) (0.01) (0.01)
Cross-subject outcomes
Take credit-bearing English by end of 1st year 0.07*** 0.05** 0.03* 0.03*** 0.06*** -0.00 0.00 0.01 0.03*** 0.05***
(0.02) (0.02) (0.02) (0.01) (0.01) (0.02) (0.02) (0.01) (0.01) (0.01)
Pass credit-bearing English by end of 1st year 0.05 0.03 0.03 0.02 0.03* 0.01 0.01 0.01 0.01 0.03*
(0.03) (0.03) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.01) (0.02)
General college outcomes
Total credits earned by end of 1st year -0.03 -0.19 0.12 -0.38 -0.21 0.43 0.58 0.57 -0.04 -0.27
(0.69) (0.60) (0.56) (0.37) (0.38) (0.44) (0.39) (0.37) (0.23) (0.30)
Total credits earned by end of 2nd year 1.35 0.08 0.60 -0.69 0.12 0.94 1.16 1.12 0.46 -0.04
(1.59) (1.40) (1.30) (0.88) (0.89) (1.17) (1.04) (0.97) (0.60) (0.80)
Return for 2nd year of college 0.01 -0.01 -0.00 -0.02 -0.01 0.00 -0.00 -0.00 0.00 0.00
(0.03) (0.03) (0.02) (0.02) (0.02) (0.02) (0.02) (0.02) (0.01) (0.02)
Note: *, ** and *** denote statistical significance at the 0.10, 0.05 and 0.01 levels, respectively. Standard errors in parentheses. Difference-in-RD estimates assume linear association between the running variable and outcomes. DD estimates use students scoring up to three points above the cutoff as the comparison and control for student ACT scores, FRL-eligibility, gender and race/ethnicity.
53
Table 9. Estimated Effect of Targeted Interventions (TI) in English on College Outcomes
2 year 4 year
Difference-in-RD DD Difference-in-RD DD
BW=3 BW=4 BW=5 ACT 15-17
ACT 12-14 BW=3 BW=4 BW=5
ACT 15-17
ACT 12-14
College English outcomes
Take remedial English by end of 1st year -0.02 -0.04** -0.04** -0.11*** -0.14*** -0.00 -0.03* -0.03** -0.10*** -0.15***
(0.02) (0.02) (0.02) (0.01) (0.01) (0.02) (0.02) (0.01) (0.01) (0.01)
Take credit-bearing English by end of 1st year 0.03 -0.01 -0.01 0.02* 0.07*** 0.01 0.00 0.02 0.03*** 0.06***
(0.03) (0.02) (0.02) (0.01) (0.01) (0.02) (0.02) (0.02) (0.01) (0.01)
Take credit-bearing English by end of 2nd year 0.03 0.00 -0.00 0.00 0.03** -0.03 -0.04* -0.01 -0.02* -0.02
(0.03) (0.02) (0.02) (0.01) (0.01) (0.03) (0.02) (0.02) (0.01) (0.01)
Pass credit-bearing English by end of 1st year 0.04 0.01 -0.00 0.01 0.04*** -0.01 -0.03 -0.00 -0.01 0.02
(0.03) (0.02) (0.02) (0.01) (0.01) (0.03) (0.02) (0.02) (0.01) (0.02)
Pass credit-bearing English by end of 2nd year 0.01 0.00 -0.01 -0.00 0.01 -0.02 -0.02 -0.01 -0.01 0.00
(0.03) (0.02) (0.02) (0.01) (0.01) (0.03) (0.02) (0.02) (0.01) (0.01)
Cross-subject outcomes
Take credit-bearing math by end of 1st year 0.03 0.00 -0.01 0.02 0.05*** 0.02 0.02 0.03 0.01 -0.01
(0.03) (0.02) (0.02) (0.01) (0.01) (0.03) (0.02) (0.02) (0.01) (0.02)
Pass credit-bearing math by end of 1st year -0.03 -0.04 -0.04 -0.02 -0.02 0.01 0.01 0.02 -0.01 -0.03**
(0.03) (0.02) (0.02) (0.01) (0.01) (0.03) (0.02) (0.02) (0.01) (0.01)
General college outcomes
Total credits earned by end of 1st year 0.62 0.05 -0.04 -0.10 0.28 -0.40 -0.42 0.07 -0.31 -0.20
(0.63) (0.52) (0.46) (0.27) (0.29) (0.52) (0.43) (0.38) (0.22) (0.27)
Total credits earned by end of 2nd year 0.85 0.08 -0.23 -0.28 0.28 -2.26 -2.29** -0.77 -0.98* -0.91
(1.45) (1.21) (1.07) (0.64) (0.66) (1.39) (1.16) (1.03) (0.58) (0.71)
Return for 2nd year of college 0.02 0.01 0.00 -0.01 -0.01 -0.03 -0.03 -0.01 -0.02* -0.03*
(0.03) (0.02) (0.02) (0.01) (0.01) (0.03) (0.02) (0.02) (0.01) (0.01)
Note: *, ** and *** denote statistical significance at the 0.10, 0.05 and 0.01 levels, respectively. Standard errors in parentheses. Difference-in-RD estimates assume linear association between the running variable and outcomes. DD estimates use students scoring up to three points above the cutoff as the comparison and control for student ACT scores, FRL-eligibility, gender and race/ethnicity.
54
Table 10. Estimated Effect of Targeted Interventions (TI) in Math on College Outcomes, by Student and School Characteristics
2 year 4 year
Difference-in-RD DD Difference-in-RD DD
BW=3 BW=4 BW=5 ACT 16-18 ACT 13-15 BW=3 BW=4 BW=5 ACT 16-18 ACT 13-15
Students who missed ACT cutoff in addition to math
Take remedial math by end of 1st year -0.06 -0.08*** -0.09*** -0.09*** -0.08*** -0.07** -0.08*** -0.08*** -0.11*** -0.15***
(0.04) (0.03) (0.03) (0.02) (0.02) (0.04) (0.03) (0.04) (0.01) (0.02)
Take credit-bearing math by end of 1st year 0.00 -0.01 -0.02 0.01 0.04** 0.04 0.03 0.04 0.04** 0.07***
(0.04) (0.03) (0.03) (0.02) (0.02) (0.04) (0.03) (0.03) (0.02) (0.02)
Pass credit-bearing math by end of 1st year -0.03 -0.04 -0.05 -0.03 -0.01 -0.02 -0.01 -0.02 -0.02 -0.02
(0.04) (0.03) (0.03) (0.02) (0.02) (0.03) (0.03) (0.03) (0.02) (0.02)
Free/reduced price lunch eligible students
Take remedial math by end of 1st year -0.02 -0.10** -0.11*** -0.13*** -0.11*** -0.09** -0.11*** -0.11*** -0.14*** -0.15***
(0.05) (0.04) (0.04) (0.02) (0.02) (0.04) (0.04) (0.03) (0.02) (0.03)
Take credit-bearing math by end of 1st year -0.04 -0.02 -0.00 0.07** 0.10*** 0.09** 0.12*** 0.12*** 0.10*** 0.08***
(0.05) (0.04) (0.04) (0.03) (0.02) (0.05) (0.04) (0.04) (0.02) (0.03)
Pass credit-bearing math by end of 1st year -0.08* -0.08* -0.07* -0.05* -0.03 0.09** 0.09*** 0.09*** 0.05** 0.03
(0.05) (0.04) (0.04) (0.03) (0.03) (0.04) (0.04) (0.03) (0.02) (0.02)
Black or Hispanic students
Take remedial math by end of 1st year -0.04 -0.10 -0.07 -0.11** -0.11** 0.03 -0.02 -0.03 -0.10*** -0.10***
(0.10) (0.08) (0.07) (0.05) (0.04) (0.06) (0.05) (0.05) (0.03) (0.03)
Take credit-bearing math by end of 1st year 0.12 0.12 0.11 0.11** 0.13*** 0.02 0.02 0.02 0.04 0.08**
(0.10) (0.08) (0.07) (0.05) (0.04) (0.06) (0.06) (0.05) (0.03) (0.04)
Pass credit-bearing math by end of 1st year 0.00 0.01 0.03 0.03 0.04 -0.02 0.01 0.00 0.01 0.01
(0.10) (0.09) (0.08) (0.06) (0.05) (0.06) (0.05) (0.05) (0.03) (0.03)
Top quartile school in % students below cutoff
Take remedial math by end of 1st year -0.07 -0.16*** -0.17*** -0.15*** -0.08*** -0.09* -0.07* -0.06 -0.12*** -0.21***
(0.06) (0.05) (0.04) (0.03) (0.03) (0.05) (0.04) (0.04) (0.02) (0.03)
Take credit-bearing math by end of 1st year -0.01 -0.01 0.01 0.05 0.09*** -0.05 -0.00 -0.00 0.05* 0.07**
(0.06) (0.05) (0.05) (0.03) (0.03) (0.06) (0.05) (0.05) (0.03) (0.03)
Pass credit-bearing math by end of 1st year -0.07 -0.08 -0.08* -0.05 -0.02 -0.04 -0.02 -0.04 0.01 0.04
(0.06) (0.05) (0.05) (0.03) (0.03) (0.05) (0.05) (0.04) (0.03) (0.03)
Note: *, ** and *** denote statistical significance at the 0.10, 0.05 and 0.01 levels, respectively. Standard errors in parentheses. Bold and italic estimates are significantly different from the corresponding effects among all students. Difference-in-RD estimates assume linear association between the running variable and outcomes. DD estimates use students scoring up to three points above the cutoff as the comparison and control for student ACT scores, FRL-eligibility, gender and race/ethnicity.
55
Table 11. Estimated Effect of Targeted Interventions (TI) in English on College Outcomes, by Student and School Characteristics
2 year 4 year
Difference-in-RD DD Difference-in-RD DD
BW=3 BW=4 BW=5 ACT 15-17 ACT 12-14 BW=3 BW=4 BW=5 ACT 15-17 ACT 12-14
Students who missed ACT cutoff in addition to English
Take remedial English by end of 1st year -0.02 -0.04* -0.04** -0.10*** -0.14*** 0.00 -0.03* -0.03** -0.10*** -0.15***
(0.02) (0.02) (0.02) (0.01) (0.01) (0.02) (0.02) (0.02) (0.01) (0.01)
Take credit-bearing English by end of 1st year 0.04 -0.01 -0.00 0.02* 0.07*** -0.00 -0.00 0.01 0.03*** 0.06***
(0.03) (0.02) (0.02) (0.01) (0.01) (0.02) (0.02) (0.02) (0.01) (0.01)
Pass credit-bearing English by end of 1st year 0.04 0.01 -0.00 0.01 0.05*** -0.03 -0.04 -0.01 -0.01 0.03
(0.03) (0.03) (0.02) (0.01) (0.01) (0.03) (0.03) (0.02) (0.01) (0.02)
Free/reduced price lunch eligible students
Take remedial English by end of 1st year -0.01 -0.01 -0.02 -0.10*** -0.14*** -0.04 -0.11*** -0.09*** -0.14*** -0.16***
(0.04) (0.03) (0.03) (0.01) (0.02) (0.03) (0.03) (0.02) (0.01) (0.02)
Take credit-bearing English by end of 1st year 0.07* 0.04 0.02 0.04** 0.08*** 0.03 0.02 0.04 0.03* 0.05**
(0.04) (0.03) (0.03) (0.02) (0.02) (0.04) (0.03) (0.03) (0.02) (0.02)
Pass credit-bearing English by end of 1st year 0.04 0.02 0.01 0.01 0.03 -0.00 -0.03 0.01 -0.02 0.00
(0.05) (0.04) (0.03) (0.02) (0.02) (0.05) (0.04) (0.04) (0.02) (0.02)
Black or Hispanic students
Take remedial English by end of 1st year -0.18** -0.19*** -0.14*** -0.14*** -0.11*** 0.05 0.00 0.01 -0.07*** -0.11***
(0.07) (0.06) (0.05) (0.03) (0.03) (0.04) (0.03) (0.03) (0.02) (0.02)
Take credit-bearing English by end of 1st year 0.24*** 0.13** 0.12** 0.09*** 0.12*** -0.01 -0.05 -0.02 0.00 0.07***
(0.08) (0.06) (0.06) (0.03) (0.03) (0.05) (0.04) (0.04) (0.02) (0.03)
Pass credit-bearing English by end of 1st year 0.13 0.08 0.06 0.04 0.09** 0.06 0.02 0.07 0.01 0.04
(0.09) (0.07) (0.07) (0.04) (0.04) (0.07) (0.06) (0.05) (0.03) (0.03)
Top quartile school in % students below cutoff
Take remedial English by end of 1st year -0.05 -0.07* -0.05 -0.12*** -0.15*** -0.00 -0.04 -0.06** -0.12*** -0.14***
(0.05) (0.04) (0.03) (0.02) (0.02) (0.04) (0.03) (0.03) (0.02) (0.02)
Take credit-bearing English by end of 1st year 0.02 0.01 0.02 0.03 0.06*** 0.07 0.06 0.07** 0.05** 0.07***
(0.05) (0.04) (0.04) (0.02) (0.02) (0.04) (0.04) (0.03) (0.02) (0.02)
Pass credit-bearing English by end of 1st year -0.02 -0.03 -0.01 0.01 0.05* -0.04 -0.02 -0.00 -0.02 -0.00
(0.06) (0.05) (0.04) (0.03) (0.03) (0.06) (0.05) (0.04) (0.03) (0.03)
Note: *, ** and *** denote statistical significance at the 0.10, 0.05 and 0.01 levels, respectively. Standard errors in parentheses. Bold and italic estimates are significantly different from the corresponding effects among all students. Difference-in-RD estimates assume linear association between the running variable and outcomes. DD estimates use students scoring up to three points above the cutoff as the comparison and control for student ACT scores, FRL-eligibility, gender and race/ethnicity.
56
Table 12. Correlation Between College Outcomes and Intervention Type, by Institution
Type and Intervention Subject: 2014 and 2015 Cohorts of 11th Grade Students
Dependent Variable
2 year 4 year
ESS Other ESS Other
Math intervention in high school
Take remedial math by end of 1st year 0.02 0.00 -0.02 -0.02
(0.02) (0.03) (0.02) (0.02)
Take credit-bearing math by end of 1st year 0.01 0.01 0.03 0.02
(0.02) (0.02) (0.02) (0.02)
Pass credit-bearing math by end of 1st year 0.00 0.02 0.04* 0.02
(0.02) (0.02) (0.02) (0.02)
English intervention in high school
Take remedial English by end of 1st year 0.05* 0.02 0.03* -0.02*
(0.02) (0.02) (0.02) (0.02)
Take credit-bearing English by end of 1st year -0.04 0.01 0.00 -0.00
(0.02) (0.02) (0.02) (0.02)
Pass credit-bearing English by end of 1st year -0.01 0.04* 0.01 -0.02
(0.02) (0.02) (0.03) (0.03)
Note: Estimated coefficients on intervention type (ESS and other) reported in table with standard errors in parentheses. Standard errors are clustered at the school level. The regression samples include all TI participants from the 2014 and 2015 cohorts of 11th grade students for whom intervention data are available. College outcomes are regressed on indicator variables for intervention type (ESS or other, with transition courses as the reference intervention type) and control variables that include student ACT scores, FRL eligibility, gender and race/ethnicity. ** and *** denote statistical significance at the 0.05 and 0.01 levels, respectively.
Top Related