Does Remediation Help All Students?

How the Effects of Postsecondary Developmental Courses Vary by

Background and Ability

Bridget Terry LongAngela BoatmanHarvard Graduate School of Education

IES ConferenceJune 2010

Florida Remediation Studies Developmental courses at the community colleges All schools use the same placement instrument

and have the same cutoff for placement Florida Administrative Dataset: Sample: All

first-time, degree-seeking CC students who began in Fall 1997, 1998, 1999, or 2000 (over 100,000 records)

Remedial and Developmental Postsecondary Courses

Lack of academic prep is a significant barrier to success –

Nationally approx. 40% of 1st year students are placed into college remediation (55-60% at CCs)

How should this problem be addressed?

Juan Carlos Calcagno and B.T. Long – "The Impact of Postsecondary Remediation Using a Regression Discontinuity Approach: Addressing Endogenous Sorting &

Noncompliance“ and “Does Remediation Help All Students? The Heterogeneous Effects of Postsecondary Developmental Courses"

Regression Discontinuity: The Intuition

Beth and Becky are observationally similar

Compare the outcomes of Beth & BeckyCompare the outcomes of Beth & Becky

Beth and Becky both take the College Placement Test (CPT)

Beth scores just above the cut-off score

Becky scores just below the cut-off score

Beth to college-level courses Becky to remediation

Endogenous Sorting:

Beth retests to place out of remediation

Crossover: Beth takes

remediation anyway

No Show: Becky never

enrolls in remediation

Negative Effects on Passing First College-Level Course

0.2

.4.6

.8

-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Reading Cutoff

Estimated Discontinuity = -0.066(0.008)

Passing First College-Level Course

0.2

.4

-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Reading Cutoff

Estimated Discontinuity = -0.025(0.004)

2 yr Degree Completion

3035

4045

-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Reading Cutoff

Estimated Discontinuity = 1.527(0.447)

Total Credits Earned

0.2

.4.6

.8

-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Reading Cutoff

Estimated Discontinuity = -0.009(0.008)

Fall-to-Fall Retention

0.2

.4

-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Reading Cutoff

Estimated Discontinuity = -0.016(0.004)

Transfer to 4 yr

2025

3035

40

-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Reading Cutoff

Estimated Discontinuity = -1.751(0.467)

Total College-Level Credits Earned

Placed in Development

al Reading

Placed in College Level

Passing the First College-Level Course by Reading CPT Score at the

Discontinuity

Positive Effects on Total Credits Earned

0.2

.4.6

.8

-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Reading Cutoff

Estimated Discontinuity = -0.066(0.008)

Passing First College-Level Course

0.2

.4

-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Reading Cutoff

Estimated Discontinuity = -0.025(0.004)

2 yr Degree Completion

3035

4045

-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Reading Cutoff

Estimated Discontinuity = 1.527(0.447)

Total Credits Earned

0.2

.4.6

.8

-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Reading Cutoff

Estimated Discontinuity = -0.009(0.008)

Fall-to-Fall Retention

0.2

.4

-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Reading Cutoff

Estimated Discontinuity = -0.016(0.004)

Transfer to 4 yr

2025

3035

40

-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Reading Cutoff

Estimated Discontinuity = -1.751(0.467)

Total College-Level Credits Earned

Total Credits Earned by Reading CPT Score at the Discontinuity

Placed in Development

al Reading

Placed in College Level

Florida Remediation Studies Being assigned to a developmental course appears

to increase total credits completed for students on the margin of passing out of the requirement

Does not increase the completion of college-level credits or eventual degree completion.

Remediation might promote early persistence but does not necessarily help students on the margin of passing the cutoff to make long-term progress

What about students of different backgrounds? The effects were: More positive for female (compared to male),

older (compared to younger), and LEP (compared to non-LEP)

More negative for low-income student (compared to non-Pell-eligible)

Does remediation work for students with far less

preparation? Past studies (FL, OH, TX) focus on students on

the margin of needing remediation -- they do not investigate the effects of remediation on students who are extremely under-prepared (i.e. don’t have an appropriate control group)

The Tennessee Case Multiple cutoffs and changes in placement

policy over time Investigate effects of different levels of remediation using multiple RDs

Sample: Fulltime students under 21 who began at a TN public college or university in the fall of 2000“Does Remediation Work for All Students? How

Remedial and Developmental Courses Affect Students With Different Levels of Academic Preparation” – A.

Boatman and B.T. Long

Remediation in Remediation in TennesseeTennessee

ACT/SAT score

determines which

placement exam (if

any) Take Take COMPASS COMPASS Arithmetic Arithmetic

examexam

Take Take COMPASS COMPASS Algebra Algebra examexam

Take Take DevelopmentDevelopmental Algebra Ial Algebra I

Go to Go to College-College-

level level coursescourses

Take Take Remedial Remedial ArithmeticArithmetic

Take Take DevelopmentDevelopmental Algebra IIal Algebra II

COMPASS Arithmetic test

COMPASS Algebra II test

CollegeCollegeLevelLevel

Devel. Devel. Alg IIAlg II

Devel. Devel. Alg IAlg I

Remed.Remed.Arith.Arith.

Score 0-29Score 0-29Score 30-100Score 30-100

Score 0-27Score 0-27Score 28-49Score 28-49Score 50-100Score 50-100

RD #1: Developmental Algebra II vs. College-level

COMPASS Arithmetic test

COMPASS Algebra II test

CollegeLevel

Devel. Alg II

Devel. Alg I

Remed.Arith.

Score 0-29Score 30-100

Score 0-27Score 28-49Score 50-100

Total Credits after 3 years

College Credits after 3 years

Completed any Degree or Certif.

FULL SAMPLE Assigned to the Lower-level

-1.4753 (2.6086)

-6.4105** (2.6360)

-0.2333* (0.1297)

Observations 490 490 490 FOUR-YEAR INSTITUTIONS ONLY

Assigned to the Lower-level

-0.9504 (3.3765)

-4.8965 (3.4042)

-0.1176 (0.1687)

Observations 263 263 263 TWO-YEAR INSTITUTIONS ONLY

Assigned to the Lower-level

-2.1376 (4.1381)

-8.3325** (4.1780)

-0.4397** (0.2080)

Observations 227 227 227

RD #1: Effects of Recommendation to Develop. Algebra II –vs– College-Level Math

Credits Accumulated Over Three Years

5

15

25

35

45

55

65

1 2 3 4 5 6

Mea

n C

redi

ts C

ompl

eted

Semester After Initial Enrollment

Total Credits - College-Level

Total Credits - Develop Alg II

College Credits - College-level

College Credits - Develop Alg II

Diff in College Credits

Not much Difference in Total Credits

4245

4851

5457

6063

66N

umbe

r of

Cre

dits

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6COMPASS Score Relative to Cutoff

Developmental Algebra II vs. College-levelCumulative College Credits After Three Years

RD #1: Effects of Recommendation to Develop. Algebra II –vs– College-Level Math

Negative Effects in

Cumulative College Credits

RD #3: Remedial Arithmetic vs. Develop Algebra I

COMPASS Arithmetic test

COMPASS Algebra II test

CollegeLevel

Devel. Alg II

Devel. Devel. Alg IAlg I

Remed.Arith.

Score 0-29Score 30-99Score 30-99

Score 0-27Score 0-27Score 28-49Score 50-100

Total Credits after 3 years

College Credits after 3 years

Completed any Degree or Certif.

FULL SAMPLE Assigned to the Lower-level

0.0524 (1.7409)

-3.0366* (1.5681

-0.1227 (0.1009)

Observations 1279 1279 1279 FOUR-YEAR INSTITUTIONS ONLY

Assigned to the Lower-level

-3.2922 (3.0729)

-5.7550** (2.9051)

-0.2475+ (0.1597)

Observations 469 469 469 TWO-YEAR INSTITUTIONS ONLY

Assigned to the Lower-level

2.2181 (2.0909)

-1.2294 (1.8187)

-0.0140 (0.1335)

Observations 810 810 810

Math Developmental/Remedial Education

Total College Credits Completed after 3 years

RD#1: College-

Level vs. Develop. Algebra II

RD #2: Develop. Algebra II vs.

Develop. Algebra I

RD #3: Develop. Algebra I vs.

Remedial Arithmetic FULL SAMPLE

Assigned to the Lower-level

-6.4105** (2.6360)

-6.8230*** (2.5605)

-3.0366* (1.5681)

Outcome Mean 54.23 54.23 36.44 Observations 490 1562 1279

FOUR-YEAR INSTITUTIONS ONLY Assigned to the Lower-level

-4.8965 (3.4042)

-5.2957 (5.9278)

-5.7550** (2.9051)

Outcome Mean 61.73 61.73 51.33 Observations 263 697 469

TWO-YEAR INSTITUTIONS ONLY Assigned to the Lower-level

-8.3325** (4.1780)

-8.1737*** (2.7701)

-1.2294 (1.8187)

Outcome Mean 46.34 46.34 28.73 Observations 227 865 810

Smaller in MagnitudeSmaller in Magnitude

College-Level vs. Developmental vs. Remedial Writing

RD #2: Develop. vs. Remedial Writing

Total Credits Completed after 1 year

1st to 2nd Semester

Persistence

Still Enrolled in Year 3

Total Credits

after 3 yrs

College Credits

after 3 yrs

Completed Degree or

Certif.

Assigned to the Lower-level

0.4092 (0.6548)

0.0664 (0.1524)

0.2197+ (0.1378)

3.9792+ (2.7198)

-0.4107 (2.5282)

0.3057* (0.1623)

FOUR-YEAR INSTITUTIONS ONLY Assigned to the Lower-level

-1.0377 (0.7709)

-0.2335 (0.2544)

0.0713 (0.1970)

-0.6326 (4.0214)

-3.2463 (3.9283)

0.2226 (0.2011)

TWO-YEAR INSTITUTIONS ONLY Assigned to the Lower-level

1.2481 (0.9836)

0.2150 (0.2002)

0.3446* (0.1927)

6.9964* (3.6959)

1.7769 (3.3119)

0.3913 (0.2530)

CollegeComposition

Developmental Writing

Remedial Writing

Score 68-100 Score 0-27Score 28-67COMPASS

English test

Grade in first college-level course MATHEMATICS WRITING

College-Level vs.

Develop. Education (RD #1)

Upper Develop. vs.

Lower Develop. (RD #2)

Develop. vs.

Remedial Education (RD #3)

College-Level vs.

Develop. Education (RD #1)

Develop. vs.

Remedial Education (RD #2)

FULL SAMPLE Assigned to the Lower-level

0.1710 (0.1565)

-0.0068 (0.2027)

-0.0352 (0.1634)

0.0342 (0.0998)

0.5389*** (0.1598)

Observations 335 803 375 782 522 FOUR-YEAR INSTITUTIONS ONLY

Assigned to the Lower-level

0.2169 (0.2025)

0.3306 (0.4126)

0.1068 (0.2462)

0.2791* (0.1464)

0.1954 (0.1803)

Observations 185 372 183 315 231 TWO-YEAR INSTITUTIONS ONLY

Assigned to the Lower-level

0.1451 (0.2422)

-0.1503 (0.2353)

-0.0617 (0.2128)

-0.1438 (0.1356)

0.8960*** (0.2765)

Observations 150 431 192 467 291

Not causal analysis, but interesting estimates…

Summary of Tennessee Results Generally negative effects if placed in a

lower math class, but the size of the effect is smaller farther down the test score distribution

Similar effects found for Developmental and Remedial Reading

In Writing, the effects are negative at the college/developmental threshold but positive at the developmental/remedial threshold

Positive effects on first college-level grade (though these results are not causal)

Implications and Remaining Questions

Remedial and Developmental courses do not affect all types of students equally (by background, ability level, or college level)

What is the best way to offer remediation? Characteristics of strong remedial programs?

What are the effects of limitations states impose on remedial course-taking (e.g., only at CCs, time limits)?

Early Placement Testing – a preventative measure?

Contact Information

Prof. Bridget Terry Long, Ph.D.Harvard Graduate School of Education, NCPR, and NBERlongbr@gse.harvard.edu

Angela Boatman, Doctoral CandidateHarvard Graduate School of Educationangela_boatman@mail.harvard.edu