Overview of the Early College High School Initiative Evaluation

Susan ColeMengli Song

Andrea BergerAmerican Institutes for Research

Presentation at the SREE 2010 Conference

March 4, 2010

About the ECHSI

Started in 2002 by the Bill & Melinda Gates Foundationo Improve postsecondary access and successo Provide opportunity to earn up to 2 years of

college credit

In fall 2009, over 200 Early College Schools (ECSs) open across the nation

About the ECHSI

Five Core Principles define an Early College School (ECS)o Target populationo Partnership with a college or universityo Integrated academic plano Student supportso Advocate for supportive policies

Evaluation Research Questions

1. What are the structural, organizational, and instructional characteristics of ECSs?

2. What are the intermediate and long-term outcomes for students attending ECSs, especially for students traditionally underserved by the postsecondary system?

Data Sources andAnalytic Methods

Qualitative datao Site visits (6 to 20 ECSs annually)

Quantitative datao School survey (entire ECS population annually)o Student survey (35 schools and 2,102 students

in 2007-08)

Analytic Methodso School survey- descriptive statistics and

regressiono Student survey- hierarchical linear modeling

RQ1: Characteristics of ECSs

157 ECSs in 2007-08 across 21 states and DC

65% have a 2-year public college partner

53% are located on a college campus

76% have admissions criteria

• Fewer than 100 students per grade on average

Source: 2007-08 ECHSI school survey

RQ1: Who do ECSs serve?

Minority- 67%

Free or Reduced-Price Lunch Eligible- 59%

Limited English Proficiency- 10%

1st Generation College Going- 46%

Sources: 2007-08 ECHSI school survey; 2007-08 ECHSI student survey

RQ1: What are characteristics of college classes in ECSs?

91% of ECSs have at least some students in college classes

61% of students have taken at least one college classo Half of these college classes are in core academic

areas

o 66% of these college classes are taken on a college campus

Source: 2007-08 ECHSI school survey; 2007-08 ECHSI student survey

RQ2: How are ECS students doing?

On average, ECSs had average daily attendance (ADA) rates over 94%.

74% of ECS students were proficient in ELA; 67% were proficient in math.

o ECSs outperformed districts in both ELA and math proficiency rates by 7%.

Sources: 2007-08 ECHSI school survey; publicly available school data

RQ2: How are ECS students doing?

Grade-to-grade progression rateso 85% on average for 9th-to-10th-grade o 87% for the 12th-to-graduation or grade 13

Cumulative Promotion Index (CPI)o For the 12 ECSs with data, the average CPI

was 66%o Exceeded districts’ CPI by an average of

14%

Sources: 2007-08 ECHSI school survey; publicly available school data

RQ2: How are ECS students doing?

Graduates earned about a semester to a year of college credit while enrolled in the ECS.

o ECS survey: about 8 college classes

o ECS transcripts: about 10 college classes

Sources: 2007-08 ECHSI school survey; 2004-05 to 2007-08 student transcript data

RQ2: How are ECS students doing?

Most ECS students enrolled in college after graduation.

The ECS average for college enrollment is equivalent to or exceeds national averages.

College Enrollment: ECS & National Average

Sources: 2007–08 ECHSI school survey; NCES, 2007

ECS(2006-07)

National(2003-04)

Postsecondary Enrollment

88% 72%

4-year IHE 45% 44%

2-year IHE 43% 28%

RQ2: How are ECS students doing?

Minor differences between subgroups on various outcomes

1st generation college-going students most consistent gapo Lower high school and college GPAso Lower educational aspirationso Lower satisfaction with the ECS

Sources: 2007-08 ECHSI student survey; 2007-08 ECHSI school survey; publicly available school data

RQ2 How are ECS students doing?

Students at ECSs located on a college campus had higher outcomes than ECSs at other locations on:

o ADA (95% and 93%)

o 9th- to 10th-grade progression rates (89% and 81%)

o Achievement proficiency rates (relative to their district)

• ELA- 14% above and 1% below

• Math- 16% above and 1% below

Sources: 2007-08 ECHSI student survey; 2007-08 ECHSI school survey; publicly available school data

Summary of findings

What do we know about ECS students?

Students are largely from populations underrepresented in postsecondary institutions.

ECS students are outperforming districts on state assessments.

Students are accumulating college credit.

On many outcomes, students in ECSs located on college campuses are doing better than student attending ECSs not located on college campuses.

Sources: 2007-08 ECHSI student survey; 2007-08 ECHSI school survey;

ECHSI Impact Study: Research Questions

1. Do ECS students have better outcomes than they would have had at other high schools?

2. Do the effects of ECSs on student outcomes differ for different types of schools?

3. Do the effects of ECSs differ for students with different background characteristics?

ECHSI Impact Study: Design and Sample

Overall Study Design: 3-year multisite RCT

Sample: – Sites: Up to 21 ECSs that use lottery-based

admission in at least one year between 2003-04 and 2007-08

– Students (up to 5 cohorts per school): Treatment: offered space in ECS through lottery Comparison: not offered space in ECS through lottery Extant data sample: about 4,600 treatment and 6,700

comparison students across 21 sites Student survey sample: about 1,800 students

ECHSI Impact Study: Outcome Measures

High School Outcomes:– High school persistence and graduation– ACT/SAT test taking and performance – College credit accrual while in high school

College Outcomes:– Highest educational attainment – Degrees and certificates earned

ECHSI Impact Study: Data Collection

Extant Data Collection:– Data: lottery records, student and school

characteristics and student outcomes– Sources: ECSs, districts, and/or subcontractors– March 2010 through November 2011

Student Survey: – Designed to capture additional student

outcome information for all sites– The only source for student outcomes for some

sites– June to November 2011

ECHSI Impact Study: Analytic Methods for RQ1

RQ1 (Overall Impact):

Intent-to-treat analyses

Multilevel models with sites as random effects (students nested within sites)– HGLM for binary outcomes– Multilevel survival analyses– HGLM for multinomial outcomes

ECHSI Impact Study: Analytic Methods for RQs2 &3

RQ 2 (Differential Impact on different types of schools):

Assessed by incorporating measures of ECS characteristics as site-level predictors into the main impact model

RQ 3 (Differential Impact on different types of students):

Assessed by incorporating ECS-by-student-characteristic interactions as student-level predictors into the main impact model