Conducted by
16
Evaluation of the DC Opportunity Scholarship Program: Achievement Impacts After Three Years of Treatment Exposure Conducted by Westat, University of Arkansas, Chesapeake Research Associates, Georgetown University Presented by Patrick Wolf, Principal Investigator “Reversion to the Mean or Does Dosage Matter?” IES Annual Research Conference June 8, 2009 Washington, DC
description
Evaluation of the DC Opportunity Scholarship Program: Achievement Impacts After Three Years of Treatment Exposure. Conducted by Westat, University of Arkansas, Chesapeake Research Associates, Georgetown University Presented by Patrick Wolf, Principal Investigator - PowerPoint PPT Presentation
Transcript of Conducted by
Evaluation of the DC Opportunity Scholarship Program:
Achievement Impacts After Three Years of Treatment Exposure
Conducted by
Westat, University of Arkansas, Chesapeake Research Associates, Georgetown University
Presented by
Patrick Wolf, Principal Investigator
“Reversion to the Mean or Does Dosage Matter?” IES Annual Research Conference
June 8, 2009Washington, DC
Presentation Overview The Program The Study Achievement Impacts in Year 3 Achievement Impacts Over Time Dosage in the Context of School
Choice
The Opportunity Scholarship Program DC School Choice Incentive Act signed in early 2004 $14 million/yr for 1,700 student vouchers worth up
to $7,500 Eligible students must be:
DC residents entering grades K-12 income <= 185 percent of poverty
Priority to applicants from “needs improvement” (SINI) schools
68 participating private schools in D.C.
The Impact Evaluation Randomized Control Trial (RCT) Design: baseline
eligible applicants assigned by lottery to be offered (“treatment” group) or not offered (“control” group) a scholarship
Impact Sample: First two cohorts in 2004 (Cohort 1) and 2005 (Cohort 2) 1,387 students assigned to treatment, 921 to control Cohort 1 outcomes lagged one year and combined with Cohort 2 Outcomes tracked annually: SAT-9 in reading and math, parent
surveys, student surveys, principal surveys (public and private)
Analysis: Regression-adjusted intent-to-treat (ITT) impact, Bloom-adjusted impact-on-treated (IOT), Instrumental Variable Analysis of effect of private schooling
DC Choice: Year 3 Achievement Impacts Overall:
Reading impact of 4.5 scale score points (.13 SD or 3.1 months of learning)
No impact in math
Subgroups by student baseline conditions:
Reading impacts for (1) non-SINI, (2) higher performing, (3) female, (4) K-8, (5) Cohort 1
No impacts for (1) SINI, (2) lower performing, (3) male, (4) 9-12, (5) Cohort 2
No impacts for any subgroups in math
Regression Adjusted Impact and Confidence Interval: Year 3 Reading
Sca
le s
core
poin
ts
Regression Adjusted Impact and Confidence Interval: Year 3 Math
DC Choice: Achievement Impacts Over Time Overall effects across 3 outcome years:
Apparent trend of cumulating impacts in reading No apparent trend in math
Subgroup reading effects across 3 outcome years: Of 5 with significant impacts in year 3, all but Cohort
1 suggest cumulating impacts Of 5 without impacts in year 3, X suggest
cumulating impacts Data not pooled across years so significance of
trends unknown
Impacts Over Three Year for
the Full SampleSca
le s
core
poin
ts
Impacts Over Three Years: SINI at Baseline Subgroups
Sca
le s
core
poin
ts
Impacts Over Three Years: Baseline Performance Subgroups
Sca
le s
core
poin
ts
Impacts Over Three Years: Gender Subgroups
Sca
le s
core
poin
ts
Impacts Over Three Years: Grade Subgroups
Sca
le s
core
poin
ts
Impacts Over Three Years: Cohort Subgroups
Sca
le s
core
poin
ts
The DC Choice Evaluation and Treatment Dosage Impact patterns differ by domain:
Math may have suffered reversion to the mean Reading impacts appear to cumulate No obvious reason for the difference
School choice interventions may require long treatment exposure to demonstrate results: Starts with adjustment to a school switch May then place student on steeper growth curve
Bring on Year 4!
Contact InfoPatrick Wolf, Ph.D.Professor and 21st Century Endowed Chair in
School ChoiceDepartment of Education ReformCollege of Education and Health Professions201 Graduate Education BuildingUniversity of ArkansasFayetteville, AR 72701Phone: 479-575-2084FAX: [email protected]
