Patrick J. WolfUniversity of Arkansas

Advocates for Parental Choice SymposiumMilwaukee, WIJune 16, 2013

Latest Research Findings on School Vouchers

About Me Long and fruitful relationship with

Catholic education Rigorous training in scientific

research methods “Wolf is a well-known advocate of

vouchers. He bases his advocacy on research…” AERA Newsletter

Department of Education Reform doctoral program

Presentation Plan DC voucher findings after 4+ years Questions/discussion Milwaukee voucher findings after 4

years Questions/discussion Florida and New York City findings Discussion of voucher program

design

Overview of the DC School Choice Incentive Act Program

“3-Sector Strategy” Student Eligibility

Grade K-12, residents of DC Family income ≤ 185% Federal poverty line

Scholarship Up to $7,500 to cover tuition, school fees, and transportation Renewable for up to 5 years

Order of Priority for Receiving a Scholarship Students attending schools in need of improvement (SINI) Students in non-SINI public schools Students already attending private school

Legislative Mandate for Evaluation

Independent, using “… the strongest possible research design for determining the effectiveness of the Program.”

Key Outcomes to be Studied Academic achievement School safety Retention, graduation, college admission “...success… in expanding school choice options”

(satisfaction) Effects on schools in DC

Annual Reports to Congress (Spring 2005-Spring 2010)

Who Was Studied? 2,308 lotteried public school applicants

2,012 students still school-age 4+ years after randomization

Characteristics of participants: Average household income below $18,000 at baseline Only 6 percent have mothers with BA SAT-9 scores: 33rd percentile in reading, 31st in math 44 percent from SINI schools between 2003 and 2005

(highest priority group for scholarships)

Study Design: Randomized Control Trial

Eligible applicants assigned by lottery to be offered (“treatment” group) or not offered (“control” group) a scholarship

Each spring, evaluation team administered: SAT-9 Parent surveys Student surveys (grades 4 and higher) Principal surveys (public and private)

Three comparisons made: Treatment versus control (pure experimental) Scholarship users versus control (adjusts for decliners) Private versus public school in year 4+ (Instrumental

Variables)

Scholarship Use Over 4+ Years

26% of treatment students used scholarship throughout the study

Participation drop-off (net): averaged 22% per year

Top reasons for dropping out of OSP Child got into a charter school (22%) Lack of space (for students transitioning from K-8)

(19%) Moved out of DC (15%) Transportation problems (14%)

Students more likely to drop out of OSP: lower initial test scores, older grade levels, male, with special needs, more siblings

Schools Attended by “Analysis Sample” in 2008-09

Treatment Control0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

55

11.5

18.4

35.3

26.6

53.2

Private Charter Traditional

Enro

llmen

t in

200

8-09

Summary of Key Findings Large increase in high school graduation rates Suggestive evidence of small gains in reading

overall No impacts apparent in math Half of subgroups show reading impacts Parents more satisfied with schools, view them

as safer No impacts on student satisfaction and reports

of safety

High School Graduation Rates, Overall Sample & SINI Subgroup, Parent Reports, 2008-09

Treatment Control Treatment Control0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

82%**

70%79%*†

66%

91%**86%*†

Full Sample SINI 2003-05 Subgroup

*Statistically significant at the 95 percent confidence level.**Statistically significant at the 99 percent confidence level.† = subgroup impact result remained statistically significant after adjustments for multiple comparisons.

IOT

ITTITT

IOT

ITT & IOT Impacts of the OSP on Reading Achievement Overall, by Years After Application

0.25 0.75 1.25 1.75 2.25 2.75 3.25 3.75 4.25 4.75-4-3-2-10123456789

1.35

3.82

5.27*4.754.46

6.88

8.03 7.94

1.03

3.17

4.46*3.9

-2.41

-0.53

0.89-0.13Sc

ale

Scor

e Po

ints

After 1 year After 2 years After three years After at least 4 years

*Statistically significant at the 95 percent confidence interval.

ITT & IOT Impacts of the OSP on Math Achievement Overall, by Years After Application

0.25 0.75 1.25 1.75 2.25 2.75 3.25 3.75 4.25-4

-3

-2

-1

0

1

2

3

4

5

6

3.58

0.29

0.950000000000001

0.850000000000001

5.74

3.534.03

4.41

2.74

0.230.81 0.700000000000

001-0.26

-3.07-2.42

-3.01

Scal

e Sc

ore

Poin

ts

After 1 year After 2 years After 3 years After at least 4 years

Effect Sizes & Months of Schooling for Statistically Significant Reading Subgroup Impacts Based on 3 Analytic Methods

Student Achievement:

Reading

Impact of the Scholarship Offer (ITT)

Impact of Scholarship Use (IOT)

Effect of Private Schooling (IV)

Effect Size Months of Schooling Effect Size Months of

Schooling Effect Size Months of Schooling  

Not SINI 2003-05 .16* 3.5 .19* 4.2 .26 5.6  

Higher performance .15* 3.9 .18* 4.6 .27 6.7  

Female .15* 3.4 .18* 4.0 .37* 8.2  

* Statistically significant at the 95 percent confidence interval.

Parent and Student Reports of Safety and an Orderly School Climate, 2008-09

Treatment Control0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

7.94**7.47

8.52**

Parent Perceptions

ITT

IOT

**Statistically significant at the 99 percent confidence level.

Treatment Control0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

6.16 6.01

Student Reports

Parent and Student Reports of School Satisfaction, 2008-09

Treatment Control0%

20%

40%

60%

80%

100%

76%**68%

78%** (IOT)

Percentage ofParents Who Gave School

Grade A or B

ITT

Treatment Control0%

20%

40%

60%

80%

100%

69% 72%

Percentage of Students Who Gave School

Grade A or B

**Statistically significant at the 99 percent confidence level.

Highlights of Qualitative Research Focus groups & interviews 2004-

2008 Findings:

Parents more satisfied if two-parent family & extensive search

Experience improved for families over time (stigma, financial policies, welcome)

Safety replaced by academics as key factor

Strict standards or accommodation?

Context DCPS proficiency and DC NAEP scores

increased by over 20 percent 2005-2009

DCPS per-pupil spending of $17,542 Reliable studies indicate graduation is

far more important than achievement: Increases earnings, health, longevity, &

marriage Decreases unemployment and incarceration

Political/Policy Developments Program closed to new applicants in March of

2009 2010 Appropriations Law cut funding, capped

program, mandated shoddy evaluation, added regs

2010 Republicans capture House, new Speaker Boehner lists OSP reauthorization as a priority

Lieberman reauthorization language added to 2011 federal budget as final element of compromise

Obama Administration initially refuses to admit new students to program, then relents

Obama Administration 2014 budget proposes to zero-out program

Reauthorization Changes Increases OSP appropriation to $20

million/year Increases voucher max to 12k for

high school and 8k for K-8 Removes most regulations on

private schools Mandates new rigorous evaluation Good for 5 more years

My 5 Minutes of Famehttp://

www.youtube.com/watch?v=5GTX-278Rpc&feature=player_embedded

Questions on DC OSP?

Milwaukee Parental Choice Program (MPCP) Nation’s first urban school voucher

program Launched in 1990 with 341 students in

7 schools Dramatically expanded in 1998 Served 23,198 students in 106 schools

in 2011-12 In 2006, State of Wisconsin called on

School Choice Demonstration Project to evaluate program over 5 years 23

MPCP Schools and Student Enrollment

1990

-91

1991

-92

1992

-93

1993

-94

1994

-95

1995

-96

1996

-97

1997

-98

1998

-99

1999

-00

2000

-01

2001

-02

2002

-03

2003

-04

2004

-05

2005

-06

2006

-07

2007

-08

2008

-09

2009

-10 -

2,500

5,000

7,500

10,000

12,500

15,000

17,500

20,000

22,500

0102030405060708090100110120130140

Total MPCP Pupil Count Participating Schools

Tota

l MPC

P St

uden

t En

rollm

ent N

umber of Participating Schools

Wisconsin 2005 Act 125 Enacted on March 10, 2006 Modified the main MPCP statute

(119.23):1. Raised enrollment cap to 22,5002. Added school accreditation requirements3. Initiated standardized testing of Choice students in

grades 4, 8, and 104. Mandated that a representative panel take the

Wisconsin Knowledge and Concepts Examinations (WKCE)

5. Required scores be sent to the SCDP, then on to the Legislative Audit Bureau

25

SCDP Longitudinal Evaluation Comprehensive – examining participating

effects on students, parents, and schools, plus systemic and community effects

Multi-method – quantitative and qualitative, with test scores, surveys, focus groups, site visits

Rigorous – careful matching on test and student characteristics for “apples-to-apples” comparisons whenever possible

Longitudinal – five years of data, fall 2006 through fall 2010, with focus on changes from baseline 26

Highlights of Quantitative Results Positive effect of program on attainment Some evidence of reading gains but

none of math gains Lots of school switching Poor performing schools in both sectors

being closed Pressure of competition has increased

MPS achievement slightly Taxpayers save over $50 million/year

Effect of 2006 MPCP Attendance on 2010-11 Educational Attainment

28-6.0

-4.0

-2.0

0.0

2.0

4.0

6.0

8.0

On Time Graduate

Five-Year Graduate

Ever Graduate

Two-Year College Enrollment

Four-Year College Enrollment

Persist in Four-Year College

7.1***

-4.1***

4.0*

-3.0*

6.0*** 6.0**

Effect of 2006 MPCP Attendance on Student Achievement: 2007 to 2010

-.2-.1

0.1

.2

2007 2008 2009 2010 2007 2008 2009 2010

Math Reading

Effe

ct (S

D U

nits

)

YearNOTE: Point estimates and confidence intervals based on results in Table 4, Model 2

Point Estimate and 90% Confidence IntervalEffect of Baseline MPCP Attendance on Student Achievement

29

Program Expansion 2011 Income limit increased to 300% of poverty Permanent income qualification No enrollment cap Students can attend non-Milwaukee private

schools Schools can require top-ups from higher-

income students Testing of all Choice students in grades 3-8 &

10 using the state test Regulations dropped regarding curriculum and

teacher training Program expanded to Racine 30

Program Expansion in 2013 Maximum of 500 students outside

Milwaukee/Racine (1000 beyond 2013) Any school district, limited to 1% of

enrollments Family income < 185% of poverty Maximum amount raised to $7,210

elementary and $7,856 high school in 2014-15

Added individual income tax deduction for private school tuition

Questions/Discussion of MPCP?

Florida Tax Credit Scholarship Program & Evaluation Launched in 2001 in response to court

ruling Served nearly 38,000 low-income

students in over 1000 schools in 2011 State Dept of Education longitudinal

evaluation by David Figlio Latest finding: program boosts reading

achievement for students close to income cut-off

33

New York City Attainment Study Privately-funded partial-tuition

scholarship program expanded in 1997

1900 students entering grades 1-5 randomly assigned

Increased college enrollment rate for African-Americans by 7.1 percentage points if offered and 8.7 if used

No overall effect or effect for Hispanic students

The Question of Policy Design: What Should Arkansas Do?1. Universal or targeted?2. Voucher, tax-credit, or ESA?3. Size of the scholarship?4. What should it cover?5. All top-off, some top-off, no top-off?6. Apply admission criteria, first come, or

lotteries?7. Hold the public schools financially harmless or

$$$ travels with the child?8. Regulations on participating schools?9. Required evaluation, testing, or financial

reporting?10. Programmatic supports for families?

For More InformationElectronic versions of all DC and Milwaukee reports

available at:http://www.uark.edu/ua/der/SCDP.html Patrick J. Wolf, Ph.D.Professor and 21st Century Endowed Chair in School ChoiceDepartment of Education ReformCollege of Education and Health Professions201 Graduate Education BuildingUniversity of ArkansasFayetteville, AR  72701Phone: 479-575-2084FAX: 479-575-3196pwolf@uark.edu