Copyright © 2015 American Institutes for Research. All rights reserved.

Elizabeth Spier, PhD Johannes Bos, PhDPrincipal Researcher Senior Vice President

FAST in Philadelphia

SEPTEMBER 2015

Keeping an RCT Relevant When Program Participation is Low

2

The Intervention

Families and Schools Together (FAST) is a school-based initiative intended to improve children’s academic and social outcomes.

For this study, FAST is being implemented in 30 persistently-low-performing schools in Philadelphia.

The School District of Philadelphia is the eighth largest in the United States, and serves many ethnically and linguistically diverse, high-poverty communities.

Families and Schools Together

3

• FAST invites students and their families into the school for a series of after-school sessions focused on improving connections within families, among families, and between families and schools

Objectives include:• Better communications and less stress at home• Better social support for families• Better behavior by the students (at home and in school)• More positive school-home engagement and collaboration

How FAST Works

4

• Investment in Innovation (i3) grant from U.S. Department of Education, focused on school turnaround.

• School turnaround is achieved by getting and keeping children on track academically and socially.

• FAST was offered universally to all families of incoming kindergarten (grade 0) students.

• A relatively high level of FAST participation is necessary to improve outcomes across the school cohort (not just for individual students).

The Current Project

5

FAST Logic Model

6

FAST

Parent-Child Relationship

Building

Parent-School Engagement

Improved Early

Learning

Improved Home

Environment

Parental Social Support

Improved Child

Behavior

Improved Classroom

Climate

Improved School Climate

School Turnaround

The Evaluation

7

Original Sampling Design

8

Random Selection

Random Assignment

60 Eligible Schools

60 Schools

30 Schools FAST 2013

12 Students Cohort 1

12 Students Cohort 2

30 Schools FAST 2016

12 Students Cohort 1

12 Students Cohort 2

The sampling design was based on these assumptions:• If FAST was offered to all incoming students and their

families, most would choose to participate.• Families who started to participate in FAST would

continue to do so, so that most students’ families (at least 60%) would attend 4 or more sessions.

• If most students in the cohort participated in FAST, then impacts (if any) would be evident at the cohort level.

Assumptions

9

Reality

10

17%

43%

40%

Completers

Non-Completers

Not expected to complete FAST

Sampling Design Amplifies Consequences

11

Kindergarten Class

Kindergarten Class

Sampling Design Amplifies Consequences

12

Kindergarten Classroom

Expected Participants

Kindergarten Class

Reality

Sampling Design Amplifies Consequences

13

Kindergarten Classroom

Expected ParticipantsStudy

Sample

Kindergarten Classroom

RealityStudy

Sample

Sampling Design Amplifies Consequences

14

Kindergarten ClassroomStudy Sample

Study Sample

Alternative Supplemental Design

15

Kindergarten Classroom Enhanced

study sample

Study Sample

• We are keeping the original RCT intact but redirected some study resources to a supplemental study to measure the effect of “treatment on the treated”

• For this study, we are using a two-level propensity score matching design that maximizes the contrast between participants and non-participants

• Note that this supplemental study answers somewhat different research questions!

Alternative Supplemental Design

16

Multi-Level Mahalanobis Matching

17

Treatment Control

Multi-Level Mahalanobis Matching

18

1 2

3

4

5

6

7 8

8 4

7

5

1

6 2

3

Treatment ControlSTEP 1Match 8

treatment schools with

highest FAST

participation rates to

comparable control schools

Multi-Level Mahalanobis Matching

19

1 2 3 4

5 6 7 8

3

4

5

6

7 8

1 2 3 4

5 6 7 8

5

1

6 2

3

Treatment ControlSTEP 2Collect

baseline data on all

kindergarten students and their families in these 16

schools

Multi-Level Mahalanobis Matching

20

Treatment ControlSTEP 3Identify

participating children in treatment

schools and match them

to similar children in

control schools

Multi-Level Mahalanobis Matching

21

Treatment ControlSTEP 4

Use resulting sample to estimate impact of

participating in FAST

(treatment on the treated)

Closing Thoughts

22

• When planning RCT with non-universal data collection, consider take-up and participation as a factor.

• Design studies to capture both individual-level “treatment on the treated” effects and school or classroom-level effects, ideally with comparable rigor for both.

• When possible, collect universal baseline data even in cases where you plan to use random subsamples for follow-up data collection (increases flexibility).

• Please stay tuned for impact results!

Elizabeth Spier, PhD Johannes Bos, PhD650-843-8226 650-843-8110espier@air.org jbos@air.org

2800 Campus Drive, Suite 200San Mateo, CA 94403General Information: 650-843-8100TTY: 650-493-2209www.air.org

23