Bringing Educational Innovation to Scale: Top-Down,

Bottom-Up, or A Third Way?

1

Doug Fuchs, Kristen McMaster, Laura Saenz, Devin Kearns, Lynn Fuchs, Loulee Yen,

Don Compton, Chris Lemons, Wen Zhang Vanderbilt University

Chris SchatschneiderFlorida State University

R305G04104 Institute of Education Sciences

Big Q and Presentation Focus

If we can think of Grade 2-6 Peer-Assisted Learning Strategies (PALS) as a “best-evidence” reading practice, how do we scale it up? More importantly, how do we scale up any worthy educational innovation?

This was the general Q of an IES Goal 4 study conducted between 2004-2010. Years 1 & 2, at kindergarten; Years 3-5, at grades 2 thru 5.

Why this question is important practically and theoretically.

2

Presentation Overview

Purpose of PALS-Reading

Grade 2-6 PALS program

Challenges involved in scaling up

Re-thinking scaling up: “top-down” & “bottom-up”

Findings

3

Purpose of PALS

Supplements the general education core program

It is a peer-mediated program, and facilitates intensive practice of core academic skills

Implemented 3 times per wk in reading; twice in math

Creates a structure for teachers to differentiate instruction: many simultaneous peer-mediated lessons rather than one teacher-directed lesson

PALS-R: K, first grade, grades 2-6, high school

PALS-M: K, first grade, grades 2-6 4

PALS Research

Originally based on “Juniper Gardens Classwide Peer Tutoring” (e.g., Greenwood et al., 1989)

More than 15 yrs of school-based research Title I and Non-Title I schools; urban and

suburban High, average, and low achievers Students with LD “Validated Practice” status (USDE, WWC, BEE)

5

Grades 2-6 PALS

Partner Reading

Paragraph Shrinking

Prediction Relay

6

Partner Reading

Conducted for 11-12 minutes Stronger reader reads aloud for 5 minutes Weaker reader reads same text aloud for 5

minutes Weaker reader retells story for 1-2 minutes Readers read quickly, correctly, and with

expression Coaches listen, correct mistakes, and mark

points (1 point for each correctly read sentence and 10 points for story retell)

7

Paragraph Shrinking

Conducted for 10 minutes For 5 minutes:

Stronger reader reads new text aloud, summarizing paragraph by paragraph

– Name the most important who or what (1 point)– Name the most important thing about the who or

what (1 point)– Shrink it to 10 or fewer words (1 point)

For next 5 minutes: Weaker reader reads new text aloud, summarizing

paragraph by paragraph (as above) Coach listens, corrects mistakes, and marks points

8

Prediction Relay

5 minutes, stronger reader read new textMakes prediction (1)Reads half page (1) Checks prediction (1)States main idea (3)Makes new predictionContinues to read

5 minutes, weaker reader continues on in new text, with the same activities

Coach listens, corrects mistakes, and marks points 9

Support to Teachers in K-PALS & Grade 2-6 PALS R&D During program development and validation, research

assistants provided in-class help 1x or 2x per wk.

Support is costly, finite, and typically disappears during scale up. Absent support, quality of implementation often suffers.

Obvious Q: How to successfully de-couple an effective intervention from a support system and “brand name” (culture)? How to sustain and scale-up?

10

Challenges to Scaling Up

Just described is the well known problem of moving an innovation from A to B.

Less well known problem: sustaining an innovation across time, or dealing with “the changing counterfactual,”

Case in point: K-PALS.

11

K-PALS:Data from 1998-99 Study

Nashville

12

Phonological Awareness Effects by Achievement Level for 1990s Data

13

Phonemic Awareness: Segmenting

-0.2

0

0.2

0.4

0.6

0.8

1

PhonemicAw areness v.

Control

Decoding v.Control

PA + Decoding v.Control

Contrast

Effe

ct S

ize

Low -achieving

Average-achieving

High-achieving

Overall

Word Reading Effects by Achievement Level for 1990s Data

14

Word Reading: WRMT-R Word Identification

-0.2

0

0.2

0.4

0.6

0.8

1

PhonemicAw areness v.

Control

Decoding v.Control

PA + Decoding v.Control

Contrast

Effe

ct S

ize

Low -achieving

Average-achieving

High-achieving

Overall

Pseudo-word Reading Effects by Achievement Level for 1990s Data

15

Word Reading: WRMT-R Word Attack

-0.2

0

0.2

0.4

0.6

0.8

1

PhonemicAw areness v.

Control

Decoding v.Control

PA + Decoding v.Control

Contrast

Effe

ct S

ize

Low -achieving

Average-achieving

High-achieving

Overall

Fluency Effects by Achievement Level for 1990s Data

16

Fluency

-0.2

0

0.2

0.4

0.6

0.8

1

PhonemicAw areness v.

Control

Decoding v.Control

PA + Decoding v.Control

Contrast

Effe

ct S

ize

Low -achieving

Average-achieving

High-achieving

Overall

Elements of K-PALS

17

Teacher-led phonological awareness

Peer practice Sound-letter

correspondenceSight words readingWord readingSentence readingBook reading

Points earned for doing activities

Explaining Vanishing Effects Over Time: Segmenting

18

Explaining Vanishing Effects: Word Identification

19

Explaining Vanishing Effects: Pseudo-Word Reading

20

Explaining Vanishing Effects: Reading Fluency

21

Grade 2-6 PALS:Years 3, 4, & 5 of IES Study

22

PALS vs. Controls, 2006-2007

23

2006-2007 (Cohort 1) Results

-0.2

0

0.2

0.4

0.6

0.8

1

Control v.s. PALS

Pre-Post Contrast

Effe

ct S

ize TOWRE

ITBS

MAZE

PALS vs. Controls, 2007-2008

24

2007-2008 (Cohort 2) Results

-0.2

0

0.2

0.4

0.6

0.8

1

Control v.s. PALS

Pre-Post Contrast

Effe

ct S

ize

TOWRE

ITBS

MAZE

Moment of Truth (for the melodramatically-inclined)

25

Fear…depression…

Thinking differently: Top-down vs. bottom-up

“Now at starbucks: A rebound”

26

Method

Top Down PALS

27

4 Activities: Partner Reading (10 Minutes)Retell (2 Minutes)Paragraph Shrinking (10 Minutes)Prediction Relay (10 Minutes)

35 Minutes per Session

Reciprocal peer tutoring

A motivational peer reinforcement system

Emphasis on fidelity – Using PALS the way it was designed, without modifications.

Bottom Up PALS

28

Core Elements of PALS: 10 minutes of Partner Reading 10 minutes of Paragraph Shrinking 35 minutes per session minimum A motivational peer reinforcement system

Customized Elements of PALS Small “tweaks” Big changes Flexibility

29

Participating Sites across 5 Years

30

Participating Teachers in Years 3-5 (Grade 2-6 PALS)

Two cohorts of 3rd-, 4th-, and 5th-grade teachers:Cohort 1

Entered study in 2006-07Cohort 2

Entered study in 2007-08Two years of study participation:

Year 1 Assigned randomly to PALS or Control

Year 2 PALS Teachers selected Top Down or Bottom Up

PALS Control Teachers remained in Control group

31

Teachers by Year 2 Study Group

32

Students by Year 2 Study Group

Student Measures in Years 3, 4, & 5

Academic Measures Test of Word Reading Efficiency (TOWRE) Woodcock Reading Mastery Test-Revised

(WRMT-R) Word Identification Subtest

Wide Range Achievement Test (WRAT) Letter and Word Identification

Subtests Comprehensive Reading Assessment

Battery (CRAB; 2 passages) Oral reading (1 min & 3 min) Comprehension (10 open-ended

questions) CBM Maze Task (2 passages)

Correct maze choices made in 2.5 min Iowa Test of Basic Skills (ITBS)

Reading Comprehension Vocabulary

Student Characteristics Demographics SWAN

Teachers rated each student’s abilities to focus attention, control activity, and inhibit impulses

Teacher ratings Teachers rated each student’s

effort in reading and behavior in the classroom

33

Teacher Measures in Years 3, 4, & 5

Classroom Measures PALS Calendars PALS Fidelity Language Arts Observation Classroom Atmosphere Rating Scale

(Wehby) Survey of Enacted Curriculum (SEC):

English and Language Arts

Teacher Characteristics Demographics Berends teacher survey (assesses

school climate, teacher professional development, teacher efficacy, etc.)

34

Procedures

Pretesting (September-October)

PALS Workshops (September-October) Year 1: All teachers attend same workshop Year 2: Separate TD and BU workshops

PALS Implementation (~18 weeks) Teachers implemented 3 times per week for 35-40 min Weekly classroom visits from project staff Three “booster” sessions for TD and BU PALS teachers Two fidelity observations

Language arts observations in PALS and Control classrooms 45-60 min Momentary time sampling of a variety of reading instructional

components Supplementary field notes

Posttesting (March-May)

35

Analysis

Create latent pretest and posttest variables combining 5 reading measures into 1

Create a latent change score Produces an “error-free” change value

Run 2-level HLM analyses Outcome: Latent change score Variables: Treatment condition (TD, BU, Control); Site (TN,

MN, TX); latent pretest score Random effects: Level 2 teacher effects; ICC = .10

Test comparability of groups on variables plausibly related to selection of TD or BU

36

Results

37

PALS vs. Controls across cohorts (2007-08, 2008-09), sites, and TD/BU

38

Effect Sizes for PALS

-.20

.00

.20

.40

.60

.80

1.00

PALS vs. Control

Effe

ct S

ize

(Hed

ges'

g)

Overall

Low

Average

High

Results: Latent Change Score

39

40

Results by Student Rating:Latent Change Score

41

Did we successfully scale up?

Yes: PALS appears to be generally effective.

BU PALS classes appeared to outperform TD PALS and Controls under the following conditions:Core elements of PALS including a 35 minute

PALS sessionSupport provided by PALS staff

42

Potential Mediators:No Differences Between TD & BU

Fidelity year 1 (ALL Teachers TD) Were teachers who later became TD or BU

better PALS implementers in their 1st year? ALL PALS activities compared.

Fidelity year 2 (TD vs. BU year) When teachers chose between TD and BU

PALS, did their PALS fidelity change? Examined core elements of PALS.

Fidelity Across Years

43

44

Classroom-based Potential Mediators: No Differences

Latent Change-Year 1 Indirect method of looking at overall teacher

efficacy AND differential effect of PALS implementation in year 1

Number of PALS Sessions Did one group implement more PALS than the

other?

Measure of Classroom EnvironmentClimate, management, behavior, etc.

Classroom-based Potential Mediators

45

46

Other Potential Mediators

No differences were found for: Scale of attitude about student needsScale of attitude of about school programsScale of attitudes about self-efficacyScale of attitudes about principal leadership