Recovering from Reading Recovery: A long-term program evaluationJoAnna Cogan-Ferchalk, Ed.S., NCSP

Matthew Ferchalk, Ed.S., NCSP

Background & Philosophy

Developed in New Zealand, by Dr. Marie Clay in the 1970's The goal of RR is to close the gap between at-risk readers and their

average peers (Schwartz, 2005) Incorporates Vygotsky’s theory of the Zone of Proximal

Development, where the student is able to read at a higher rate with help from the teacher than would be possible alone (Clay & Cazden, 1990)

Whole language background-- phonics is not taught explicitly to children It is taught “incidentally” if an opportunity arises (Schwartz, 2004)

Details of print and letter/sound relationships should be gained through writing, not reading instruction. This knowledge is considered limited when compared to comprehension of the text (Clay, 1993; Grossen et al., 1997)

(CREC, 2004; see also Saginaw Public Schools, 1992)

How it works

Students are selected for RR by teacher ratings and by screening with the Observational Survey (the lowest students are not always enrolled)

Students receive daily 30 minute lessons during their first grade year

Instruction includes use of magnetic letters to help write words, and teaching to read for meaning

Most students graduate from RR after 12-20 weeks ELCO’s RR teachers have adopted a more structured

phonics element

School District Demographics

Approximately 2400 students 16% special education placement rate Rural, farming community in Pennsylvania Reading Recovery has been in place for

11 years

Literature on RR Several studies have found RR to be moderately effective (Marina &

Gilman, 2003; Pinnell, Lyons, DeFord, Byrk, & Seltzer, 1994; Schwartz, 2005) There are complaints that these studies are flawed (Center & Wheldall,

1992; Heibert, 1994; Hoff, 2002; Rasinski, 1995) Unequal treatment time Not using the lowest readers

Other studies have found that students do not attain the same reading proficiency as their peers (Huggins, 1999), which has been attributed to lack of explicit phonics instruction (Pressley, 1998)

When explicit instruction in phonics is added to RR researchers have found greater improvement than with traditional RR (Chapman, Tunmer, & Prochnow, 2001; Iverson & Tunmer, 1993)

Few studies have examined long term effects of RR. Those that do typically high rates of special education placement in later years (Ramaswami, 1994)

Research Questions How do students who have been through the RR

program score on the PSSAs and curriculum-based assessments (CBA) in reading?

If students demonstrate improvement in reading scores after RR, do these gains persist over time?

Do students who have been through RR make the same rate of reading progress as their peers who did not need supplemental instruction?

What is the prevalence of placement into programs such as Title I and Special Education after students have graduated or were discontinued from RR?

Method Review existing data for students who

were enrolled in RR and are currently in 5th, 6th, and 7th grades8 students per elementary building per year

(n= 96 students) 3rd and 5th grade PSSA reading scores CBA reading scores from 1st to 5th grade Special programs placement rate following RR

Review existing data from a randomly selected comparison group (n=96)

5th Grade PSSA data, Cohort 1

Reading Recovery Below Basic 24% Basic 38% Proficient 19% Advanced 19%

Total Proficient 38%

District wide data Below Basic 14% Basic 22% Proficient 30% Advanced 34%

Total Proficient 64%

3rd Grade PSSA Data, Cohort 2

Reading Recovery 1st quartile 28% 2nd quartile 22% 3rd quartile 28% 4th quartile 22%

Upper quartiles 50%

District wide data 1st quartile 14% 2nd quartile 20% 3rd quartile 34.5% 4th quartile 31.5%

Upper quartiles 66%

5th Grade PSSA Data, Cohort 2

Reading Recovery Below Basic 27% Basic 14% Proficient 50% Advanced 9%

Total Proficient 59%

District wide data Below Basic 18% Basic 12% Proficient 47% Advanced 23%

Total Proficient 70%

Across time PSSA, Cohort 2

PSSA scores from 3rd to 5th grade declined from 1298 to 1238

The difference was not significant at 0.05

pssa3 pssa5

800

1000

1200

1400

1600

3rd Grade PSSA Data, Cohort 3

Reading Recovery 1st quartile 35% 2nd quartile 30% 3rd quartile 22% 4th quartile 13%

Upper quartiles 35%

District wide data 1st quartile 17% 2nd quartile 22% 3rd quartile 31% 4th quartile 30%

Upper quartiles 61%

CBA Cohort 1

Short-term, 2nd grade, 75% of the

comparison group were at mastery or instructional levels

Compared to 9% of RR students

Long-term, 5th grade 90% of the

comparison group were at mastery or instructional 81% mastery

Compared to 80% of RR students 20% mastery

Reading CBA Data Cohort 1

0

10

20

30

40

50

60

70

80

90

100

Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Comp 1 Comp 2 Comp 3 Comp 4 Comp 5

GradeReading Recovery Comparison Group

Pe

rce

nt

Pro

fic

ien

t

Advanced

Proficient

CBA Cohort 2

Short-term, 2nd grade 82% of the

comparison group were at mastery or instructional levels

Compared to 32% of RR students

Long-term, 5th grade 95% of the

comparison group were at mastery or instructional levels 42% mastery

Compared to 82% of RR students 29% mastery

Cohort 2

0

10

20

30

40

50

60

70

80

90

100

Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Comp 1 Comp 2 Comp 3 Comp 4 Comp 5

GradeReading Recovery Comparison Group

Pe

rce

nt

Pro

fic

ien

t

Advanced

Proficient

CBA Cohort 3

Short-term, 2nd grade 43% of the

comparison group were at mastery or instructional levels

None of the RR students had reached an instructional level

Long-term, 4th grade 90% of the

comparison group were at mastery or instructional levels 71% mastery

Compared to 67% of RR students 25% mastery

Cohort 3

0

10

20

30

40

50

60

70

80

90

100

Grade 1 Grade 2 Grade 3 Grade 4 Comp 1 Comp 2 Comp 3 Comp 4

GradeReading Recovery Comparison Group

Pe

rce

nt

Pro

fic

ien

t

Advanced

Proficient

Special programs placement rates

Special Education Cohort 1

19% Cohort 2

22.7% Cohort 3

12%

District wide rate is approximately 16%

Title I Cohort 1

1 year- 100% 2 years- 33.3% 3 years- 14.3%

Cohort 2 1 year- 100% 2 years- 31.8% 3 years- 27.3%

Cohort 3 1 year- 100% 2 years- 40% 3 years- no data

How do students who have been through the RR program in the ELCO school district score on standardized tests of achievement and curriculum-based assessments (CBA) in reading?

On the PSSA exams, fewer students in RR scored in the proficient range as compared to district wide dataThis held true across all cohorts

On CBA data, fewer students in RR scored at the instructional and mastery levels than did students not in RRThis held true across all cohorts

Do students who have been through RR make the same rate of reading progress as their peers who did not need supplemental instruction?

When examining the CBA data it appears that RR students started out well below their peers, and remained below them over the course of elementary school RR’s goal is to close this gap in achievement

Over time more RR students scored in the instructional and mastery levels on the CBA measures; however, they did not approach the level of mastery attained by their peers

If students demonstrate improvement in reading scores after RR, do these gains persist over time?

In general, students did not demonstrate initial gains in reading (as demonstrated by CBA)

By 5th grade, students in RR were still behind their peers (as demonstrated by CBA & PSSA scores)

A jump was noted in fourth grade CBA scores; however this cannot be attributed to RR as it occurred three years after the intervention. This may also be the result of CBA scores no longer being kept for students in special education.

What is the prevalence of placement into programs such as Title I and Special Education after students have graduated or were discontinued from RR?

In cohorts 1 & 2, students were placed into special education at a higher rate than the district average (19 & 22.7% vs. 16%)

In cohort 3, the rate was lower (12% vs. 16%) 100% of students in RR received Title I services

in 1st grade 40-60% of students in RR received Title I

services in 2nd and 3rd grade

Conclusions & Implications The goal of RR (bringing at-risk students to the level of their

peers in a short period of time) does not appear to have been met

Given that RR is not designed for the lowest readers, it excludes students who are in need of help, and may lead to misleading reports of effectiveness

The practice of dropping students if they do not make adequate progress leaves some students without much needed intervention

School psychologists are in a unique position to recommend programs that have a broader research base and can provide more benefits to more students (i.e.- PHONICS!!!)

We can also take an active role as a researcher and/or program evaluator to keep our districts informed of beneficial programs

Limitations Quality of data kept in the school No “pre RR” reading levels of students available Different data sets exists for each cohort Too many confounding variables to accurately

determine the effects of RR 100% of students in RR received Title I for at least

one year, and many more received years of support

Attrition When students in RR were placed in special

education, their CBA scores were not recorded

Recommendations for future research More long-term research is needed to

examine the effects or RR Research should include initial reading

levels of at-risk students

References Capital Region Education Council. (2004). Who we are and what we do. Retrieved

online September 26, from http://www.crec.org/rr/index.shtml#what. Center, Y., & Wheldall, K. (1992). Evaluating the effectiveness of Reading Recovery:

A critique. Educational Psychology, 12(3/4) Clay, M. M. (1985). Observation survey of early literacy achievement. Auckland, New

Zealand: Heinemann. Clay, M. M. (1993). Reading Recovery: A guideline for teachers in training.

Portsmouth, NH: Heinemann. Clay, M. M. (1998). An observation survey of early literacy achievement. Auckland,

New Zealand: Heinemann. Clay, M. M. & Cazden, C. B. (1990). A Vygotskyan interpretation of Reading

Recovery. In L. Moll (Ed.), Vygotsky and education: Instructional implications and applications of sociohistorical psychology (pp. 206-222). Cambridge, England: Cambridge University Press.

Chapman, J. W., Tunmer, W. E., & Prochnow, J. E. (2001). Does success in the Reading Recovery program depend on developing proficiency in phonological processing skills? A longitudinal study in a Whole Language instructional context. Scientific Studies of Reading, 5(2), 141-177.

Dyer, P. C. (1992). Reading Recovery: A cost-effectiveness and educational outcomes analysis. Spectrum: Journal of Research in Education, 10(1), 110-119.

Elbaum, B., Vaughn, S., Hughes, M. T. & Moody, S. W. (2000). How effective are one-to-one tutoring programs in reading for elementary students at risk for reading failure? A meta-analysis of the intervention research. Journal of Educational Psychology, 92(4), 605-619.

Evidence-based research on Reading Recovery. (2002). Report to House and Senate Education Committees. Retrieved July 1, 2002, from, http://www.nrrf.org/rrletter_5-02.pdf.

Glynn, T., & Crooks, T. (1992). Reading recovery in context: Implementation and outcome. Educational Psychology, 12(3/4), 249-262.

Grossen, B., Coulter, G., & Ruggles, B. (1997). Reading Recovery: An evaluation of benefits and costs. Retrieved on July 1, 2002, from, http://darkwing.uoregon.edu/~bgrossen/rr.htm.

Hoff, D. J. (2002). Researchers urge officials to reject Reading Recovery. Education Week, 21(39), 1-3.

Huggins, R. (1999). Longitudinal study of the Reading Recovery program: 1994-1998. Urban Education. (ERIC Document Reproduction Service No. 430 067).

Iversen, S. & Tunmer, W. E. (1993). Phonological processing skills and the Reading Recovery program. Journal of Educational Psychology, 85, 112-126.

Marina, B., & Gilman, D. A. (2003). Is Reading Recovery worth the cost? (ERIC Document Reproduction Service No. ED 473 957).

Moore, P., Lysy, D. C., Bamford, R., & Cook, L. A. (1999). A look at Reading Recovery programs in Maine schools: A supplement to the State of Maine Reading Recovery report and evaluation, 1997-1998. Maine University: Center for Early Literacy. (ERIC Document Reproduction Service No. ED 463 533).

Munoz, M.A. (1999) Reading Recovery and the at-risk first grade students. (ERIC Document Reproduction Service No. ED 471 186).

Reading Recovery Executive Summary, 1984 to 1998. (1999). Columbus, OH: Reading Recovery Council of North America. (ERIC Document Reproduction Service No. ED 434 318).

Report of the National Reading Panel: Teaching Children to Read. (April 2000). National Institute of Child Health and Human Development. NIH Pub. No. 00-4769. Retrieved July 1, 2002, from, http://www.nichd.nih.gov/publications/nrp//findings.htm.

Pinnell, G. S., Lyons, C. A., DeFord, D. E., Bryk, A. S., & Seltzer, M. (1994). Comparing instructional models for the literacy education of high-risk first graders. Reading Research Quarterly, 29, 9-39.

Pressley, M. (1998). Reading instruction that works: The case for balanced teaching. New York: Guilford.

Ramaswami, S. (1994). The differential impact of Reading Recovery on achievement of first graders in the Newark School District. Newark, NJ: Newark Board of Education, Office of Planning, Evaluation and Testing. (ERIC Document Reproduction Service No. ED 374 180).

Rasinski, T. V. (1995). Commentary on the effects of Reading Recovery: A response to Pinnell, Lyons, DeFord, Bryk, and Seltzer. Reading Research Quarterly, 30, 264-270.

Saginaw Public Schools, Department of Evaluation Services. (1992). Compensatory education product evaluation: Reading Recovery Program 1991-92. Saginaw: Author. (ERIC Document Reproduction Service No. ED 350 587).

Shanahan, T., & Barr, R. (1995). Reading Recovery: An independent evaluation of the effects of an early instructional intervention for at-risk learners. Reading Research Quarterly, 30, 958-995.

Schawrtz, R. M. (2005). Literacy learning of at-risk first-grade students in the Reading Recovery early intervention. Journal of Educational Psychology, 97(2), 257-267.

Tunmer, W. E., & Chapman, J. W., (2001). The Reading Recovery approach to preventative early intervention: As good as it gets? Palmerston North, New Zealand: Massey University.