Evaluation of South Carolina’s Reading First (SCRF) Initiative
description
Transcript of Evaluation of South Carolina’s Reading First (SCRF) Initiative
Evaluation of South Carolina’s Reading First
(SCRF) Initiative
Achievement and Survey Results from the First Three Years
Presented at the American Educational Research Association’s Annual Meeting
March 27, 2008
Tammiee S. Dickenson, PhD
Robert L. Johnson, PhD
Heather L. Bennett, MSW
Katie A. Sesso-Dahlke, MSW
Brett Ermer
Mugdha Galande
Joanna Gilmore
Jessalyn Smith
Diane M. Monrad, PhD Sarah J. Gareau, MEdJennifer GayDiana MindrilaAnita Rawls
Pam WillsPatricia BranhamBecca DriggersDiane Dunham, EdS
Overview
SCRF evaluation
Data sources
Achievement data
Participant group survey results
Conclusions
SCDE
SCEPCOPE
Planning
Data Collection
Data Analysis
Reporting
SCRF Evaluation
Participants
2004-2005: served 52 schools in 24 districts (approximately 12,000 students)
2005-2006: served 51 schools in 23 districts (approximately 11,000 students)
2006-2007: served 49 schools in 23 districts (approximately 10,500 students)
Data Sources
Stanford Reading First assessment
Palmetto Achievement Challenge Test (PACT)
Participant group surveys
School Leadership Team (SLT) surveys
Professional development workshop surveys
Annual SLT Presentation
Results presented from previous year
Stanford Reading First assessment results
Cohort analysis of all students in previous year
Matched analysis of students in all years
Individual school reports provided
Participant group survey results
Formative feedback for program improvement
Stanford Reading First Assessment
Measures achievement in grades 1 through 3
Sections:
Multiple choice (components – phonemic awareness; phonics; vocabulary development; reading fluency; and reading comprehension strategies)
Oral fluency (components – speaking vocabulary and oral reading fluency)
Results: At Grade Level (AGL) Grade 1
Fall Spring Annual Gain
Year 1 17.0% 48.2% 31.2%
Year 2 18.8% 51.2% 32.4%
Year 3 21.1% 54.3% 33.2%
Percentage of SCRF Grade 1 Students Scoring At Grade Level (AGL) on Stanford Reading First and Annual Gains
Results: At Grade Level (AGL) Grade 2
Fall Spring Annual Gain
Year 1 30.8% 35.5% 4.7%
Year 2 35.7% 40.4% 4.7%
Year 3 38.0% 44.6% 6.6%
Percentage of SCRF Grade 2 Students Scoring At Grade Level (AGL) on Stanford Reading First and Annual Gains
Results: At Grade Level (AGL) Grade 3
Fall Spring Annual Gain
Year 1 21.0% 41.5% 20.5%
Year 2 25.2% 46.2% 21.0%
Year 3 27.6% 49.0% 21.4%
Percentage of SCRF Grade 3 Students Scoring At Grade Level (AGL) on Stanford Reading First and Annual Gains
Average NCE Scores on Stanford Reading First for SCRF Students with AGL Reference Line
0
10
20
30
40
50
60
Fall Gr1 Spr Gr1 Fall Gr2 Spr Gr2 Fall Gr3 Spr Gr3
SESAT2 Primary 1 Primary 2 Primary 3
NC
E S
co
re
At Grade Level SCRF Students
Average Scale Scores on Stanford Reading First for SCRF Students and the Norm Group
400
450
500
550
600
650
700
Fall Gr1 Spr Gr1 Fall Gr2 Spr Gr2 Fall Gr3 Spr Gr3
SESAT2 Primary 1 Primary 2 Primary 3
Scal
e sco
re
Norm Group SCRF Students
Participant Group Surveys
Respondent groups: interventionists, literacy coaches, principals, and teachers
Sections: implementation, level of support, roles and responsibilities, professional development, and program effectiveness
Comparison across years (2005-2006 compared to 2006-2007)
Results: Participant Group Surveys Better understanding of program goals Reported a high rate of involvement in professional
development activities and found those activities helpful
Decrease in the need for professional development about the program and use of assessments
Increased need for professional development on effective instructional strategies to use for students below grade level
Implementation Survey Items
0% 20% 40% 60% 80% 100%
I understand my rolesand responsibilities
within the SCRF Initiative.
I understand the goals ofthe SCRF Initiative.
I support the SCRFInitiative.
% Agree or Strongly Agree
Principal Literacy Coach Interventionist Teacher
Implementation Survey Items (Cont.)
0% 20% 40% 60% 80% 100%
Teachers, interventionists,administrators, and the
school-based SCRF literacycoach(es) are working
together to implement theSCRF Initiative.
SLT members regularlyshare important information
with our faculty about theSCRF Initiative.
% Agree or Strongly Agree
Principal Literacy Coach Interventionist Teacher
Describe One Benefit of the SCRF Initiative
Commonly cited benefits across groupsFocus on assessment
Increased awareness of students’ needs/strengths/weaknesses
Progress monitoring; Dominie Improved use of assessments; use of assessment data to
guide instruction
Resources Additional books and materials Extra funding/money
Professional development; study groups
Increased collaboration
Learned new strategies
Results: Participant Recommendations
Recommendations
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Changes to study groups Changes related toassessments
Increase writing emphasis
Per
cent
age
Teacher Interventionist Literacy Coach Principal
Results: Effectiveness Ratings
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
Not Effective Somewhat Effective Effective Very Effective
Effectiveness Rating
Per
cent
age
Teacher Interventionist Literacy coach Principal
83.1% of teachers rated SCRF as effective or very effective. 88.1% of interventionists rated SCRF as effective or very effective. 81.8% of literacy coaches rated SCRF as effective or very effective. 90.9% of principals rated SCRF as effective or very effective.
Conclusions: Achievement
Students performed better on the Stanford Reading First assessment in the 2006-2007 school year as compared with the 2005-2006 school year.
Students’ scores on the Stanford Reading First assessment improved from fall to spring in all three years for all grade levels.
The largest gains were made by students in grade 1 and the lowest gains occurred in grade 2 in 2006-2007, which is also consistent with the last two years.
Conclusions: Participant Groups Program participants have a better
understanding of program goals.
Participant groups find professional development useful.
Over 80% of each participant group rated the initiative as either effective or very effective in Years 2 and 3.
Recommendations
More time for sustained reading for students in grade 2
Additional strategies for working with students below grade level
Participant groups’ suggestions regarding study groups
Contact InformationSouth Carolina Educational Policy CenterUniversity of South Carolina, College of EducationWardlaw Suite 010, Columbia, SC 29208
P: 803-777-8244
F: 803-777-0220 E: [email protected] (Diane Monrad)
Office of Program EvaluationUniversity of South Carolina, College of EducationWardlaw Suite 023, Columbia, SC 29208
P: 803-777-3402F: 803-777-8838E: [email protected] (Tammiee Dickenson)