Minnesota Reading Corps Mid-Year Evaluation · IGDI Rhyming IGDI Picture ... assessment windows and...

MRC Evaluation of 20

Minnesota Reading Corps Mid-Year Evaluation 9/1/2008 – 12/31/2008 Statewide Report

Prepared By:

Kerry Bollman, SSP, NCSP Academic Collaborative Planner, Reading Center Director

Saint Croix River Education District [email protected]

Benjamin Silberglitt, PhD

Senior Consultant, Assessment and Implementation Technology and Information Educational Services

[email protected]

David Heistad, Ph.D Director, Research Evaluation and Assessment

Minneapolis Public School District [email protected]


Background of Minnesota Reading Corps Minnesota Reading Corps (MRC) is an AmeriCorps program that provides trained literacy tutors (Members) for children age three to grade three. Some MRC Members work with preschoolers and focus on integrating talking, reading, and writing into all activities. Other Members provide supplemental literacy skills tutoring for children in kindergarten to third grade. Still others recruit, train, and manage community volunteers to expand the capacity of the program. MRC Members and community volunteers are trained in specific research-based, leveled literacy instructional protocols, and are supported by expert coaches. Members use reliable, valid assessment tools to monitor student progress on a regular basis, and with help from their expert coaches, use data from assessments to inform tutoring strategies for each student. Use of specific research based instructional techniques and technically adequate assessment tools for decision making make the MRC program both highly unique and highly coveted across the literacy landscape. The vision of the Minnesota Reading Corps is to impact literacy in the state of MN through children, AmeriCorps members and communities as follows:

All children in MN, ages 3 to grade 3, who qualify for MRC, will have access to MRC and will meet reading standards by third grade.

AmeriCorps members, through the training, development and service opportunity provided by MRC, will pursue education related careers and/or continue to be ambassadors for children's literacy throughout their lives.

Schools and community institutions/organizations, through their experiences with MRC, will understand and adopt the MRC methods for increasing literacy; those institutions will, in turn, promote MRC literacy methods to their colleagues.

Evaluation Design The evaluation for the Minnesota Reading Corps 2008-2009 program year includes six broad questions as listed below. Additions to the current year evaluation include more thorough analysis of performance for participating students relative to their performance level at the start of tutoring, ongoing follow-up on progress for students who previously participated in MRC services, analysis of performance on the 2009 Minnesota Comprehensive Assessment II for participating MRC students, and enhanced measurement of Pre-K site systems change.

1. What is the current impact of the MRC on the state of Minnesota in terms of numbers of

students and programs receiving support? 2. Are the data collection tools used with children being implemented with fidelity? 3. Are the interventions used with children being implemented with fidelity?


4. Is the performance of MRC-participating students in terms of their literacy improvement consistent with expectations?

5. Are the organizations with which the MRC is working changing to adopt the practices of the

MRC? 6. What is the impact of the MRC experience on the AmeriCorps Members? Assessment Data Collection The assessment data listed below will be collected during the 2008-2009 school year. Minnesota Reading Corps Members collect all student assessment data. Internal and Master Coaches collect fidelity data for assessments and interventions. Internal Coaches, Site Supervisors, and Members complete survey data regarding perceptions of systems change and impact. Internal and Master Coaches collect Pre-K environmental observation data. Student Data for Pre-school Programs:

Program participants are strongly encouraged, but not required, to collect benchmark data on

three-year-old students in classrooms served by Minnesota Reading Corps Members.

Fall (Oct 6 -21) Winter (Jan 19-30) Spring (April 20 – May 1)

Age 3 on or before

Sept 1st *

IGDI Rhyming IGDI Picture Naming IGDI Alliteration



Age 4

on or before Sept 1st

IGDI Rhyming IGDI Picture Naming IGDI Alliteration Letter Naming Fluency Letter Sound Fluency



Age 5

on or before Sept 1st but not enrolled in K





Student Data for K-3 Programs:

Additional Data Observations of assessment fidelity – three times per year Observations of intervention fidelity – nine times per year in K-3 programs only Early Language and Literacy Classroom Observation – two times per year in Pre-K programs only End of year MRC Member surveys End of year MRC site surveys Evaluation Report

What is the current impact of the MRC on the state of Minnesota in terms of numbers of students and programs receiving support? In the tables below, the number of Minnesota Reading Corps Members, full or part time, serving during the 2008-2009 school year, who collected data for students during one or more benchmark assessment windows and submitted the data for evaluation is recorded, along with the number of students receiving MRC services for whom data are recorded. Numbers of participating students are compiled according to the following criteria: • Number of preschool students with data from 1 benchmark window • Number of preschool students with data from 3 benchmark windows (end of year) • Number of K-3 students benchmarked as a follow up from previous year’s tutoring • Number of K-3 students with at least 1 week of data on at least 1 measure • Number of K-3 students with at least 5 weeks of data on at least 1 measure • Number of K-3 students with at least 10 weeks of data on at least 1 measure • Number of K-3 students with at least 20 weeks of data on at least 1 measure

Fall (Sept 8 – 19) Winter (Jan 12 -23) Spring (May 4 -15) Kindergarten Letter Naming Fluency

Letter Sound Fluency Letter Naming Fluency Letter Sound Fluency Nonsense Word Fluency

Letter Naming Fluency Letter Sound Fluency Nonsense Word Fluency

Grade 1 Letter Naming Fluency Letter Sound Fluency Nonsense Word Fluency

Nonsense Word Fluency Oral Reading Fluency (3

passages)

Oral Reading Fluency (3 passages)

Grade 2 Oral Reading Fluency (3

passages) Oral Reading Fluency (3


passages) Grade 3 Oral Reading Fluency (3



passages)


Table 1: Pre-Kindergarten Participation

Region

Number of Members

Number of Students with 1 or More Assessments

in Window

Number of Students with 3 Assessments in

Window Central – St. Cloud 10 153 0 Metro – St. Paul 47 747 0 North Central – Grand Rapids 13 187 0 North East - Duluth 3 46 0 North West - Moorhead 12 184 0 Southern - Rochester 18 322 0 Total 103 1,639 0 Table 2: Kindergarten-Grade 3 Participation

Region

Number

of Members

Benchmarked as Follow up

1 or More Assessments

5 or More Assessments on at least 1

Measure

10 or More Assessments on at least 1

Measure

20 or more Assessments on at least 1

Measure Central – St. Cloud 36 90 730 564 237 0 Metro – St. Paul 108 532 2,137 1,722 749 0 North Central – Grand Rapids

10 71 123 95 39 0

North East - Duluth 22 258 457 364 167 0 North West - Moorhead 18 199 333 273 96 0 Southern - Rochester 28 97 567 474 237 0 Total 222 1,247 4,347 3,492 1,525 0 In order to more fully describe the population of children served by the Minnesota Reading Corps program, data regarding gender, ethnicity, special education entitlement and primary language spoken were collected by Reading Corps Members. Demographic data for students participating in the Minnesota Reading Corps program with at least one data point collected through the year are summarized below. It is noted that the ethnicity categories used for the 2008-2009 evaluation reflect revisions to categories used in the Federal No Child Left Behind law since the time of the 2007-2008 MRC evaluation report.


Table 3: Pre-Kindergarten – Grade 3 Participant Demographic Data

Gender

Ethnicity Special Education

Entitlement

Primary Language

Pre-K

52% Male 47% Female 1% Unknown

2% American Indian / Alaska 19% African American 5% Asian 0% Filipino 14% Hispanic / Latino 0% Pacific Islander 1% Unknown 6% Multiple / No Response 31% White 22% Not Reported

92% General Education 8% Students with IFSPs

66% English as Primary Language 29% ELL 5% Unknown

K-3

48% Male 44% Female 8% Unknown

2% American Indian / Alaska 13% African American 3% Asian 0.03% Filipino 5% Hispanic / Latino 0.1% Pacific Islander 32% Unknown 1% Multiple / No Response 42% White

97% General Education 2% Title 1 1% Students with IEPs

94% English as Primary Language 6% ELL

Are the data collection tools used with children being implemented with fidelity? Analysis of fidelity with which student assessment data are conducted is a critical initial aspect to the evaluation of the MRC program so that results from evaluation of these data may be reported with confidence. In order to accomplish this, a series of Accuracy of Implementation Rating Scales (AIRS) have been compiled from each Minnesota Reading Corps (MRC) site. MRC Internal Coaches and new Master Coaches were trained in August 2008 to administer and score assessment measures, and to conduct observations of Reading Corps members as they administer and score these measures. The AIRS are structured observational protocols that provide an opportunity for observers to certify that each aspect of a standardized administration for each assessment measure has been fully conducted. Master and/or Internal Coaches completed a minimum of 1 AIRS for each Reading Corps Member for each type of assessment the member conducted at least 3 times each year, around the benchmark data collection periods. The table below documents the number of AIRS assessments compiled and percent fidelity documented for each measure. In addition to reporting these data in aggregate format to document high fidelity of assessment procedures across the state, this observation system also provided Members with immediate feedback regarding the quality of their own assessment skills, and an opportunity to receive clarification or re-training as needed in a timely manner.


Table 4: Fidelity of Assessment Data Collection Procedures

Measure

Picture

Naming

Alliteration

Rhyming

Letter

Naming

Letter

Sounds

Nonsense

Words

Oral Reading Fluency

Total AIRS Collected

124

124

134

235

239

122

172

Fidelity Range

Reported

24% - 100%

28% - 100%

11% - 100%

15% - 100%

23% - 100%

23% - 100%

48% - 100%

Median % Fidelity

Reported

100%

97%

100%

100%

100%

100%

95%

Mean % Fidelity

Reported

91%

83%

86%

90%

86%

92%

94%

Standard Deviation

0.16

0.24

0.24

0.17

0.23

0.13

0.07

Are the interventions used with children being implemented with fidelity? As with the assessment tools, analysis of the level of fidelity with which the student intervention protocols are followed is a critical initial aspect to the evaluation of the MRC program so that results of student growth analysis may be attributed to accurate implementation of intervention scripts. In order to accomplish this, a series of intervention integrity observations have been compiled from each MRC site. MRC Internal and Master Coaches were trained in August 2008 to evaluate implementation integrity for each of the MRC interventions. The integrity checklists provide an opportunity for observers to certify that each aspect of a standardized administration for each intervention has been fully conducted. Master and/ or Internal Coaches completed a minimum of 1 intervention integrity checklist for each MRC member during each monthly visit, for a possible total of 9 checklists per Member per year. The interested reader is referred to Ehrhardt, Barnett, Lentz, Stollar, & Reifin, (1996) for a description of how to use scripts to improve intervention integrity. The table below documents the number of integrity checklists compiled and percent fidelity documented for each intervention:


Table 5: Fidelity of Intervention Implementation Procedures

Intervention

Rep

eate

d R

eadi

ng

Due

t R

eadi

ng

New

scas

ter

Rea

din

g

Rep

eate

d R

eadi

ng w

/ Q

ues

tion

Gen

erat

ion

Stop

Go

Pen

cil T

ap

Gre

at L

eaps

Let

ter

Soun

d

Iden

tifi

cati

on

Wor

d B

lend

ing

Pho

nem

e B

lend

ing

Pho

nem

e Se

gmen

ting

Total Fidelity Checks

Collected

237

189

14

6

5

23

140

73

78

29

14

Fidelity Range

Reported

38% - 100%

21% - 100%

55% - 100%

100% - 100%

90% - 100%

60% - 100%

6% - 100%

33% - 100%

20% - 100%

63% - 100%

56% - 100%

Median % Fidelity

Reported

93%

94%

91%

100%

100%

88%

83%

100%

100%

100%

94%

Mean % Fidelity

Reported

90%

91%

90%

100%

96%

90%

78%

91%

92%

96%

88%

Standard Deviation

0.14

0.13

0.13

0.00

0.05

0.11

0.20

0.14

0.15

0.09

0.15

Is the performance of MRC-participating students in terms of their literacy improvement consistent with expectations? The following sections document growth and achievements of children age 3 to grade 3 who participated in the MRC program during the 2008-2009 school year. It is important to acknowledge that MRC participating students are also supported by a variety of resources, most notably the instruction and guidance provided by their schools and families. This evaluation is not intended to address or control for the variables related to these resources, nor to suggest that student progress or lack thereof must be attributed solely to the service being provided through the efforts of the MRC program. This design’s purpose is to focus on the desired literacy outcomes for all children. Pre-Kindergarten Student Performance The five measurement tools utilized for the Pre-K Reading Corps program are listed below. For a description of the technical characteristics of these assessments, the interested reader is referred to the 2007-2008 Minnesota Reading Corps program evaluation. For each assessment tool, a target score was identified as the goal for the end of the year. These target scores were based on the target scores used in Minneapolis Public Schools for incoming kindergarten students, and upon 50th %ile scores for incoming kindergarten students within school districts served by Saint Croix River Education District. Prior to use of these targets for the current project, these targets were reviewed by the researchers at the University of Minnesota who originally created the Individual Growth and Development Indicators. The measures and target scores for this project are listed below:


Measure Spring Target Score Rhyming 12

Picture Naming 26 Alliteration 8

Letter Sound Fluency 8 Letter Naming Fluency 14

Pre-Kindergarten student performance on fall IGDI measures is listed in the table below for all students with a birth-date reported, and data collected during the fall assessment window. Score ranges are also reported. Students score “NA” when they do not complete the sample items sufficiently to warrant participation in the assessment. A score of “0” indicates adequate performance on the sample items, but no accurate responses during the assessment. Table 6: Pre-Kindergarten Participant Performance on IGDIs: Fall Benchmark

Measure

Output

Three Year Olds

Four Year Olds

Five Year Olds

Fall Number Students Tested 349 1,065 35 Range of Scores NA-33 NA-40 0-38

Picture Naming

Fall (Number) Percent Students Above Spring Target (17) 4.9% (182) 17.08% (11) 31.4% Fall Number Students Tested 349 1,065 35 Range of Scores NA-11 NA-20 NA-9

Alliteration


Rhyming


Letter Naming Fluency


Letter Sound Fluency

Fall (Number) Percent Students Above Spring Target (3) 0.9% (92) 8.64% (3) 8.6% The figure below shows the normative performance of all 4-year-old students participating in the IGDI measures during the fall benchmark window of the 2008-2009 school year. As a clarification, it is noted that students who did not successfully complete the sample items on each assessment measure in order to continue on to the actual assessment were given a score of “NA” which is recorded in this figure as a -6. The NA score is distinguished from a score of 0, which reflects a performance in which the student’s appropriate responses to the sample items warranted continuation with the assessment in accordance with standardized procedure, but responses during the timed assessment yielded no correct answers.


Figure 1: Normative Performance of 4-Year-Olds on IGDI Measures: Fall

* Green Diamond = 90th %ile score; Mid-Blue X = 75th %ile score; Navy Triangle = 50th %ile score; Light Blue Circle= 25th %ile score; Red Square = 10th %ile score. A cross-cohort analysis of performance by 4-year-old students has been compiled across the six years of the Minnesota Reading Corps program. The following figure shows the percent of 4-year-old MRC participants meeting the assessments’ spring target scores at fall, winter, and spring assessment times across years. It is noted that the 2006-2007 data was analyzed by an outside agency, and only includes students enrolled in Headstart MRC classrooms. Across successive school years, fairly stable fall performance is noted, along with an overall increase in percent of students meeting spring target scores by the spring benchmark window. Data from the current year again show performance roughly equivalent with or slightly above that of previous years on fall measures.


Figure 2: Cross-Cohort Percent Above Target on Early Literacy Measures

Pre-Kindergarten Matched Sample Analysis

In order to address the evaluation question regarding growth across years for students participating in the Minnesota Reading Corps program, analysis has been completed for a subset of MRC participating students who were enrolled in PICA Head Start programming during the 2007-2008 school year, and continued on to elementary school within Minneapolis Public School (MPS) system as kindergarteners in fall of 2008. For comparison purposes, an additional group of students was identified who did not receive intervention from a Reading Corps member as preschoolers, but also enrolled in MPS in fall of 2008. Minneapolis Beginning of Kindergarten Assessment Beginning of Kindergarten literacy was assessed using the Minneapolis Beginning of Kindergarten Assessment (BKA). This assessment is individually administrated by retired school teachers who are trained to be reliable data collectors. The assessment includes 5 literacy domains: phonological awareness (initial letter sounds and rhyming), alphabetic principles (letter names and letter sounds), vocabulary (picture naming), concepts of print (front of the book, right to left sweep, etc.), and oral comprehension.


The BKA has scores with a high level of reliability (Betts J., Heistad D., Pickart M., and Anderson, A., 2005). The overall total test score, called the Early Literacy Composite has internal consistency reliability at or above 0.90. Similarly high internal consistency reliability estimates have been found among the three norm groups (below 5 ½, between 5 ½ and 6, and above 6 years of age as measured by age at time of BKA). The test-retest reliability was found to be 0.92. The scores on the BKA have been found to have strong validity evidence (content, criterion and construct). The predictive validity of the BKA to 1st grade oral reading (0.80), 1st grade comprehension (0.66) and to 2nd grade reading (0.66) as measured by the Northwest Achievement Levels Tests (NALT) is also moderately strong. The BKA Early Literacy Composite is also highly correlated with the End of Kindergarten Assessment (EKA) Early Literacy Composite (0.74). Matched Sample Methodology In order to locate matched samples for comparison, files of all kindergarten students were sorted hierarchically on these variables in the MPS data system in fall 2008 using the following sort order:

• Home Language • Special Education Status • Special Education Disability Category • Free or reduced price Lunch • Racial Ethnic Category • English Language Learner Status • Gender • Residential Zip Code • Student Birth date (note: matched students needed to be the same age within 6 months

for a match. Students in the comparison groups were matched on at least 8 of 9 variables. The best match was chosen by computer as the student directly above or directly below the Reading Corps student with the most matches. Ties (e.g. if the student above and below in the file each had a match on 9 variables) were broken based on birth date closest to the Reading Corps student. Matching between Reading Corps students and non-Reading Corps students were as follows:

• 48 pairs (73%) had 9 out of 9 variables matched • 18 pairs (27%) had 8 out of 9 variables matched

Demographic data for students in this analysis are listed below:

• 52% Male • 96% Free/reduced lunch • 0% Special Education students • 56% ELL students • 51% Lives with both parents


Table 7: Racial/Ethnic Background of Study Participants

Racial/Ethnic Background Reading Corps Non- Reading Corps Matched Sample

American Indian 3 (4.5%) 3 (4.5%) African American 37 (56%) 37 (56%)

Asian 1 (1.5%) 1 (1.5%) Hispanic 23 (35%) 23 (35%)

White 2 (3%) 2 (3%) Total 66 66

Table 8: Home Language of Study Participants

Home Language Reading Corps Non- Reading Corps Matched Sample

English 26 (39%) 27 (41%) Hmong 1 (1.5%) 1 (1.5%) Spanish 25 (38%) 25 (38%) Somali 14 (21%) 13 (19%) Total 66 66

Results of Matched Sample Analysis The Dependent t-test comparing mean total literacy scores on the Minneapolis Public Schools Beginning Kindergarten Assessment (BKA) for students in Reading Corps relative to students not in Reading Corps was statistically significant t (65)= 2.96; p=.004. The effect size (in standard deviation units) for Reading Corps vs. non-Reading Corps was .53 which is a moderate effect. In conclusion, the difference between Reading Corps students and non-Reading Corps students was a large difference which was statistically significant at a 99.5% confidence level. The Minneapolis Public School system has identified a score of 80 as the target for the literacy portion of the Beginning Kindergarten Assessment, representing a performance that predicts future success in reading development and adequate performance on state assessments of reading. As seen in Figure 3, the average total literacy score of 127.6 for Reading Corps participating students within the PICA Headstart program who continued on to Kindergarten within the Minneapolis Public Schools system well exceeds this target score.


Figure 3: Performance on Minneapolis BKA for Reading Corps and Non-Reading Corps Students

Kindergarten-Grade 3 Student Performance The four assessment tools utilized for the K-3 Reading Corps program are listed below. For a description of the technical characteristics of these assessments, the interested reader is referred to the 2007-2008 Minnesota Reading Corps program evaluation. For each assessment tool, a target score was identified as the goal for the beginning, middle, and end of the year. These target scores were based on research conducted at the St. Croix River Education District, which documented the predictive and concurrent validity of these measures with the Minnesota Comprehensive Assessment II. As a result of the strong correlations between performance on the selected fluency measures and on the MCAII, a series of cut scores has been identified. The table below specifies assessments given at each grade level and the cut scores for each assessment during several points throughout the school year. These cut scores, or target scores, define levels of performance on the fluency measures that strongly predict future success on the grade 3 Minnesota Comprehensive Assessment II. For example, a student who reads a Grade 1 passage in the winter of first grade at a rate of 20 words read correctly per minute has an 75% chance of earning a score in the “meets standards” or “exceeds standards” ranges on the 3rd grade MCAII two years later.


Grade Measure Fall Target Winter Target Spring Target

K Letter Sound Fluency 10 21 41 1 Nonsense Word Fluency 32 52 N/A 1 Oral Reading Fluency N/A 22 52 2 Oral Reading Fluency 43 72 90 3 Oral Reading Fluency 70 91 109

The target scores as listed above for each assessment used as a part of ongoing student literacy measurement in Reading Corps grow across years from age 3 to grade 3, define a pathway to success. Through consideration of the inherent growth that would occur for a child who met each of the targets, an expectation of growth rate at each grade level can be defined. For example, the fall grade 2 target score is 43 on oral reading fluency. The spring grade 2 target score on this measure is 90. To grow from 43 to 90 in one academic year, a student would need to gain 1.31 words correct per minute per week on the oral reading fluency assessment. Thus, 1.31 words growth per week becomes the expectation for 2nd grade growth rates. Because our targets are connected to the statewide assessment rather than normative performance of other students in local districts, we have a consistent and meaningful comparison across the state. Students participating in the Minnesota Reading Corps program are monitored frequently. The primary purpose of this progress monitoring is to enable those providing support to the student the ability to evaluate the effectiveness of current reading instruction, and to make data-based decisions regarding changes in instruction. For the purposes of outcomes evaluation, the progress monitoring data also provides a means for comparing the rate of growth of participating Minnesota Reading Corps students to the expected grade level growth rate. Students are selected for participation in the Reading Corps program because they are identified as having below grade level skills in reading. These students who achieve higher growth rates than those indicated by our targets are “catching up” to grade level expectation by making more than one year’s growth in one year’s time. For the state-wide Minnesota Reading Corps project, one measure of our success is the extent to which our participating students are achieving this primary goal. In the table below, a comparison between weekly growth rate expectations and the average weekly growth rates on program assessment measures of children participating in K-3 Reading Corps programs who have at least 3 data points collected per measure is listed. The current analysis includes all data collected between 9/1/2008 and 12/31/2008. Notably in all grade levels, the average growth rate of Minnesota Reading Corps participants exceeded the target growth rate (note that in Grade 1, while mean growth of participants in the second half of the year, on Oral Reading Fluency, was slightly below target growth, the mean growth in the first half of the year, on Nonsense Word Fluency, exceeded target growth). In other words, the average growth rate for Minnesota Reading Corps participants exceeded a rate of one year’s growth in one year’s time. This is significant as it demonstrates that participating students are actually catching up to grade level expectations.


Table 9: Kindergarten-Grade 3 Participant Growth

Grade K Grade 1 Grade 2 Grade 3

Letter Sound Fluency

Nonsense Word Fluency

(Fall to Winter)

Oral Reading Fluency (Winter

to Spring)**

Oral Reading

Fluency

Oral Reading

Fluency Reading Corps Mean

Growth Rate

2.36

2.48

N/A

2.10

2.30 Target

Growth Rate

1.15

1.11

1.67

1.31

1.08 Number

of Students

466

1,196

N/A

1,176

1,167 * Only students with 3 or more data points on the given measure were included in growth rate calculations ** Students in this group may have also participated in Grade 1 (NWF) The table below examines K-3 students who have at least 3 data points collected per measure, represented by the “Total # of Students”. The table presents the percentage of these students whose individual growth rates exceeded the target growth rates for that grade level and measure. This calculation represents the portion of the population participating in a Reading Corps intervention that had growth rates in excess of one year of growth in one years time. Percentages are given by region, and overall. Table 10: Kindergarten – Grade 3 Percentage of Students Above Growth Targets by Region

* TOTAL represents the total number of slopes analyzed, not the total number of students, as students in Grade 1 may have participated in two categories ** Students in this group may have also participated in Grade 1 (NWF) In the table below, the percent of K-3 MRC participating students with at least 3 progress monitor data points who have graduated from tutoring (exit) so far this year is reported. A 3 data point minimum was selected in order to remove from the analysis students who were erroneously included in the data set for a short time, but who did not have at least a reasonable minimum length of service prior to discontinuation.

Re Grade K

(LSF) Grade 1 (NWF)

Grade 1 (R-CBM)**

Grade 2 (R-CBM)

Grade 3 (R-CBM) TOTAL*

% Above Target 88.50% 91.01% N/A 80.95% 98.32% 89.97% Central – St. Cloud Total # of Students 113 178 N/A 168 179 638

% Above Target 89.58% 91.02% N/A 75.16% 94.43% 86.89% Metro – St. Paul Total # of Students 240 557 N/A 620 574 1,191

% Above Target 100.00% N/A 91.18% 96.49% 95.50% North Central – Grand Rapids Total # of Students 0 20 N/A 34 75 111

% Above Target 87.50% 94.81% N/A 75.42% 96.00% 89.28% North East - Duluth Total # of Students 32 154 N/A 118 125 429

% Above Target 96.30% 95.41% N/A 73.96% 97.33% 89.25% North West - Moorhead Total # of Students 27 109 N/A 96 75 307

% Above Target 88.89% 91.57% N/A 86.43% 95.54% 91.12% Southern - Rochester Total # of Students 54 178 N/A 140 157 529

% Above Target 89.48% 92.14% N/A 77.72% 95.63% 88.61% TOTAL Total # of Students 466 1,196 N/A 1,176 1,167 4,005


Table 11: Number and Percentage of students with 3 weeks of progress monitoring data who exit

Region Grade Number Exited Percent Exited Central - Saint Cloud K 38 33.63% Central - Saint Cloud 1 43 24.16% Central - Saint Cloud 2 34 20.24% Central - Saint Cloud 3 23 12.85% Metro – St. Paul K 36 15.00% Metro – St. Paul 1 58 10.41% Metro – St. Paul 2 73 11.77% Metro – St. Paul 3 91 15.85% North Central - Grand Rapids 1 3 15.00% North Central - Grand Rapids 2 0 0.00% North Central - Grand Rapids 3 10 17.54% North East - Duluth K 2 6.25% North East - Duluth 1 27 17.53% North East - Duluth 2 17 14.41% North East - Duluth 3 20 16.00% North West - Moorhead K 0 0.00% North West - Moorhead 1 14 12.84% North West - Moorhead 2 13 13.54% North West - Moorhead 3 9 12.00% Southern - Rochester K 0 0.00% Southern - Rochester 1 18 10.11% Southern - Rochester 2 3 2.14% Southern - Rochester 3 0 0.00% TOTAL K 76 16.31% TOTAL 1 163 13.63% TOTAL 2 140 11.90% TOTAL 3 153 13.11% TOTAL TOTAL 532 13.28%

In order to evaluate longer-term outcomes for students previously enrolled in the Minnesota Reading Corps program, additional analysis has been conducted on current performance of students who participated in MRC during the 2007-2008 school year. These participants were divided into two groups. The first group included all 2007-2008 MRC participants who graduated from tutoring prior to the end of the school year based on program exit criteria, or had a spring benchmark score that met or exceeded the spring target score for that grade level (Success in Prior Year = 1). The second group included all 2007-2008 MRC participants who did not meet either of those criteria (Success in Prior Year = 0). In the table below, average scores for the fall benchmark period during the 2008-2009 school year is reported as well as the percent of these students whose current year scores meet or exceed current enrollment grade targets.


Table 12: 2008-2009 fall mean scores and percent above target for 2007-2008 participants

Region Grade Measure

Success in Prior Year?

Number Tested

Average Score (Fall

Benchmark)

Percent Above Target (Fall Benchmark)

Central - Saint Cloud 1 NWF 0 15 48.7 7% Central - Saint Cloud 1 NWF 1 35 61.6 49% Central - Saint Cloud 2 R-CBM 0 12 51.1 0% Central - Saint Cloud 2 R-CBM 1 6 69.9 17% Central - Saint Cloud 3 R-CBM 0 9 19.8 0% Central - Saint Cloud 3 R-CBM 1 13 33.7 31% Metro – St. Paul 1 NWF 0 35 29.0 6% Metro – St. Paul 1 NWF 1 140 58.2 34% Metro – St. Paul 2 R-CBM 0 54 18.9 13% Metro – St. Paul 2 R-CBM 1 101 27.6 45% Metro – St. Paul 3 R-CBM 0 84 18.0 5% Metro – St. Paul 3 R-CBM 1 103 23.3 28% North Central - Grand Rapids 1 NWF 0 11 14.7 0% North Central - Grand Rapids 1 NWF 1 8 14.6 0% North Central - Grand Rapids 2 R-CBM 0 4 21.0 0% North Central - Grand Rapids 2 R-CBM 1 16 14.2 56% North Central - Grand Rapids 3 R-CBM 0 7 21.5 0% North Central - Grand Rapids 3 R-CBM 1 18 47.1 22% North East - Duluth 1 NWF 0 16 14.6 0% North East - Duluth 1 NWF 1 69 27.6 33% North East - Duluth 2 R-CBM 0 32 24.6 13% North East - Duluth 2 R-CBM 1 73 36.4 33% North East - Duluth 3 R-CBM 0 33 43.1 3% North East - Duluth 3 R-CBM 1 35 64.5 26% North West - Moorhead 1 NWF 0 14 41.5 14% North West - Moorhead 1 NWF 1 35 42.0 37% North West - Moorhead 2 R-CBM 0 31 63.7 19% North West - Moorhead 2 R-CBM 1 56 19.1 52% North West - Moorhead 3 R-CBM 0 36 30.8 0% North West - Moorhead 3 R-CBM 1 40 31.0 28% Southern - Rochester 1 NWF 0 6 26.4 0% Southern - Rochester 1 NWF 1 11 44.5 36% Southern - Rochester 2 R-CBM 0 22 40.1 45% Southern - Rochester 2 R-CBM 1 21 59.8 76% Southern - Rochester 3 R-CBM 0 20 17.0 10% Southern - Rochester 3 R-CBM 1 17 31.5 47%


The table below displays data from above in aggregate form, with the data collapsed by grade level to show overall percent of students who met or exceeded current fall targets by level of success during the 2007-2008 school year, by region, and across the state. In total 797 participating MRC students in this analysis either met exit criteria or scored at or above spring target during the 2007-2008 school year. 36.89% of these students continued to score at or above target on current fall measures during the 2008-2009 school year. By contrast, 441 participating MRC students did not meet exit criteria or spring target scores during the 2007-2008 school year. Only 8.84% of these students scored at or above target on current fall measures during the 2008-2009 school year: Table 13: Percent of students who met or exceeded fall 2008 targets by success in prior years

Region Success in Prior Year? Number Tested Percent Above Target

(Fall Benchmark) Central - Saint Cloud 0 36 3% Central - Saint Cloud 1 54 41% Metro – St. Paul 0 173 8% Metro – St. Paul 1 344 35% North Central - Grand Rapids 0 22 0% North Central - Grand Rapids 1 42 31% North East - Duluth 0 81 6% North East - Duluth 1 177 32% North West - Moorhead 0 81 10% North West - Moorhead 1 131 40% Southern - Rochester 0 48 25% Southern - Rochester 1 49 57% TOTAL 0 441 8.84% TOTAL 1 797 36.89% Are the organizations with which the MRC is working changing to adopt the practices of the MRC? At the pre-K level, Internal and Master coaches utilize the Early Literacy and Language Classroom Observation tool (ELLCO) to evaluate changes in the quality of the classroom environment in MRC Pre-K classrooms across the school year. This tool is offered through Brookes Publishing, and a technical report is available regarding the psychometric properties of this instrument on the company website at http://www.brookespublishing.com/store/books/smith-ellco/index.htm. Observations focus on 5 key literacy elements: classroom structure, curriculum, the language environment, books and book reading, and print and early writing supports. In the fall, coaches review observation data with teachers and MRC Members serving in the participating classrooms, and set goals for improving the literacy and language environment. Coaching during the school year is related to making progress toward these identified classroom goals. Spring observations are completed to identify and measure changes across the year in classroom environment. The tables below list numbers of participating classrooms as well as fall scores on the ELLCO tool.


Table 14: Participating Classrooms in ELLCO tool Assessment Region Number of Classrooms with ELLCO Score Central – St. Cloud 4 Metro – St. Paul 34 North Central – Grand Rapids 12 North East – Duluth 2 North West – Moorhead 13 Southern – Rochester 19 Total 84 Table 14: Median Scores on ELLCO tool

Scale Possible Total Score

Median Fall Score

Median Spring Score

Classroom Structure Subtotal 20 16 Curriculum Subtotal 15 10 Language Environment Subtotal 20 13.75 Books and Book Reading Subtotal 25 19 Print and Early Writing Subtotal 15 10 Internal Coaches and Site Supervisors will participate in electronic surveys toward the end of the school year in order to share their perceptions about change in their local systems resulting from participation in the MRC program. These data will be compiled in the final report. What is the impact of the MRC experience on the AmeriCorps Members? Minnesota Reading Corps members will participate in an electronic survey toward the end of the school year in order to share their perceptions of impact of the MRC program. These data will be compiled for the final report.

Minnesota Reading Corps Mid-Year Evaluation · IGDI Rhyming IGDI Picture ... assessment windows and...

Documents

Transcript of Minnesota Reading Corps Mid-Year Evaluation · IGDI Rhyming IGDI Picture ... assessment windows and...