Evaluation Results

2002-2003

Missouri Reading InitiativeMissouri Reading Initiative

MRI’s Evaluation Activities:

Surveys*Teacher Beliefs and Practices (pre/post)Annual Participant Questionnaire

*Data CollectionTest Scores

Standardized TestsClassroom Assessments (DRA)MAP

DemographicsSpecial Education Information

MAP Analyses*For schools beginning in 2002

MAP ANALYSES:

Map analyses compare schools that have finished the MRI program with randomly chosen samples of non-MRI elementary

schools

Results indicate MRI schools generally outperform non-MRI schools

Notes for MAP Analyses Note: With all the following MAP Analyses charts the

numbers are not as important as the comparative performance between MRI and non-MRI schools. This is because:

1. There is a lot of variation in the data from year to year and school to school

2. Statewide 2003 MAP scores went down from 2002 3. The calculation of the baseline is slightly different:

For the 2002 schools 1999 was the baseline For the 2003 schools an average of 1999/2000 was the

baseline

Notes for Chart 11. This chart compares MRI and non-MRI schools

performance on MAP Communication Arts Index

2. Baseline year is 1999; Outcome year is 2002

3. Each sample has 15 schools = number of schools that finished MRI Spring 2002

4. Total random sample = 150 (large enough number to satisfy statistical significance, high confidence levels)

% Change in MAP Commuincation Arts Index 1999-2002 for MRI and 10 Samples

1.6

3.7

3.0

-0.1

1.3

3.1

1.8 1.9

0.5

1.2

1.8

6.5

-1.0

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

Samples:1-10

Sample Average N=150

MRI N=15

Notes for Chart 21. This chart compares MRI and non-MRI schools

performance on MAP Reading Index

2. Baseline year is 1999; Outcome year is 2002

3. Each sample has 15 schools = number of schools that finished MRI Spring 2002

4. Total random sample = 150 (large enough number to satisfy statistical significance, high confidence levels)

% Change in MAP 3rd Grade Reading Index 1999-2002

MRI and 10 Samples

3.0

4.5

8.5

2.83.4

6.5

5.35.6

2.8

3.9

4.6

13.1

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

Samples:1-10

Sample Average N=150

MRI N=15

Notes for Chart 31. This chart compares MRI and non-MRI schools

performance on MAP Communication Arts Index 2003

2. Baseline year is an average of 1999/2000 (smoothes out variations); Outcome year is 2003

3. Each sample has 20 schools = number of schools that finished MRI Spring 2003

4. Total random sample 200 (large enough number to satisfy statistical significance, high confidence levels)

MAP Communication Arts Index 1999/2000 to 2003

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

5.0

6.0

Samples

Total sample: N=200Mean = 0.9

MRI: N=20Mean = 4.8

0.2 0.3

1.4

-0.3

1.21.6

3.0

2.3

-0.1

-1.1

95% Confidence Level=/- 1.0%

Samples: 1-10

Notes for Chart 41. This chart compares MRI and non-MRI schools

performance on MAP Reading Index 2003

2. Baseline year is an average of 1999/2000 (smoothes out variations); Outcome year is 2003

3. Each sample has 20 schools = number of schools that finished MRI Spring 2003

4. Total sample 200 (large enough number to satisfy statistical significance, high confidence levels)

MAP Reading Index 1999/2000 to 2003

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

Samples

Total sample: N=200Mean = 1.6

MRI: N = 20Mean = 7.3

1.4

-0.3

3.8

-0.7

-0.1

2.8

3.6

4.3

0.3 0.4

95% Conficence Level +/- 1.5%

Samples: 1-10

Notes for Chart 5 Left Side: For 20 MRI schools, there was an average 19.5%

decrease (the desirable direction) in the number of students scoring in the bottom two categories (Step 1 and Progressing) of the 2003 3rd Grade Communication Arts test as compared to a 2.8% increase (not the desirable direction) for 200 randomly selected Missouri elementary schools.

Right Side: For 20 MRI schools, there was an average 32.5% increase (the desirable direction) in the number of students scoring in the top two categories (Proficient and Advanced) of the 2003 3rd Grade Communication Arts test as compared to a 6.6% increase for 200 randomly selected Missouri elementary schools.

2003 MAP 3rd Communication Arts CategoriesAverage Change in Percentage of Students

Baseline=99/00 Average

-30.0

-20.0

-10.0

0.0

10.0

20.0

30.0

40.0

Perc

enta

ge C

hang

e

2.8%

-19.5%

6.6%

32.5%

CA categories 1&2

"Down" is the desireable

direction

CA categores 4&5

"Up" is the desireable direction

Red : Random Sample N=200

Black: MRI

N=20

Notes for Chart 6 Left Side: 75% of MRI schools had decreases (the desirable

direction) in the number of students who scored in the two lowest categories (Step 1 and Progressing) on the 2003 MAP 3rd Grade Communication Arts test as compared to 48% of the schools in the random sample.

Right Side: 75% of MRI schools had increases (the desirable direction) in the number of students who scored in the two lowest categories (Proficient and Advanced) on the 2003 MAP 3rd Grade Communication Arts test as compared to 44.5% of the schools in the random sample.

2003 MAP 3rd Grade Communication ArtsPercentage of Schools that went in the Desireable Direction

Baseline=99/00 Average

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

Perce

ntag

e: Ba

selin

e to 2

003

CA categories 1&2

"Down" is the desireable direction

CA categores 4&5

"Up" is the desireable direction

48%

75%

44.5%

75%

Red : Random Sample N=200

Black: MRI

N=20

2003 MAP: Average Yearly Progress

Statewide: 49.7% Met AYP MRI 81% Met AYP (N=60/74)

Of the 14 that did not meet AYP: Six met AYP for All Students

3 - Did not meet AYP for FRL 1 - Did not meet AYP for IEP2 - <95% of eligible group (30+)

Participant Survey

Participants rate the usefulness of component utilization,practice change, "buy in", attitudes toward the program and trainer, etc.

Results drive program change; e.g., Orientation

Notes for Participant Survey Slide This slide introduces the survey and its uses. The

example is the Table in the next slide which demonstrates how the survey is often used. In this case:

1. 2002 survey respondents identified the problem of being “overwhelmed”

2. Program responded by redesigning orientation and other details

3. Program satisfaction for 1st year schools improved from 02 to 03 surveys

Notes for School Rating Slide The year (e.g., 01-02; 02-03) indicates the year of the survey The number (e.g., 1st, 2nd, 3rd) indicates the MRI year for those schools

This slide provides evidence for two statements:1. There is a higher degree of satisfaction expressed by participants the

longer they are in the program. For example, 68% (23/34) of schools who began in 01-02 rated the program better than 4.0 as in their 2nd year as opposed compared to 47% (16/34) in their 1st year. At the same time only 9% (3/34) of those schools rated the program below 3.5 in their 2nd year as opposed to 29% (10/34) of the same schools in their 1st year.

2. There has been an increase in satisfaction each year by schools at the same point of the program. For example 72% (13/18) of 1st year schools in 02-03 rated the program better than 4.0 as compared to 47% (16/34) of 1st year schools in 01/02. (see previous slide notes)

School Ratings: 01-03 Reflecting on the effectiveness of the MRI program

as a whole, how would you rate it?

Category1st

01-02

1st

02-03

2nd

01-02

2nd

02-03

3rd

02-03

2.0-2.49 1 0 0 0 0

2.5-2.99 4 1 3 1 0

3.0-3.49 5 2 0 2 1

3.5-3.99 8 2 6 8 4

4.0-4.49 12 9 7 15 8

4.5-5.0 4 4 7 8 9

TOTALS 34 18 23 34 22

Special Education

We track the effects of MRI on Special education in two ways:

1. Beginning with schools that started MRI in the Fall of 2002, all students with IEPs are identified and the type of IEP is described (Reading, Math, Speech, etc.)

2. Annual reports are made from every school about their IEP evaluation process (see following slide)

Notes for Special Education Table This Table is for schools that were in their 3rd year 2001-2002 Many schools do not have this data, or it is not easily

accessed so many schools did not report. 9 of 18 in 2002 5 0f 23 in 2003 The data we do have all points to a decrease in referrals,

evaluations, and assignment of IEP over the time schools participate in MRI.

We are collecting this data from the onset of schools that began in 2002-2003 so that we will have a complete report in 2005.

Special Education Results

Special Education Results:

School 98-99 99-00 00-01 01-02 Ref Eval IEP Ref Eval IEP Ref Eval IEP Ref Eval IEP #1 na na na 14 14 11 12 12 12 5 5 4

#2 na na na 22 14 13 18 13 10 15 13 2

#3 4 2 2 7 5 2 6 5 1 2 0 0

#4 na na na na na na na na na 10 5 2

#5 na na na na na 12 17 12 7 7 7 2

#6 na na na na na na 58 na na 44 na 7

#7 1 1 1 1 1 0 1 1 1 2 2 2

#8 15 3 3 7 4 1 10 3 2 7 2 2

#9 na na na 7 7 5 5 5 3 4 3 2

Ref. = Number of student referrals to site teacher support team Eval. = Number of student referrals special education evaluation IEP = Number of IEP placements

DRA GRADE LEVEL (GL) KEY

(<GL) = (>GL) 1st Grade Fall <3 3-4 >4

1st Grade Spring <16 16-18 >18

2nd Grade Fall <16 16-18 >18

2nd Grade Spring <24 24-28 >28

3rd Grade Fall <24 24-28 >28

3rd Grade Spring <34 34-38 >38

MRI DRAs: Matched PairsMEDIANS

Fall and Spring 2002-2003

0

5

10

15

20

25

30

35

40

1st GradeN=722

2nd GradeN=915

3rd GradeN=865

FALL SPRING FALL SPRING FALL SPRING FALL SPRING FALL SPRING FALL SPRING