Evaluation of the Michigan Literacy Progress Profile … of the Michigan Literacy Progress Profile...
Transcript of Evaluation of the Michigan Literacy Progress Profile … of the Michigan Literacy Progress Profile...
Evaluation of the MichiganLiteracy Progress Profile
(MLPP)
Robert D. CarpenterUniversity of Michigan
Center for the Improvement of Early Reading Achievement (CIERA)
Contact information:e-mail: [email protected]
CIERA: www.ciera.org
Evaluation TeamMichigan State University University of Michigan
Dr. P. David Pearson Dr. Scott Paris, ProfessorDistinguished Professor of Education CIERA, Department of Psychology,
Combined Program inEducation and Psychology
Gina Cervetti Robert CarpenterSharman Siebenthal Alison ParisBrigette Laier Graduate ResearchersGraduate Researchers CIERA, Combined Program in
Education and Psychology
Denise LarabellResearch Assistant
Ingham Intermediate School District
Tamara Bashore-Berg, SupervisorDepartment of Planning and Evaluation
Kathy Humphrey, Evaluation SpecialistDepartment of Planning and Evaluation
Jonathan Flukes, Evaluation AssistantDepartment of Planning and Evaluation
Kelly Trout, Evaluation AssistantDepartment of Planning and Evaluation
Allen Strozewski, Computer TechnicianDepartment of Planning and Evaluation
Chris Quinn, SecretaryDepartment of Planning and Evaluation
Overview
• Overall Evaluation
• Year 1 Test/Retest Reliability
• Year 1 Concurrent Validity
• Year 1 Summary
• Year 2 Overview
Overall Evaluation
Primary Goal:
To evaluate the reliability, validity,and utility of the MLPP
Evaluation Timeline
–Year 1•Test/Retest Reliability•Concurrent Validity
–Year 2•Predictive Validity•Consequential Validity
Purposes of Year 1
• To collect baseline data for long-termpredictive validity of the MLPP
• Examine the test/retest reliability of theMLPP enabling skills
• Examine the test/retest reliability of theMLPP oral reading assessments
• To test the concurrent validity of both theenabling and oral reading tasks
Method
• Data were collected in the spring andfall of 2001
• For test/retest reliability studentswere tested on the same tasks 2-4weeks apart by teachers andresearchers
• Concurrent validity established usingteacher and researcher MLPP dataand researcher collected TPRI andGates data
Sample
• Four sites across the state wereselected because they representedrural, urban, and suburbancommunities and were strongimplementors of the MLPP
• More than 700 K-3 studentsapproximately 25% in each grade
• 41 % Female
• 68 % Caucasian
UM Specific Procedures
• Met administrators and teachers toexplain project in spring 2001
• Retested students using the sameMLPP assessments and proceduresas teachers 2-4 weeks after theteacher administration (spring 2001)
• Tested students using the TPRI(spring 2001) and Gates (fall 2001) atMilan during a similar 4-week window
MLPP EnablingComponents
• Letter Identification
• Letter-Sound Identification
• Hearing and Recording Sounds
• Phonological Awareness–Rhyme–Onset Rim and Blending–Segmentation
• Concepts of Print
• Known Words
MLPP Oral ReadingFramework
• Sight Word/Decodable Word List
• Oral Reading–Accuracy–Fluency–Retelling–Comprehension
Test/Retest Correlationsfor MLPP Enabling TasksTask Milan Willow
RunLetter ID .94*** .97***
Letter Sound ID .91*** .74***
Phonemic Awareness
Rhyming .47* .73***
Blending .78*** .91***
Segmentation .10 .68***
H and R Sounds .93***
Concepts of Print .65*** .33
Known Words .73***
* = p< .05; ** = p< .01; *** = p< .001
Letter IdentificationMilan Scatterplot of Teacher and Researcher Scores
Researcher administration
5550454035302520151050
Tea
cher
Adm
inis
trat
ion
55
50
45
40
35
30
25
20
15
10
5
0
Grade
First Grade
Kindergarten
Willow Run Scatterplot of Teacher and Researcher Scores
Researcher Administration
5550454035302520151050
Tea
cher
Ad
min
istr
atio
n
55
50
45
40
35
30
25
20
15
10
5
0
Overall Regression Line
Agreement Line
Regression LineAgreement Line
Letter Sound IDMilan Scatterplot of Teacher and Researcher Scores
Researcher Administration
302520151050-5
Teac
her A
dmin
istra
tion
30
25
20
15
10
5
0
-5
Grade
First Grade
Kindergarten
Willow Run Scatterplot of Teacher and Researcher Scores
Researcher Administration
302520151050-5
Tea
cher
Adm
inis
trat
ion
30
25
20
15
10
5
0
-5
Overall Regression LineAgreement Line
Regression Line
Agreement Line
Phonemic Awareness- RhymingMilan Scatterplot of Teacher and Researcher Scores First Grade Only
Researcher Administration
181614121086420
Tea
cher
Adm
inis
trat
ion
18
16
14
12
10
8
6
4
2
0
Willow Run Scatterplot of Teacher and Researcher Scores
Researcher Administration
181614121086420
Teac
her A
dmin
istra
tion
18
16
14
12
10
8
6
4
2
0
Grade
First Grade
KindergartenOverall Regression Line
Agreement Line
Regression Line
Agreement Line
Phonemic Awareness- BlendingMilan Scatterplot of Teacher and Researcher Scores
Researcher Administration
181614121086420
Tea
cher
Ad
min
istr
atio
n
18
16
14
12
10
8
6
4
2
0
Willow Run Scatterplot of Teacher and Researcher Scores
Researcher Administration
181614121086420
Tea
cher
Adm
inis
trat
ion
18
16
14
12
10
8
6
4
2
0
Regression Line
Agreement Line
Regression LineAgreement Line
Phonemic Awareness- SegmentationMilan Scatterplot of Teacher and Researcher Scores
Researcher Administration
1086420
Tea
cher
Ad
min
istr
atio
n
26
24
22
20
18
16
14
12
10
8
6
4
2
0
Willow Run Scatterplot of Teacher and Researcher Scores
Researcher Administration
1086420-2
Tea
cher
Ad
min
istr
atio
n
10
8
6
4
2
0
-2
Regression Line
Agreement Line
Regression LineAgreement Line
Hearing and Recording SoundsWillow Run Scatterplot of Teacher and Researcher Scores
Researcher Administration
4035302520151050
Tea
cher
Ad
min
istr
atio
n
40
35
30
25
20
15
10
5
0Regression LineAgreement Line
Concepts of Print Milan Scatterplot of Teacher and Researcher Scores
Researcher Administration
242220181614121086420
Tea
cher
Adm
inis
trat
ion
24
22
20
18
16
14
12
10
8
6
4
2
0
Grade
First Grade
Kindergarten
Willow Run Scatterplot of Teacher and Researcher Scores
Researcher Administration
242220181614121086420
Tea
cher
Ad
min
istr
atio
n
24
22
20
18
16
14
12
10
8
6
4
2
0
Overall Regression LineAgreement Line
Regression Line
Agreement Line
Known WordsWillow Run Scatterplot of Teacher and Researcher Scores
Researcher Administration
65605550454035302520151050
Tea
cher
Ad
min
istr
atio
n
65
60
55
50
45
40
35
30
25
20
15
10
5
0
Regression Line
Agreement Line
Test/Retest Correlationsfor MLPP Oral Reading
Task Milan WillowRun
Word Recognition(Highest List)
.66*** .78***
Word Recognition(Scale)
.94*** .98***
Accuracy .51*** .46***
Fluency .54**
Rate .78***
Comprehension .35*
Retelling .53*** .10
* = p< .05; ** = p< .01; *** = p< .001
Highest Sight Word ListMilan Scatterplot of Teacher and Researcher Scores
Researcher Administration
211815129630
Teac
her A
dmin
istra
tion
21
18
15
12
9
6
3
0
Grade
Third Grade
Second Grade
Willow Run Scatterplot of Teacher and Researcher Scores
Researcher Administration
211815129630
Teac
her A
dmin
istra
tion
21
18
15
12
9
6
3
0
Grade
Third Grade
Second Grade
First GradeOverall Regression Line
Agreement Line
Overall Regression Line
Agreement Line
Total Sight Word ScoreFigure 19. Milan Scatterplot of Teacher and Researcher Scores
Researcher Administration
180160140120100806040200
Teac
her A
dmin
istra
tion
180
160
140
120
100
80
60
40
20
0
Grade
Third Grade
Second Grade
Figure 20. Willow Run Scatterplot of Teacher and Researcher Scores
Researcher Administration
120100806040200
Teac
her A
dmin
istra
tion
120
100
80
60
40
20
0
Grade
Third Grade
Second Grade
First Grade
Overall Regression Line
Agreement Line
Overall Regression Line
Agreement Line
AccuracyMilan Scatterplot of Teacher and Researcher Scores
Researcher Administration
1101009080706050
Teac
her A
dmin
istra
tion
110
100
90
80
70
60
50
Grade
Third Grade
Second Grade
Willow Run Scatterplot of Teacher and Researcher Scores in Decimals
Researcher Administration
1.101.00.90.80.70.60.50
Teac
her A
dmin
istra
tion
1.10
1.00
.90
.80
.70
.60
.50
Grade
Third Grade
Second Grade
First GradeOverall Regression Line
Agreement Line
Overall Regression LineAgreement Line
Fluency
Highest Passage Scatterplot of Teacher and Researcher Scores
Researcher Administration
543210
Teac
her A
dmin
istra
tion
5
4
3
2
1
0
Grade
Third Grade
Second GradeOverall Regression Line
Agreement Line
Reading Rate
Scatterplot of Highest Passage Reading Times
Researcher Administration
350300250200150100500
Teac
her A
dmin
istra
tion
350
300
250
200
150
100
50
0
Grade
Third Grade
Second GradeOverall Regression Line
Agreement Line
Comprehension
Scatterplot for Highest Passage
Researcher Administration
1109070503010-10
Teac
her A
dmin
istra
tion
110
90
70
50
30
10
-10
Grade
Third Grade
Second GradeOverall Regression Line
Agreement Line
RetellingMilan Scatterplot of Highest Passage Percent Propositions
Researcher Administration
80706050403020100
Teac
her A
dmin
istra
tion
80
70
60
50
40
30
20
10
0
Grade
Third Grade
Second Grade
Willow Run Scatterplot of Retelling Rubric Scores
Researcher Administration
4.54.03.53.02.52.01.51.0.5
Teac
her A
dmin
istra
tion
4.5
4.0
3.5
3.0
2.5
2.0
1.5
1.0
.5
Grade
Third Grade
Second Grade
First Grade
Overall Regression Line
Agreement Line
Overall Regression LineAgreement Line
Concurrent Validityfor MLPP Tasks
Task TPRI GMRT
Letter ID .94*** .82**
Phonemic Awareness .19
Rhyming .43**
Blending .81***
Word Recognition .79***
Accuracy .41*
Comprehension .07 .71***
* = p< .05; ** = p< .01; *** = p< .001
Summary ofYear 1 Findings
• High reliability for enabling skills
• Moderate reliability for oral reading tasks
• Moderate concurrent validity with theTPRI and the GMRT
Year 2
• We are currently collecting MEAP dataon 3rd grade students from year one inaddition to the MEAP data ofparticipants in previous MLPP studies tomeasure predictive validity
• Spring 2002 we will collect surveys andinterviews of students, teachers andadministrators to determine theconsequential validity of the MLPP