Using MEAP To Evaluate Teachers MERA Fall 2011 Conference Dave Treder
description
Transcript of Using MEAP To Evaluate Teachers MERA Fall 2011 Conference Dave Treder
Using MEAPTo Evaluate
Teachers
MERA Fall 2011 ConferenceDave Treder
Study Methods
• Looked at MEAP growth/change over two consecutive years (grade 4 MEAP to grade 5 MEAP):
Both Fall 07 to Fall 08 and Fall 08 to Fall 09
2007-08 School Year 2008-09 School YearSchools 71 72Teachers 188 192Students 4576 4751
With MEAP Scores, Yr1 & Yr2Schools 71 72Teachers 186 189Students 4223 4318
(92%) (91%)
Counts-- 20 of the 21 school districts In Genesee ISD
Description of Study Data
• 4th Grade Teachers • 20 of the 21 Districts In Genesee ISD• Based on the Districts’ SIS system• Business rules:
- If schedules weren’t used in the district, the homeroom teacher was coded as the math and reading teacher – in the districts that used schedules in the elementary grades, math and reading classes were used to designate the math and reading teacher
- Teachers were included if they were the teacher of record at the beginning of the school year (1st semester grades) and the end of the school year (last semester grades)
- Students included if they had the same teacher at the beginning of the school year (1st semester grades) and the end of the school year (2nd semester grades)
Number of 4th Grade Teachers, by District:F07-08, F08-09, and in the District Both Years
District Number N0708 N0809 Both Years1 36 37 122 24 22 173 15 15 144 12 12 115 11 14 96 11 12 107 9 9 88 9 9 69 9 9 810 9 9 811 7 6 512 5 6 513 5 5 314 5 5 415 4 3 316 4 4 417 4 4 418 3 3 319 2 3 220 2 2 1 TOTAL 186 189 137 74%
Clas
s Size
F07-08 Class Size Frequency
What was the size of the classrooms, for the teachers in this study?(2007-08 School Year)
Study MethodsMethod 11) Compute by-student residuals (non-linear regression, student predicted - actual scale score)2) Compute Mean Teacher residual
Method 21) Compute by-student Performance Level (PL) Change Index2) Compute Mean Teacher PL change index
• For each method, compute indices for F07-F08 and F08-F09.
• Look at the relationship between the two years.Also Tried:1) Converting Scales Scores to Z Scores2) Compute by-student ‘’change”: Year2 Z_Score – Year1 Z_Score3) Compute mean teacher change
Computing By-Student Residuals (Predicted – Actual)Math F07-F08
Scal
e Sc
ore,
Gra
de 5
Scale Score, Grade 4
How Much of the Variance can be Explained at the Teacher Level?By Teacher, Mean Residual, Fall07-08
Math: 15%
Reading: 1.5%
What is the Relationship Between Teacher Level Mean Residuals, F07-08 & F08-09?
Correlation:Math: .32 Reading: .19
Method 1 – General Results
Rank – F07 – F08
Ran
k –
F08
– F0
9
Math –Teacher Ranks (Mean Residual), F07-08 X F08-F09
How do Teachers’ Ranks Compare, F07-08 to F08-F09?
MATH Within District Relationship - Quintile Rank – SS residual
How do Teachers’ Ranks Compare, Within Districts, F07-08 to F08-F09?
Rank – F07 – F08
Ran
k –
F08
– F0
9
Reading –Teacher Ranks (Mean Residual), F07-08 X F08-F09
How do Teachers’ Ranks Compare, F07-08 to F08-F09?
READING Within District Relationship - Quintile Rank – SS residual
How do Teachers’ Ranks Compare, Within Districts, F07-08 to F08-F09?
State Top-to-Bottom RankingPerformance Level Change IndexFOR EACH THE SCHOOL, THE PERFORMANCE LEVEL CHANGE SCORES ARE SUMMED ACROSS STUDENTS AND AN AVERAGE IS TAKEN TO CREATE THE TWO-YEAR AVERAGE PERFORMANCE LEVEL CHANGE INDEX. THE INDEX FOR EACH SCHOOL IS THEN COMPARED TO THE REST OF THE SCHOOLS IN THE STATE TO CREATE A SCHOOL-LEVEL PERFORMANCE LEVEL CHANGE Z-SCORE. THAT Z-SCORE IS THEN MULTIPLIED BY 1/4 TO CONTRIBUTE TO THE OVERALL SCHOOL INDEX IN THE CONTENT AREA.
Method 21) Compute by-student Performance Level (PL) Change Index2) Compute Mean Teacher PL change index
-- Computed with the New Cut Score Sub-Level Cut Scores --
2011 Top-to-Bottom PL Change Business Rules
-60
-50
-40
-30
-20
-10
0
10
20
30
40
50
60
ELEM_MS_RD_IMP_8_9
-60 -50 -40 -30 -20 -10 0 10 20 30 40 50 60
N = 2500r = .032p = .11
Relationship between: PL Change/Growth F07-to-F08 & PL Change/Growth F08-to-F09
Top-to-Bottom List*- School-Level Correlation - Reading, Grds 3-8F0
7 to
F08
F08 to F09*Original Top-to-Bottom Rules: (N Imp – N Dec )/ N Matched
ELEM/MS PL Change & Achievement Status
How much of the Variance can be Explained at the Teacher Level?By Teacher, Mean Residual, Fall07-08
Math: 15%
Reading: 4%
What is the Relationship Between Teacher Level Mean Residuals, F07-08 & F08-09? Correlation
Math: .42 Reading: .15
Method 2 – General Results
PL Change Index – F07 – F08
PL
Cha
nge
Inde
x –
F08
– F0
9
R = .42
Math –By Teacher PL Change Index, F07-08 X F08-F09
Relationship Between Teacher PL Change Index, F07 to F08 X F08 to F09
Mean MEAP Scl Score Z– F07
PL
Cha
nge
Inde
x –
F07
– F0
8
R = -.46
MATH –By Teacher, Mean MEAP Scl Score Z– F07 X PL Change IndexF07-08
What’s the Relationship Between a Teacher’s Classroom Starting Point (Mean Scale Score) and Their PL Change Index?
2011 State Top-to-bottom Math – Pl Change X Achievement Status(Old Cut Scores, New Rules)
-3
-2
-1
0
1
2
3
4
5
-5 -4 -3 -2 -1 0 1 2 3
PL C
hang
e Z
scor
e
Achievement Status z score
R = - .81
MATH –Teacher Ranks (PL Change Index), F07-08 X F08-F09
Rank – F07 – F08
Ran
k –
F08
– F0
9How do Teachers’ Ranks Compare, F07-08 to F08-F09?
MATH –Within District Relationship quintile rank - PL Change Index
How do Teachers’ Ranks Compare, Within Districts, F07-08 to F08-F09?
READING –Teacher Ranks (PL Change Index), F07-08 X F08-F09
Rank – F07 – F08
Ran
k –
F08
– F0
9How do Teachers’ Ranks Compare, F07-08 to F08-F09?
READING –Within District Relationship quintile rank - PL Change Index
How do Teachers’ Ranks Compare, Within Districts, F07-08 to F08-F09?
PL
Cha
nge
Inde
x –
F07
– F0
8
Mean Residual
Relationship Between Teacher Mean Residual & PL Change Index
READING –By Teacher, Mean Residual X PL Change Index (F07to 08)
R = .88
PL
Cha
nge
Inde
x –
F07
– F0
8
Mean Residual – F07 – F08
Relationship Between Teacher Mean Residual & PL Change Index
MATH –By Teacher, Mean Residual X PL Change Index (F07to 08)
R = .77
…Error rates for teacher-level analyses will be about 26 percent if three years of data are use for estimation. This means that in a typical performance measurement system, more than 1 in 4 teachers who are truly average in performance will be erroneously identified for special treatment, and more than 1 in 4 teachers who [exceed] average performance by will be overlooked.
-17%-16%-6%
-12%
6%
-6%
11%3% 1%
-4%
-29%
-3% -2%-13%
-4%
13%11%
-4%-13%-14%
-50%-40%-30%-20%-10%
0%10%20%30%40%50%
F06-
F07
F07-
F08
F08-
F09
F09-
F10
F06-
F07
F07-
F08
F08-
F09
F09-
F10
F06-
F07
F07-
F08
F08-
F09
F09-
F10
F06-
F07
F07-
F08
F08-
F09
F09-
F10
F06-
F07
F07-
F08
F08-
F09
F09-
F10
Reading
MATH
-23%-12%
-22%-25%
-12%-9%-17%
-12%
4% 8%16%
22%
1%12% 10%
-1%
-41%-33%
23%12%
-50%-40%-30%-20%-10%
0%10%20%30%40%50%
F06-
F07
F07-
F08
F08-
F09
F09-
F10
F06-
F07
F07-
F08
F08-
F09
F09-
F10
F06-
F07
F07-
F08
F08-
F09
F09-
F10
F06-
F07
F07-
F08
F08-
F09
F09-
F10
F06-
F07
F07-
F08
F08-
F09
F09-
F10
Math
Gr3-Gr4 Gr4-Gr5 Gr5-Gr6 Gr6-Gr7 Gr7-Gr8
MEAP PL Change Index, Math & Reading, F06 -- F10 (GISD Students)
Is Your PL Change Index Helped (or Hurt), based on the Grade You Teach? (Or, On the Grade Levels in Your School)?
Original Top-to-Bottom Rules: (N Imp – N Dec )/ N Matched
PL C
hang
e In
dex
Is Your PL Change Index Helped (or Hurt), based on the Grade You Teach? (Or, On the Grade Levels in Your School?
Math
Reading
(OLD Cut Scores, NEW PL Index Rules)MEAP PL Change Index, Math & Reading, 08-09 & 09-10 (GISD Students)
PL C
hang
e In
dex
MATH
READING
Old & New Cut Scores
School-Level Std Dev - Math: .21 (08-09) .27 (09-10) Reading: .16 (08-09) .18 (09-10)
Math
Reading
Reading
Is Your PL Change Index Helped (or Hurt), based on the Grade You Teach? (Or, On the Grade Levels in Your School?
MEAP PL Change Index, Math & Reading, 08-09 & 09-10 (GISD Students)(NEW Cut Scores, NEW PL Index Rules)
PL C
hang
e In
dex
Developing a Growth/Progress Measure, Based on a Single Summative Assessment, that:
1) is stable across years;2) doesn’t punish high performing students/
classes/schools;3) incorporates the need to expect low performing
students/classes/schools to “grow” at a greater rate than low performing students/classes/schools.
Ain’t easy
Conclusion