Prepared by Steven H. Shaha, PhD, DBA
description
Transcript of Prepared by Steven H. Shaha, PhD, DBA
PD 360 Impact Assessment:Updated Findings
regarding the Impact of PD 360 on
Student Proficiency Rates
Prepared by Steven H. Shaha, PhD, DBA
1
PD 360 Impact AssessmentPD 360 Impact AssessmentExecutive Summary: Initial FindingsExecutive Summary: Initial Findings
• Statistically significant* advantages were verified favoring schools with PD 360 versus District Benchmarks.– 4th Grade and 8th Grade for:
• Math proficient and advanced (p<.001)
• Reading proficient and advanced (p<.001)
2* Statistical significance establishes genuine differences between groups and verifies that impacts were “real” and not merely due to chance and, in this case, due to any pre-existing biases in group differences. The appropriate p-values are included with all differences explained herein.
Impacts of PD 360 for4th Grade
3
Note to Reader: To better dramatize the magnitude of the consistently favorable impact of PD 360, graphics included hereafter represent a variety of perspectives and a sampling of different interpretive insights, and not an exhaustive nor uniformly arrayed set of results.
53.554.054.555.055.556.056.557.057.558.0
Prior Year District Benchmark
Post-PD 360 Year
Math Proficient - 4th Grade
High Users
Low Users
Improvements in Math Proficiency for PD 360
4
High Utilizing Schools improved High Utilizing Schools improved by 4.9% over the District by 4.9% over the District benchmark benchmark (p<.001)
and Low Utilizing School and Low Utilizing School improved by 2.0%improved by 2.0% (p<.01)
High Utilizing Schools improved High Utilizing Schools improved by 4.9% over the District by 4.9% over the District benchmark benchmark (p<.001)
and Low Utilizing School and Low Utilizing School improved by 2.0%improved by 2.0% (p<.01)
33.033.534.034.535.035.536.036.537.037.538.038.5
Prior Year District Benchmark
Post-PD 360 Year
Math Advanced - 4th Grade
High Users
District Avg
Improvements in Math Advanced for PD 360
High Utilizing Schools High Utilizing Schools significantly outperformed significantly outperformed collective district change collective district change between years to create a between years to create a performance gap of 9.0% performance gap of 9.0% (p<.001)
High Utilizing Schools High Utilizing Schools significantly outperformed significantly outperformed collective district change collective district change between years to create a between years to create a performance gap of 9.0% performance gap of 9.0% (p<.001)
5
High Utilizing Schools High Utilizing Schools significantly outperformed the significantly outperformed the collective district benchmark collective district benchmark by 1.4% by 1.4% (p<.05)
While the district fell by 7.0% While the district fell by 7.0% (p<.001)
High Utilizing Schools High Utilizing Schools significantly outperformed the significantly outperformed the collective district benchmark collective district benchmark by 1.4% by 1.4% (p<.05)
While the district fell by 7.0% While the district fell by 7.0% (p<.001)
60.0
62.0
64.0
66.0
68.0
70.0
72.0
Prior Year District Benchmark
Post-PD 360 Year
Reading Proficient - 4th Grade
High Users
Low Users
Improvements in Reading Proficiency for PD 360
High Utilizing Schools significantly High Utilizing Schools significantly outperformed the collective outperformed the collective district benchmark by 10.3% district benchmark by 10.3% (p<.001)
High Utilizing Schools significantly High Utilizing Schools significantly outperformed the collective outperformed the collective district benchmark by 10.3% district benchmark by 10.3% (p<.001)
6
Low Utilizing Schools significantly Low Utilizing Schools significantly outperformed the collective outperformed the collective district benchmark by 7.2% district benchmark by 7.2% (p<.001)
Low Utilizing Schools significantly Low Utilizing Schools significantly outperformed the collective outperformed the collective district benchmark by 7.2% district benchmark by 7.2% (p<.001)
High Utilizing Schools significantly High Utilizing Schools significantly outperformed the Low Utilizing outperformed the Low Utilizing Schools by 2.9% Schools by 2.9% (p<.01)
High Utilizing Schools significantly High Utilizing Schools significantly outperformed the Low Utilizing outperformed the Low Utilizing Schools by 2.9% Schools by 2.9% (p<.01)
56.0
58.0
60.0
62.0
64.0
66.0
68.0
70.0
72.0
Prior Year District Benchmark
Post-PD 360 Year
Reading Proficient - 4th Grade
High Users
Low Users
District Avg
Improvements in Reading Proficiency for PD 360
7
High Utilizing Schools created a High Utilizing Schools created a performance gap of 14.0% performance gap of 14.0% (p<.001)
High Utilizing Schools created a High Utilizing Schools created a performance gap of 14.0% performance gap of 14.0% (p<.001)
High Utilizing Schools significantly High Utilizing Schools significantly outperformed the collective outperformed the collective district benchmark by 10.3% district benchmark by 10.3% (p<.001)
While the district fell by 3.2% While the district fell by 3.2% (p<.001)
High Utilizing Schools significantly High Utilizing Schools significantly outperformed the collective outperformed the collective district benchmark by 10.3% district benchmark by 10.3% (p<.001)
While the district fell by 3.2% While the district fell by 3.2% (p<.001)
Low Utilizing Schools significantly Low Utilizing Schools significantly outperformed the collective outperformed the collective district benchmark by 7.2% district benchmark by 7.2% (p<.001)
Low Utilizing Schools significantly Low Utilizing Schools significantly outperformed the collective outperformed the collective district benchmark by 7.2% district benchmark by 7.2% (p<.001)
Impacts of PD 360 for8th Grade
8
Note to Reader: To better dramatize the magnitude of the consistently favorable impact of PD 360, graphics included hereafter represent a variety of perspectives and a sampling of different interpretive insights, and not an exhaustive nor uniformly arrayed set of results.
50.0
51.0
52.0
53.0
54.0
55.0
56.0
Prior Year District Benchmark
Post-PD 360 Year
Math Proficient - 8th Grade
High Users
Low Users
Improvements in Math Proficiency for PD 360
9
High Utilizing Schools improved High Utilizing Schools improved by 4.9% over the District by 4.9% over the District benchmark benchmark (p<.001)
and Low Utilizing School and Low Utilizing School improved by 2.6%improved by 2.6% (p<.01)
High Utilizing Schools improved High Utilizing Schools improved by 4.9% over the District by 4.9% over the District benchmark benchmark (p<.001)
and Low Utilizing School and Low Utilizing School improved by 2.6%improved by 2.6% (p<.01)
20.021.022.023.024.025.026.027.028.029.030.0
Prior Year District Benchmark
Post-PD 360 Year
Math Advanced - 8th Grade
High Users
Low Users
Improvements in Math Advanced for PD 360
10
High Utilizing Schools improved High Utilizing Schools improved by 28.8% over the District by 28.8% over the District benchmark benchmark (p<.001)
and Low Utilizing School and Low Utilizing School improved by 5.6%improved by 5.6% (p<.001)
High Utilizing Schools improved High Utilizing Schools improved by 28.8% over the District by 28.8% over the District benchmark benchmark (p<.001)
and Low Utilizing School and Low Utilizing School improved by 5.6%improved by 5.6% (p<.001)
56.0
58.0
60.0
62.0
64.0
66.0
68.0
70.0
Prior Year District Benchmark
Post-PD 360 Year
Reading Proficient - 8th Grade
PD 360 Users
District Avg
Improvements in Reading Proficiency for PD 360
11
High Utilizing Schools High Utilizing Schools significantly outperformed significantly outperformed collective district change collective district change between years to create a between years to create a performance gap of 11.9% performance gap of 11.9% (p<.001)
High Utilizing Schools High Utilizing Schools significantly outperformed significantly outperformed collective district change collective district change between years to create a between years to create a performance gap of 11.9% performance gap of 11.9% (p<.001)
PD 360 Utilizing Schools PD 360 Utilizing Schools significantly outperformed the significantly outperformed the collective district benchmark collective district benchmark by 8.3% by 8.3% (p<.001)
While the district fell by 3.2% While the district fell by 3.2% (p<.001)
PD 360 Utilizing Schools PD 360 Utilizing Schools significantly outperformed the significantly outperformed the collective district benchmark collective district benchmark by 8.3% by 8.3% (p<.001)
While the district fell by 3.2% While the district fell by 3.2% (p<.001)
30.030.531.031.532.032.533.033.534.034.5
Prior Year District Benchmark
Post-PD 360 Year
Reading Advanced - 8th Grade
High Users
District Avg
Improvements in Reading Advanced for PD 360
12
High Utilizing Schools High Utilizing Schools significantly outperformed significantly outperformed collective district change collective district change between years to create a between years to create a performance gap of 8.1% performance gap of 8.1% (p<.001)
High Utilizing Schools High Utilizing Schools significantly outperformed significantly outperformed collective district change collective district change between years to create a between years to create a performance gap of 8.1% performance gap of 8.1% (p<.001)
High Utilizing Schools High Utilizing Schools significantly outperformed the significantly outperformed the collective district benchmark collective district benchmark by 5.7% by 5.7% (p<.001)
While the district fell by 2.2% While the district fell by 2.2% (p<.05)
High Utilizing Schools High Utilizing Schools significantly outperformed the significantly outperformed the collective district benchmark collective district benchmark by 5.7% by 5.7% (p<.001)
While the district fell by 2.2% While the district fell by 2.2% (p<.05)
PD 360 Impact Assessment:Former Findings
regarding the Impact of PD 360 on
Student Proficiency Rates
Prepared by Steven H. Shaha, PhD, DBA
13
PD 360 Impact AssessmentPD 360 Impact AssessmentExecutive Summary: Initial FindingsExecutive Summary: Initial Findings
• Statistically significant and resounding advantages were identified favoring schools with PD 360.
• The higher the quantified utilization, the greater the statistical advantage.– High utilization led to performance advantages ranging
from twicefrom twice to 15 timesto 15 times the gains in proficiency rates versus district averages (p<.01 and p<.001, respectively).
– Low utilization led to gains in proficiency rates from 10% from 10%
to twice to twice the district averages (p<.01 and p<.001, respectively).
14
Improvements in Proficiency Rates Correlated with PD 360
0.00
0.50
1.00
1.50
2.00
2.50
3.00
School Math Change School Reading Change
Comparative Change forLow vs. High Utilization
Low Utilization
High Utilization
Districts Average
Low Utilization
High Utilization
Districts Average
Percent Advantage for High vs.
Low Utilization
Statistical Significance
School Math Change 1.18 2.54 0.16 115.3% p<.001 2.2times greater improvement
School Reading Change 1.22 2.13 1.12 74.4% p<.001 1.7times greater improvement
An advantage of:
High Utilization results in more the TWICE the gains in Math proficiency rates than does Low Utilization (p<.001)
High Utilization results in nearly TWICE the gains in Reading proficiency rates than does Low Utilization (p<.001)
High Utilizers Outperformed all othersHigh Utilizers Outperformed all others
But even low utilizers outperformed no PD 360
15
0.00
0.50
1.00
1.50
2.00
2.50
3.00
Math Change Reading Change
Axi
s Ti
tle
Comparative Change forHigh Utilization vs. Districts Average
High Utilization Schools
Districts Average
High Utilization
SchoolsDistricts Average
Percent Advantage
Statistical Significance
Math Change 2.54 0.16 1484.7% p<.001 15.8times greater improvement
Reading Change 2.13 1.12 90.6% p<.001 1.9times greater improvement
An advantage of:
High Utilization results in more the FIFTEEN times the gains in Math proficiency rates than the Districts average (p<.001)
High Utilization results in nearly TWICE the gains in Reading proficiency rates than the Districts average (p<.001)
High Utilizers High Utilizers outperformed no PD 360
15.8 times greater gains in Math vs. District
90.6% greater gains in Reading vs. District
16
Improvements in Proficiency Rates Correlated with PD 360
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
Math Change Reading Change
Comparative Change forLow Utilization vs. Districts Average
Low Utilization Schools
Districts Average
Low Utilization results in more the SEVEN times the gains in Math proficiency rates than the Districts average (p<.001)
Low Utilization results in nearly 10% greater gains in Reading proficiency rates than the Districts average (p<.01)
But even Low Utilizers Low Utilizers outperformed no PD 360
7.4 times greater gains in Math vs. District
9.3% greater gains in Reading vs. District
Low Utilization
SchoolsDistricts Average
Percent Advantage
Statistical Significance
Math Change 1.18 0.16 636.1% p<.001 7.4times greater improvement
Reading Change 1.22 1.12 9.3% p<.01 1.1times greater improvement
An advantage of:
17
Improvements in Proficiency Rates Correlated with PD 360
High Min/User Schools
Districts Average
Percent Advantage
Statistical Significance
Math Change 1.91 0.16 1096.6% p<.001 12.0times greater improvement
Reading Change 1.92 1.12 71.5% p<.001 1.7times greater improvement
An advantage of:
0.00
0.50
1.00
1.50
2.00
2.50
Math Change Reading Change
Comparative Change forHigh Min/User Schools vs. Districts
High Min/User Schools
Districts Average
High Minutes per User result in more the TWELVE times the gains in Math proficiency rates than Districts avg. (p<.001)
High Minutes per User result in nearly TWICE the gains in Reading proficiency rates than the Districts average (p<.001)
More Minutes per User More Minutes per User resulted in Higher Performance
12.0 times greater gains in Math vs. District
71.5% greater gains in Reading vs. District
18
Improvements in Proficiency Rates Correlated with PD 360
0.000.200.400.600.801.001.201.401.601.80
Math Change Reading Change
Comparative Change forLow Min/User Schools vs. Districts
Low Min/User Schools
Districts Average
Low Min/User Schools
Districts Average
Percent Advantage
Statistical Significance
Math Change 1.67 0.16 943.4% p<.001 10.4times greater improvement
Reading Change 1.49 1.12 32.8% p<.001 1.3times greater improvement
An advantage of:
Low Minutes per User result in more the TEN times the gains in Math proficiency rates than the Districts average (p<.001)
Low Minutes per User result in more than 30% greater gains in Reading proficiency rates than the Districts average (p<.001)
But even Low Minutes per User Low Minutes per User outperformed no PD 360
10.4 times greater gains in Math vs. District
32.8% greater gains in Reading vs. District
19
Improvements in Proficiency Rates Correlated with PD 360
High Pct of Users that
Viewed Schools
Districts Average
Percent Advantage
Statistical Significance
Math Change 2.17 0.16 1254.2% p<.001 13.5times greater improvement
Reading Change 2.19 1.12 95.8% p<.001 2.0times greater improvement
An advantage of:
0.00
0.50
1.00
1.50
2.00
2.50
Math Change Reading Change
Comparative Change forHigh Pct Viewer Schools vs. Districts
High Pct of Users that Viewed Schools
Districts Average
High Percentage of Users that Viewed resulted in more the THIRTEEN times the gains in Math proficiency rates than the Districts average (p<.001)
High Pct of Users that Viewed resulted in TWICE the gains in Reading proficiency rates than the Districts average (p<.001)
Higher percentage of Viewers Higher percentage of Viewers resulted in Higher Performance
13.5 times greater gains in Math vs. District
95.8% greater gains in Reading vs. District
20
Improvements in Proficiency Rates Correlated with PD 360
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
Math Change Reading Change
Comparative Change forLow Pct Viewer Schools vs. Districts
Low Pct of Users that Viewed Schools
Districts Average
Low Pct of Users that
Viewed Schools
Districts Average
Percent Advantage
Statistical Significance
Math Change 1.50 0.16 836.5% p<.001 9.4times greater improvement
Reading Change 1.38 1.12 23.6% p<.001 1.2times greater improvement
An advantage of:
Low Percent of Users that Viewed resulted in more the NINE times the gains in Math proficiency rates than the Districts average (p<.001)
Low Percent of Users that Viewed resulted in more than 20% greater gains in Reading proficiency rates than the Districts average (p<.001)
But even Low percentage of Viewers Low percentage of Viewers outperformed no PD 360
9.4 times greater gains in Math vs. District
23.6% greater gains in Reading vs. District
21
Improvements in Proficiency Rates Correlated with PD 360
0.00
0.50
1.00
1.50
2.00
2.50
Math Change Reading Change
Comparative Change forSchools with High # of Viewers vs.
Districts
High Pct of Users that Viewed Schools
Districts Average
High Number of Users that
Viewed Schools
Districts Average
Percent Advantage
Statistical Significance
Math Change 2.14 0.16 1237.1% p<.001 13.4times greater improvement
Reading Change 2.16 1.12 92.5% p<.001 1.9times greater improvement
An advantage of:
High Number of Users that Viewed resulted in more the THIRTEEN times the gains in Math proficiency rates than the Districts average (p<.001)
High Number of Users that Viewed resulted in nearly TWICE the gains in Reading proficiency rates than the Districts average (p<.001)
More Viewers More Viewers resulted in Higher Performance
13.4 times greater gains in Math vs. District
92.5% greater gains in Reading vs. District
22
Improvements in Proficiency Rates Correlated with PD 360
0.000.200.400.600.801.001.201.401.601.80
Math Change Reading Change
Comparative Change forSchools with Low # of Viewers vs.
Districts
Low Number of Users that Viewed Schools
Districts Average
Low Number of Users that
Viewed Schools
Districts Average
Percent Advantage
Statistical Significance
Math Change 1.55 0.16 870.0% p<.001 9.7times greater improvement
Reading Change 1.47 1.12 31.6% p<.001 1.3times greater improvement
An advantage of:
Low Number of Users that Viewed resulted in nearly TEN times the gains in Math proficiency rates than the Districts average (p<.001)
Low Number of Users that Viewed resulted in more than 30% greater gains in Reading proficiency rates than the Districts average (p<.001)
But even fewer viewers outperformed no PD 360
9.7 times greater gains in Math vs. District
31.6% greater gains in Reading vs. District
23
Improvements in Proficiency Rates Correlated with PD 360
24
Sample Demographics State
State
TXTNMNKYFLCOAZAKP
erc
en
t
30
20
10
0
School Level
107 57.2 57.2 57.2
29 15.5 15.5 72.7
29 15.5 15.5 88.2
14 7.5 7.5 95.7
7 3.7 3.7 99.5
1 .5 .5 100.0
187 100.0 100.0
Elementary
Middle - JH (incl 6-8)
El-Mid (e.g. K-8)
High School
Mid-Hi (e.g. 6-12)
El-High (e.g. K-12)
Total
ValidFrequency Percent Valid Percent
CumulativePercent
• Eight States• 187 Schools
Improvements in Proficiency Rates Correlated with PD 360