School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement...
-
Upload
octavia-henry -
Category
Documents
-
view
216 -
download
3
Transcript of School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement...
![Page 1: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/1.jpg)
School Improvement Network Impact Assessment:Higher Engagement Schools
versus Lower Engagement Schools
Steven H. Shaha, PhD, DBAProfessor, Center for Public Policy and Administration
Independent EvaluatorDecember 2012
![Page 2: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/2.jpg)
Overarching Research Question:
Does engagement in PD 360 and Observation 360,
tools within the Educator Effectiveness System,
significantly affect student success
and school-wide metrics?
![Page 3: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/3.jpg)
Sample Description• High Video Utilizers– 39 States– 211 Districts– 734 Schools
• Metrics:• Educator
Engagement• Student Success• School Impacts
![Page 4: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/4.jpg)
• 32 data elements collected or computed through PD 360 and Observation 360
• Contrasted higher engagement schools versus lower engagement schools– Improvement in percentages of students who tested
advanced or proficient in math and reading– Classified into four quartiles– Analyses of highest and lowest quartiles only
Study of Educator Engagement
![Page 5: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/5.jpg)
Metrics for Differentiating Advantages for Higher Engagement Organizations:
• Focus Objectives Set Up• Observations Performed• Percent Registered Users• Percent of Users in Communities
• Minutes Viewed• Forums Viewed• Programs Viewed• Segments Viewed• Links Viewed
• Follow-up Questions Answered• Reflection Questions Answered• Focus Objectives Set Up• Forums Posted• Downloaded Files• Uploaded Files• Participation in Communities
Leadership, Implementation
and Accountability
Educator Participation
Educator Engagement
These are the 15 metrics for which higher engagement schools were significantly higher than their lower engagement counterparts
![Page 6: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/6.jpg)
Sample of Differentiating Metrics of Utilization and Engagement
Links Viewed
Minutes Viewed Follow-up Questions Answered
Observations Performed
Passive participation (e.g. video viewing alone) is LESS influential than Active engagement
63.8% advantage (p<.001)
39.0% advantage (p<.001)
70.3% advantage (p<.001)
4.3% advantage (p<.001)
![Page 7: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/7.jpg)
Passive participation (e.g. video viewing alone) is LESS influential than Active engagement
Uploaded Files
Downloaded Files Forums Viewed
Forums Posted
Sample of Differentiating Metrics of Utilization versus Engagement
47.3% advantage (p<.001)
30.5% advantage (p<.001)79.5% advantage (p<.001)
68.6% advantage (p<.001)
![Page 8: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/8.jpg)
Who Cares?Who cares if educators used it
more?
Did it make a difference for kids and schools?
![Page 9: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/9.jpg)
Student Success:• Performance on standardized tests– Percent either proficient or advanced in the
following subjects:• Reading• Math
Study of Student Success
![Page 10: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/10.jpg)
ImprovedStudent Performance
4.9% gain for lower engagement schools (p<.01)
![Page 11: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/11.jpg)
ImprovedStudent Performance
4.9% gain for lower engagement schools (p<.01)
18.0% gain for higher engagement schools (p<.001)
Closed the Gap:267% advantage in gains for higher engagement schools (p<.001)
Nearly 4 times the impact
![Page 12: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/12.jpg)
ImprovedStudent Performance
0.5% gain for lower engagement schools (p=ns)
Actually Important Gains:For every 200 students, 1 more performed at proficient or advanced levelthan in the previous year
![Page 13: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/13.jpg)
ImprovedStudent Performance
0.5% gain for lower engagement schools (p=ns)
18.9% gain for higher engagement schools (p<.001)
Surpassed the Gap:3,520% advantage in gains for higher engagement schools (p<.001)
36 times greater impact
![Page 14: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/14.jpg)
• Performance on key indicators from Internet (when publicly available) and structured telephone interviews:– Dropout Rates– Student Discipline Rates– Teacher Retention Rates– College-Bound Rates
Metrics of School Impact
![Page 15: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/15.jpg)
Improved Dropout Rates
4.9% improvement for lower engagement schools (p<.01)
For every 100 students, 5 fewer dropped out than in the previous year
Figures reflect rounding, projections reflect correct math.
![Page 16: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/16.jpg)
Improved Dropout Rates
20.0% improvement for higher engagement schools (p<.001)
309.1% advantage in gains for higher engagement schools (p<.001)
Executive SummaryHigher engagement schools
began statistically equal, then significantly
outperformed their counterparts (p<.01)
4.9% improvement for lower engagement schools (p<.01)
For every 100 students, 20 fewer dropped out than in the previous year.
![Page 17: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/17.jpg)
Improved Student Discipline Rates
Figures reflect rounding, projections reflect correct math.Y-Axis is inverted to reflect improvement as intuitively upward trend.
7.4% fewer disciplinary incidents for lower engagement schools (p<.01)
![Page 18: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/18.jpg)
33.2% fewer disciplinary incidents for higher engagement schools (p<.001)
Improved Student Discipline Rates
Executive SummaryHigher engagement schools significantly outperformed their counterparts (p<.01)
7.4% fewer disciplinary incidents for lower engagement schools (p<.01)
351% advantage in gains for higher engagement schools (p<.001)
4 ½ times the impact
For every 100 students, 33 fewer problem students than in the previous year.
![Page 19: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/19.jpg)
Figures reflect rounding, projections reflect correct math.
ImprovedTeacher Retention Rates
1.7% more teachers stayed for lower engagement schools (p<.01)
![Page 20: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/20.jpg)
ImprovedTeacher Retention Rates
Executive SummaryHigher engagement schools significantly outperformed their counterparts (p<.01)
65.9% advantage in gains for higher engagement schools (p<.001)
Nearly twice the impact
1.7% more teachers stayed for lower engagement schools (p<.01)
2.8% more teachers stayed for higher engagement schools (p<.01)
For every 100 teachers, nearly 3 fewer left than in the previous year.
Figures reflect rounding, projections reflect correct math.
![Page 21: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/21.jpg)
Improved College-Bound Rates
No decrease or gain for lower engagement schools (ns)
Figures reflect rounding, projections reflect correct math.
Percentage of students schools report as being college-bound.
![Page 22: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/22.jpg)
Improved College-Bound Rates
Executive SummaryHigher engagement
schools began statistically equal, then significantly
outperformed their counterparts (p<.01)
Incalculable advantage in gains for higher engagement schools (p<.001)
9.6% improvement for higher engagement schools (p<.001)
No decrease or gain for lower engagement schools (ns)
For every 100 students, 10 more were college-bound than in the previous year.
Figures reflect rounding, projections reflect correct math.
![Page 23: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/23.jpg)
• Dropout Rates– Approx. 15 fewer dropouts per 100 students than lower engagement
school counterparts
• Student Discipline Rates– Approx. 33 fewer students “in the office” per 100 students than for
lower engagement school counterparts
• Teacher Retention Rates– Approx. 3 fewer teachers leaving per 100 teachers, which is 1 fewer
than for lower engagement school counterparts
• College-Bound Rates– Approx. 10 more college-bound students per 100 students than for
lower engagement school counterparts
Summary of School Impacts
![Page 24: School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for.](https://reader035.fdocuments.us/reader035/viewer/2022070415/5697bf9c1a28abf838c93446/html5/thumbnails/24.jpg)
Student Success
School Impacts
Leadership, Implementation
and Accountability
Educator Participation
Educator Engagement
A Model for Educational Success