Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration

School Improvement Network Impact Assessment:Higher Engagement Schools

versus Lower Engagement Schools

Steven H. Shaha, PhD, DBAProfessor, Center for Public Policy and Administration

Independent EvaluatorDecember 2012

Overarching Research Question:

Does engagement in PD 360 and Observation 360,

tools within the Educator Effectiveness System,

significantly affect student success

and school-wide metrics?

Sample Description• High Video Utilizers– 39 States– 211 Districts– 734 Schools

• Metrics:• Educator

Engagement• Student Success• School Impacts

• 32 data elements collected or computed through PD 360 and Observation 360

• Contrasted higher engagement schools versus lower engagement schools– Improvement in percentages of students who tested

advanced or proficient in math and reading– Classified into four quartiles– Analyses of highest and lowest quartiles only

Study of Educator Engagement

Metrics for Differentiating Advantages for Higher Engagement Organizations:

• Focus Objectives Set Up• Observations Performed• Percent Registered Users• Percent of Users in Communities

• Minutes Viewed• Forums Viewed• Programs Viewed• Segments Viewed• Links Viewed

• Follow-up Questions Answered• Reflection Questions Answered• Focus Objectives Set Up• Forums Posted• Downloaded Files• Uploaded Files• Participation in Communities

Leadership, Implementation

and Accountability

Educator Participation

Educator Engagement

These are the 15 metrics for which higher engagement schools were significantly higher than their lower engagement counterparts

Sample of Differentiating Metrics of Utilization and Engagement

Links Viewed

Minutes Viewed Follow-up Questions Answered

Observations Performed

Passive participation (e.g. video viewing alone) is LESS influential than Active engagement

63.8% advantage (p<.001)




Passive participation (e.g. video viewing alone) is LESS influential than Active engagement

Uploaded Files

Downloaded Files Forums Viewed

Forums Posted

Sample of Differentiating Metrics of Utilization versus Engagement


30.5% advantage (p<.001)79.5% advantage (p<.001)


Who Cares?Who cares if educators used it

more?

Did it make a difference for kids and schools?

Student Success:• Performance on standardized tests– Percent either proficient or advanced in the

following subjects:• Reading• Math

Study of Student Success

ImprovedStudent Performance

4.9% gain for lower engagement schools (p<.01)


4.9% gain for lower engagement schools (p<.01)

18.0% gain for higher engagement schools (p<.001)

Closed the Gap:267% advantage in gains for higher engagement schools (p<.001)

Nearly 4 times the impact


0.5% gain for lower engagement schools (p=ns)

Actually Important Gains:For every 200 students, 1 more performed at proficient or advanced levelthan in the previous year


0.5% gain for lower engagement schools (p=ns)

18.9% gain for higher engagement schools (p<.001)

Surpassed the Gap:3,520% advantage in gains for higher engagement schools (p<.001)

36 times greater impact

• Performance on key indicators from Internet (when publicly available) and structured telephone interviews:– Dropout Rates– Student Discipline Rates– Teacher Retention Rates– College-Bound Rates

Metrics of School Impact

Improved Dropout Rates

4.9% improvement for lower engagement schools (p<.01)

For every 100 students, 5 fewer dropped out than in the previous year

Figures reflect rounding, projections reflect correct math.

Improved Dropout Rates

20.0% improvement for higher engagement schools (p<.001)

309.1% advantage in gains for higher engagement schools (p<.001)

Executive SummaryHigher engagement schools

began statistically equal, then significantly

outperformed their counterparts (p<.01)

4.9% improvement for lower engagement schools (p<.01)

For every 100 students, 20 fewer dropped out than in the previous year.

Improved Student Discipline Rates

Figures reflect rounding, projections reflect correct math.Y-Axis is inverted to reflect improvement as intuitively upward trend.

7.4% fewer disciplinary incidents for lower engagement schools (p<.01)

33.2% fewer disciplinary incidents for higher engagement schools (p<.001)

Improved Student Discipline Rates

Executive SummaryHigher engagement schools significantly outperformed their counterparts (p<.01)

7.4% fewer disciplinary incidents for lower engagement schools (p<.01)

351% advantage in gains for higher engagement schools (p<.001)

4 ½ times the impact

For every 100 students, 33 fewer problem students than in the previous year.


ImprovedTeacher Retention Rates

1.7% more teachers stayed for lower engagement schools (p<.01)

ImprovedTeacher Retention Rates

Executive SummaryHigher engagement schools significantly outperformed their counterparts (p<.01)

65.9% advantage in gains for higher engagement schools (p<.001)

Nearly twice the impact

1.7% more teachers stayed for lower engagement schools (p<.01)

2.8% more teachers stayed for higher engagement schools (p<.01)

For every 100 teachers, nearly 3 fewer left than in the previous year.


Improved College-Bound Rates

No decrease or gain for lower engagement schools (ns)


Percentage of students schools report as being college-bound.

Improved College-Bound Rates

Executive SummaryHigher engagement

schools began statistically equal, then significantly

outperformed their counterparts (p<.01)

Incalculable advantage in gains for higher engagement schools (p<.001)

9.6% improvement for higher engagement schools (p<.001)

No decrease or gain for lower engagement schools (ns)

For every 100 students, 10 more were college-bound than in the previous year.


• Dropout Rates– Approx. 15 fewer dropouts per 100 students than lower engagement

school counterparts

• Student Discipline Rates– Approx. 33 fewer students “in the office” per 100 students than for

lower engagement school counterparts

• Teacher Retention Rates– Approx. 3 fewer teachers leaving per 100 teachers, which is 1 fewer than

for lower engagement school counterparts

• College-Bound Rates– Approx. 10 more college-bound students per 100 students than for

lower engagement school counterparts

Summary of School Impacts

Student Success

School Impacts

Leadership, Implementation

and Accountability

Educator Participation

Educator Engagement

A Model for Educational Success

Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration

Documents

Transcript of Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration