North Carolina New Teacher Support Program - UNC Public ......North Carolina Department of Public...

North Carolina New Teacher Support

Program

Final Race to the Top Evaluation Report

Authors:

Kevin Bastian and Julie Marks

Education Policy Initiative at Carolina, The University of North

Carolina at Chapel Hill

August 2015

Consortium for Educational Research and Evaluation–North Carolina

NC NTSP: Final RttT Evaluation Report

August 2015

Consortium for Educational Research and Evaluation–North Carolina

Table of Contents

Executive Summary ........................................................................................................................ 3

Introduction ..................................................................................................................................... 6

Data and Analyses........................................................................................................................... 8

Evaluation Sample ...................................................................................................................... 8

Data Sources ............................................................................................................................... 9

Analysis..................................................................................................................................... 10

Findings......................................................................................................................................... 11

Implementation ......................................................................................................................... 11

To What Extent was the NC NTSP Implemented as Intended? ........................................... 11

To What Extent did the NC NTSP Reach its Target Population? ........................................ 11

Knowledge, Skills, and Attitudes.............................................................................................. 13

How do Teachers Perceive the Impact of NC NTSP Components on their Knowledge,

Skills, and Attitudes about Teaching? .................................................................................. 13

How do Teachers Perceive the Impact of NC NTSP Components on Self-Efficacy and Job

Satisfaction? .......................................................................................................................... 15

Teacher Effectiveness ............................................................................................................... 16

To What Extent does the NC NTSP Impact Teacher Effectiveness, as Measured by Teacher

Value Added to Student Achievement? ................................................................................ 16

To What Extent does the NC NTSP Impact Teacher Effectiveness, as Measured by the

North Carolina Educator Evaluation System (NCEES) Teacher Evaluation Ratings? ........ 18

Retention ................................................................................................................................... 20

To What Extent does the NC NTSP Impact the Retention of Novice Teachers a) in the Same

Schools or LEAs, and b) in the State? .................................................................................. 20

Conclusions ................................................................................................................................... 22

Summary of Findings ................................................................................................................ 22

Implementation ..................................................................................................................... 22

Teacher Knowledge, Skills, and Attitudes ............................................................................ 22

Teacher Effectiveness ........................................................................................................... 22

Teacher Retention ................................................................................................................. 22

Limitations ................................................................................................................................ 23

Recommendations and Next Steps............................................................................................ 23

Appendix A. Evaluation Sample................................................................................................... 25

Appendix B. Data Sources ............................................................................................................ 30

Appendix C. To What Extent was the NC NTSP Implemented as it was Intended? ................... 42


August 2015

Consortium for Educational Research and Evaluation–North Carolina 2

Appendix D. To What Extent did the NC NTSP Reach its Target Population? ........................... 44

Appendix E. How do Teachers Perceive the Impact of the NC NTSP Components on their

Confidence, Knowledge, and Skills in Teaching? ........................................................................ 52

Appendix F. How do Teachers Perceive the Impact of the NC NTSP on their Self-Efficacy and

Job Satisfaction? ........................................................................................................................... 58

Appendix G. To What Extent does the NC NTSP Impact Teacher Effectiveness as Measured by

Teacher Value-Added to Student Achievement? .......................................................................... 61

Appendix H. To What Extent does the NC NTSP Impact Teacher Effectiveness as Measured by

the North Carolina Educator Evaluation System (NCEES) Teacher Evaluation Ratings? .......... 82

Appendix I. To What Extent does the NC NTSP Impact the Retention of Novice Teachers to the

Same School, LEA, and the State? ............................................................................................... 90


August 2015


NORTH CAROLINA NEW TEACHER SUPPORT PROGRAM: FINAL RACE TO THE

TOP EVALUATION REPORT

Executive Summary

Overview

The North Carolina New Teacher Support Program (NC NTSP) was developed to provide

induction supports to beginning teachers in North Carolina’s lowest-achieving schools and to

meet two high-priority needs identified by the state’s Race to the Top (RttT) grant: (1) helping

teachers to succeed during their initial years in teaching; and (2) retaining qualified teachers,

particularly in high-need schools. The NC NTSP aims to improve the instructional knowledge,

skills, attitudes, effectiveness, and retention of participating teachers through the provision of

three support components: institutes (multi-day trainings); instructional coaching; and

professional development. This report reflects findings from an independent external evaluation

of the NC NTSP conducted by the Education Policy Initiative at Carolina as part of the statewide

RttT evaluation undertaken by the Consortium for Educational Research and Evaluation–North

Carolina (CERE–NC).

Data and Methods

This final evaluation report draws upon the following data sources: (1) participation records from

each of the components of the NC NTSP; (2) Perception of Success Inventory for Beginning

Teachers (PSI-BT) survey responses; and (3) student demographics and test scores, classroom

rosters, teacher evaluation ratings, EVAAS scores, certified salaries, school personnel, and

school characteristics files provided by the North Carolina Department of Public Instruction.

With these data sources, the Evaluation Team used a comparison group design to examine: levels

of participation in NC NTSP components; NC NTSP teachers’ perceptions of the impact of

program components on their confidence, knowledge, and skills for teaching; teachers’

perceptions of their self-efficacy and job satisfaction; and the impact of the NC NTSP on teacher

value-added, teacher evaluation ratings, and teacher retention.

Summary of Findings

This report reflects evaluation findings for the two full years of program implementation

spanning the 2012-13 and 2013-14 school years. The study design and analysis were developed

to address evaluation questions across four overarching areas: implementation; teacher

knowledge, skills, and attitudes; teacher effectiveness; and teacher retention.

Implementation

To what extent was the NC NTSP implemented as intended, and to what extent did it reach its

target population? Over the course of the four-year grant period, the NC NTSP was developed,

staffed, and implemented from the ground up and has grown to scale, serving over 1,100 teachers

in 114 schools as of the 2013-14 school year. There were substantial differences in program

participation and implementation across the four NC NTSP regions. As the NC NTSP doubled in


August 2015


size in 2013-14, there was a drop in attendance at institutes and professional development

sessions and fewer instructional coach visits per teacher.

Teacher Knowledge, Skills, and Attitudes

How do teachers perceive the impact of NC NTSP components on their confidence, knowledge,

skills, and attitudes toward teaching? There was a statistically significant difference in the

proportion of NC NTSP evaluation sample respondents who felt the program components had a

positive impact on their teaching, compared with similar services provided by their own school.

This significant difference also existed between NC NTSP respondents’ perceptions of program

utility and comparison sample perceptions of analogous school-provided services.

Teacher Effectiveness

To what extent does the NC NTSP impact teacher effectiveness as measured by teacher value-

added to student achievement (EVAAS) and the North Carolina Educator Evaluation System

(NCEES) teacher evaluation ratings? Overall, NC NTSP evaluation sample teachers had

significantly higher EVAAS estimates than comparison sample teachers in fifth and eighth grade

science. When assessing results by cohort, positive and significant EVAAS results were

concentrated within NC NTSP Cohort 1 teachers, while NC NTSP Cohort 2 teachers were

generally no more or less effective. Regarding teacher evaluation ratings, there were no

significant differences between NC NTSP evaluation sample teachers and comparison teachers

in overall models. By cohort, NC NTSP Cohort 1 teachers had significantly higher evaluation

ratings on four standards in 2013-14.

Teacher Retention

To what extent does the NC NTSP impact the retention of novice teachers in the same schools or

local education agencies (LEAs) and in the state? Overall, NC NTSP evaluation sample teachers

were significantly more likely to return to teaching in North Carolina public schools, to the same

LEA, and to the same low-performing school. NC NTSP teachers from both cohorts were

significantly more likely than comparison sample teachers to return to the same low-performing

school.

Limitations

Two primary limitations should be taken into account when interpreting the findings in this

report: (1) the ability to isolate the impact of the NC NTSP is diminished due to the lack of a

comparison group of teachers working in schools similar to those served by the NC NTSP in all

ways other than participating in the program; and (2) given the time required for program

development and scaling up, the length of the evaluation period is not adequate to assess the

effectiveness of a consistent program model implemented as intended for first-, second-, and

third-year teachers over the course of three years.


August 2015


Recommendations and Next Steps

There are three overarching recommendations for the NC NTSP after the close of the RttT grant:

1. The impacts of the NC NTSP on teacher value-added to student achievement and teacher retention support a recommendation to sustain the program beyond the end of the RttT grant;

2. Findings from this evaluation should be used to explore the disparities in program implementation by region, and moving forward, implementation fidelity should be formally

monitored; and

3. Strategies should be explored to counter the decline in program participation and effectiveness seen in 2013-14, such as requiring participating schools and Local Education

Agencies (districts) to enforce mandatory participant attendance, and/or concentrating

resources toward instructional coaching (the most intensive program component).


August 2015


Introduction

North Carolina’s $400 million, four-year Race to the Top (RttT) grant was built upon a

comprehensive plan to strengthen the education workforce, with the goal of having a great

teacher in every classroom and a great principal leading every school. With this end in mind, the

North Carolina Department of Public Instruction (NCDPI) identified the state’s highest-priority

needs, including the recruitment and preparation, equitable distribution, professional

development, and induction and retention of high-quality teachers.1

Data from the 2010 Teacher Working Conditions Survey illustrated the need for induction

supports prior to RttT, showing that while nearly all new teachers (93%) are assigned a mentor,

almost half (47%) do not have time during the day to meet with their mentors, and one in eight

indicate that they never received any additional support as new teachers.1,2

These supports are

particularly needed in high-need schools, where novice teachers are concentrated and where the

teacher turnover rate is often greatest.

To help teachers succeed during their early-career years and persist in the state’s highest-need

schools, NCDPI partnered with the UNC General Administration (UNC-GA) to develop and

implement the North Carolina New Teacher Support Program (NC NTSP). The NC NTSP is a

comprehensive induction program for early-career teachers designed to increase competency in

goal-setting, backwards planning and assessment, data-driven decision-making, classroom

management, and strategies for success in the school and community. The goal of the NC NTSP

is to help participating novice teachers acquire the knowledge and skills necessary to increase the

quality of their instruction, raise student achievement, and persist in teaching.

The NC NTSP drew from the successful induction model utilized by Teach For America to

create a three part program comprised of 1) institutes (multi-day training sessions); 2) intensive

face-to-face and virtual instructional coaching; and 3) professional development sessions held

throughout the academic year. Implementation of these beginning teacher supports is organized

through a central NC NTSP office and regional anchor sites located at four UNC system

institutions—East Carolina University (ECU), the UNC Center for School Leadership

Development (UNC-CSLD), UNC Charlotte (UNCC), and UNCG Greensboro (UNCG).

As part of the statewide evaluation of RttT conducted by the Consortium for Education Research

and Evaluation–North Carolina (CERE–NC),3 the Education Policy Initiative at Carolina (EPIC)

was tasked with conducting an independent external evaluation of the NC NTSP from the 2011-

12 to the 2013-14 school years. Over this time period, EPIC has collected and analyzed data on

program implementation and participation, short-term and intermediate outcomes, and

summative impact findings to address the following evaluation questions:

1 NC RttT Proposal, 2010

2 Teacher Working Conditions Survey, 2010

3 The Consortium for Educational Research and Evaluation–North Carolina (http://cerenc.org) is a partnership of:

the SERVE Center, University of North Carolina at Greensboro; the Education Policy Initiative at Carolina,

University of North Carolina at Chapel Hill; and the Friday Institute for Educational Innovation, North Carolina

State University.

http://cerenc.org/


August 2015


Implementation

1. To what extent was the NC NTSP implemented as it was intended?

2. To what extent did the NC NTSP reach its target population?


3. How do teachers perceive the impact of NC NTSP components on their confidence, knowledge, and skills in teaching?

4. How do teachers perceive the impact of NC NTSP components on their self-efficacy and job satisfaction?


5. To what extent does the NC NTSP impact teacher effectiveness as measured by teacher value-added to student achievement?

6. To what extent does the NC NTSP impact teacher effectiveness as measured by the North Carolina Educator Evaluation System (NCEES) teacher evaluation ratings?

Teacher Retention

7. To what extent does the NC NTSP impact the retention of novice teachers in the same schools or LEAs and in the state?


August 2015


Data and Analyses

Evaluation Sample

For the 2012-13 academic year, the inclusion criteria for the NC NTSP evaluation sample was all

first- and second-year teachers who began receiving NC NTSP services by December 2012 and

who worked in schools that were eligible for and agreed to participate in the NC NTSP.

The exclusion criteria for the 2012-13 NC NTSP evaluation sample were as follows:

Teachers who began receiving NC NTSP supports in January 2013 or later;

Novice teachers receiving NC NTSP supports in non-RttT schools;

The small number of third-year teachers served by the program in 2012-13; and

Teach for America (TFA) corps members.

For the 2013-14 academic year, the Evaluation Team defined the NC NTSP evaluation sample as

all first-, second-, and third-year teachers who began receiving NC NTSP supports by December

2013 and who worked in schools that were eligible for and agreed to participate in the NC NTSP.

While this sample includes third-year teachers who were fully served by the program in 2013-14,

it again excludes teachers who began receiving NC NTSP supports in January 2014 or later,

novice teachers who received NC NTSP supports in non-RttT schools, and TFA corps members.

In the 2012-13 academic year, the NC NTSP evaluation sample consisted of 344 teachers

working in 59 schools and 16 LEAs, with a large majority (72%) of this treatment sample being

first-year teachers. In the 2013-14 academic year, the NC NTSP evaluation sample consisted of

808 teachers working in 91 schools and 25 LEAs. Approximately 50% of the treatment sample

were first-year teachers, 36% were second-year teachers, and 15% were in their third year of

teaching.

To best isolate the impact of the NC NTSP, the Evaluation Team used a comparison group

design to compare findings between NC NTSP program participants and comparison teachers

who did not receive the intervention. An optimal comparison group would be comprised of

teachers who were identical to the evaluation sample in every way other than participating in the

NC NTSP; however, identifying such a group poses a particular challenge for evaluations of

programs that are universally offered within the eligible target population, leaving two primary

options for comparison groups: (1) those who were not eligible for the program but are thought

to be similar due to characteristics that place them narrowly outside the program criteria, and (2)

those who are eligible for the program but did not participate.

To assess the impact of the NC NTSP, the Evaluation Team created two comparison groups. The

group included in the main body of this final evaluation report meets the criteria of Option One

above—schools that were in the bottom decile of school performance in 2011-12 but that were

not eligible for and did not receive NC NTSP services. The key limitation of this sample is that it

is comprised of schools that did not receive any RttT services, meaning comparisons to this

group do not allow for isolation of the impact of the NC NTSP from other RttT programs.


August 2015


In the 2012-13 school year, this comparison group (labeled Non-RttT Comparison throughout the

report) consisted of all of the first- and second-year teachers who began working in these schools

by December 2012. In the 2013-14 school year, this comparison group consisted of all of the

first-, second-, and third-year teachers who began working in these schools by December 2013.4

Overall, the Non-RttT Comparison evaluation sample included 1,033 teachers working in 147

schools and 48 LEAs in the 2012-13 school year and 1,507 teachers working in 149 schools and

48 LEAs in the 2013-14 school year.

Table 1 presents school characteristics for the NC NTSP evaluation sample and the Non-RttT

Comparison evaluation sample. Across 2012-13 and 2013-14, NC NTSP teachers worked in

schools with more students qualifying for subsidized school meals, more racial and ethnic

minority students, a higher short-term suspension rate, and a lower school performance

composite than did comparison teachers. In 2012-13, NC NTSP schools also had a significantly

higher percentage of novice teachers. Please see Appendix A for more details on the NC NTSP

evaluation sample, both comparison samples,5 and additional characteristics of the treatment and

comparison sample schools.

Table 1. School Characteristics for the NC NTSP and Non-RttT Comparison Group

Characteristic

2012-13 School Year 2013-14 School Year

NC NTSP

Non-RttT

Comparison NC NTSP

Non-RttT

Comparison

Free and Reduced-Price

Lunch Percentage 93.58 85.93

** 91.78 87.40**

Racial/Ethnic

Minority Percentage 90.87 78.37

** 89.63 78.67**

Short-Term Suspension Rate

(Per 100 students) 48.86 33.52

** 38.14 25.50**

Performance Composite 20.04 23.25** 31.07 34.29

**

Novice Teacher Percentage 32.99 27.19** 32.93 30.29

Note: +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.

Data Sources

The Evaluation Team collected the data presented in this final evaluation report from the UNC-

GA implementation team, the NC NTSP evaluation sample teachers and comparison sample

teachers, and administrative datasets supplied by NCDPI. Specifically, this final evaluation

report draws upon the following data sources: (1) participation records from each of the

components of the NC NTSP (institutes, instructional coaching, and professional development);

(2) the Perception of Success Inventory for Beginning Teachers (PSI-BT) survey responses of

NC NTSP evaluation sample teachers and Non-RttT Comparison sample teachers; and (3)

student test scores, student demographics, classroom rosters, teacher evaluation ratings, EVAAS

scores, certified salaries, school personnel, and school characteristics files provided by NCDPI.

4 Like the NC NTSP evaluation sample, the Non-RttT Comparison evaluation sample excludes TFA corps members.

5 The second sample is identified as the NC NTSP Eligible Comparison sample.


August 2015


With these data sources, the Evaluation Team examined: levels of participation in NC NTSP

components; NC NTSP teachers’ perceptions of the impact of program components on their

confidence in, knowledge of, and skills for teaching; their self-efficacy and job satisfaction; and

the impact of the NC NTSP on teacher value-added measures, teacher evaluation ratings, and

teacher retention. Please see Appendix B for more details on program documents provided by the

UNC-GA implementation team and for PSI-BT survey items, response rates, and respondent

characteristics.

Analysis

This summative report utilizes data collected over a two-year period from two cohorts and across

four regional sites. While the NC NTSP was implemented using a structured program model

across regions and over time, natural variation is to be expected based upon the different needs

and contextual factors within each region. Furthermore, due to the annual scaling-up of the

program, different cohorts were exposed to different levels of program intensity. To account for

these differences, the Evaluation Team will present impact results for teacher value-added

measures, teacher evaluation ratings, and teacher retention in the following groupings:

Overall: To determine the overall impact of the NC NTSP, the Evaluation Team will present results for the NC NTSP and Non-RttT Comparison evaluation sample teachers using pooled

data from the 2012-13 and 2013-14 school-years.

By Region: To examine any regional differences in outcomes, the Evaluation Team will present results, separately, for each of the four regions served by the anchor institutions at

East Carolina University (ECU), the University of North Carolina Center for School

Leadership Development (UNC-CSLD), the University of North Carolina at Charlotte

(UNCC), and the University of North Carolina at Greensboro (UNCG). These results use

pooled data from 2012-13 and 2013-14.

By Cohort: To examine the impact of the NC NTSP based on when a teacher entered the program, the Evaluation Team will present results, separately, for teachers who entered the

program in 2012-13 (Cohort 1) and those who entered the program in 2013-14 (Cohort 2).

Appendices G through I show value-added, evaluation rating, and retention results in reference

to a second comparison group (the NC NTSP Eligible Comparison sample), display results for

first-year teachers only, and include results from analyses with a more comprehensive set of NC

NTSP and comparison sample teachers.


August 2015


Findings

Implementation

To What Extent was the NC NTSP Implemented as Intended?

After the receipt of RttT grant funds, the NC NTSP was developed from the ground up in the

2010-11 school year. As such, the model was scaled up over time as the program increased

capacity and there was increased buy-in from Local Education Agencies (LEAs)6 and schools

across the state.

As shown in Table 2, there was a 30-fold increase in the number of teachers served by the NC

NTSP over the course of the grant period. By 2013-14, the NC NTSP provided induction

services to 1,108 teachers in 114 schools and 27 LEAs. Overall, the NC NTSP provided

induction services to novice teachers in 73 RttT schools in 2012-13 and 91 RttT schools in 2013-

14.7 By the end of the RttT funding period, this coverage includes over three-quarters of the

lowest-achieving schools the program was tasked to serve.

Please see Appendix C for more details on the implementation of the NC NTSP.

Table 2. NC NTSP Program Implementation

Year Teachers Served Schools Served LEAs Served

2010-11* N/A N/A N/A

2011-12 35 13 5

2012-13 542 77 19

2013-14 1108 114 27

* Note: The 2010-11 school year was used for program development, and focused on program

planning, staffing, identification of anchor institutes, and school recruitment.

To What Extent did the NC NTSP Reach its Target Population?

The NC NTSP is comprised of three components: institutes (multi-day trainings); instructional

coaching; and professional development. In 2012-13, institutes were held at one centralized

location at the University of North Carolina at Chapel Hill; in 2013-14, there were regional

institutes held prior to the start of the school year, followed by a centralized institute in

September. All instructional coaching interactions and professional development opportunities

were organized and implemented at the regional level.

6 LEA is North Carolina’s term for traditional school districts and charter schools.

7 There were a small number of schools served by the NC NTSP that were not in the RttT sample. Please see

Appendix C for more details.


August 2015


As shown in Table 3, the attendance and reach for all three NC NTSP components notably

declined between the 2012-13 and 2013-14 school years. In addition, there was notable

variability across regions in program attendance and reach.

In the 2012-13 school year, nearly 55% of all NC NTSP evaluation sample participants attended

an institute. There was considerable variation between regions, however, with almost 75% of

teachers from the ECU region attending an institute, compared with only 40% of teachers in the

UNCG region. In 2013-14, institute attendance decreased by over 50% across all regions and fell

to 21% overall.

Table 3. NC NTSP Evaluation Sample Participation

NC NTSP

Overall

ECU

Region

UNC-CSLD

Region

UNCC

Region

UNCG

Region

NC NTSP Institute

Attended a

2012-13 Institute 54.94% 74.24% 66.25% 46.67% 39.74%

Attended a

2013-14 Institute 20.54% 27.51% 18.99% 16.48% 19.50%

NC NTSP Instructional Coaching

Average Number of

In-Person Coaching

Visits Per Month in

2012-13

3.73 6.02 2.10 3.39 4.01

Average Number of

In-Person Coaching

Visits Per Month in

2013-14

2.39 3.82 1.65 2.56 1.77

NC NTSP Professional Development

Average Number of

PD Sessions Attended

in 2012-13

2.22 2.41 3.59 1.17 2.26

Average Number of

PD Sessions Attended

in 2013-14

0.76 1.03 1.22 0.14 0.52

The middle panel of Table 3 reflects a decrease in the frequency of instructional coaching visits

between the 2012-13 and 2013-14 school years and shows substantial variation in the average

number of in-person coaching visits between regions. Program-wide, teachers averaged 1.3

fewer in-person instructional coaching visits per month in 2013-14.8 Coaching visits declined

across all regions, with the largest decreases in the UNCG and ECU regions. In the 2012-13 and

2013-14 school years, the ECU region had the highest average number of instructional coaching

visits, while the UNC-CSLD had the lowest average number of instructional coaching visits.

8 The drop in the average number of in-person coaching visits was not due to the NC NTSP serving third-year

teachers in 2013-14 (a teacher group that averaged fewer coaching visits overall); rather, the average number of in-

person instructional coaching visits declined for first- and second-year teachers.


August 2015


To further illustrate the variation in program intensity across regions, Figure 1 displays the

average number of total contact hours (in-person and virtual) between instructional coaches and

NC NTSP evaluation sample teachers in the 2013-14 school-year.9 Overall, teachers in the ECU

and UNCC regions averaged 41 and 34 contact hours, respectively, while teachers in the UNC-

CSLD and UNCG regions averaged approximately 10 contact hours.

Finally, the bottom panel of Table 3 (above) reports the average number of NC NTSP

professional development sessions participants attended. Overall, professional development

attendance was low, with a marked decline in the 2013-14 school year. In 2012-13, NC NTSP

teachers averaged 2.2 professional development sessions (out of six), with a high of 3.59

sessions in the UNC-CSLD region.10

In 2013-14, NC NTSP teachers averaged 0.76 professional

development sessions (out of six), with a high of 1.22 in the UNC-CSLD region and a low of

0.14 in the UNCC region. Please see Appendix D for more detailed information on participation

in NC NTSP components.

Figure 1. Average Total Contact Hours with Instructional Coaches (2013-14)

Knowledge, Skills, and Attitudes

How do Teachers Perceive the Impact of NC NTSP Components on their Knowledge, Skills, and

Attitudes about Teaching?

A survey assessing teachers’ perceptions of their knowledge, skills, and attitudes was

administered to the NC NTSP evaluation sample and the Non-RttT Comparison evaluation

sample teachers. The Evaluation Team asked questions to reflect items specific to NC NTSP

participants alone (e.g., related to institutes or instructional coaching), as well as items that

allowed NC NTSP and comparison sample teachers to respond (e.g., concerning analogous

support services provided by their schools or LEAs). Respondents were asked the extent to

which they agreed that each NC NTSP component (or analogous school-/LEA-provided

9 The UNC-GA program implementers only tracked and reported contact hours for the 2013-14 school year.

10 In 2012-13, several LEAs and schools in the UNC-CSLD region required their NC NTSP teachers to attend the

NC NTSP professional development sessions.

41.16

11.35

33.61

10.20

0

5

10

15

20

25

30

35

40

45

ECU Region UNC-CSLD Region UNCC Region UNCG Region


August 2015


component) had been helpful in developing their confidence, knowledge, and skills in teaching.

Given the potential for response bias, due in part to lower than desirable response rates (see

Appendix B), survey results should be interpreted with caution.

The top panel of Table 4 presents the summative survey item for NC NTSP institutes. In 2012-13

and 2013-14, approximately 80% of NC NTSP survey respondents agreed or strongly agreed that

the NC NTSP institute was helpful in developing their confidence, knowledge, and skills in

teaching. Within years, however, there was considerable variation in responses by region,

ranging from 59% to 94% in 2012-13 and from 65% to 100% in 2013-14.

The middle panels of Table 4 display responses to two items related to NC NTSP instructional

coaching. In both the 2012-13 and 2013-14 school years, 77% of the NC NTSP evaluation

sample respondents agreed or strongly agreed that the program’s instructional coaching was

helpful in developing their confidence, knowledge, and skills in teaching. In 2012-13 and 2013-

14, approximately 60% of NC NTSP evaluation sample respondents attributed “quite a bit” or a

“great deal” of their teaching success to help from their NC NTSP instructional coaches.

Table 4. NC NTSP Summative Survey Items

NC NTSP

Overall

ECU

Region

UNC-CSLD

Region

UNCC

Region

UNCG

Region

NC NTSP Institute was helpful in developing my confidence, knowledge, and skills in teaching

Percentage Agree or

Strongly Agree in 2012-13 80.00 81.58 59.09 93.93 76.47

Percentage Agree or


NC NTSP Instructional Coaching was helpful in developing my confidence, knowledge, and skills in

teaching

Percentage Agree or


Percentage Agree or


Of the success you have had as a beginning teacher, what amount would you attribute to help from

your NC NTSP Instructional Coach

Percentage Responding

Quite a Bit or a Great Deal

in 2012-13

62.43 53.19 45.45 66.67 83.78

Percentage Responding

Quite a Bit or a Great Deal

in 2013-14

58.20 65.98 52.71 52.88 55.21

NC NTSP Professional Development was helpful in developing my confidence, knowledge, and skills

in teaching

Percentage Agree or



August 2015


As a basis of comparison, Figure 2 displays the responses of NC NTSP evaluation sample

teachers to comparable items about their school-provided mentors. While 77% of NC NTSP

respondents indicated that their NC NTSP instructional coaches were helpful in developing their

confidence, knowledge, and skills in teaching, only 60% of NC NTSP respondents answered

similarly about their school-provided mentors. Likewise, while approximately 60% of NC NTSP

respondents attributed “quite a bit” or a “great deal” of their teaching success to their NC NTSP

instructional coaches, less than half of the NC NTSP respondents answered similarly for their

school-provided mentors. Both of these differences were statistically significant.

The last panel of Table 4 (above) presents the summative survey item for NC NTSP professional

development. In 2012-13, 87% of NC NTSP evaluation sample respondents agreed or strongly

agreed that the program’s professional development sessions were helpful in developing their

confidence, knowledge, and skills in teaching.11

Please see Appendix E for more details on NC NTSP teachers’ perceptions of program utility.

Figure 2. Teacher Perceptions of NC NTSP Quality Relative to School-Provided Support

How do Teachers Perceive the Impact of NC NTSP Components on Self-Efficacy and Job

Satisfaction?

The Evaluation Team assessed teachers’ perceptions of self-efficacy and job satisfaction with

five previously-validated items on the PSI-BT. As shown in Table 5 (following page), in both

2012-13 and 2013-14, the percentages of NC NTSP evaluation sample respondents who agreed

or strongly agreed with the self-efficacy and job satisfaction items were higher than the

percentages of Non-RttT Comparison evaluation sample respondents who did so. In 2012-13, NC

11

Due to an error in the response categories recorded for the 2013-14 summative professional development

question, the Evaluation Team is unable to present 2013-14 results.

49.09

60.10

45.92

60.38

58.10

77.05

62.43

77.58

0 10 20 30 40 50 60 70 80 90

Attributes "Quite a Bit" or "A Great Deal" of

Success to Coach/Mentor

Agrees/Strongly Agrees that Coach/Mentor Helpful

in Developing Confidence, Knowledge, and Skills in

Teaching

NC NTSP-Provided 2012-13 NC NTSP-Provided 2013-14

School-Provided 2012-13 School-Provided 2013-14


August 2015


NTSP evaluation sample respondents registered significantly higher levels of agreement (84% to

78%) with the self-efficacy item, “I feel inspired to instruct students to the best of my ability.” In

2013-14, NC NTSP evaluation sample survey respondents rated their self-efficacy and job

satisfaction significantly higher than did comparison sample respondents on all five items. These

differences should be interpreted carefully, however, due to the low response rate for Non-RttT

Comparison teachers and the potential for bias in the sample of teachers who responded.

See Appendix B for more details on these response rates and differences between NC NTSP

teachers who did and did not respond to the PSI-BT, and see Appendix F for more details on the

NC NTSP and Non-RttT Comparison evaluation sample responses to the self-efficacy and job

satisfaction items.

Table 5. Self-Efficacy and Job Satisfaction Survey Items

Survey Items

Percentage Agree or Strongly Agree

NC NTSP

2012-13

Comparison

2012-13

NC NTSP

2013-14

Comparison

2013-14

I am able to successfully

teach students with a

variety of ability levels

78.92% 72.64% 84.07% 74.16%

I am able to motivate all

students 60.24% 54.07% 68.87% 54.50%

I feel inspired to instruct

students to the

best of my ability

84.15%

77.92% 81.52% 75.14%

In general, I am satisfied

with my current job 59.15% 54.55% 56.76% 47.75%

I consider teaching to be my

ideal career 70.73% 67.20% 68.07% 60.68%


To What Extent does the NC NTSP Impact Teacher Effectiveness, as Measured by Teacher Value

Added to Student Achievement?

To assess the contributions of NC NTSP teachers to student achievement, the Evaluation Team

analyzed EVAAS teacher effectiveness estimates. The Evaluation Team combined EVAAS data

from elementary and middle grades to run separate models for mathematics, reading, and science

(End-of-Grade [EOG] science exams in grades five and eight), and combined EVAAS data from

End-of-Course (EOC) exams (Biology, English II, and Math I) to run a single EOC model. For

these models, the Evaluation Team made teacher EVAAS estimates the outcome variable and

regressed this measure of value added to student achievement on a set of school characteristics

and teacher experience. The Evaluation Team chose to control for these variables due to the

significant differences in school characteristics between NC NTSP and comparison sample

schools, and because these contextual variables are not accounted for in EVAAS models. Results

from these models express the adjusted-average differences in student achievement—in normal


August 2015


curve equivalency units—between students taught by NC NTSP teachers and students taught by

Non-RttT Comparison sample teachers.

The top portion of Table 6 displays overall EVAAS results for NC NTSP evaluation sample

teachers in comparison to Non-RttT Comparison evaluation sample teachers. Over the 2012-13

and 2013-14 school years, students taught by NC NTSP evaluation sample teachers in fifth and

eighth grade science made significantly larger achievement gains than did students taught by

Non-RttT Comparison evaluation sample teachers. There were no significant differences for

EOG mathematics, EOG reading, and all EOC exams.

The bottom portion of Table 6 illustrates the heterogeneity of value-added results across regions.

Students taught by NC NTSP evaluation sample teachers in the ECU region made significantly

larger achievement gains in mathematics, reading, and fifth and eighth grade science than did

students taught by Non-RttT Comparison evaluation sample teachers. Conversely, students

taught by NC NTSP evaluation sample teachers in the ECU region made significantly smaller

achievement gains on EOC exams. NC NTSP evaluation sample teachers in the UNCC region

were significantly more effective in elementary and middle grades mathematics, and NC NTSP

evaluation sample teachers in the UNCG region were significantly more effective in courses with

EOC exams.

Table 6. NC NTSP Overall and Regional EVAAS Results (2012-13 and 2013-14)

Elementary

and Middle

Grades

Mathematics

Elementary

and Middle

Grades

Reading

5th

and 8th

Grade

Science

EOC

Exams

NC

NT

SP

vs.

Non

-

Rtt

T C

om

pari

son

Gro

up (

2012-1

3 a

nd

2013-1

4)

NC NTSP Overall 0.828 0.259 1.011+ 0.083

ECU Region 2.219**

1.116**

2.772*

-0.652*

UNC-CSLD Region -0.007 -0.161 1.052 -0.727

UNCC Region 1.915+ 0.236 0.234 -0.370

UNCG Region -1.033 -0.336 -0.051 1.214*



August 2015


Table 7 presents considerable variability in teacher value added to student achievement by NC

NTSP cohort. NC NTSP Cohort 1 teachers—first served by the program in 2012-13—were

significantly more effective than Non-RttT Comparison evaluation sample teachers in elementary

and middle grades mathematics and reading in 2012-13 and when pooling data for two years

(2012-13 and 2013-14). Teachers from both NC NTSP cohorts were more effective in fifth and

eighth grade science in 2013-14. There were no other significant differences for NC NTSP

Cohort 2 teachers.

Table 7. NC NTSP EVAAS Results by Cohort

Elementary and Middle Grades

Math

Elementary and Middle Grades

Reading

Cohort 2012-13 2013-14

2012-13

2013-14 2012-13 2013-14

2012-13

2013-14

NC NTSP

Cohort 1 3.054

** 0.639 1.483+

1.670** 0.307 0.759

*

NC NTSP

Cohort 2 --- -0.189 -0.011 --- -0.183 -0.273

5th

and 8th

Grade Science EOC Exams

NC NTSP

Cohort 1 0.971 1.208

+ 1.127 0.693 -0.088 0.029

NC NTSP

Cohort 2 --- 1.261

+ 0.862 --- -0.122 0.202


Please see Appendix G for: (1) more details on the value-added methodology; (2) a complete set

of EVAAS results, including both comparison groups and models focusing on first-year teachers

only; (3) an alternative set of value-added models with student-level data; and (4) value-added

results with a more comprehensive set of NC NTSP and comparison sample teachers.

To What Extent does the NC NTSP Impact Teacher Effectiveness, as Measured by the North

Carolina Educator Evaluation System (NCEES) Teacher Evaluation Ratings?

Since many important aspects of teaching will not be fully captured by measures of teachers’

value added to student achievement, the Evaluation Team analyzed teachers’ evaluation ratings

on the five North Carolina Professional Teaching Standards (NCPTS), all of which are directly

assessed by school administrators. For these analyses, the Evaluation Team estimated models for

which the outcome variable was a teacher’s evaluation rating on a one-to-five scale (where one

was Not Demonstrated and five was Distinguished) and controlled for teacher experience and

school characteristics. Results from these models estimate the odds of NC NTSP evaluation

sample teachers receiving higher evaluation ratings than their Non-RttT Comparison evaluation

sample peers. Statistically significant results greater than one indicate higher evaluation ratings;

statistically significant results less than one indicate lower evaluation ratings.


August 2015


Over the 2012-13 and 2013-14 school years, the top panel of Table 8 shows no significant

evaluation rating differences between NC NTSP evaluation sample teachers and Non-RttT

Comparison evaluation sample teachers. When analyzing the data by region, the bottom panel of

Table 8 shows that NC NTSP teachers in the UNCC region had significantly higher evaluation

ratings on Leadership (Standard 1) and Reflecting on Practice (Standard 5).

Table 8. Overall and Regional NC NTSP Evaluation Rating Results (2012-13 and 2013-14)

Standard

1

Standard

2

Standard

3

Standard

4

Standard

5

NC

NT

SP

vs

Non

-Rtt

T

Com

pari

son

(2012-1

3 a

nd

2013

-14)

NC NTSP Overall 1.072 0.960 0.993 0.975 0.996

ECU Region 0.878 0.921 0.666 0.655 0.941

UNC-CSLD Region 0.834 0.719 0.906 0.896 0.886

UNCC Region 1.822+ 1.601 1.613 1.588 1.735

+

UNCG Region 0.928 0.771 0.931 0.913 0.675


To determine whether evaluation ratings differ by NC NTSP cohort, Table 9 presents separate

evaluation rating results for Cohorts 1 and 2. For NC NTSP Cohort 1, there were no significant

differences in 2012-13; however, in their second year in the program (2013-14), NC NTSP

Cohort 1 teachers had significantly higher evaluation ratings on Leadership (Standard 1),

Content Knowledge (Standard 3), Facilitating Student Learning (Standard 4), and Reflecting on

Practice (Standard 5). In contrast, NC NTSP Cohort 2 teachers had significantly lower evaluation

ratings for Classroom Environment (Standard 2) in 2013-14 and when pooling data for two years

(2012-13 and 2013-14).

Please see Appendix H for: (1) more details on the evaluation rating models; (2) a complete set

of evaluation rating results, including both comparison groups and models focusing on first-year

teachers only; and (3) evaluation rating results with a more comprehensive set of NC NTSP and

comparison sample teachers.

Table 9. NC NTSP Evaluation Rating Results by Cohort

Cohort

Standard

1

Standard

2

Standard

3

Standard

4

Standard

5

2012-13 School Year

NC NTSP Cohort 1 1.082 1.128 0.997 0.842 0.980

NC NTSP Cohort 2 --- --- --- --- ---

2013-14 School Year

NC NTSP Cohort 1 1.722* 1.388 1.709

+ 1.844

* 1.572

+

NC NTSP Cohort 2 0.845 0.700+ 0.748 0.811 0.801

2012-13 and 2013-14 School Years

NC NTSP Cohort 1 1.376 1.267 1.323 1.203 1.257

NC NTSP Cohort 2 0.779 0.658* 0.694 0.752 0.730



August 2015


Retention

To What Extent does the NC NTSP Impact the Retention of Novice Teachers a) in the Same

Schools or LEAs, and b) in the State?

To determine whether NC NTSP evaluation sample teachers were more likely to remain in

teaching than their Non-RttT Comparison evaluation sample peers, the Evaluation Team

estimated models for three types of retention: (1) returning to any North Carolina public school

in the following school year (2013-14 and 2014-15); (2) returning to the same LEA in the

following school year; and (3) returning to the same low-performing school in the following

school year. For these analyses, the outcome variable was a “1” if the teacher returned in the

following school year and a “0” if the teacher did not return. In these models, the Evaluation

Team controlled for teacher experience and school characteristics and, post-estimation,

converted the results to predicted retention probabilities to facilitate easier interpretation of the

results.

Overall, when pooling data from the 2012-13 and 2013-14 school years, NC NTSP evaluation

sample teachers were significantly more likely to return to a North Carolina public school and to

their same LEA than were Non-RttT Comparison evaluation sample teachers. However, given

the need to keep teachers in high-need environments, the most important retention results are for

teachers returning to the same low-performing school. As shown in Figure 3 (following page),

the program-wide predicted probability of school-level retention was 73%—significantly higher

than for the Non-RttT Comparison evaluation sample (66%). These school-level predicted

probabilities of retention were above 70% in all four NC NTSP regions and were significantly

higher than those for Non-RttT Comparison evaluation sample teachers in the ECU, UNCC, and

UNCG regions.

Examining school-level retention by cohort, Figure 4 (following page) indicates that NC NTSP

Cohort 1 teachers had significantly higher retention rates than did Non-RttT Comparison

evaluation sample teachers. When school-level retention was calculated for Cohort 2 in the 2014-

15 school year, results demonstrated that they also had significantly higher retention rates than

did Non-RttT Comparison evaluation sample teachers.

Please see Appendix I for: (1) more details on the retention models; (2) a complete set of

retention results, including both comparison groups and models focusing on first-year teachers

only; and (3) retention results with a more comprehensive set of NC NTSP and comparison

sample teachers.


August 2015


Figure 3. NC NTSP School-Level Retention (2013-14 and 2014-15)


Figure 4. NC NTSP School-Level Retention Results by Cohort


NC NTSP Cohort 2 does not have a bar for returning in 2013-14 because they entered the program in 2013-14.

73.04** 77.26**

70.22 72.66+ 72.35+

65.78

0

20

40

60

80

NC NTSP

Overall

ECU Region UNC-CSLD

Region

UNCC Region UNCG Region Non-RttT

Comparison

Pre

dic

ted

Pro

ba

bil

ity

of

Ret

urn

ing

74.11+ 75.76** 74.18** 71.12* 71.67*

67.14 64.92 65.78

0

20

40

60

80

Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15

Pre

dic

ted

Pro

ba

bil

ity

of

Ret

urn

ing

NC NTSP Cohort 1 NC NTSP Cohort 2 Non-RttT Comparison


August 2015


Conclusions

Summary of Findings

Implementation

Over the course of the four-year grant period, the NC NTSP was developed, staffed, and

implemented from the ground up and has grown to scale, serving over 1,100 teachers in 114

schools as of the 2013-14 school year. There were substantial differences in program

participation and implementation across the four NC NTSP regions. Furthermore, as the NC

NTSP doubled in size in 2013-14, there was a drop in attendance at institutes and professional

development sessions, as well as fewer instructional coach visits per teacher.


A large majority of NC NTSP evaluation sample teachers who responded to the PSI-BT survey

felt that the program components helped their confidence, knowledge, and skills in teaching. In

addition, there was a statistically significant difference in the proportion of NC NTSP evaluation

sample respondents who felt the program components had a positive impact on their teaching,

compared to similar services provided by their own school. This significant difference also

existed between NC NTSP perceptions of program utility and Non-RttT Comparison sample

perceptions of analogous school-provided services.


Overall, NC NTSP evaluation sample teachers were significantly more effective than Non-RttT

Comparison evaluation sample teachers in fifth and eighth grade science, based on students’

End-of-Grade exam results. When assessing results by cohort, significant EVAAS results were

concentrated within NC NTSP Cohort 1 teachers—Cohort 1 teachers were significantly more

effective in elementary and middle grades mathematics and reading (both in 2012-13 in isolation

and with two years of pooled data) and were more effective in fifth and eighth grade science in

2013-14. There was only one positive and significant result for Cohort 2 teachers, who were

significantly more effective than Non-RttT Comparison evaluation sample teachers in fifth and

eighth grade science in 2013-14.

In terms of teacher evaluation ratings, there were no significant differences between NC NTSP

evaluation sample teachers and Non-RttT Comparison evaluation sample teachers in the overall

models. By cohort, NC NTSP Cohort 1 teachers had significantly higher evaluation ratings on

four North Carolina Professional Teaching Standards (Leadership, Content Knowledge,

Facilitating Student Learning, and Reflecting on Practice) in the 2013-14 school year, while

Cohort 2 teachers had significantly lower evaluation ratings on a fifth Standard (Classroom

Environment).

Teacher Retention

Overall, NC NTSP evaluation sample teachers were significantly more likely to return to

teaching in North Carolina public schools, to the same LEA, and to the same low-performing


August 2015


schools in comparison to Non-RttT Comparison evaluation sample teachers. Focusing on school-

level retention, NC NTSP teachers from both cohorts were significantly more likely to return to

the same low-performing school than were Non-RttT Comparison evaluation sample teachers.

Limitations

Two primary limitations should be taken into account when interpreting the findings in this

report:

1. The Evaluation Team’s ability to isolate the impact of the NC NTSP is diminished by two factors. First, the NC NTSP serves schools that received the highest proportion of RttT

interventions, some of which—District and School Transformation—are providing support

services similar to those provided by the NC NTSP. Thus, choosing comparison sample

schools outside of this eligible sample, like the Non-RttT Comparison group, precludes the

ability to hold these additional RttT interventions constant. Second, the NC NTSP is

universally offered in all of North Carolina’s lowest-achieving schools, meaning

comparisons with schools that are eligible yet non-participating may not take into account

potential confounding factors related to selection into the program. If there are differences

between the NC NTSP sample and the comparison groups that the Evaluation Team does

not control for and that influence the outcomes of interest, then the evaluation results will

be biased.

2. A second challenge of this evaluation lies in the fact that the data used in these analyses represent three program components (institutes, instructional coaching, and professional

development) implemented across four regional sites with two cohorts of teachers and in a

model that has greatly expanded in scope over the course of the evaluation period. Simply

put, given the time required for program development and scaling up, a longer evaluation

period is needed to assess adequately the effectiveness of a consistent program model

implemented as intended for first-, second-, and third-year teachers over the course of three

years.

Recommendations and Next Steps

There are three overarching recommendations for the NC NTSP after the close of the RttT grant:

1. Given the overwhelming need for effective teachers who remain in low-performing schools, the impacts of the NC NTSP on teacher value added to student achievement and teacher

retention support a recommendation to sustain the program beyond the end of the RttT

grant.

2. Findings from this evaluation should be used to explore the disparities in program implementation by region. Going forward, the Evaluation Team recommends that program

implementation fidelity be monitored formally to obtain reasonable consistency between

regions.

3. Further study is required to determine whether the decline in program participation and effectiveness in the 2013-14 school year is an artifact of the substantial scaling-up of the

program between 2012-13 and 2013-14. If the efficacy of the program remains diluted

when implemented at scale, the program implementers should explore options such as: (1)


August 2015


making agreements with participating LEAs and schools to require attendance at institutes

and professional development; (2) concentrating program resources on higher-intensity

instructional coaching; and/or (3) limiting program size to ensure a high-intensity program.


August 2015


Appendix A. Evaluation Sample

The Evaluation Team defined the NC NTSP evaluation sample for the 2012-13 academic year as

all first- and second-year teachers who began receiving NC NTSP services by December 2012

and who worked in schools that were eligible for and agreed to participate in the NC NTSP (RttT

schools).12

This excludes teachers who began receiving NC NTSP supports in January 2013 or

later, novice teachers receiving NC NTSP supports in non-RttT schools, the small number of

third-year teachers served by the program in 2012-13, and TFA corps members. For the 2013-14

academic year, the Evaluation Team defined the NC NTSP evaluation sample as all first-,

second-, and third-year teachers who began receiving NC NTSP supports by December 2013 and

who worked in schools that were eligible for (RttT schools) and agreed to participate in the NC

NTSP. While this includes third-year teachers, who were fully served by the program in 2013-

14, this again excludes teachers who began receiving NC NTSP supports in January 2014 or

later, novice teachers receiving NC NTSP supports in non-RttT schools, and TFA corps

members. For this final RttT evaluation report, the Evaluation Team excluded these groups

because (1) estimates of program performance need to be based upon a sample of teachers who

received NC NTSP supports for a majority of the school year; (2) the primary objective of the

Evaluation Team is to evaluate the performance of the program in its intended treatment area—

RttT schools; and (3) TFA corps members are significantly more likely to exit teaching after

their two-year service commitment, are significantly more effective, on average, than other

novice teachers (particularly in mathematics and science courses), and already receive induction

services from TFA, and thus, did not fully participate in the NC NTSP.

As shown in the left panel of Table A1 (following page), in the 2012-13 academic year the NC

NTSP evaluation sample consisted of 344 teachers working in 59 schools and 16 LEAs. A large

majority (72%) of this treatment sample were first-year teachers, and as stated above, all these

teachers worked in RttT schools and entered the program by December 2012. The right panel of

Table A1 shows that in the 2013-14 academic year the NC NTSP evaluation sample consisted of

808 teachers working in 91 schools and 25 LEAs. Nearly 50% of this treatment sample were

first-year teachers, 36% were second-year teachers, and 15% were in their third year of teaching.

All these teachers worked in RttT schools and entered the NC NTSP by December 2013.

12

The directive of the NC NTSP is to provide comprehensive induction services to novice teachers employed in the

state’s lowest-performing schools—schools that in the year before RttT began were either in the lowest 5% of all

schools in terms of student achievement or had graduation rates below 60%. We refer to these schools as “RttT

schools.”


August 2015


Table A1. NC NTSP Evaluation Sample

2012-13

NC NTSP

Evaluation Sample

2013-14

NC NTSP

Evaluation Sample

Teacher Count 344 808

Teachers by Region

ECU 66 189

UNC-CSLD 80 237

UNCC 120 182

UNCG 78 200

1st Year Teacher % 71.80 48.51

2nd

Year Teacher % 28.20 36.14

3rd

Year Teacher % n/a 15.35

Teachers in RttT Schools 344 808

Teachers Served by NC NTSP

Before January 344 808

TFA Corps Members n/a n/a

School Count 59 91

Schools by Region

ECU 12 23

UNC-CSLD 15 29

UNCC 18 19

UNCG 14 20

LEA Count 16 25

LEAs by Region

ECU 5 9

UNC-CSLD 4 5

UNCC 4 6

UNCG 3 5

To assess the impact of the NC NTSP, the Evaluation Team implemented a comparison group

design to contrast the outcomes of NC NTSP teachers with those of other novice teachers

working in low-performing schools. Isolating the effects of the NC NTSP on participating

teachers is particularly challenging because several other RttT interventions were also

concentrated in RttT schools over the same time period. The most notable was the District and

School Transformation (DST) initiative, which provided regular professional development and

coaching designed to improve the effectiveness and retention of teachers in RttT schools.

To address these challenges, the Evaluation Team created two different comparison groups to

better isolate the impact of the NC NTSP. For the first group, the Evaluation Team used school

performance composite data from the 2011-12 school year to identify schools in the bottom

decile of performance that were not eligible for (non-RttT schools) and did not receive NC NTSP

services. In the 2012-13 school year this comparison group, labeled Non-RttT Comparison,

consisted of all the first- and second-year teachers who began working in these schools by

December 2012. In the 2013-14 school year this comparison group consisted of all the first-,


August 2015


second-, and third-year teachers who began working in these schools by December 2013.13

Like

the NC NTSP evaluation sample, this Non-RttT Comparison evaluation sample excludes TFA

corps members and those beginning work in these schools after December. Overall, the left panel

of Table A2 shows that the evaluation sample for the Non-RttT Comparison group consisted of

1,033 teachers working in 147 schools and 48 LEAs in the 2012-13 school year; in 2013-14, the

Non-RttT Comparison evaluation sample consisted of 1,507 teachers working in 149 schools and

48 LEAs.

For the second comparison group, the Evaluation Team identified RttT schools (eligible for the

NC NTSP) that did not participate in the NC NTSP. In the 2012-13 school year this comparison

group, labeled NC NTSP Eligible Comparison, consisted of all the first- and second-year

teachers who began working in these schools by December 2012. In the 2013-14 school year this

comparison group consisted of all the first-, second-, and third-year teachers who began working

in these schools by December 2013. As with the evaluation samples for the NC NTSP and the

Non-RttT Comparison, the evaluation sample for the NC NTSP Eligible Comparison group

excludes TFA corps members and those beginning work in these schools after December.

Overall, the right panel of Table A2 shows that the evaluation sample for the NC NTSP Eligible

Comparison group consisted of 201 teachers working in 32 schools and 18 LEAs in the 2012-13

school year. In 2013-14 the NC NTSP expanded to serve an additional set of RttT schools.

Therefore, in the 2013-14 school year, the evaluation sample for the NC NTSP Eligible

Comparison group was reduced in size and consisted of 169 teachers working in 16 schools and

11 LEAs. In the main body of the final evaluation report, the Evaluation Team only reports

results for the Non-RttT Comparison sample; throughout the report appendices, the Evaluation

Team provides results for both the Non-RttT Comparison and the NC NTSP Eligible Comparison

groups.

Table A2. Evaluation Sample Comparison Groups

Like the NC NTSP sample, both of these comparison groups consist of novice teachers working

in low-performing schools. The Non-RttT Comparison group provides a larger sample for

13

Because the NC NTSP fully served third year teachers in the 2013-14 school year, the Evaluation Team included

third year teachers in both comparison samples in 2013-14.

Characteristics

Non-RttT Comparison NC NTSP Eligible Comparison

2012-13

Evaluation

Sample

2013-14

Evaluation

Sample

2012-13

Evaluation

Sample

2013-14

Evaluation

Sample

Teacher Count 1033 1507 201 169

1st Year Teacher % 60.47 40.69 60.70 39.05

2nd

Year Teacher % 39.53 34.77 39.30 33.14

3rd

Year Teacher % n/a 24.53 n/a 27.81

Teachers Hired Before

January 1033 1507 201 169

TFA Corps Members n/a n/a n/a n/a

School Count 147 149 32 16

LEA Count 48 48 18 11


August 2015


analyses; however, since it is comprised of schools that did not receive RttT services,

comparisons to this group do not allow for isolation of the impact of the NC NTSP from other

RttT programs. The NC NTSP Eligible Comparison group addresses this concern by comparing

NC NTSP teachers to novice teachers who received other RttT supports (primarily through

DST). Because this group is much smaller, it provides less statistical power for detecting

differences in outcomes. In addition, it is unknown why these schools declined to participate in

the NC NTSP (in 2012-13 and 2013-14) and why some schools chose to enter the program in

2013-14. Therefore, analyses may not take into account other factors related to non-participation.

Throughout subsequent appendices, the Evaluation Team presents results for the NC NTSP

evaluation sample and the Non-RttT Comparison and NC NTSP Eligible Comparison evaluation

samples. In appendices focused on teacher value-added to student achievement, teacher

evaluation ratings, and teacher retention, the Evaluation Team presents two additional sets of

results: (1) those for all teachers served by the NC NTSP and all novice teachers in comparison

sample schools, and (2) those for an amended NC NTSP evaluation sample that includes teachers

served in non-RttT schools who entered the NC NTSP by December of the academic year and

were not TFA corps members.

Table A3 (following page) presents school characteristics for the NC NTSP evaluation sample

and for the evaluation samples for each of the comparison groups in 2012-13 and 2013-14. In

2012-13, NC NTSP teachers worked in schools with (1) more students qualifying for subsidized

school meals; (2) more racial and ethnic minority students; (3) higher short-term suspension rates

and higher violent acts rates (in comparison to NC NTSP Eligible Comparison schools); (4)

higher percentages of novice teachers; and (5) lower performance composites (in comparison to

the Non-RttT Comparison schools). In 2013-14, NC NTSP teachers worked in schools with (1)

more students qualifying for subsidized school meals (in comparison to Non-RttT Comparison

schools); (2) more racial and ethnic minority students; (3) higher short-term suspension rates (in

comparison to Non-RttT Comparison schools); and (4) lower performance composites.


August 2015


Table A3. School Characteristics for the NC NTSP and Comparison Groups

Characteristic

2012-13 School Year 2013-14 School Year

NC NTSP

Sample

Non-RttT

Comparison

Group

NC NTSP

Eligible

Comparison

Group

NC NTSP

Sample

Non-RttT

Comparison

Group

NC NTSP

Eligible

Comparison

Group

Free and Reduced-Price

Lunch Percentage 93.58 85.93

** 87.16

+ 91.78 87.40** 86.54

Racial/Ethnic

Minority Percentage 90.87 78.37

** 82.31

* 89.63 78.67**

80.55+

Short-Term Suspension

Rate (Per 100 Students) 48.86 33.52

** 27.56

** 38.14 25.50** 26.30

Violent Acts Rate

(Per 1000 Students) 12.24 10.96 4.82

** 8.71 10.46 6.30

Total Per-Pupil

Expenditures $11,001 $10,052

+ $11,136 $10,534 $10,307 $10,214

Performance Composite 20.04 23.25** 22.55 31.07 34.29

** 39.94

*

Novice Teacher

Percentage 32.99 27.19

** 26.59

** 32.93 30.29 32.63

School Level

Elementary/Elementary-

Middle Combination

35

(59.32%)

89

(60.54%)

23

(71.88%)

57

(62.64%)

94

(63.09%)

10

(62.50%)

Middle School 10

(16.95%)

24

(16.33%)

2

(6.25%)

13

(14.29%)

23

(15.44%)

1

(6.25%)

High School 14

(23.73%)

33

(22.45%)

7

(21.88%)

21

(23.08%)

31

(20.81%)

5

(31.25%)

K-12 School 0

(0.00%)

1

(0.68%)

0

(0.00%)

0

(0.00%)

1

(0.67%)

0

(0.00%)

School Count 59 147 32 91 149 16

Note: This table displays school characteristics for schools in the NC NTSP, Non-RttT Comparison, and NC NTSP Eligible Comparison samples. +, *, and

** indicate statistically significant differences between NC NTSP schools and Non-RttT Comparison/NCNTSP Eligible Comparison schools at the 0.10,

0.05, and 0.01 levels, respectively.


August 2015


Appendix B. Data Sources

The Evaluation Team collected the data presented in this final evaluation report from the UNC-

GA implementation team, NC NTSP evaluation sample teachers and comparison sample

teachers, and administrative datasets supplied by the North Carolina Department of Public

Instruction (NCDPI). Specifically, this final evaluation report draws upon the following data

sources: (1) participation records from each of the components of the NC NTSP—institutes,

instructional coaching, and professional development; (2) survey responses by NC NTSP

evaluation sample teachers and Non-RttT Comparison sample teachers; and (3) student test

scores, student demographics, classroom rosters, teacher evaluation ratings, certified salaries,

school personnel, and school characteristics files provided by NCDPI. With these data sources

the Evaluation Team examined levels of participation in NC NTSP components, NC NTSP

teachers’ perceptions of program utility and self-efficacy/job satisfaction, and the impact of the

NC NTSP on teacher value-added to student achievement, teacher evaluation ratings, and teacher

retention. Below, we elaborate on the data sources used in this final evaluation report.

NC NTSP Program Documents

To examine the evaluation sample’s participation in the components of the NC NTSP, the UNC-

GA program implementers supplied the Evaluation Team with requested program documents.

For the 2012-13 year this included (1) the curricula, agendas, and rosters of attendees for the

2012 NC NTSP Summer and Winter Institutes; (2) counts of NC NTSP instructional coach visits

with each teacher; and (3) the agendas and rosters of attendees for NC NTSP professional

development sessions. Data for the 2013-14 year included (1) the curricula, agendas, and rosters

of attendees for the 2013 NC NTSP Regional Institutes and Statewide Institute; (2) counts of in-

person and virtual NC NTSP instructional coach visits with each teacher; (3) contact hours—in-

person and virtual—between NC NTSP teachers and program instructional coaches; and (4) the

agendas and rosters of attendees for NC NTSP professional development sessions.

Survey Responses by NC NTSP and Non-RttT Comparison Sample Teachers

To assess (1) the perceptions of NC NTSP teachers regarding the focus and utility of NC NTSP

components, and (2) the perceptions of both NC NTSP and Non-RttT Comparison sample

teachers regarding school-provided novice teacher supports, school context, teacher practices,

self-efficacy, and job satisfaction, the Evaluation Team partnered with North Carolina State

University’s (NCSU) College of Education to administer the Perceptions of Success Inventory—

Beginning Teachers (PSI-BT) survey in the spring of 2013 and 2014 (see the end of Appendix B

for a complete survey).14

For each survey administration the sample included NC NTSP and

Non-RttT Comparison sample teachers in the evaluation sample. Both sets of teachers completed

the regular PSI-BT items. In addition, NC NTSP teachers completed items developed by the

Evaluation Team assessing the components—institutes, instructional coaching, professional

development—of the NC NTSP. Researchers at NCSU used both online and paper-based

mediums to administer the PSI-BT in spring 2013; in spring 2014 NCSU researchers

administered the survey online only.

14

Please see http://ncsu.edu/succeed/beginning-teachers/ for more information on the development and use of the

PSI-BT instrument.

http://ncsu.edu/succeed/beginning-teachers/


August 2015


For the 2012-13 school year, the left panel of Table B1 presents response rates on the PSI-BT for

the NC NTSP and Non-RttT Comparison samples overall, and for each NC NTSP region. Nearly

half of the NC NTSP teachers in the evaluation sample responded to the survey (49.26%), with

higher response rates for those in the ECU region. To encourage responses in the Non-RttT

Comparison sample, the Evaluation Team provided a $5 financial incentive to survey

completers; approximately 41% of Non-RttT Comparison sample teachers responded to the

survey. The right panel of Table B1 displays PSI-BT response rates for the 2013-14 school year.

Slightly more than half of the NC NTSP teachers in the evaluation sample responded to the

survey (51.12%), with higher response rates for the ECU region and lower response rates for the

UNC-CSLD region. As in 2012-13, the Evaluation Team offered a $5 financial incentive to Non-

RttT Comparison sample teachers to complete the survey but only 17% responded in the 2013-14

school year.

Table B1. PSI-BT Response Rates

Survey Group

2012-13 PSI-BT Administration 2013-14 PSI-BT Administration

Survey

Respondents

Administered

Surveys

Response

Rate

Survey

Respondents

Administered

Surveys

Response

Rate

NC NTSP Overall 167 339 49.26% 409 800 51.12%

ECU 47 66 71.21% 147 187 78.61%

UNC-CSLD 33 77 42.86% 78 236 33.05%

UNCC 49 118 41.53% 88 179 49.16%

UNCG 38 78 48.72% 96 198 48.48%

Non-RttT

Comparison Group 308 759 40.58% 179 1053 16.99%

Due to the possibility for bias in the respondent sample, Table B2 (following page) displays data

on participation in NC NTSP components for those NC NTSP teachers who did and did not

respond to the survey (both overall and by region). In the 2012-13 school year, NC NTSP

teachers who responded to the PSI-BT were more likely to attend an institute (66% for

respondents versus 45% for non-respondents), received more instructional coaching visits per

month (4.18 visits for respondents versus 3.33 for non-respondents), and attended more

professional development sessions (2.87 PD sessions for respondents versus 1.60 PD sessions for

non-respondents). In the 2013-14 school year, differences in program participation between PSI-

BT respondents and non-respondents were reduced in magnitude; however, respondents were

still more likely to attend an institute, to receive more instructional coaching visits, and to attend

more professional development sessions. Given these differences, it is important to carefully

interpret all survey results.


August 2015


Table B2. Participation in NC NTSP Components, by PSI-BT Response Status

Region

Institute Attendance

Average Instructional

Coaching Visits

Per Month

Number of Professional

Development Sessions

Attended

Respond

Non

Respond Respond

Non

Respond Respond

Non

Respond

2012

-13 P

SI-

BT

Ad

min

istr

ati

on

NC NTSP

Overall 66.47% 45.35% 4.18 3.33 2.87 1.60

ECU 80.85% 57.89% 6.35 5.20 2.89 1.21

UNC-CSLD 66.67% 70.45% 2.26 1.94 4.15 3.32

UNCC 67.35% 33.33% 3.53 3.31 1.86 0.67

UNCG 47.37% 32.50% 4.02 4.01 3.03 1.53

2013-1

4 P

SI-

BT

Ad

min

istr

ati

on

NC NTSP

Overall 23.47% 17.14% 2.86 1.92 0.93 0.58

ECU 27.89% 25.00% 4.11 2.80 1.17 0.45

UNC-CSLD 21.79% 17.72% 1.81 1.59 1.60 1.04

UNCC 19.32% 13.19% 2.81 2.36 0.10 0.15

UNCG 21.88% 16.67% 1.87 1.70 0.79 0.27

Administrative Data from NCDPI

To assess the impact of the NC NTSP on teacher value-added to student achievement, teacher

evaluation ratings on the North Carolina Professional Teaching Standards (NCPTS), and teacher

retention, the Evaluation Team used administrative datasets provided by NCDPI. Specifically, to

estimate teacher value-added, the Evaluation Team used student test scores and demographics,

classroom rosters, school personnel, and school characteristics files. This allows for the

connection of students to their prior scores, demographics, teachers, and classroom peers;

teachers to their characteristics; and students and teachers to the characteristics of their schools.

To examine teacher evaluation ratings, the Evaluation Team used ratings on each of the five

NCPTS that are directly assessed by school administrators. Finally, for teacher retention, the

Evaluation Team used certified salary files to determine whether individuals returned to a

teaching position—overall, within the same LEA, or within the school school—in North

Carolina public schools.


August 2015


PSI-BT Survey

Unless otherwise noted, PSI-BT items have a six-point scale: strongly disagree, disagree,

slightly disagree, slightly agree, agree, and strongly agree.

Domain One: School- or LEA-Provided Mentor Support

Directions: Please respond to the survey questions below conce

North Carolina New Teacher Support Program - UNC Public ......North Carolina Department of Public...

Documents

Transcript of North Carolina New Teacher Support Program - UNC Public ......North Carolina Department of Public...