District Response to ProgramImprovement Accountability Sanctionsand Technical Assistance in California

December 1, 2011 – CERA Presentation

Presented by: Theresa Westover and Mary Stump

1

Evaluation Context• Federal: ESEA mandates increasing sanctions for districts

not making AYP. – In addition, ESEA requires states provide provide technical

assistance to all LEAs identified for improvement (NCLB, PL 107-110, TI, Sec 1116(c)

• California: Sanction F/Corrective Action 6 – Curriculum AB 519 – Chaptered in 2008– Required independent evaluation 2009-12

• Evaluation focus: DAIT Process and Student Achievement Changes – Mixed Methods– Cohort 1: 43 Districts in PI Year 3 assigned corrective action 6 in

2008-09 & their DAITs– Cohort 2: 30 Districts in PI3, assigned Corrective Action 6 in

2009-10 & their DAITs

2

Program Improvement – The Top-Down Accountability Era

3

Today’s focus:1. Brief look: what districts are in program

improvement and focus of evaluation

2. How the DAITs engaged with their districts

3. What actions the district and DAIT took - the focus of the improvement efforts

4. The barriers and facilitators encountered along the way

4

Cohort 1: Districts in PI (N=43)Broad geographic and size range

5

PI3 Districts: High Need Students, Less Experienced Teachers

• PI3 districts have more disadvantaged students than do districts that are not in PI

• Within PI3 districts, those with DAIT treatment appear to have the most disadvantaged students in California

• Districts with assigned DAITs have a significantly different teacher workforce in terms of average and median experience levels and certification rates.

(Details in Year 1 Report, available at cees.ucdavis.edu)

6

There are significant disparities in poverty and ELL designations between PI districts

Non-PI Districts

PI 1 Districts

PI 2 Districts

All PI 3 Districts

PI 3 Districts – No Aid

PI 3 Districts

– TA

PI 3 Districts –

Contracted DAIT

PI 3 Districts – Assigned

DAIT

PI3 districts service more Hispanic and fewer white students

Non-PI Districts

PI 1 Districts

PI 2 Districts

All PI 3 Districts

PI 3 Districts – No Aid

PI 3 Districts

– TA

PI 3 Districts –

Contracted DAIT

PI 3 Districts – Assigned

DAIT

Methods and Data (Year 2 Report)

For each of the 43 Cohort I districts we:– Analyzed weaknesses and recommendations in

the capacity studies– Issued surveys to both the DAIT providers and the

Districts– Interviewed both the DAIT provider and the

District leadership team• Response rates for surveys and interviews: 100% for

DAIT providers, close to 90% for districts

9

Qualitative data

CEES selected areas of evaluation inquiry based on:

• legislation requirements• the guidelines provided by CDE to DAIT

providers (the DAIT “strands”)• California’s 9 Essential Program Components

(EPCs) http://www.cde.ca.gov/ta/lp/vl/essentialcomp.asp

10

DAIT Process in California

11

DAIT Recommendations in Capacity Studies

12

DAIT engagement• 97% of providers remained engaged in their districts

for 2 or more years• Over 90% agreed on surveys that the DAIT:– Effectively diagnosed district needs & priorities– Provided support for the LEA plan/addendum– Was provided access and information needed to

understand district needs– Was able to effectively engage the district leadership to

address needed changes• Most providers met at least monthly w/ district and

reported having successfully established open & cooperative relations

13

Focus of improvement efforts

• Open ended interview responses indicated that high priority areas for improvement included:– Supports for English learners (50% mentioned)– Focus on math curriculum/instruction (33%)– Data based decision making/data systems (40%)

• These areas were reflected in survey ratings as showing improvement over the 2 years

14

Changes in implementation of EL/ELD supports

Practice/Policy

Percent Full/Substantial Implementation 2008-09

Percent Full/Substantial Implementation 2009-10

Change

Administrative procedures to implement and monitor district ELD programs

45.2 80.5 +35.3%

All teachers participate in ELD/SDAIE professional development

53.6 73.8 +20.2

District ensures that teachers use SBE adopted/approved materials for ELD

58.5 81.0 +22.5

District ensures that ELs have access to grade level core instruction in Math

75.6 97.0 +21.4

15

Changes in data systems/data-based decision making

Practice/Policy

Percent Full/Substantial

Implementation2008-09

Percent Full/Substantial

Implementation2009-10

Change

District administrators regularly use data to monitor student progress

47.6 83.3 +35.7%

District has a system of regular data collection to determine the effectiveness of its academic program

69.0 92.9 +23.9

All teachers are provided collaboration time specifically for examining student data to inform instruction

61.9 88.1 +26.2

District staff adhere to established criteria for student entry and exit into interventions

35.7 64.3 +28.6

16

Overall Changes in Implementation Ratings (4 point scale, with 4=full implementation)

17

Barriers and Facilitators• Research in organizational change and

district/school improvement demonstrate that organizational change takes time and needs to be responsive to contextual factors.

• Interviews confirmed that the district improvement efforts are highly contextual – not only the content of work but the way it was undertaken and the success, or lack thereof, of the efforts

18

District leadership factors

• Tenure of district leadership (Supt & cabinet)• Leadership’s willingness to engage in reform • Leadership style• Existing relationships among district staff,

between district staff and school board, teacher unions and other stakeholders

• Change in district leadership over the course of the engagement

19

District history, setting, and culture

• Tradition of school site accountability to district• History of student achievement, mobility &

demographics• Location & size of district• Responsiveness to external pressure/mandates• Culture around expectations for student

achievement and student subgroups

20

Local stakeholders

• Local political climate and relations among stakeholder groups

• School Board stability, level of involvement in district policies and practice, political positions

• Teacher union contractual language, relationship w/ Board and district, history of negotiations w/ district

• Legal actions

21

District resources, management and structures

• Fiscal resources• Human resources – e.g. expertise, staffing

levels• Existing structures and practices– Data & assessment systems– Practices around monitoring classroom instruction– Communication & accountability structures

district:schools– Allocation of responsibilities among district staff

22

Most significant changes

• Supports for under-performing students, esp. ELs

• Improved instructional materials in ELA & Math, including teacher & administrator PD

• Teacher support – coaches, professional learning communities (PLCs)

• Data systems and use of data for decision making

23

Sustainability

• Unclear at this point• Districts/DAITs identified a number of threats:– Fiscal crisis is a major threat to sustainability– Shifting priorities back from district to school level– Lack of on-going support and accountability for

change once DAIT is gone

24

Preliminary Recommendations• Continue to support district level capacity building and

technical assistance• Increase or maintain accountability structures for both

districts & DAITs• Assess district readiness for change and act more

quickly to intervene when necessary• Educate stakeholders – especially local boards• Simplify and consolidate federal and state mandated

reporting requirements• Provide additional support and resources to assist

district capacity building

25

Resources and Information About Program Improvement in California

1. California Department of EducationCDE Website for PI: http://www.cde.ca.gov/ta/ac/ti/programimprov.asp and specifically for PI3 resources: http://www.cde.ca.gov/ta/ac/ti/pilearesources.aspCDE/SBE criteria for determining level of LEA need for technical assistance:

EC Section 52055.57(d)Criteria identified in Item 16 on the SBE Agenda--November 18-19, 2009 Web page at http://www.cde.ca.gov/be/ag/ag/yr09/agenda200911.asp

2. California Comprehensive Center at WestEd Webinars, history and assessment instruments: http://www.cacompcenter.org/cs/cacc/print/htdocs/cacc/esea-requirements.htm 3. Center for Education and Evaluation Services, UC Davis - Theresa Westover & USC Rossier School of Education, Katharine Strunk.Interim Evaluation Reports on AB519 – Enactment of Corrective Action 6 in California: http://education.ucdavis.edu/select-publications-and-reports

4. The Use and Efficacy of Capacity-Building Assistance for Low-Performing Districts: The Case of California’s District Assistance and Intervention Teams Paper prepared for the Annual Research Conference of the American Education Finance and Policy Association, March 23-25, 2011. www.aefpweb.org/.../Strunk_Westover_and_McEachin_AEFP_2011

26

Contact Information:Theresa Westover

tnwestover@ucdavis.edu530-754-9523

Mary Stumpmlstump@ucdavis.edu

530-752-2809

For more information, please visit the Center for Education and Evaluation Services website at: cees.ucdavis.edu

27