ACCOUNTABILITY & EVALUATION

Overview of this Session

• Defining Accountability & Evaluation

• MTSSS & Program Evaluation Issues

• Example of Evaluation in MTSSS

• Discussion– Current status– Next steps

What does…

• Accountability mean to you?

• Evaluation mean to you?

Accountability in Florida

• Increasing accountability focus the last decade

• Examples include:– School grading– AYP– Special education rules– DA– FEAPS & Teacher evaluation systems

Impact of Accountability

Criticisms• Lack of educator

involvement• Controversy• Consequence driven• Compliance driven• Conflicting

requirements• Duck and cover

approach

Positives• Establishes and

maintains standards for performance

• Reinforces use of data to monitor student outcomes

• Reinforces need to examine resource use

• Student outcome rather than process focus

• Success stories

(Hall & Hord)

Accountability & Evaluation Issues

• Compliance driven versus informative evaluation– Evaluation often done to meet accountability

requirements– Evaluation can serve to help integrate and improve

school and district services

• Evaluation is fundamental to MTSSS• MTSSS has the potential to:

– Be viewed as one more thing we have to do OR– Help address accountability & evaluation demands

through the multi-tier framework

Mission and Vision

Multi-Tiered System of Student Supports - Inter-Project Collaboration

The collaborative vision of the Florida Problem-Solving/Response to Intervention (FL PS/RtI) and the Florida Positive Behavior Support/Response to Intervention for Behavior (FLPBS/RtI:B) Projects is to:• Enhance the capacity of all Florida school districts to successfully

implement and sustain a multi-tiered system of student supports with fidelity in every school;

• Accelerate and maximize student academic and social-emotional outcomes through the application of data-based problem solving utilized by effective leadership at all levels of the educational system;

• Inform the development, implementation, and ongoing evaluation of an integrated, aligned, and sustainable system of service delivery that prepares all students for post-secondary education and/or successful employment within our global society.

Important MTSSS Evaluation Issues

• Stakeholders should be involved in all aspects of planning and carrying out the evaluation process as well as in decision-making

• Goals through planning should drive the process

• Information obtained to:– Determine where you currently are (needs)– Take ongoing looks at how things are working– Make decisions about what to keep doing and

what to change or eliminate

MTSSS Eval Issues cont.

• The data you collect should be driven by the questions you want to answer– Are students meeting expectations?

Academically? Behaviorally? Social-emotionally?– Are we implementing MTSSS with fidelity?– Do we have the capacity to implement

successfully?– Do staff buy into implementing MTSSS?

*Example questions

EXAMPLE QUESTIONS AND DATA SOURCES

How Are Students Performing?

Examples of data sources• Academics

– FCAT– FAIR– Core K-12– End of Course Exams

• Behavior– Attendance– Tardies– Suspensions– Discipline referrals

• Global Outcomes– Graduation Rates

www.flrtib.org

Are Schools Implementing MTSSS with Fidelity?

Examples of data sources• Curriculum and Instruction/Intervention

– Principal walkthroughs– Lesson plans– Intervention Documentation Worksheets

• Components of MTSSS and Data-Based Problem-Solving*– BOQ, PIC, BAT– SAPSI, Tier I & II CCCs, Tier III CCCs

* See http://flpbs.fmhi.usf.edu/ and http://floridarti.usf.edu for more information

Ro

les

Rep

rese

nte

d

Pro

ble

m Id

enti

fica

tio

n

Pro

ble

m A

nal

ysis

Inte

rven

tio

n D

evel

o...

Pro

gra

m E

valu

atio

n/..

.0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0.680.76

0.54

0.28

0.64

Tiers I & II Observation Checklist

Percent Present

Roles Present and Problem-Solving Steps

Per

cen

tag

e o

f R

ole

s/C

om

po

nen

ts P

rese

nt

Do We Have the Capacity to Implement MTSSS with Fidelity?

Examples of data sources• Leadership Team structure and functioning

– Organizational charts– Minutes/meeting summaries– SAPSI, BOQ, PIC

• Staff knowledge and skills– FEAPs & teacher evaluation system– Staff development evaluations– Work samples

• Resources allocated to match needs– SIP, DIP– Master calendar/schedule– School rosters– Resource maps

Marzano Evaluation Domains

• Classroom Strategies and Behaviors• Preparing and Planning• Reflecting on Teaching• Collegiality and Professionalism

Do Staff Buy Into Implementing MTSSS?

Examples of data sources• Leadership vision and commitment– SAPSI, BOQ, PIC– Required and non-required plans

• Staff buy in– SAPSI, BOQ, PIC– District/school staff and climate surveys– Dialogue– Brief interviews with key personnel

District commitment SBLT support Faculty involvement SBLT present Data to assess commitment

1 2 3 4 5

0

1

2

3

Sunshine Elementary: Self-Assessment of Problem Solving Implementation (SAPSI) Data Consensus Building

BOY

EOY

Item Description

Indi

cato

r Sta

tus

3= Maintaining2= Achieved1= In Progress0= Not Started

Discussion

• What are you currently doing to examine these areas in your district or school?– What are the critical questions you ask?– What data sources do you have to answer them?– What questions do you already have that you cannot

answer with available data?– How do you use the data you collect to inform decisions?

• What areas need to be addressed as you return to your districts to plan? What are the priorities?– What critical questions do you need to start asking?– What data sources do you need?– How can you better use the data to inform decisions?

Our Plan for Moving Forward

• Learn more about your needs for evaluation support through needs assessment and DAPP protocol

• Develop resources for evaluating critical domains necessary to implement MTSSS such as:– Leadership– Coaching– Fidelity of implementation

• What additional supports can we provide?