Day Treatment Program Evaluation

2010-11 Academic YearProject Summary

Day Program Overview

• The program provides youth (14 -19) an alternative learning opportunity in a smaller and more focused academic environment, with therapeutic and life skills support.

• The DTP has a capacity for serving 24 students.

• 3 classrooms, each staffed with a TDSB teacher and a Child and Youth Worker (CYW)

• Offers high school credit courses in a range of subjects

• Provides support and treatment to youth enrolled in the program

Evaluation Overview

• Centre of Excellence grant to carry out evaluation plan (plan developed with funding from MCYS)

• Purpose: To assess the degree to which the program is producing

positive outcomes for students enrolled in the program To determine what program areas/activities/processes, if any,

might be improved To utilize the findings of the evaluation to implement changes

that will improve program effectiveness and inform decisions about future program planning

• Mixed methods evaluation: quantitative and qualitative Program outcome data (e.g. attendance, CAFAS scores,

credits etc.) Stakeholders interviews (teachers, staff, youth, families)

• Steering committee oversaw the progress of evaluation

Program Evaluation Questions - Outcomes• Is the DYS day treatment program admitting the right

type of student?

• Are DYS day treatment program students committed to the program?

• Are DYS day treatment program students doing better in school?

• Are DYS day treatment program students earning educational credits?

• Are DYS day treatment program students' behavioural/emotional challenges improving?

• Have DYS day treatment program students found the program helpful?

Program Evaluation Questions (continued) Program Processes/Components

• In what ways do the key components of the DTP admission and intake process help to facilitate (or hinder) the enrolment of youth most likely to benefit from the program?

• In what ways do the key program components help to facilitate (or hinder) the success of youth enrolled in the program?

• In what ways do the key elements of the DTP discharge planning process help to facilitate (or hinder) a youth's preparedness for their post-DTP experience?  

Are Students meeting program admission criteria?

Client is:   YES NO% Met Criteria

1 A resident of the Toronto Area 17 0 100%2 Between the ages of 14 and 19 17 0 100%

3Have a secondary school system designation 17 0 100%

4 Conversant in English 17 0 100%

5

Experiencing psychiatric, behaviour symptom(s) that disrupt school functioning and/or educational challenge(s) 17 0 100%

6Formally diagnosed with M.I.D. (Mild Intellectual Disability) 0 17 100%

7 A danger to others 0 17 100%8 In a wheelchair 0 17 100%

Target: 100% of DTP students will meet admission criteria

Are Students meeting program admission criteria?

Insights from Stakeholder interviews:• the program’s admission and intake process might

be improved by having teachers more directly involved in this area of the program

• Concern expressed for the program policy of allowing both predominately internalizing youth to coexist in the same program with predominately behavioral youth

“we have to decide if we are going to be a program that’s going to do one thing well or we are going to try to be all things to all people…there is a distinction between students who are physically and verbally aggressive and those that have severe anxiety or depression.”

Are students committed to and engaged in the program?

• This suggests that the program was not successful in ensuring participant commitment in at least 20% of the youth during the 2010-11 school year.

> 80% > 70%* < 50%0%

20%

40%

60%

80%

100%

44%

69%

22%

DTP Students’ Attendance RatesN = 32

* Includes those students who have reached the target of >80% attendance rate

Target: 75% of DTP students attend at least 80% of scheduled school days

Are students committed to and engaged in the program?

Insights from Stakeholder Interviews:

• Highlighted deficiencies with our measurement approach to participant commitment and engagement (e.g. multidimensional nature of student engagement---cognitive, emotional & behavioral types of engagement)

• Highlighted other potential indicators of participants’ commitment to the program: completion of school related work attainment of high school credits participation in extracurricular activities establishment of positive relationships with program peers and

staff

Are Students functioning better in a school based environment?

Insights from Stakeholders

• Program activities that may contribute to improved client outcomes: Emphasis on goal setting and working towards goals Small class sizes Availability of a child and youth workers in the classroom setting Individualized programming and flexibility Strength based approach to both the academic and treatment component of the

program

Target: 80% of DTP students will show at least a 25% improvement in the CAFAS school subscale

25% (or more) improvement no improvement0%

20%40%60%80%

100%

38%

63%

Exiting DTP Students’ Level of ImprovementN = 16

Are students earning educational credits?

• Overall, the day treatment program shows a 58% percent credit achievement rate for the entire 2010-11 academic year.

at least 50% of cred-its achieved

All credits achieved No credits achieved0%

20%

40%

60%

80%

100%

56%

25%31%

Percentage of DTP Students Achieving CreditsN = 32

Target: 60% of DTP students will achieve at least 50% of attempted credits.

Have Students improved their behavioral/emotional functioning?

Target: 80% of DTP students will show at least a 25% improvement in their CAFAS ‘moods and emotions’ subscale or their ‘behavior towards others’ subscale

• A third of the group showing improved emotional or behavioral functioning was rated as having no significant dysfunction at program exit.

at least a 25% im-provement

No improvement0%

20%

40%

60%

80%

100%

56%44%

Exiting DTP Students’ Level of ImprovementN = 16

Have Students improved their behavioral/emotional functioning?

• Insights from Stakeholders the most important goal for program participants

is for them to work towards treatment goals and learn strategies that will help them cope with their issues and challenges

DTP teachers have limited involvement in creating or reviewing treatment goals. They would like greater input into the process.

Client Satisfaction: Do students have a positive view of their program experience?• Overall Response rate = 72% of all program participants

~78% of respondents report overall program satisfaction levels that are either good or excellent.

~77% of respondents would recommend DYS’programs to others.

Over 85% of respondents report that their primary program workers are a good match for them and that they’ve had a positive relationship with their primary workers.

Over 75% of respondents indicate that they felt empowered and respected in decision making related to the service that they received.

~48% of respondents agreed or strongly agreed that their personal goals were successfully achieved during their stay in the day treatment program (not inconsistent with our outcome data…moderate levels of success)

Further suggestions for improving the DTP: Stakeholder Insights

• Life skills worker to provide skill-building activities for program youth.

• Youth psychiatrist to help youth with treatment issues.

• Access to a guidance counselor with more specialized knowledge of the available school programs in the GTA.

• Nutrition program or incentives (e.g. breakfast program).

• Code of conduct with consistent rules, expectations and consequences for program participants.

• More opportunities for youth to participate in physical activities or gym class, clubs, sports teams and extracurricular activities.

• Greater access to George Harvey and its resources.

• Follow up with youth who leave the program.