Outcome Monitoring for Ontario Substance Use Services: Let’s call it

a “work in progress”

Dr. Brian RushProfessor

University of Toronto, Dept. of Psychiatry Scientist Emeritus,

Centre for Addiction and Mental HealthHealth Systems and Health Equity Research Group

Toronto, Canada

Why is Outcome Monitoring Important? Accountability (service or system) - promises

made with public funds include positive health benefits

Quality Improvement (service or system) – using feedback to improve what we do

Research - (service or system) – finding out things that others will be interested in and sharing it with them

What do we mean by benefits/outcomes? Recovery – defined as …… ? By whom…..?

Some elements: Reduction in alcohol or drug use (abstinence or

reduced use?) Improvements in self-efficacy (confidence), mental

health, health and social circumstances (eg. relationships, work, social network)

Improvement in overall quality of life/wellbeing Reduction in health care utilization (e.g., emerg visits) Reduction in criminal behaviour

What do we mean by outcome “monitoring”? Three models:

Post-Discharge/Post-Intake Follow-up Model Within-Treatment Model Hybrid Model

Rush, Martin and Corea, 2008

Post-treatment Follow-up Model Continues to be the standard (reflex) approach to

outcome assessment in the addictions field - Typically involves follow-up by trained third party at 3,

6, or 12 months after discharge Comprehensive assessment of outcome post-

treatment Resource intensive if not adapted (e.g., peer follow-

up workers) Others adaptations include: simplified outcome

measures, sampling, follow-up by telephone/IVR or clinicians

Some design considerations for Post-Treatment Follow-up Model Are the baseline evaluation tools embedded in the

clinical tools at intake and assessment? What measures will we use How long will we follow people? How will we locate them or will we just check status at

discharge? Is that sufficient? Do we have resources and expertise to manage all

this? How can we keep it simple and still meet outcome

expectations that may be placed on us?

Within Treatment Model Involves “...monitoring of an individual’s progress over

the course of treatment and feedback of this information...”

Compatible with good clinical practice No post-discharge follow-up (less costly, practical) Seems best suited to long-term (community) treatment Basis of new monitoring system in the UK using TOP

Hybrid Model Combines features of the Post-treatment Follow-up

and Within-Treatment Models;

Uses quarterly follow-up by program staff to collect outcome data and assess the need for further treatment;

Linkage manager uses MI techniques to assist clients to re-enter treatment;

Who is doing this routinely/systematically? Interest is always very high but when the rubber

hits the road it’s a different story:

Pretty much no one in Ontario or Canada doing this on a routine or systematic basis

DTFP gave us the opportunity to test feasibility and potential for scale up across the province

DTFP in Ontario was and remains an Opportunity for System Enhancement

The DTFP is a federal contribution program designed to support provinces and territories in their efforts to strengthen evidence-informed substance abuse treatment systems; and services. Itsupports sustainable improvement in the quality and organization of substance abuse treatment systems through investments in the following areas:

LINKAGE & EXCHANGE

STRENGTHENING EVALUATION & PERFORMANCE MEASUREMENT

IMPLEMENTATION OF EVIDENCE

INFORMED PRACTICE

April 1, 2011 and ending March 31, 2013.* One year extension to March 31, 2014.

Initial Ontario Systems Projects Ontario Systems Projects by investment areaProject Project Lead/Organization

IMPLEMENTATION OF EVIDENCE INFORMED PRACTICE

Best Practice Assessment Procedures Project Brian Rush, CAMH

Peer Support Services: Best Practices Project Janis Cramp, Addictions Ontario

Trauma and Substance Abuse: Developing Best Practice Standards

Nancy Bradley, Jean Tweed Centre

STRENGTHENING EVALUATION & PERFORMANCE MEASUREMENT

Client Outcome Monitoring Project Brian Rush, CAMH

Client Satisfaction Project Brian Rush, CAMH

Addictions Treatment Costs Project Brian Rush, CAMH

Youth Services System Review Project Gloria Chaim & Joanna Henderson, CAMH

Addiction Treatment Data Elements and Provincial Service Categories Project

Claudio Rocca, DATIS, CAMH

Residential Support Services Evaluation Project Janis Cramp, Addictions Ontario

Supportive Housing Evaluation Project David Kelly, OFCMHAP

LINKAGE & EXCHANGE

Ontario Mental Health and Addiction Knowledge Exchange Network Project

Heather Bullock, CAMH

The “Making Gains” Project David Kelly, OFCMHAP

Conceptual FrameworkDevelopmental Perspective

Assessment

Sta

ge o

f C

lient

Eng

agem

ent

DIVERSITY LENS

Screening

Stage 1(case finding)

Stage 2(case definition)

Treatment & Support

Stage 1(within-treatment

monitoring)

Transitional Youth

Stage 1(placement matching)

Stage 2(modality matching)

Child Adolescent Older Adult Adult

Recovery Monitoring

Stage 2(post-treatment

monitoring)

What did we try out? Objective: to assess feasibility of a comprehensive outcome monitoring system for addiction services iin Ontario based on:

Hybrid model Within-treatment monitoring Post intake follow up with “return-to-treatment”

protocol

DTFP Advisory Committee and (shared) Working Group with screening and assessment project

Where? Five programs in the pilot - same as the

screening and assessment pilot Thames Valley Addiction Services Four Counties Addiction Services Team Rideauwood Addiction and Family Services Addiction Centre (Hastings/Prince Edward) Manitoulin Community Withdrawal Management

Each experienced their own successes and challenges in recruitment and working with the follow-up team

Important to note this was in addition to their commitment to pilot the new screening and assessment tools

The basic process…. Recruitment/exploration of interest at pilot agency Informed consent via central team phone interview –

no payment for participation Locator information and baseline interview (GAIN-

Q3 MI collected by pilot agency – passed forward Tracking and 3 – 6 month telephone interviews by

central team Return-to-treatment protocol by central team Follow-up results shared with the program staff Monthly within-treatment monitoring by agency staff

Follow-up rates?

The majority of recruited clients came from two of the pilot sites

200 clients recruited and 148 completed a baseline interview

99 of these or 67% completed 3 month interview

89 or 69% of those due for a six month follow-up participated

Representativeness of those we found? Ok by age, gender, employment and mental

health status

BUT

Those followed up were more stable in terms of being in a relationship, high school education, with housing

Some key lessons learned re: process of outcome monitoring Extensive feedback obtained from managers and

staff, as well as tracking notes/logs from the follow-up team

In general most clients appreciated the interest in checking on how they were doing

High value to clinicians in the follow-up feedback Also the results from within-treatment monitoring “Return-to-treatment” protocol highly valued as well

Lessons-learned (con’t) Agency-level challenges - just following the many

steps in the process – again this was on top of the new screening and assessment piece

Locator information should be collected by follow-up team

Cell phone charges – need better use of technology

How to engage the more marginalized? Locating some clients can be very challenging – couldn’t implement the “community tracking” component

Despite the challenges what do the follow-up data tell us?

We analysed the results for the 117 clients followed for both the 3 and 6 months…

Analysis of baseline and follow-up data: Substance Use in previous 90 days (n=117):

Substance UseIndicator

Baseline 3 months 6 months

% of group totally abstinent

7.3 25.8 28.0

# of days abstinent

44.9 56.4 68.6

% of days with marijuana use

39.3 30.0 28.0

% of days with cocaine or crack use

22.4 8.6 9.8

Analysis of baseline and follow-up data - % scoring high in problem domain for previous 90 days

Problem Domain Baseline 3 months 6 months

Substance Use 74.6 43.6 34.1

Int. Mental Health 73.9 61.9 56.1

Ext. Mental health 45.2 34.0 25.1

Physical health 22.8 17.5 14.3

Stress 49.1 48.5 28.6

Risk Behaviour 22.8 17.5 14.3

Analysis of baseline and follow-up data - % using Hospital Inpatient services in previous 90 days

Hospital Service Used Baseline 3 months 6 months

Physical health 12.9 5.2 4.7Mental Health 11.3 5.3 6.1Hospital use for – any reason, including substance use

17.2 6.2 8.1

For those with ANY previous use of emergency or hospital services (n=22)• Days of hospital use dropped from 160 to 77 to 30 days at 6-months• Expressed in cost reduction: $80,080 to $15,390 or about a 5-fold decline in

cost

So where to from here…….

?

Value of GAIN-Q3 MI for Outcome Monitoring Although more piloting of Outcome Monitoring is

needed (especially for residential services and enhanced community tracking) we can see the GAIN Q3 MI provides a solid foundation for outcome monitoring

So the GAIN Q3 “works” beyond the baseline client

profiling and individual treatment planning

However….

Difficult but important choices made for other work with the current DTFP funding

Need to pursue post-intake/discharge outcome monitoring with other resources

Need fulsome discussion of how this might work provincially or regionally and stay in scope of potential resources

Continue Within-Treatment Monitoring in context of Screening and Assessment implementation

Build provincial performance measurement system - mazimize use of existing data – DATIS, OPOC-MHA, ICES, RAI-MH, OWTOM-A etc

Support individual provider, network and LHIN work

And stay positive because its time will come…

Thank you!