Outcome Monitoring for Ontario Substance Use Services: Let’s call it
a “work in progress”
Dr. Brian RushProfessor
University of Toronto, Dept. of Psychiatry Scientist Emeritus,
Centre for Addiction and Mental HealthHealth Systems and Health Equity Research Group
Toronto, Canada
Why is Outcome Monitoring Important? Accountability (service or system) - promises
made with public funds include positive health benefits
Quality Improvement (service or system) – using feedback to improve what we do
Research - (service or system) – finding out things that others will be interested in and sharing it with them
What do we mean by benefits/outcomes? Recovery – defined as …… ? By whom…..?
Some elements: Reduction in alcohol or drug use (abstinence or
reduced use?) Improvements in self-efficacy (confidence), mental
health, health and social circumstances (eg. relationships, work, social network)
Improvement in overall quality of life/wellbeing Reduction in health care utilization (e.g., emerg visits) Reduction in criminal behaviour
What do we mean by outcome “monitoring”? Three models:
Post-Discharge/Post-Intake Follow-up Model Within-Treatment Model Hybrid Model
Rush, Martin and Corea, 2008
Post-treatment Follow-up Model Continues to be the standard (reflex) approach to
outcome assessment in the addictions field - Typically involves follow-up by trained third party at 3,
6, or 12 months after discharge Comprehensive assessment of outcome post-
treatment Resource intensive if not adapted (e.g., peer follow-
up workers) Others adaptations include: simplified outcome
measures, sampling, follow-up by telephone/IVR or clinicians
Some design considerations for Post-Treatment Follow-up Model Are the baseline evaluation tools embedded in the
clinical tools at intake and assessment? What measures will we use How long will we follow people? How will we locate them or will we just check status at
discharge? Is that sufficient? Do we have resources and expertise to manage all
this? How can we keep it simple and still meet outcome
expectations that may be placed on us?
Within Treatment Model Involves “...monitoring of an individual’s progress over
the course of treatment and feedback of this information...”
Compatible with good clinical practice No post-discharge follow-up (less costly, practical) Seems best suited to long-term (community) treatment Basis of new monitoring system in the UK using TOP
Hybrid Model Combines features of the Post-treatment Follow-up
and Within-Treatment Models;
Uses quarterly follow-up by program staff to collect outcome data and assess the need for further treatment;
Linkage manager uses MI techniques to assist clients to re-enter treatment;
Who is doing this routinely/systematically? Interest is always very high but when the rubber
hits the road it’s a different story:
Pretty much no one in Ontario or Canada doing this on a routine or systematic basis
DTFP gave us the opportunity to test feasibility and potential for scale up across the province
DTFP in Ontario was and remains an Opportunity for System Enhancement
The DTFP is a federal contribution program designed to support provinces and territories in their efforts to strengthen evidence-informed substance abuse treatment systems; and services. Itsupports sustainable improvement in the quality and organization of substance abuse treatment systems through investments in the following areas:
LINKAGE & EXCHANGE
STRENGTHENING EVALUATION & PERFORMANCE MEASUREMENT
IMPLEMENTATION OF EVIDENCE
INFORMED PRACTICE
April 1, 2011 and ending March 31, 2013.* One year extension to March 31, 2014.
Initial Ontario Systems Projects Ontario Systems Projects by investment areaProject Project Lead/Organization
IMPLEMENTATION OF EVIDENCE INFORMED PRACTICE
Best Practice Assessment Procedures Project Brian Rush, CAMH
Peer Support Services: Best Practices Project Janis Cramp, Addictions Ontario
Trauma and Substance Abuse: Developing Best Practice Standards
Nancy Bradley, Jean Tweed Centre
STRENGTHENING EVALUATION & PERFORMANCE MEASUREMENT
Client Outcome Monitoring Project Brian Rush, CAMH
Client Satisfaction Project Brian Rush, CAMH
Addictions Treatment Costs Project Brian Rush, CAMH
Youth Services System Review Project Gloria Chaim & Joanna Henderson, CAMH
Addiction Treatment Data Elements and Provincial Service Categories Project
Claudio Rocca, DATIS, CAMH
Residential Support Services Evaluation Project Janis Cramp, Addictions Ontario
Supportive Housing Evaluation Project David Kelly, OFCMHAP
LINKAGE & EXCHANGE
Ontario Mental Health and Addiction Knowledge Exchange Network Project
Heather Bullock, CAMH
The “Making Gains” Project David Kelly, OFCMHAP
Conceptual FrameworkDevelopmental Perspective
Assessment
Sta
ge o
f C
lient
Eng
agem
ent
DIVERSITY LENS
Screening
Stage 1(case finding)
Stage 2(case definition)
Treatment & Support
Stage 1(within-treatment
monitoring)
Transitional Youth
Stage 1(placement matching)
Stage 2(modality matching)
Child Adolescent Older Adult Adult
Recovery Monitoring
Stage 2(post-treatment
monitoring)
What did we try out? Objective: to assess feasibility of a comprehensive outcome monitoring system for addiction services iin Ontario based on:
Hybrid model Within-treatment monitoring Post intake follow up with “return-to-treatment”
protocol
DTFP Advisory Committee and (shared) Working Group with screening and assessment project
Where? Five programs in the pilot - same as the
screening and assessment pilot Thames Valley Addiction Services Four Counties Addiction Services Team Rideauwood Addiction and Family Services Addiction Centre (Hastings/Prince Edward) Manitoulin Community Withdrawal Management
Each experienced their own successes and challenges in recruitment and working with the follow-up team
Important to note this was in addition to their commitment to pilot the new screening and assessment tools
The basic process…. Recruitment/exploration of interest at pilot agency Informed consent via central team phone interview –
no payment for participation Locator information and baseline interview (GAIN-
Q3 MI collected by pilot agency – passed forward Tracking and 3 – 6 month telephone interviews by
central team Return-to-treatment protocol by central team Follow-up results shared with the program staff Monthly within-treatment monitoring by agency staff
Follow-up rates?
The majority of recruited clients came from two of the pilot sites
200 clients recruited and 148 completed a baseline interview
99 of these or 67% completed 3 month interview
89 or 69% of those due for a six month follow-up participated
Representativeness of those we found? Ok by age, gender, employment and mental
health status
BUT
Those followed up were more stable in terms of being in a relationship, high school education, with housing
Some key lessons learned re: process of outcome monitoring Extensive feedback obtained from managers and
staff, as well as tracking notes/logs from the follow-up team
In general most clients appreciated the interest in checking on how they were doing
High value to clinicians in the follow-up feedback Also the results from within-treatment monitoring “Return-to-treatment” protocol highly valued as well
Lessons-learned (con’t) Agency-level challenges - just following the many
steps in the process – again this was on top of the new screening and assessment piece
Locator information should be collected by follow-up team
Cell phone charges – need better use of technology
How to engage the more marginalized? Locating some clients can be very challenging – couldn’t implement the “community tracking” component
Despite the challenges what do the follow-up data tell us?
We analysed the results for the 117 clients followed for both the 3 and 6 months…
Analysis of baseline and follow-up data: Substance Use in previous 90 days (n=117):
Substance UseIndicator
Baseline 3 months 6 months
% of group totally abstinent
7.3 25.8 28.0
# of days abstinent
44.9 56.4 68.6
% of days with marijuana use
39.3 30.0 28.0
% of days with cocaine or crack use
22.4 8.6 9.8
Analysis of baseline and follow-up data - % scoring high in problem domain for previous 90 days
Problem Domain Baseline 3 months 6 months
Substance Use 74.6 43.6 34.1
Int. Mental Health 73.9 61.9 56.1
Ext. Mental health 45.2 34.0 25.1
Physical health 22.8 17.5 14.3
Stress 49.1 48.5 28.6
Risk Behaviour 22.8 17.5 14.3
Analysis of baseline and follow-up data - % using Hospital Inpatient services in previous 90 days
Hospital Service Used Baseline 3 months 6 months
Physical health 12.9 5.2 4.7Mental Health 11.3 5.3 6.1Hospital use for – any reason, including substance use
17.2 6.2 8.1
For those with ANY previous use of emergency or hospital services (n=22)• Days of hospital use dropped from 160 to 77 to 30 days at 6-months• Expressed in cost reduction: $80,080 to $15,390 or about a 5-fold decline in
cost
So where to from here…….
?
Value of GAIN-Q3 MI for Outcome Monitoring Although more piloting of Outcome Monitoring is
needed (especially for residential services and enhanced community tracking) we can see the GAIN Q3 MI provides a solid foundation for outcome monitoring
So the GAIN Q3 “works” beyond the baseline client
profiling and individual treatment planning
However….
Difficult but important choices made for other work with the current DTFP funding
Need to pursue post-intake/discharge outcome monitoring with other resources
Need fulsome discussion of how this might work provincially or regionally and stay in scope of potential resources
Continue Within-Treatment Monitoring in context of Screening and Assessment implementation
Build provincial performance measurement system - mazimize use of existing data – DATIS, OPOC-MHA, ICES, RAI-MH, OWTOM-A etc
Support individual provider, network and LHIN work
And stay positive because its time will come…
Thank you!
Top Related