Dr. Brian Rush

Centre for Addiction and Mental Health, Toronto, OntarioandDepartments of Psychiatry and School of Public Health, University of Toronto

EVALUATION AND QUALITY IMPROVEMENT IN SUBSTANCE USE TREATMENT AND SUPPORT: GOING FROM “SIMPLE” TO “COMPLEX” INTERVENTIONS.

SOME BASICS ABOUT EVALUATION

WHAT DO WE MEAN BY “EVALUATION”?

Evaluation involves the systematic collection of feedback about a program or intervention. The feedback is used in making decisions.

need assessment

participant satisfaction

outcome evaluation

economic evaluation

Different Types of Evaluation

Process

Need

Satisfaction

Outcome

Economic

WHY DO WE DO IT?

Accountability (service or system) - promises we make with public funds

Quality Improvement (service or system) – using feedback to improve what we do

Research/testing theory(service or system) – finding out things that others will be interested in and sharing it with them

Precontemplation – never think about it; actively avoid; only if pressured; don’t like to ask too many questions

Contemplation – do the minimum stats; ok but how do I start? Interested but afraid of the answers; went to a workshop once but no time to do it; bought a book about evaluation

Action – have a logic model and evaluation plan for one of my services; been talking to a researcher to help out; found a good way to start and made a proposal to the administration

Maintenance – wake up every morning thinking about how to evaluate my service; have a plan for each service; routine

DOES ANY ONE CARE ABOUT IT?

need assessment

participant satisfaction

outcome evaluation

economic evaluation

Establishing a Healthy Culture for Evaluation

The Importance of a Healthy Evaluation Culture

Process

Need

Satisfaction

Outcome

Economic

Treatment modality (CBT, motivational interviewing, naltrexone, methadone)

Service/program level (e.g., inpatient unit, day or evening drop-in service; family group, opiate substitution service)

Organization level (e.g., hospital, NGO, therapeutic community)

System/network level (e.g., substance use and mental health services, primary and urgent care, hospital and community networks, justice, education)

EVALUATION CAN BE DONE AT DIFFERENT LEVELS

PURPOSES AND LEVELS OF EVALUATION

Accountability/ Monitoring

Quality Improvement

Research/ Development

Service/ Intervention

Program/ Organization

System/Network

THE IMPORTANCE OF “EVIDENCE”

GOING FROM THEORY TO PRACTICE IN THE COMMUNITY

THE IMPORTANCE OF “EVIDENCE” IN SUBSTANCE USE TREATMENT AND

SUPPORT

Increase chances of really helping people (and at reasonable cost to society)

Research evidence does play a critical role in treated safely and respectfully

Research evidence is needed to ensure one’s “approach” gains the acceptance of professionals, funders and the community as a whole

Current challenge – there are many kinds of research and many ways of “knowing” something

THE “EVIDENCE PYRAMID” BEHIND CURRENT MEDICAL

PRACTICE

Meta-Analysis

Systematic Reviews

RCT”s

Cohort Studies

Case Control Studies

Case Reports

Animal Research

ADAPTATION OF THE PYRAMID FOR

QUALITATIVE EVIDENCE

I - Generalizable conceptual studies

II- Descriptive Studies

III- Single Case Studies

BUT THIS IS THE PYRAMID THAT”RULES”

Meta-Analysis

Systematic Reviews

RCT”s

Cohort Studies

Case Control Studies

Case Reports

Animal Research

EVOLVING MODELS OF RESEARCH EVIDENCE

More recognition of the limitations of RCT’s Who is actually represented in the research populations?

Does this research design really work for highly complex and system-level interventions?

More recognition now of “practice-based evidence” to complement evidence-based practice

More recognition of the challenges with this model in working with indigenous populations and studies of traditional medicine - a different view on the nature of “knowing”

CHALLENGE(S) MEASURING “SYSTEMS CHANGE”

CHALLENGE(S) MEASURING “SYSTEMS CHANGE”, “INTEGRATION”,

“COLLABORATION”

in most intervention research OUTCOME = intervention MINUS “noise” internal validity trumps external validity

in systems or complexity–based research OUTCOME = intervention PLUS context nothing can be understood outside of the context in which it exists

CHALLENGE #1

The goals or anticipated benefits of a systems change are not always shared or explicit among the key actors

HE SAYS:

“THIS IS THE PART I ALWAYS HATE!”

Substance Abuse

Treatment and Support

System

Mental Health

Treatment and Support

System

A process is underway in many countries to improve this problem

The personal and clinicalexperience of co-occurring disorders

Mental Disorders

Overlap in Canadian General Population

Substance Use

Disorders15-20%

Mental Disorders

Substance Use Services (Ontario)

Substance Use

Disorders70- 80%

Mental Disorders

Overall Mental Health System (Ontario)

Substance Use

Disorders15-20%

Mental Disorders

Young males in inpatient MH settings

Substance Use

Disorders55%

Mental Disorders

To help keep things in perspective…

Where are we with the justice system?

Substance Use

Disorders80-90%

WITH RESPECT TO THE INTEGRATION OF MENTAL HEALTH AND ADDICTIONS IN CANADA AND

ELSEWHERE …...

The train has clearly left the station!

ANTICIPATED BENEFITS OF “INTEGRATION”

To reduce stigma and discriminationTo improve access to servicesTo get better retentionTo improve continuity of careTo improve outcomes

??? ….. improved prevention of SU (e.g. addressing mental health and determinants of MH at young age)

DOES THE INTEGRATION OF HEALTH CARE SYSTEMS HAVE PREVENTION

IMPLICATIONS?

Prevention

Resources

Treatment

Resources

OTHER POSSIBLE GOALS???

Increased efficiency/less managementMore competitive positioning for resourcesInter-disciplinary issues of world

view/power/control (e.g., medical – non-medical)

Leveling the playing field (wages, credentialing, workforce mobility)

Expectations and perceived benefits are essentially “values based”

CHALLENGE # 2

“Systems Integration” does not mean the same thing to the various actors

MANY FIND IT HELPFUL TO CONSIDER “INTEGRATION” ALONG A CONTINUUM…

Full segregation

Co-operation

SO MANY WAYS TO CONSIDER “INTEGRATION”

Services vs Systems integration

Functional vs Horizontal

Normative integration (values based)

Information continuity: the use of information on past events and personal circumstances to make current care appropriate for each individual

Management continuity: a consistent and coherent approach to the management of a health condition that is responsive to a person’s changing needs

Relational continuity: an ongoing therapeutic relationship between a person and one or more providers

DIFFERENT ASPECTS OF INTEGRATION OR “CONTINUITY OF CARE”

SO MANY WAYS TO PUT “INTEGRATION” INTO PRACTICE

integrated clinical teams

centralized access and intake

shared electronic records

linkage managers/case management

the devil is in the details

every situation is unique

CHALLENGE # 3

“It is not clear how we should go about achieving “Systems Integration”

THE MECHANISMS/PROCESSES TO ACHIEVE IMPROVED “INTEGRATION” ARE NOT CLEAR OR EVIDENCE-BASED

May have a model in mind and its about “best practice” implementation – implementation science calls for active implementation supports

May be driven by a defined quality improvement process (Plan, Do, Study, Act) or change management process

Top down (Big world) vs bottom up (small world)Muddling along

Whatever the initial vision or model one typically works with the existing services and sub-systems as starting point(s) – leads to a basic strategy of “incrementalism” and ”contextual adaptation”

ESSENTIALLY A MIRACLE OCCURS….

THE MECHANISMS/PROCESSES TO ACHIEVE IMPROVED “INTEGRATION” ARE NOT CLEAR OR EVIDENCE-BASED

Whatever the initial vision or model, one typically works with the existing services and sub-systems as starting point(s) – leads to a basic process of “incrementalism” and ”contextual adaptation”

CHALLENGE # 4

“Health outcomes are often distal not proximal to the change efforts”

INTERMEDIARY OUTCOMES

(Process)Outcomes

- Access- Continuity of care- Information moves

Integration activity or

intervention

(Health)Outcomes

- SU- MH- QOL

CHALLENGE # 5

“Systems are never static – always responding to external context – can you really isolate the intervention from its dynamic context”

SYSTEMS ARE ALWAYS RESPONDING TO THE

ENVIRONMENT (TO SURVIVE)

SYSTEMS ARE ALWAYS RESPONDING TO THE

ENVIRONMENT (TO SURVIVE)

Aging population

Resource

AvailabilityNew needs

Political

reform

Health Determinants

CHALLENGE # 6

“Systems are never what they appear to be”

ARCHEOLOGICAL SYSTEMS AND ORGANIZATIONS

SecretsMemories

Myths

Old warriorsLies Truths

NEED TO DIG DEEP TO UNDERSTAND

SecretsMemories

Myths

Old warriorsLies Truths

CHALLENGE # 7

“Implementation of any systems change is never smooth – rarely do you end up with what you set out to achieve”

CHALLENGE # 8

“Routine information systems will always present challenges for evaluation of system change”

CHALLENGES WITH ROUTINE INFORMATION SYSTEMS

health data rarely collected for research and evaluation purposes

challenges with data linkage

mental health and addictions information can be lost

regionalization challenges commonality in data elements and quality

IS THERE ANY ROOM FOR OPTIMISM IN THE EVALUATION OF

INTEGRATION?

EMERGENT REALIST EVALUATION MODEL

Intervention + context = outcome

Systems analysis and complexity science

Mixed methods - qualitative and quantitative

Realist research synthesis and contribution analysis

Working on it and sharing the experiences

Thank you and good luck in your personal and professional work!

BUT WHAT DO WE DO?