Knowledge Translation&

Common Child and Youth Mental Disorders (CCYMD)

Offord Centre for Child Studies, November 26, 2015

Plan for Today

KT Refresher•KT & evidence-informed decision making•Synthesizing research evidence: the science & the art•Diffusion, dissemination or implementation?•Knowledge to Action Cycle•Implementation Science

KT & CCYM Disorders: Are We Ready for Prime Time?•SR, MA & PG: Do they meet international quality standards?•Knowledge repositories & tools: Users guide•Effective Knowledge Implementation Strategies?

Team

Alexa Bagnell Daniel Gorman Lynn Miller

Gail Bernstein Daphne Korczak Christopher Mushquash

Peter Braunberger Stanley Kutcher Amanda Newton

Jeffrey Bridge Paul Links Anne Rhodes

Melissa Brouwers Ellen Lipman Maureen Rice

Amy Cheung Katharina Manassis Robert Santos

Stephanie Duda Ian Manion Peter Szatmari

Jane Garland John McLennan Lehana Thabane

Common Child and Youth Mental Disorders (CCYMD)

Anxiety Disorders: Agoraphobia, generalized anxiety disorder, social phobia, specific phobia, panic disorder, separation anxiety disorder

Mood Disorders: Major depressive disorder, dysthymia

Disruptive Behaviour Disorders: Attention deficit hyperactivity disorder, oppositional defiant disorder, conduct disorder

What is Knowledge Translation?

CIHR Definition*:1. Process: Dynamic and iterative; includes synthesis, dissemination, exchange and ethically sound application of knowledge.

2.Three goals: improve the health of Canadians, provide more effective health services and products and strengthen the health-care system.

* CIHR. Knowledge translation and commercialization. (Updated Nov 17, 2015) http://www.cihr-irsc.gc.ca/e/29529.html

But Many Different Terms Out There…

McKibbon et al (2010). A cross-sectional study of the number & frequency of terms used to refer to knowledge translation in a body of health literature in 2006: A Tower of Babel? Implementation Science; 5:16.

• 100 different terms in 581 articles• CIHR definition used widely.

Integrated Knowledge Translation (iKT)*

1. KUs involved in all stages of research*: • increase relevance • facilitate uptake

2. Process: takes place within complex system of interactions between researchers and knowledge-users (KUs) which may vary in intensity, complexity and level of engagement depending on the nature of the research and findings, as well as the needs of the particular knowledge user.

* CIHR (2015). Guide to Knowledge Translation Planning at CIHR: Integrated and End-of-Grant Approaches. http://www.cihr-irsc.gc.ca/e/45321.html

* Gagnon ML. Moving knowledge to action through dissemination and exchange. Journal of Clinical Epidemiology, 2011

1. Process:• Systematic, transparent use of research evidence in

health practice and policy decision making

2. Principles:• Research evidence provides guidance not prescription• Research evidence doesn’t make decisions – people do• Primum non nocere: Above all, do no harm

What is Evidence-Informed Policy & Practice?

Evidence-informed Decision Making: 3 Pillars

First Step: Synthesize Research Knowledge

• Major focus/energy to date• Synthesis can:

1. Inform research agenda: • Identify research gaps & priorities • Provide rationale for new research

2. Facilitate research knowledge use in health services decisions:

• Strengthen health provider knowledge and intentions• Better care process decisions• Improve health outcomes

• Science and art

Targets of

Change

Research Evidence Pyramid: The Science

Why Synthesize Body of Knowledge?

• Ioannidis, PLoS 2005*: Need adequately powered, low bias evidence:

“ … most research questions are addressed by many teams, and it is misleading to emphasize the statistically significant findings of any single team. What matters is the totality of the evidence...”

• Ioannidis, JAMA 2005†: • 45/49 highly cited studies claim intervention effective• 16% contradicted by subsequent studies• 16% found bigger effects• 44% replicated• 24% remained largely unchallenged

* Ioannidis JPA. Why most published research findings are false. PLoS. 2005.

† Ioannidis JPA. Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. JAMA. 2005.

Research Evidence Synthesis: The Art

1. What: Raw or digested?• Primary studies• SR/MA• PGs• Synopses & tools• Curated/quality assessed• Actionable recommendations

2. Who: Tailored and user friendly?• Practitioners• Patients• Managers• Policy-makers

3. How: Communication format/medium?1. e-Resources2. Reminders and updates3. Face-to-face4. Paper5. Combination6. Push, pull, push & pull

From Knowledge to Action

Knowledge Users*Knowledge Producers

* Policy-makers; Managers; Clinicians; Patients & Families

Knowledge To Action Cycle*

* Graham ID, et al. Lost in knowledge translation: Time for a map? The Journal of Continuing Education in the Health Professions. 2006.

Action!

Diffusion • Passive spread of evidence

Dissemination

• Active, tailored and planned distribution of evidence to target groups to encourage adoption of innovations

Implementation

• Active efforts to put evidence into practice

Knowledge To Action Cycle*

* Graham ID, et al. Lost in knowledge translation: Time for a map? The Journal of Continuing Education in the Health Professions. 2006.

Implementation Science

“Scientific study of methods to promote the systematic uptake of research findings & other evidence-based practices into routine practice to improve the quality and effectiveness of health services and care”*

• Identify barriers and facilitators to practice & policy change• Create interventions to promote research uptake• Theories, models & frameworks

* Eccles, MP & Mittman BS. Welcome to Implementation Science. Implementation Science, 2006. * Nilsen P. Making sense of implementation theories, models and frameworks. Implementation Science, 2015.

Theories, Models & Frameworks*

Three aims: • Describe translation process• Understand or explain what influences implementation

outcomes• Evaluate implementation

Theories • Clear explanation of how and why specific relationships lead to specific events

Models • Simplification of a phenomenon• Descriptive rather than explanatory

Frameworks • Describe empirical phenomena by fitting them into categories

• Not explanatory

Common Sense

• “Informal” theories

* Nilsen P. Making sense of implementation theories, models and frameworks. Implementation Science, 2015.

Consolidated Framework for Advancing Implementation

Science (CFIR)

• Damschroder, 2009*:• Identified 19 theories/models/frameworks• Created CFIR • Overarching typology to promote implementation theory

development and verification about what works where and why across multiple contexts

*Damschroder LJ, et al. Fostering implementation of health services research findings into practice: Consolidated framework for advancing implementation science. Implementation Science, 2009.

CFIR: Domains & Constructs

Intervention Characteristic

s

Outer Setting

Inner Setting

Characteristics of

Individuals Involved

Process of Implementation

• Intervention Source• Evidence Strength

& Quality• Relative Advantage• Adaptability• Trialability• Complexity• Design Quality &

Packaging• Cost

• Patient Needs & Resources

• Cosmopolitanism• Peer Pressure• External Policies

& Incentives

• Structural Characteristics• Networks &

Communications• Culture• Implementation Climate: Tension for change Compatibility Relative Priority Organizational Incentives &

Rewards Goals & Feedback Learning Climate• Readiness for

Implementation: Leadership Engagement Available Resources Access to

Information & Knowledge

• Knowledge & Beliefs about the Intervention

• Self-efficacy• Individual Stage of

Change• Individual

Identification with Organization

• Other Personal Attributes

• Planning• Engaging: Opinion Leaders Formally

Appointed Internal

Implement- ation Leaders Champions External Change Agents• Executing• Reflecting & Evaluating

Effective Knowledge Implementation Interventions

• Implementation processes combined with high quality research knowledge

• Grimshaw et al, 2012*: What we know so farProfessional behaviour changePolicy makers and senior health service managers

* Grimshaw J, et al. Knowledge translation of research findings. Implementation Science, 2012.

Effective Knowledge Implementation Strategies

Professional Behaviour Change Strategies•Printed educational materials•Educational meetings•Educational outreach•Local Opinion Leaders•Audit and Feedback•Computerized Reminders•Tailored Interventions

Policy-makers and Senior Health Service Managers•? but many innovative approaches developed and worthy of testing

KT & CCYM Disorders: Are We Ready for Prime Time?

1. Research Synthesis: SR/MA & PG Quality?2. Dissemination: Knowledge Repositories & Tools

– Coverage & Quality?3. Implementation: Effective Strategies?

Do Systematic Reviews and Meta-analyses About CCYM Disorders Meet

International Quality Standards?

• Cochrane & PRISMA Systematic review methods• Research librarian created & conducted search• Two independent reviewers• Inclusion criteria:

• Systematic review or meta-analysis• Prevention or treatment• Anxiety, depression, suicide related behaviors• Aged ≤18 years• English language• 2000 – 2012

PRISMA Flow Diagram

Unique Records Identified & Screened

Eligible Reviews

35 14Anxiet

yDepressi

onSRB

20 16Anxiety

& Depressi

on

4194

85

Quality Assessment Methods

AMSTAR*• Assessing the Methodological Quality of Systematic

Reviews• 11 items• Minimum quality defined as:

• Systematic reviews: 5/9• Meta-analyses: 6/11

• 2 raters; Disagreements resolved by consensus

*Shea BJ, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology, 2009.

How Many Eligible Reviews Meet Minimum Quality

Standards?

Disorder (n)

% Meet Minimum Quality Standards

Anxiety (20) 40.0

Depression (35) 60.0

Anxiety & Depression (16) 37.5

Suicide Related Behaviors (14) 50.0

Total Reviews (85) 49.4

AMSTAR Criteria Failure Rate

Risk of Bias Criteria Failure Rate (%)

Research Question & Inclusion Criteria Stated 5.9

Duplicate Study Selection & Data Extraction 74.1

Comprehensive Literature Search Performed 44.7

Grey Literature & Other Languages Considered 61.2

Included & Excluded Studies Listed 56.5

Characteristics of Included Studies Provided 16.5

Study Quality Assessed & Documented 67.1

Study Quality Considered in Conclusions 62.4

Appropriate Meta-analysis Methods* 9.5

Publication Bias Assessed* 47.6

Funding Source & Conflict of Interest Reported 68.2

* Data based on reviews containing meta-analyses only (n=42)

Did Introduction of AMSTAR in 2007 Improve Review Quality?

No significant difference in AMSTAR scores 2007-2009 compared to 2010-2012; t(54)= -1.08, p = 0.286.

Does Journal Impact Factor Predict Review Quality?

Impact Factor

nMean

AMSTAR Score (sd)

≤ 3.000 29 4.64 (2.23)

3.001 – 4.000

14 4.75 (2.05)

4.001 – 6.000

16 5.75 (2.81)

6.001 – 8.000

19 7.47 (3.47)

≥ 8.001 3 6.83 (1.89)

Not Reported 4 4.00 (2.48)Correlation between AMSTAR Score & Impact Factor: r = 0.25, p<0.05

Do Practice Guideline Development Methods Meet International Quality

Standards?*

• Cochrane and PRISMA systematic review methods• Research librarian created & conducted search• Two independent reviewersStep 1: Find CYMH PGs (published or updated 2009-2014)

• Journals (4) of leading CYMH professional organizations• National Guideline Clearinghouse web-site• Websites of organizations who produce PGs: NICE, SIGN,

USPSTFStep 2: Find CYMH development methods

• Eligible PGs reviewed to identify cited or associated development methods

*Bennett K, et al. Practitioner Review: On the trustworthiness of clinical practice guidelines: A systematic review of the quality of methods used to develop guidelines in child and youth mental health. Journal of Child Psychology and Psychiatry (Forthcoming).

PRISMA Flow Diagram

PGs

PG Development Methods

PGs Identified

Unique PGs Screened

Eligible PGs

PG Development Methods Screened

Eligible PG Development Methods Sets

PG Quality Rating Methods

AGREE II*: • Appraisal of Guidelines for Research and Evaluation• Validated PG quality assessment tool• 6 domains

Scoring Options:

*Brouwers M, et al. AGREE II: Advancing guideline development, reporting and evaluation in healthcare. CMAJ, 2010.

Score

Interpretation

0 Domain/standard not mentioned

1Domain/standard optional - low confidence that the standard would be adhered to by PG developers

2Domain/standard optional - moderate confidence that the standard would be adhered to by PG developers

3Domain/standard mandatory - high confidence that the standard would be adhered to by PG developers

AGREE II Ratings

PG Development Methods

AGREE II DomainScore

Frequency

Scope & Purpose

Stakeholder

Involvement

Rigour of Develop-

ment

Clarity of

Present-ion

Applica-abiilty

Editorial Indepen-

dence0 1 2 3

NICE 3 3 3 3 3 3 0 0 0 6

SIGN 3 3 3 3 3 3 0 0 0 6

USPSTF 3 2 3 3 1 3 0 1 1 4

AAP 2 2 1 3 1 2 0 2 3 1

AACAP 2 1 1 1 0 1 1 4 1 0

AACAP = American Academy of Child and Adolescent PsychiatryAAP = American Academy of PediatricsUSPSTF = U.S. Preventive Services Task ForceNICE = National Institute for Health and Care ExcellenceSIGN = Scottish Intercollegiate Guidelines Network

0 = not mentioned 1 = optional - low confidence 2 = optional - moderate confidence 3 = mandatory - high confidence

Summary

Source

Development Methods

Meet Internation

al Standards?

Percent Of Available

PGsRepresente

d

PG Development Organizations:

31.5%

NICE & SIGN All 22.9%

USPSTF Most 8.6%

Specialty Societies: 21.4%

AACAP No 17.1%

AAP No 4.3%

Other or not eligible: ? 47.1%

68.5%

Conclusions

1. Up to 69% of PGs developed using methods that don’t align with AGREE II

2. Do professional specialty societies have the resources required to produce high quality PGs?

3. Quality of individual PGs needs to be assessed (underway)

4. Need strategies to:• Guide users to high quality PGs quickly• Facilitate adherence by PG developers to quality standards• Avoid harm and wasted resources

Dissemination: e-Resources/Technologies

Repositories and Clearinghouses:• What’s out there?• How good are they?• Do they meet user needs?• Pilot study findings

Research Repositories: ‘Best in Class’

• Rx for Change • EvidenceUpDates• HealthEvidence.org • HealthSystemsEvidence.org • Cochrane Database of Systematic Reviews

‘Best In Class’ Knowledge Repositories

Rx for Change EvidenceUpdates HealthEvidence.orgHealthSystemsEvidence

.org

Cochrane Database of Systematic

Reviews

Source University of Ottawa Jeremy Grimshaw

McMaster University Brian Haynes

McMaster UniversityMaureen Dobbins

McMaster Health ForumJohn Lavis

The Cochrane Collaboration

Purpose Behaviour change strategies for drug

prescribing and health technology use

Evidence-based medicine

Evidence-based public health decision-

making

Evidence-based health system strategies

Evidence-based medicine

Target Audience

Health Care Professionals & Policy-

makersPhysicians

Public Health Practitioners/

Managers/Policy-makers

Health System Policy-makers/Stakeholders

Health Care Decision Makers (Clinicians,

Patients, Researchers, Policy-makers, etc.)

Coverage of CCYMD† 0 61/?

2.5%(109/4,401)

80/?0.34%

(31/9,156)

Content•Original• Synopses

Original (SR)&

Synopses (SR)Original (PS, SR) Original (SR) Original (PS, SR) Original (SR)

Quality Appraised

X

Quality Appraisal Method

AMSTAREvidence Update Quality

Inclusion CriteriaOxman OQAQ tool* AMSTAR N/A

Searchable Database

Updates/Alerts Service

X

(Tailored)

(Tailored)

(Tailored)X

* OQAQ = Overview Quality Assessment Questionnaire (Oxman & Guyatt, Journal of Clinical Epidemiology, 1991)† # of hits when searching for ‘child and adolescent mental health’

CCYMD Repositories

• SAMHSA National Registry of Evidence-based Programs and Practices (NREPP)

• California Evidence-based Clearinghouse for Child Welfare (CEBC)

• Child Trends LINKS database• Canadian Best Practices Portal• What Works Clearinghouse• Ontario Centre of Excellence for CYMH Evidence

In-Sight and Policy Ready Papers• U-Mind

CCYMD Knowledge Repositories 1NREPP CEBC

What Works/LINKS database

Canadian Best Practices Portal

Source Substance Abuse & Mental Health Services

Administration (SAMHSA)

California Dept. of Social Services

Child Trends (US-based NPO)

Public Health Agency of Canada

Purpose Evidence-based Mental Health & Substance Abuse Interventions

Evidence-based Practices for Child Welfare System

Evidence-based programs relevant to

Child Health & Education

Successful Disease Prevention & Health

Promotion Interventions

Target Audience

Public

State-wide agencies, Counties, Public &

Private Organizations & Individuals

?Health Professionals & Public Health Decision-

makers

Coverage of CCYMD 5.8%(22/378)

70/? 1,138/? 14/?

Content•Original•Synopses

Synopses Synopses Synopses Synopses

Quality Appraised X X

Quality AppraisalMethod NREPP Quality of

Research Criteria

CEBC Scientific Rating Scale & CEBC

Measurement Tools Rating Scale

N/A N/A

Searchable Database

Updates/Alerts Service

(General)

(General)

(General)

X

CCYMD Knowledge Repositories 2

What Works Clearing-house

Evidence In-Sight Policy Ready Papers U-Mind

SourceInstitute of Education

Sciences, U.S. Dept. of Education

Ontario Centre of Excellence for Child & Youth Mental Health

Ontario Centre of Excellence for Child & Youth Mental Health

Kids’ Health Links Foundation &

Lutherwood Institute for Children and Youth

Mental Health

PurposeEvidence-based

EducationEvidence-based Practice

in CYMHEvidence-based Policy in

CYMH

CYMH Professional Resources & Practical

Tools (Content Shared by Members?)

Target AudienceEducators

Community Agency CYMH Professionals

Policy-makers & Decision-makers

CYMH Professionals

Coverage of CCYMD0

100%(60/60)

100%(16/16)

100%

Content•Original•Synopses

Synopses Synopses

(Rapid Reviews Upon Request)

Synopses Neither

(see Purpose)

Quality Appraised X X X

Quality AppraisalMethod

WWC Group Design Standards

N/A N/A N/A

Searchable Database X

Updates/Alerts Service

X X X

(General)

Summary

• What is good enough?• Who are the users?• What are their needs?

Effective Knowledge Implementation Interventions: 2

SRs So FarNovins et al (2013)*:

• 73 eligible articles (16 RCTS)based on 44 host studies • Inner context factors:

• Strongest evidence: fidelity monitoring & supervision• Less evidence for organizational climate/culture but better

sustainment and youth outcomes

• Outer context factors:• Strongest evidence: training and use of special technologies

Barwick et al (2012)†:• 12 intervention studies• Poor quality; self report of behavior change

*Novins DK, et al. Dissemination and implementation of evidence-based practices for child and adolescent mental health: A systematic review. JAACAP, 2013.† Barwick MA, et al. Knowledge translation efforts in child and youth mental health: A systematic review. Journal of Evidence-based Social Work, 2012.

What Are We Doing Re Implementation?

1. CIHR Dissemination Grant:• Disseminating Child and Youth Mental Health Practice

Guidelines: The Development of a User-Informed, Social Media Integrated, Mobile Website

2. Tailored PG implementation tool

Questions & Thank-you

kbennett@mcmaster.ca