Research Questions and Design Considerations: Effectiveness to Implementation

Geoffrey M. Curran, PhDDirector, Center for Implementation Research

Professor, Departments of Pharmacy Practice and PsychiatryUniversity of Arkansas for Medical Sciences

Research Health Scientist, Central Arkansas Veterans Healthcare System

Goals for the session• Discuss some Whens and Hows of getting into D&I research• Present older and newer conceptions of the “research pipeline”

– Efficacy… Effectiveness… D&I– We are moving away from “pass the baton” to something more integrated and coordinated– Designing for D&I: Thinking ahead for implementation and sustainability

• Discuss the concept of “hybrid designs” which combine elements of clinical/preventive effectiveness and implementation research– Type 1: Explore Implementabilty of an intervention while we are testing its effectiveness – Type 1: Learn what is needed to support implementation in the “real world” (towards

implementation strategies)– Type 2: Test implementation strategies during effectiveness trials (simultaneous look at both)– Type 3: Test implementation strategies while also documenting clinical/prevention intervention

outcomes (evaluating them as they relate to uptake and fidelity)• Present examples of hybrid design studies• Questions

Who am I?• Sociologist by training (1996)• Most of the last 20 years in a Department of Psychiatry

– Last 2 years also in a Department of Pharmacy Practice

• Began doing implementation research in the VA in 1998– Quality Enhancement Research Initiative (QUERI)– Implement EBPs while studying how best to implement

• NIDA, NIMH, NIDDK, NIMHD implementation research grants– Testing implementation strategies in support of adoption of EBPs

• Focus as well on methods and design in implementation research

Who are you? (Based on pre-conference survey and my own guessing…)

• People who have done mostly “clinical research” who are curious about “implementation research”

• People who would like to get their effective clinical interventions into widespread use

• People who want to support implementation of best practices in the real world right now

• People already doing implementation research who want to learn more, take it further, or get a refresher

• People who feel their science is too orderly and boring• Other?

Implementation Research

Effectiveness Research

Improved processes, outcomes

Efficacy Research

Traditional Research Pipeline (when do we do implementation research?)

• “Finish” efficacy research • “Finish” effectiveness research• Then start implementation research…

EfficacyEfficacy trial research question(s)

– Does intervention beat control under ideal conditions?– Is this idea worth pursuing any further?

Efficacy trial design considerations– Highly controlled conditions– Highly trained researchers under tight control– Highly selected sample– Favors internal validity

EffectivenessEffectiveness trial key research questions*

– Does intervention beat control under more routine conditions?– How do interventions compare to each other (and under which

circumstances is one better than the other)?– Can we afford this intervention? Which one is more cost effective?

Effectiveness trial design considerations– Seek to replicate general practice conditions– Broader and more complex sample– Favors external validity (generalizability)

*I’ll revisit these questions shortly…

D & I D&I key research questions*

– What barriers and facilitators to implementation exist?– What implementation strategies are indicated?– Does implementation intervention/strategy beat control?

D&I research design considerations/options– Observational “Diagnostic” study of implementation context – Pilot test of an implementation strategy– Implementation trial of 1 or more strategies– Favor internal or external validity depending on evidence, context…

*Focusing here for now on questions and designs associated with preparing for and then testing implementation strategies in support of specific interventions… so ignoring for now, for example, large observational studies of uptake initiated by policy, mandates, etc.)

Relay race analogy: “Here… GO! GO! GO!”

“Here… GO! GO! GO!”

Intervention “ready for” dissemination and implementation

“Here… GO! GO! GO!”

“Implementation”

“Clinical” Researcher

Researcher

Intervention “ready for” dissemination and implementation

Early VA QUERI Experience: early 2000’s

• QUERI RFA for implementation research – Mandated proof of “strong evidence base” for intervention

• Adaptation for context was OK, but not departing from “core EBPs”– No real guidance at this point on how to do this…

– No expectation to collect effectiveness outcomes• Reflected (and reified) traditional pipeline thinking• Over time, the RFA was revised to reflect newer IS thinking and

realities of VA – E.g., VA mandates for practices with “weak” evidence base and wanting

research on how best to implement them (“um, we’re gonna need a new RFA”)

What is it like now?• Most agree that efficacy, effectiveness, and implementation research shouldn’t be so

separate and sequential, and doing so overlooks complexity, leads to some wrong answers, and slows us down…– Wells (1999) and Glasgow et al (2003) recommend principles/approaches for “closing the gap”

between efficacy and effectiveness research– Green (2006) argued that most of our EBPs are not built on practice-based evidence– Curran et al (2012) propose “hybrid designs” to combine elements of effectiveness and

implementation research– Brownson et al (2013), Owen et al (2012) offer/summarize principles for “designing

[interventions] for dissemination” – Chambers et al (2013) “Dynamic Sustainability Framework” challenges assumptions of “voltage

drop” and “program drift”– Wiltsey-Stirman et al (2013) propose a framework for modifications and adaptations of EBPs– Chambers and Norton (2016) argue for a focus on science of adaptation and purposive

collection of data on outcomes associated with adaptation (embrace that we need to fit interventions to settings… a lot)

• While I cannot say that the pipeline has been radically altered as of yet, looking around this room shows us that many people are interested in these ideas and want to get in the D&I research enterprise!

What to do now?• I will focus most of the rest of the talk on describing hybrid

effectiveness-implementation designs• But, I first want to “plug” and briefly describe the related concept

of “designing for dissemination/implementation”– Could be a useful initial step into the D&I space for persons who have

spent most of their time doing effectiveness research• Ross leads a breakout session on this topic at 11:35am today!

– Bring D&I thinking and planning into what you are already doing– This extends the scope of research questions for effectiveness!– Also, consider “Re-design” for D&I if your are considering taking an EBP

out of its comfort zone…• E.g., new type of provider, different setting, different culture

Implementation Research

Effectiveness Research

Improved processes, outcomes

Design for Dissemination-Implementation

• Build for implementability as early as you can-- Or, start there in a new effort to translate a “proven” EBP

• “End-user” input and partnership before trial (PCORI anyone?)-- Better interventions and increased buy-in

• Use of implementation frameworks in design of components-- See CFIR framework, construct 1 (trialability, adaptability, complexity, cost…)

• Focus on service delivery issues– staffing, technology, dose -- What are the hard-to-reach places? Look at underserved locations

Effectiveness research questions revisited for D4D&IEffectiveness trial research questions

– Does intervention beat control under more routine conditions?– How do interventions compare to each other (and under which circumstances is

one better than the other)?– Can we afford this intervention? Which one is more cost effective?– What components can be adaptable for multiple settings?– What different staffing models can be used to deliver the intervention?– What technologies could be used to speed uptake, support implementation into

new settings?– How would the intervention need to be specified/changed to be implemented

in low resource conditions?– And more…

Implementation Research

Effectiveness Research

Improved processes, outcomes

Efficacy Research

OK, now on to hybrid designs…

Spatially speaking, hybrids “fit” in here…

Why Hybrid Designs?• The speed of moving research findings into routine adoption can

be improved by considering hybrid designs that combine elements of effectiveness and implementation research

• Don’t wait for “perfect” effectiveness data before moving to implementation research

• We can “backfill” effectiveness data while we test implementation strategies

• How do clinical/prevention outcomes relate to level of adoption and rate of fidelity?– How will we know this without data from “both sides”?

Curran et al., 2012

Some terms before we cover the types

• We use the term intervention to refer to the clinical/prevention practice we have an interest in exploring

• We use the term strategy to refer to the implementation-support activities/tools we have an interest in exploring

• Both are interventions whose effectiveness we are interested in, but in the hybrid design context it can get confusing…

• We usually use the term effectiveness only when referring to the clinical outcomes.

• We often use the term impact to describe the implementation outcomes.

ClinicalEffectiveness

ResearchImplementation

Research

Hybrid Type 1

Hybrid Type 2

Hybrid Type 3

Hybrid Type 1: test clinical/prevention intervention, observe/gather information on implementation

Hybrid Type 2: test clinical/prevention intervention, test/study implementation strategy

Hybrid Type 3: test implementation strategies, observe/gather information on clinical/prevention outcomes

Types of Hybrids

Research aims by hybrid typesStudy Characteristic Hybrid Type I Hybrid Type II Hybrid Type III

Research Aims Primary Aim:Determine effectiveness of an intervention

Secondary Aim: Better understand context for implementation

Primary Aim:Determine effectiveness of an intervention

Co-Primary* Aim:Determine feasibility and/or (potential) impact of an implementation strategy

*or “secondary”…

Primary Aim:Determine impact of an implementation strategy

Secondary Aim: Assess clinical outcomes associated with implementation

Definition:• Test clinical/prevention intervention and explore implementation-related factors

(80%/20%...)Description: • Conventional effectiveness study “plus”:

• Describe implementation experience (worked/didn’t; barriers/facilitators)• Interview/survey/observe participants regarding implementation experience• What is needed to support implementation in the real world?

Indications (circa 2012):• Clinical/prevention effectiveness evidence remains limited, so intensive focus on

implementation might be premature…BUT• Effectiveness study conditions offer ideal opportunity to explore implementation

issues, plan implementation strategies for next stage

Hybrid Type 1 Designs

Design Characteristics

• The original definition of a type 1 emphasized secondary aims/questions and exploratory data collection and analysis preparatory to implementation activity

• Current review we are doing indicates that this is the common model of type 1

• However, some type 1 studies are doing more intense focus on “implementability”– E.g., adapting intervention for better presumed uptake before the trial

and measuring “implementation potential” as a more equal/central element (if not explicitly measuring uptake and fidelity [which would be more a type 2…])

Example 1: CALM study • Curran et al., 2012, Implementation Science• Large effectiveness trial of anxiety intervention in primary care

– 4 cities, 17 clinics, 1004 patients– Care managers using software tool with patients to navigate Tx manual– Care managers were local nurses/social workers already working in the clinics– Intervention was designed with “future implementation in mind”

• Qualitative process evaluation alongside trial– 47 interviews with providers, nurses, front office, and anxiety care managers– Most interviews done on the phone– Interview guide informed by an implementation framework (PARIHS)

• (these days, that link needs to be very explicit…)

CALM study process evaluation• Interview Guide

1. What worked and what didn’t work?2. How did CALM operate in your clinic? Adaptations?3. How did CALM affect workload, burden, and space?4. How was CALM received by you and others in your site and how did

that change over time?5. Were there “champions” or “opinion leaders” for CALM and if so, what

happened with them?6. How did the communication between the care manager, the external

psychiatrist, and local PCPs work?7. What outcomes are/were you seeing?8. What changes should be made to CALM?9. What are the prospects for CALM being sustained in your clinic and

why/why not?

Example 2: Talking Health study

• Zoellner et al., 2014, Contemp Clin Trials– Patient-level RCT of intervention to reduce consumption of sugar-

sweetened beverages– RE-AIM framework guided evaluation

• Process evaluation: Reach, implementation• Health behaviors evaluation: Effectiveness, maintenance• Interviews assessed perceptions of intervention components (small

group sessions, personal action plans, drink diaries/exercise logs, teach back call, IVR calls, resources provided)

• Adoption not measured: research staff delivered intervention

Definition:• Test clinical/prevention intervention and test/study implementation strategy

(50/50? 60/40? 72/28?)Description: • Dual-focus study:

– Clinical/Prevention Effectiveness trial within either:• Implementation trial (so, a comparative effectiveness factorial type design)• Pilot (non-randomized) study of implementation strategy

Indications (circa 2012):• Clinical/prevention effectiveness data available, though perhaps not for

context/population of interest for this trial• Data on barriers and facilitators to implementation available• Implementation momentum in terms of system/policy demands?

Hybrid Type 2 Designs

Design Characteristics

• The original definition of a type 2 described possibilities of dual focused, dual randomized, factorial designs & randomized effectiveness trials nested in pilots of an implementation strategy– Majority of currently published Type 2s are the latter– Dual randomized designs used non-complex interventions/strats

• When looking at the aims or hypotheses of existing studies, most have primary focus on intervention outcomes

Design Characteristics

• Important to have an explicitly described implementation strategy that is distinct from the intervention

• Measure adoption, fidelity…• Important to be clear about intervention components versus

implementation strategy components– This isn’t always easy to decide– E.g., delivery format…

• Is delivering the intervention over the telephone an intervention component or an implementation strategy?

Example: Brief CBT with pilot impl strat

• Cully et al., 2012, Implementation Science– Clinical trial of brief cognitive behavioral therapy in treating depression

and anxiety– Patient randomization only; Pilot study of implementation strategy

(online training, audit and feedback, facilitation)– Intent-to-treat analysis of clinical outcomes– Feasibility, acceptability, and “preliminary effectiveness” data collected

on implementation strategy• Measured knowledge acquisition, fidelity to model• Qualitative data on implementability, time spent, etc.

– Measured sustainability of provision of brief CBT after trial– Preparatory to implementation trial of strategy

Example 2: HiTIDES study

• Pyne et al., 2009; Curran et al., 2011• Patient randomized trial of collaborative care for depression

in 3 HIV clinics• Pilot test of implementation strategies

– EBQI– External facilitation/coaching, audit and feedback, problem solving,

etc• FORMATIVE evaluation before, during, and after

implementation

What is EBQI?• Rubenstien et al., 2006; 2014• Partnership process to adapt intervention and specify implementation

strategy (pre-implementation)– Local clinicians suggest intervention adaptations for their context and

guide/approve implementation strategy selection– Clinical experts say go/no go on adaptations– Implementation experts recommend implementation strategies– Leaders/administrators lead and support

• Builds trust, understanding • Maintain “evidence-based factors” while locally adapting• Other similar models out there (see Powell et al., 2015; Stirman et al.,

2013)

What is Formative Evaluation (FE)?

• Build the plane while we fly it! (Stetler et al., 2006)• The first iteration of the adapted intervention and (especially)

implementation strategy likely not optimal• Adapt intervention and implementation strategy based on routine

data collection on outcomes, feasibility, satisfaction, recommendations– Implementation strategy usually gets adapted more; good time for it as

this is a pilot study of the strategy…• BEGIN with “diagnostic” needs assessment, barriers/facilitators,

preferences, etc. before attempting implementation (This bit can also be the last aim of a Type 1…!)

Type 3…?

• We’re not really going over this type today…• BUT

– These designs primarily compare implementation strategies– Randomization usually occurs at provider/clinic/system level– Main outcomes are adoption, fidelity, sustainability– Secondary outcomes are clinical outcomes (non-randomized…)– Works best with “easy” to access clinical outcomes

• Not so great for MH outcomes that usually require primary data collection

• I am happy to provide more info on these another time…

New thinking on hybrid designs• New thinking on “lack of fixed-ness” of interventions contributed to

changing views on when and why of hybrid-type designs• Hybrid type 1 as “default” for effectiveness research?

– Certainly for comparative effectiveness research• We expect dual-randomized type 2 trials to be rare

– However, stepped-wedge designs (randomization to same strategy by time) can work well with Type 2 and we are seeing these more

• Hybrid type 3 implementation trials… seek balance– When wouldn’t we want clinical/prevention outcomes data?– Balance of evidence(s), resources, time, expertise

Challenges for later…

• If we use Formative Evaluation during a study and revise the intervention and/or implementation strategy, how do we know when to measure outcomes? What about power?

• What do we do when we expect sites to need different implementation strategies to succeed?– Adaptive designs are fun…!

• When do we start to care about how much implementation strategies cost?

Resources• http://cancercontrol.cancer.gov/IS/

– Great website for introductory and advanced help– Seminars, tool, publications, grant examples– Wonderful team of consultants (David Chambers is lead)

• http://impsci.tracs.unc.edu/– “Get informed, get funded, get published…”– Lots of grant examples, trainings, tools, theory stuff

• http://www.societyforimplementationresearchcollaboration.org/– Measures resources, working groups, conference

• Implementation Science journal– Just start reading…– Searchable by key word

Thanks to:

• Brian Mittman, Mark Bauer, Sara Landes, Jeff Pyne, Cheryl Stetler, David Chambers, JoAnn Kirchner, Amy Kilbourne, Enola Proctor, Ross Brownson

• And all of you!