Models, Methods, & Measures:Examining the adoption, implementation

and sustained use of innovations

in the Ohio Mental Health System

Phyllis C. Panzano, Ph.D.

University of South Florida

Decision Support Services, Inc.

September 20-21, 2010

Improving Implementation Research Methods for Behavioral and Social Science

IDARP*

CLIFFNotes

* Innovation Diffusion and Adoption Research Project, Panzano & Roth* Innovation Diffusion and Adoption Research Project, Panzano & Roth

ODMH; MacArthur Network on Mental Health PolicyODMH; MacArthur Network on Mental Health Policy

For More Details:• ODMH Publications:

– New Research in Mental Health Volume 15 - 17, – IDARP Bulletins 1-7

• Journal Articles: – Panzano, P.C., and Roth, D. (2006). The decision to adopt evidence-

based and other innovative mental health practices: Risky Business? Psychiatric Services. Vol. 57, pp. 1153 – 1161.

– Seffrin, B, Panzano, PC, & Roth, D (2009). What gets noticed: How barrier and facilitator perceptions relate to the adoption and implementation of innovative mental health practices, Community Mental Health Journal., On-line version currently available through Springer Science and Business Media, LLC 2008.

– Massatti, RM, Sweeney, HA, Panzano, PC, and Roth, D. (2008). The de-adoption of innovative mental health practices (IMHP): Why organizations choose not to sustain an IMHP; Administration and Policy in Mental Health and Mental Health Services Research, 35:50 – 65.

– Carstens, C, Panzano, PC, Massatti, RM, Sweeney, HA, and Roth, D. (2007). A naturalistic study of MST dissemination in thirteen Ohio Communities; Journal of Behavioral Health Services and Research.

• Dissertation:– Vaidyanathan, V. (2004). Looking beyond the adoption decision in

innovation research: Investigating innovation implementation. The Ohio State University, Columbus, OH.

IDARP Catalyst: Funding of CCOEs to facilitate adoption & assimilation of effective & salient practices

ODMH Research ContextE

vid

en

ce B

ase

Political Salience

Advance Directives MH/Schools

MH/Criminal Justice OMAPOMAP

Family Psycho-education Cluster-based PlanningCluster-based Planning

MSTMSTIDDT

History: Policy (S & R), Strategic Initiative (Hospital Closing), Law (MH Act of 1988) Implementation & Impact Studies

What factors and processes influence the adoption and assimilation of evidence-based (EBPs) and promising practices by behavioral healthcare organizations*?

ODMH Research Question

2001 2002 2003 2004 2005 2006 2007 2008 2009

T1: Adoption

T2: ImplementationT2: Implementation

T3: Implementation

T4: Implementation

ODMH funds 8 CCOEsODMH funds 8 CCOEs

n = 85

n = 50

n = 38

n = 34

* User-based model

Core Research Streams: Innovation development, diffusion, adoption,

implementation

(e.g., Damanpour; Fixsen et al; Frambach & Schillewaert; Greenhalgh; Hickson et al; Real & Poole; Rogers; Van de Ven; Yin)

Strategic decision making; decision making under risk

(e.g., Dutton & Jackson; March & Shapira; Panzano & Billings; Sitkin & Weingart; Staw et al, Tversky & Kahneman)

Health care, innovation & public sector planning

(e.g, Meyer & Goes; Nutt; Yin)

Prominent Paradigm Upper Echelon Theory (Hambrick & Mason)

Extensive Relevant Literature

Design, Methods, Measures

1. Design: Longitudinal (up to 4 rounds), primarily concurrent and prospective; observational field study

2. Focal entities: 85 Innovation decision processes & 50 implementation efforts involving 4 possible practices

3. Practices:

a. All 8 CCOEs volunteered; resources ltd study to 4

b. Selection: structured decision process involving OSU faculty, ODMH Director’s and Medical Director’s offices, IDARP researchers

c. Variability on key innovation-level variables (e.g., evidence, complexity, cost); salience also important

Design, Methods, Measures

4. Selected 2 team-delivered EBPs (IDDT and MST) as primary; 2 individually-delivered PPs (CBP and OMAP) as secondary

5. Recruitment of Sites for IDARP

6. Methods: interviews (structured; process reconstruction), surveys (organization & CCOE), archival data

7. Key informants: Top decision-makers (CEO, CCO, CFO), implementation managers, primary CCOE liaisons

IDARP: Guiding Models and Key Sources for Measures

1. Adoption Decision: 1 Model Card Shark Model DVs: Adopt/not Adopt; Decision Stage (e.g., Yin)

2. Implementation: 4 Models Launch; Russian Doll, Dilbert; and Glove Models DVs:

Continued Use vs De-adoption Implementation Effectiveness (e.g., fidelity,

commitment) Innovation Effectiveness (e.g., satisfaction, outcomes) Decision stage: extent assimilated; plans to persist

Model 1: The Adoption Decision

The Card Shark Model

Decision making under risk (e.g., Prospect theory)

Strategic Issue Diagnosis

Climate and Leadership for Innovation

The decision to adopt depends on calculated

risk; the size of your chip stack does matter!

Under Risk A Decision

• IMPLEMENTATION

UNDERWAY

• JUST DECIDED TO

ADOPT

• STILL CONSIDERING

• NEVER WILL ADOPT

Perceived

Risk of

Adopting

Capacity to

Manage or

Absorb Risk

Risk-taking

Propensity

-.50

.40

.28

ANTECEDENTS

DECISION STAGE

Panzano & Roth (2006) Psychiatric Services

Adoption as Decision Under Risk: Some Key Sources for Measures

1. Survey Scales

Risk & Antecedents: Sitkin and Pablo; Sitkin and Weingart; Panzano & Billings; Bourgeois; Khandwalla

Expectancies: Dutton & Jackson; Thomas & McDaniel Innovation Attributes: Moore & Benbaset; Mathieson and Davies;

Venkatesh & Davis; Rogers; Tornatsky & Klein Climate & Culture: Amabile; Bass; Jung et al, Klein &

Sorra: Makri et al; Marsick & Watson; Siegel Attitudes: Aarons; Chatman & O’Reilly; Dunham

2. Interview Questions Decision Stage: Nutt, Meyer & Goes; Yin

Model 2: Implementation Success

The Launch Model

Initial conditions … prior to and at takeoff… have

important impacts on the course of events.

Organizational change

Implementation strategy

Planning Process frameworks

INITIATION Decision IMPLEMENTATION SuccessSuccess

TimeTimeTimeTime

Model 2: Factors from earlier stages impact success

Time 1Time 1 Time 1Time 1 Time 2Time 2

Initiation-Phase Effects

Expected Benefits +++

Relative Advantage +++

Results Demonstrability +++

Trust in CCOE (purveyor) +++

SUCCESS *

* Assimilation scale; Global positive outcome scale

Decision-Phase Effects

Objective decision +++

Information access +++

Internal influence +++

Commitment +++

SUC C ESS

* Assimilation scale; Global positive outcome scale

*

Model 3: Implementation Success

The Russian Doll ModelSurrounding

conditions and circumstances

influence implementation

success.

The Meso Paradigm

Levels Issues in Organizational Research

Social Ecology Theory

Level 4: Inter-organizational

Level 3: Adopting organization

Level 2: Project level

Level 1: Innovation level

Dependent Variables:

• Implementation effectiveness

• Innovation effectiveness

Level 5: Environment

IOR – Quality of communication (R2 = .13)

Org – Learning culture (R2 = .23)

Project – Leadership Commitment (R2 = .38)

EBP

Time 2

Positive outcomes R2 = .38

Environment Time 1

Model 4: Implementation Success

The Dilbert Model

Projects can rise and fall depending on how

soundly they’re managed.

Climate for implementation

Project management

Climate for Implementation:

Top management support Goal Clarity Dedicated resources Access to training & TA Rewards/recognition for implementing Removal of obstacles Performance monitoring Freedom to express doubts

Holahan et al; Klein, Conn and Sorra; Vaidyanathan, 2004

CLIMATE AND SUCCESS

IMPLEMENTATION EFFECTIVENESS

Climate for Implementation

INNOVATION EFFECTIVENESS

.75

.45

Vaidyanathan, 2004

Time 1 Time1

CLIMATE AND SUCCESS

IMPLEMENTATION EFFECTIVENESS

Climate for Implementation

INNOVATION EFFECTIVENESS

+++

+++

NS

Time 1 Time 2

Panzano et al, 2006

Model 5: Sustained Use

If the Glove Still Fits, Keep-wearing-it-model

External and Internal Developments

Influence Goodness-of-Fit.

Strategic Issue Diagnosis & Management

Project Management

Sustained Use Model

Fit Climate

Success

Panzano and Roth, 2007

Such as…Such as…

CompatibiliCompatibilityty

CapacityCapacity

Perceived Perceived RiskRisk

Use History

Developments

Continued Use Continued Use vs De-vs De-

adoptionadoption

Degree Degree AssimilatedAssimilated

ImplementatiImplementation on

EffectivenessEffectiveness

Innovation Innovation EffectivenessEffectiveness

Top 2 Reasons for De-adopting1

Financial resources Community & network issues Staffing Tx Practice Compatibility Effectiveness Purveyor (CCOE) Barriers Technology integration

problems

1 12 matched pairs of sustainers vs deadopter sites from organizational surveys and interviews; Massati, Sweeney, Panzano and Roth, 2008

Fit and climate measures differentiate sustainers from de-

adopters Support from external organizations to

continue Degree of ongoing support from top

management & organization as a whole Compatibility of practice with org values Positive attitudes about practice among staff Capacity: Know – how and skill at implementing Access to TA during implementation Current & projected resource availability

Sustainer Model: Sustainer Model: CCOE-based CCOE-based

Tentative Revision*Tentative Revision*

Fit

Climate

ImplementationEffectiveness

(e.g. fidelity)

InnovationEffectiveness(e.g., outcomes)

Sustain/Assimilate

T3 and T4 CCOE Surveys; n = 34 projects still underway at T4;

Panzano & Knudsen et al, 2010

Implementation Models: Some Key Sources of Measures

Interview Protocols and Structured Questions

Yin, 1979

Hickson et al, 1986

Nutt, 2004

Van de Ven et al’s survey from the Minnesota Innovation Studies (2000)

Some Key Sources of Measures

1. Survey Scales (organization and CCOE) Innovation Attributes: see citations for Adoption

Model

Attitudes: see citations for Adoption Model plus Dooley et al; Shore et al

Inter-organizational relationships: Oliver; Ring and Van de Ven, Granner and Phillips; El Ansari

Organizational structure, size, resources: Hall; Kimberly; Sutcliffe

Environmental uncertainty: Sutcliffe; Milliken

Politicality: Thomas, Shankster, Mathieu; Dean and Sharfman

Some Key Sources of Measures

1. Surveys (cont’d) Leadership: Bass; Makri et al and Championship:

Howell and Higgins

Culture and/or climate: Glisson; Fixsen et al; Holahan et al; Klein and Sorra

Fidelity and reinvention: Dusenbury; Rice and Rogers; Van de Ven at al; practice-specific measures

Implementation outcomes: Hickson; Linton; Nutt; Real and Poole, Van de Ven; Yin;

2. Archival Measures ODMH databases (e.g., Medicaid)

Agency association database

Concluding Thoughts

Models Value Messages

Design, Methods, MeasuresStrengthsWeaknessesAlternatives