Joint Meeting on Adolescent Treatment EffectivenessWashington, DC – April 10, 2012

Pay-for-performance as a method to improve delivery of high-quality care: Results from the Reinforcing Therapist Performance (RTP)

experiment

Bryan R. Garner, Susan H. Godley, Michael L. Dennis, Brooke D. Hunter, Christin M. L. Bair, & Mark D. Godley

Chestnut Health SystemsBloomington-Normal, IL

R01AA017625 (PI: Garner)

Contract #270-07-0191

Opinions are those of the authors and not official positions of the government

Learning Objectives

To learn more about…

1. Methods to improve the implementation of evidence-based practices

2. The effectiveness of pay-for-performance methods

3. The relationship between performance measures and client outcomes

Group of treatment organizations that are: Receiving the same level of funding. Implementing the same evidence-based

treatment. Receiving the same comprehensive training

model. Documenting treatment implementation using

the same web-based program. Documenting client outcomes using the same

evidence-based assessment instrument.

AAFT

AAFT as the foundation of the Reinforcing Therapist Performance (RTP) Experiment

AAFT

A-CRA Clinician Training and Certification Process

Feedback Monitor Clinician Fidelity

RecordTherapySessions

Achieve Basic A-CRA Certification

In 9 procedures

Treatment Manual

and Training

Workshop

Participant in Coaching Calls with Supervisors/Coaches

Adapted from Godley, Garner, Smith, Meyers, & Godley (2011)

AAFT

A-CRA Training Workshop

A-CRA Treatment Manual

+

A-CRA Clinician Training and Certification Process

AAFT

Feedback Monitor Clinician Fidelity

RecordTherapySessions

Achieve Basic A-CRA Certification

In 9 procedures

Treatment Manual

and Training

Workshop

Participant in Coaching Calls with Supervisors/Coaches

A-CRA Clinician Training and Certification Process

AAFT

Digital Session Recordings

(DSRs)

A-CRA Clinician Training and Certification Process

Upload DSR to EBTx.org website

AAFT

Feedback Monitor Clinician Fidelity

RecordTherapySessions

Achieve Basic A-CRA Certification

In 9 procedures

Treatment Manual

and Training

Workshop

Participant in Coaching Calls with Supervisors/Coaches

A-CRA Clinician Training and Certification Process

AAT 1:First Year

AAFT 1:Second Year

AAFT 1:Third Year

15 sites funded in October

2006

AAFT 2:First Year

AAFT 2:Second Year

AAFT 2:Third Year

17 sites funded in October

2007

Improving the ImplementationProcess within AAFT

How might we improve the process?

Recommended Pay-for-Performance (P4P) as a promising method to improve the delivery of high quality of care.

“The literature evaluating the effectiveness of pay for performance consists of fewer than 20 studies, yielding mixed conclusions on overall impact.”

Rewarding Provider Performance: Aligning Incentives in Medicare (IOM, 2007)

AAFT 1:First Year

AAFT 1:Second Year

AAFT 1:Third Year

15 sites funded in October

2006

AAFT 2:First Year

AAFT 2:Second Year

AAFT 2:Third Year

17 sites funded in October

2007

RTP study funded in October 2008

How might we improve the process?

Improving the ImplementationProcess within AAFT

AAFT

RTP

Aims: Examine the effectiveness and cost-effectiveness of providing monetary incentives to therapists as a method to improve treatment service delivery and subsequent treatment outcomes.

Design: Randomly assigned AAFT grantees and their therapists to either: 1) Implementation-As-Usual (IAU) or 2) Pay-For-Performance (P4P)

$50 for each month that a randomly selected digital session recording (DSR) has at least one A-CRA procedure rated at or above the minimum level of competence required for A-CRA certification.

$200 for each adolescent who they deliver an empirically derived level of A-CRA treatment that has been shown to be significantly related improved treatment outcomes (Garner et al., 2009, 2010).

Targets: Monthly A-CRA Competence Target A-CRA

Aims and Design of RTP Experiment

TargetA-CRA

6-month Remission

Status

+

+

Tre

atm

ent I

mpl

emen

tatio

n P

erfo

rman

ce M

easu

res

TreatmentEffectiveness

A-CRACompetence

RTP hypotheses to be tested

+

Assignment to ExperimentalPay-For-Performance

Intervention

Sample size & Data structure

Therapist 104

Site 29Level 3:

29 Sites

Level 2: 105

Therapists

Level 1: 986 Clients

…

Therapist 105

1 2 3 4 5 6 7 8 9

Therapist 1

Site 1

Therapist 3Therapist 2 …

… 986985983982 984

Therapist Characteristics (N=105)

a p < .05; b See Donnellan, Oswald, Baird, & Lucas (2006) for more information about Mini-International Personality Item Pool (Mini-IPIP)

% or Mean (SD)

IAU(n = 52)

P4P(n = 53)

Therapists (N=105)

Female 67% 81%

White 58% 51%

Age 37.2 (11.7) 36.0 (10.7)

Master’s degree or higher 60% 49%

Months experience 60.3 (80.2) 35.9 (37.5)a

In Recovery 6% 6%

Mini-IPIP personality factorsb

Extraversion 3.6 (0.73) 3.5 (0.77)

Agreeableness 4.4 (0.47) 4.4 (0.50)

Conscientiousness 3.9 (0.73) 4.2 (0.59)a

Neuroticism 2.5 (0.63) 2.5 (0.69)

Intellect/Imagination 4.0 (0.67) 3.9 (0.72)

Client Characteristics (N=986)

% or Mean (SD)

IAU(n = 534)

P4P(n = 452)

Clients (N=986)

Female 22% 27%a

White 36% 28%a

Age 15.8 (1.34) 15.9 (1.47)Criminal Justice Involved 64% 68%Prior substance use treatment 37% 28%a

Percent-of-days-abstinent .58 (.37) .66 (.34)a

a p < .05;

AAFT

RTP

Aims: Examine the effectiveness and cost-effectiveness of providing monetary incentives to therapists as a method to improve treatment service delivery and subsequent treatment outcomes.

Design: Randomly assigned AAFT grantees and their therapists to either: 1) Implementation-As-Usual (IAU) or 2) Pay-For-Performance (P4P)

$50 for each month that a randomly selected digital session recording (DSR) has at least one A-CRA procedure rated at or above the minimum level of competence required for A-CRA certification.

$200 for each adolescent who they deliver an empirically derived level of A-CRA treatment that has been shown to be significantly related improved treatment outcomes (Garner et al., 2009, 2010).

Targets: Monthly A-CRA Competence Target A-CRA

Results

Results: A-CRA Competence

0%

5%

10%

15%

20%

25%

IAU P4P

9.2%

23.7%

Odds Ratio = 2.2p = .02

AAFT

RTP

Aims: Examine the effectiveness and cost-effectiveness of providing monetary incentives to therapists as a method to improve treatment service delivery and subsequent treatment outcomes.

Design: Randomly assigned AAFT grantees and their therapists to either: 1) Implementation-As-Usual (IAU) or 2) Pay-For-Performance (P4P)

$50 for each month that a randomly selected digital session recording (DSR) has at least one A-CRA procedure rated at or above the minimum level of competence required for A-CRA certification.

$200 for each adolescent who they deliver an empirically derived level of A-CRA treatment that has been shown to be significantly related improved treatment outcomes (Garner et al., 2009, 2010).

Targets: Monthly A-CRA Competence Target A-CRA

Results

Results: Target A-CRA

0%

5%

10%

15%

20%

IAU P4P

3.3%

15.9%

Odds Ratio = 5.2p = .01

AAFT

RTP

Aims: Examine the effectiveness and cost-effectiveness of providing monetary incentives to therapists as a method to improve treatment service delivery and subsequent treatment outcomes.

Design: Randomly assigned AAFT grantees and their therapists to either: 1) Implementation-As-Usual (IAU) or 2) Pay-For-Performance (P4P)

$50 for each month that a randomly selected digital session recording (DSR) has at least one A-CRA procedure rated at or above the minimum level of competence required for A-CRA certification.

$200 for each adolescent who they deliver an empirically derived level of A-CRA treatment that has been shown to be significantly related improved treatment outcomes (Garner et al., 2009, 2010).

Targets: Monthly A-CRA Competence Target A-CRA

Results

Results: Remission Status

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

IAU P4P

50.8%

41.8%

No Significant Difference

Average Remission Status was 24% in

Cannabis Youth Treatment (CYT) study

(Dennis et al., 2004)

Results: Remission Status

?

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

IAU P4P

65%56%

Poor Follow-up Rates

Remission Status

Unknownfor 44%

Remission Status

Unknownfor 35%

Differential Client Attrition!

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

IAU P4P

65%56%

Poor Follow-up Rates

Remission Status

Unknownfor 44%

Remission Status

Unknownfor 35%

No SignificantDifferences

Significant Differences

Follow-up group was

significantly more severe at baseline

TargetA-CRA

6-month Remission

Status Tre

atm

ent I

mpl

emen

tatio

n P

erfo

rman

ce M

easu

res

TreatmentEffectiveness+

A-CRACompetence

+

Assignment to ExperimentalPay-For-Performance

Intervention

Not Supported

Supported

Post hoc analyses

Take home messages

1. Modest-sized incentives can have large impacts on improving treatment implementation

2. Despite the large impact of P4P on the two treatment implementation measures, there remained considerable room for improvement even in the P4P condition.

Next Step

Examine the extent to which the improvements in A-CRA Competence and Target A-CRA were Cost-Effective.

Thank You.