Joint Meeting on Adolescent Treatment Effectiveness Washington, DC – April 10, 2012...
-
Upload
damien-robart -
Category
Documents
-
view
214 -
download
0
Transcript of Joint Meeting on Adolescent Treatment Effectiveness Washington, DC – April 10, 2012...
Joint Meeting on Adolescent Treatment EffectivenessWashington, DC – April 10, 2012
Pay-for-performance as a method to improve delivery of high-quality care: Results from the Reinforcing Therapist Performance (RTP)
experiment
Bryan R. Garner, Susan H. Godley, Michael L. Dennis, Brooke D. Hunter, Christin M. L. Bair, & Mark D. Godley
Chestnut Health SystemsBloomington-Normal, IL
R01AA017625 (PI: Garner)
Contract #270-07-0191
Opinions are those of the authors and not official positions of the government
Learning Objectives
To learn more about…
1. Methods to improve the implementation of evidence-based practices
2. The effectiveness of pay-for-performance methods
3. The relationship between performance measures and client outcomes
Group of treatment organizations that are: Receiving the same level of funding. Implementing the same evidence-based
treatment. Receiving the same comprehensive training
model. Documenting treatment implementation using
the same web-based program. Documenting client outcomes using the same
evidence-based assessment instrument.
AAFT
AAFT as the foundation of the Reinforcing Therapist Performance (RTP) Experiment
AAFT
A-CRA Clinician Training and Certification Process
Feedback Monitor Clinician Fidelity
RecordTherapySessions
Achieve Basic A-CRA Certification
In 9 procedures
Treatment Manual
and Training
Workshop
Participant in Coaching Calls with Supervisors/Coaches
Adapted from Godley, Garner, Smith, Meyers, & Godley (2011)
AAFT
A-CRA Training Workshop
A-CRA Treatment Manual
+
A-CRA Clinician Training and Certification Process
AAFT
Feedback Monitor Clinician Fidelity
RecordTherapySessions
Achieve Basic A-CRA Certification
In 9 procedures
Treatment Manual
and Training
Workshop
Participant in Coaching Calls with Supervisors/Coaches
A-CRA Clinician Training and Certification Process
AAFT
Digital Session Recordings
(DSRs)
A-CRA Clinician Training and Certification Process
Upload DSR to EBTx.org website
AAFT
Feedback Monitor Clinician Fidelity
RecordTherapySessions
Achieve Basic A-CRA Certification
In 9 procedures
Treatment Manual
and Training
Workshop
Participant in Coaching Calls with Supervisors/Coaches
A-CRA Clinician Training and Certification Process
AAT 1:First Year
AAFT 1:Second Year
AAFT 1:Third Year
15 sites funded in October
2006
AAFT 2:First Year
AAFT 2:Second Year
AAFT 2:Third Year
17 sites funded in October
2007
Improving the ImplementationProcess within AAFT
How might we improve the process?
Recommended Pay-for-Performance (P4P) as a promising method to improve the delivery of high quality of care.
“The literature evaluating the effectiveness of pay for performance consists of fewer than 20 studies, yielding mixed conclusions on overall impact.”
Rewarding Provider Performance: Aligning Incentives in Medicare (IOM, 2007)
AAFT 1:First Year
AAFT 1:Second Year
AAFT 1:Third Year
15 sites funded in October
2006
AAFT 2:First Year
AAFT 2:Second Year
AAFT 2:Third Year
17 sites funded in October
2007
RTP study funded in October 2008
How might we improve the process?
Improving the ImplementationProcess within AAFT
AAFT
RTP
Aims: Examine the effectiveness and cost-effectiveness of providing monetary incentives to therapists as a method to improve treatment service delivery and subsequent treatment outcomes.
Design: Randomly assigned AAFT grantees and their therapists to either: 1) Implementation-As-Usual (IAU) or 2) Pay-For-Performance (P4P)
$50 for each month that a randomly selected digital session recording (DSR) has at least one A-CRA procedure rated at or above the minimum level of competence required for A-CRA certification.
$200 for each adolescent who they deliver an empirically derived level of A-CRA treatment that has been shown to be significantly related improved treatment outcomes (Garner et al., 2009, 2010).
Targets: Monthly A-CRA Competence Target A-CRA
Aims and Design of RTP Experiment
TargetA-CRA
6-month Remission
Status
+
+
Tre
atm
ent I
mpl
emen
tatio
n P
erfo
rman
ce M
easu
res
TreatmentEffectiveness
A-CRACompetence
RTP hypotheses to be tested
+
Assignment to ExperimentalPay-For-Performance
Intervention
Sample size & Data structure
Therapist 104
Site 29Level 3:
29 Sites
Level 2: 105
Therapists
Level 1: 986 Clients
…
Therapist 105
1 2 3 4 5 6 7 8 9
Therapist 1
Site 1
Therapist 3Therapist 2 …
… 986985983982 984
Therapist Characteristics (N=105)
a p < .05; b See Donnellan, Oswald, Baird, & Lucas (2006) for more information about Mini-International Personality Item Pool (Mini-IPIP)
% or Mean (SD)
IAU(n = 52)
P4P(n = 53)
Therapists (N=105)
Female 67% 81%
White 58% 51%
Age 37.2 (11.7) 36.0 (10.7)
Master’s degree or higher 60% 49%
Months experience 60.3 (80.2) 35.9 (37.5)a
In Recovery 6% 6%
Mini-IPIP personality factorsb
Extraversion 3.6 (0.73) 3.5 (0.77)
Agreeableness 4.4 (0.47) 4.4 (0.50)
Conscientiousness 3.9 (0.73) 4.2 (0.59)a
Neuroticism 2.5 (0.63) 2.5 (0.69)
Intellect/Imagination 4.0 (0.67) 3.9 (0.72)
Client Characteristics (N=986)
% or Mean (SD)
IAU(n = 534)
P4P(n = 452)
Clients (N=986)
Female 22% 27%a
White 36% 28%a
Age 15.8 (1.34) 15.9 (1.47)Criminal Justice Involved 64% 68%Prior substance use treatment 37% 28%a
Percent-of-days-abstinent .58 (.37) .66 (.34)a
a p < .05;
AAFT
RTP
Aims: Examine the effectiveness and cost-effectiveness of providing monetary incentives to therapists as a method to improve treatment service delivery and subsequent treatment outcomes.
Design: Randomly assigned AAFT grantees and their therapists to either: 1) Implementation-As-Usual (IAU) or 2) Pay-For-Performance (P4P)
$50 for each month that a randomly selected digital session recording (DSR) has at least one A-CRA procedure rated at or above the minimum level of competence required for A-CRA certification.
$200 for each adolescent who they deliver an empirically derived level of A-CRA treatment that has been shown to be significantly related improved treatment outcomes (Garner et al., 2009, 2010).
Targets: Monthly A-CRA Competence Target A-CRA
Results
Results: A-CRA Competence
0%
5%
10%
15%
20%
25%
IAU P4P
9.2%
23.7%
Odds Ratio = 2.2p = .02
AAFT
RTP
Aims: Examine the effectiveness and cost-effectiveness of providing monetary incentives to therapists as a method to improve treatment service delivery and subsequent treatment outcomes.
Design: Randomly assigned AAFT grantees and their therapists to either: 1) Implementation-As-Usual (IAU) or 2) Pay-For-Performance (P4P)
$50 for each month that a randomly selected digital session recording (DSR) has at least one A-CRA procedure rated at or above the minimum level of competence required for A-CRA certification.
$200 for each adolescent who they deliver an empirically derived level of A-CRA treatment that has been shown to be significantly related improved treatment outcomes (Garner et al., 2009, 2010).
Targets: Monthly A-CRA Competence Target A-CRA
Results
Results: Target A-CRA
0%
5%
10%
15%
20%
IAU P4P
3.3%
15.9%
Odds Ratio = 5.2p = .01
AAFT
RTP
Aims: Examine the effectiveness and cost-effectiveness of providing monetary incentives to therapists as a method to improve treatment service delivery and subsequent treatment outcomes.
Design: Randomly assigned AAFT grantees and their therapists to either: 1) Implementation-As-Usual (IAU) or 2) Pay-For-Performance (P4P)
$50 for each month that a randomly selected digital session recording (DSR) has at least one A-CRA procedure rated at or above the minimum level of competence required for A-CRA certification.
$200 for each adolescent who they deliver an empirically derived level of A-CRA treatment that has been shown to be significantly related improved treatment outcomes (Garner et al., 2009, 2010).
Targets: Monthly A-CRA Competence Target A-CRA
Results
Results: Remission Status
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
IAU P4P
50.8%
41.8%
No Significant Difference
Average Remission Status was 24% in
Cannabis Youth Treatment (CYT) study
(Dennis et al., 2004)
Results: Remission Status
?
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
IAU P4P
65%56%
Poor Follow-up Rates
Remission Status
Unknownfor 44%
Remission Status
Unknownfor 35%
Differential Client Attrition!
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
IAU P4P
65%56%
Poor Follow-up Rates
Remission Status
Unknownfor 44%
Remission Status
Unknownfor 35%
No SignificantDifferences
Significant Differences
Follow-up group was
significantly more severe at baseline
TargetA-CRA
6-month Remission
Status Tre
atm
ent I
mpl
emen
tatio
n P
erfo
rman
ce M
easu
res
TreatmentEffectiveness+
A-CRACompetence
+
Assignment to ExperimentalPay-For-Performance
Intervention
Not Supported
Supported
Post hoc analyses
Take home messages
1. Modest-sized incentives can have large impacts on improving treatment implementation
2. Despite the large impact of P4P on the two treatment implementation measures, there remained considerable room for improvement even in the P4P condition.
Next Step
Examine the extent to which the improvements in A-CRA Competence and Target A-CRA were Cost-Effective.
Thank You.