BIOE 301 Lecture Twenty: Clinical Trials Mike Cordray [email protected].
Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt...
-
date post
20-Dec-2015 -
Category
Documents
-
view
215 -
download
0
Transcript of Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt...
![Page 1: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/1.jpg)
Analyzing Intervention Fidelity and Achieved Relative Strength
David S. CordrayVanderbilt University
NCER/IES RCT Training Institute,2010
![Page 2: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/2.jpg)
Perspectives on Interventionion Fidelity: When is Analysis Needed?
• Monitoring implementation, retraining, maximize fidelity to the intervention model.
• Program-specific models (within Tx, only)• Interpreting results from ITT analyses;
– Cause-Effect congruity;– Lack of Cause-Effect congruity.
• Analysis of variation in delivery/receipt of intervention components.
• Compliance-based analyses (TOT, LATE)• Mediation – instrumental variables approach
![Page 3: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/3.jpg)
Program Specific Models
• Analyses conducted within the intervention, only
• RAOutputs Outcomes
• No variance, no covariance
• Component-wise assessment of fidelity
• Configural judgment about implementation fidelity
![Page 4: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/4.jpg)
Achieved Relative Strength =.15
Infidelity
“Infidelity” tC
t tx
cY
tY
TTX’ TTx
TC
.45
.40
.35
.30
.25
.20
.15
.10
.05
.00
Treatment Strength
100
90
85
80
75
70
65
60
55
50
Outcome
TY
CY
Intervention Exposure
True Fidelity
Tx ContaminationAugmentation of C
Intervention Exposure
Positive Infidelity
'TY
Intervention Differentiation
Review: Concepts and Definitions
Variability
Variability
![Page 5: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/5.jpg)
Analysis Type I
Congruity of Cause-Effect
![Page 6: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/6.jpg)
Common Cause-Effect Scenarios
The Cause The Effect
Low High
Low Low/Low = Cause-Effect
Congruity
Low/High = ????
High High/Low =DampeningProcess ????
High/High = Cause-Effect
Congruity
![Page 7: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/7.jpg)
Cause-Effect Congruity: High/High Example
• Fantuzzo, King & Heller (1992) studied the effects of reciprocal peer tutoring on mathematics and school adjustment. – 2 X 2 factorial design crossing levels of structured
peer tutoring and group reward– 45 min. 2-3 per week; 60-90 sessions
• Fidelity assessments: – Observations (via checklist) of students and staff,
rated the adherence of group members to scripted features of each condition;
• 50% random checks of sessions– Mid-year, knowledge tests to index the level of
understanding of students about the intervention components in each of the four conditions.
![Page 8: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/8.jpg)
Fantuzzo et al. Continued
• Fidelity results:– Adherence (via observations):
• 90-100% across conditions, • 95% overall
– Student understanding (via 15 item test):• 82% SD=11% (range 47-100%); ANOVA=ns• Reward+structure condition: 84% Control: 86%
• Effects on mathematics computation: ES= (7.7-5.0)/1.71 = 1.58
• Congruity=High/High; no additional analyses needed
![Page 9: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/9.jpg)
Exposure and Achieved Relative Strength
• Fantuzzo et al. example is:– Relatively rare;– Incorporates intervention differentiation, yielding
fidelity indices for all conditions.• More commonly, intervention exposure is
assessed:– Yielding scales of the degree to which individuals
experience the intervention components in both conditions
– The achieved relative strength index is used for establishing the differences between conditions on causal components
![Page 10: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/10.jpg)
Average ARS Index
Where,
= mean for group 1 (tTx )
= mean for group 2 (tC)
ST = pooled within groups standard deviation
nTx = treatment sample size
nC = control sample size
n = average cluster size
p = Intra-class correlation (ICC)
N = total sample size
1 2 3 2( 1)( ) (1 ) 1
4( ) 9 2T Tx C
X X n pg ARS
S n n N
1X
2X
Group Difference Sample Size Adjustment
Clustering Adjustment
![Page 11: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/11.jpg)
A Partial Example of the Meaning of ARSI
Randomized Group Assignment
Professional Development
Differentiated Instruction
Improved Student Outcomes
![Page 12: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/12.jpg)
70
60
50
40 0 1
30 0 0 0 1 2 2 2 4 7 8 8 8 9
20 0 1 4 4 4 5 6 6 6 6 7 7 7 7 7 7 9
10 2 3 8 9
00
70 1 1 1
60 0 0 0 1 1 1 1 2 5 6 7 7 7 78 9 9 9 9
50 4 5 6 7 7 7 7 7 8 8 8 9
40 7 9
30
20
10
00
70
60
50
40 0 1
30 0 0 0 1 2 2 2 4 7 8 8 8 9
20 0 1 4 4 4 5 6 6 6 6 7 7 7 7 7 7 9
10 2 3 8 9
00
70 1 1 1
60 0 0 0 1 1 1 1 2 5 6 7 7 7 78 9 9 9 9
50 4 5 6 7 7 7 7 7 8 8 8 9
40 7 9
30
20
10
00
Hours of Professional Development
Hours of Professional Development
Mean= 28.2
SD=7.04
Mean=61.8
SD=6.14
Control Intervention
ARSI:
= (61.8-28.2)/6.61
=5.08
U3= 99%
Very Large Group Difference, Limited Overlap Between Conditions
![Page 13: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/13.jpg)
Cohen’s U3 Index: Very Large Group Separation
Control Mean
50th percentile
Intervention Mean
U3=99th Percentile
ARSI=5.08
![Page 14: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/14.jpg)
70
60
50
40 0 1
30 0 0 0 1 2 2 2 4 7 8 8 8 9
20 0 1 4 4 4 5 6 6 6 6 7 7 7 7 7 7 9
10 2 3 8 9
00
70
60
50
40 0 0 0
30 0 0 0 0 0 1 4 5 6 6 6 6 7 8 8 8 8
20 3 4 5 6 6 6 6 7 7 7 8 9 9 9
10 6 8
00
70
60
50
40 0 1
30 0 0 0 1 2 2 2 4 7 8 8 8 9
20 0 1 4 4 4 5 6 6 6 6 7 7 7 7 7 7 9
10 2 3 8 9
00
70
60
50
40 0 0 0
30 0 0 0 0 0 1 4 5 6 6 6 6 7 8 8 8 8
20 3 4 5 6 6 6 6 7 7 7 8 9 9 9
10 6 8
00
Hours of Professional Development
Hours of Professional Development Mean= 28.2
SD=7.04
Mean=30.8
SD=6.14
Control Intervention
ARSI:
= (30.8-28.2)/6.61
=0.39
U3= 66%
Small Group Differences, Substantial Overlap
![Page 15: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/15.jpg)
Main points….
• Analysis of intervention fidelity and achieve relative strength is a natural counterpart to estimating ESs in ITT studies.
• They provide an interpretive framework for explaining outcome effects.
• When ES and ARSI are discordant, serve as the basis for additional analysis.
• Next section focuses on analysis of variation
![Page 16: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/16.jpg)
Analysis II
Linking Variation in Treatment Receipt/Delivery to Outcomes
![Page 17: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/17.jpg)
Variation in Treatment Receipt/Delivery Within Groups: Regression Models
Using Fidelity/ARS Indicators • Rather than relying on the 0,1 coding of groups,
fidelity indicators replace the group variable. • New question being answered. • Value of fidelity indices will depend on their
strength of the relationship with the outcome;• The greater the group difference, on average,
the less informative fidelity indicators will be; and• High predictability requires reliable indices
![Page 18: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/18.jpg)
Using Group, Fidelity Indicators, or Both: A Simple Example
Randomized Group Assignment
Fidelity Indicator= Hours of Professional Development
Outcome= Differentiated Instruction
Improved Student Outcomes
![Page 19: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/19.jpg)
The “Value Added” of Implementation Fidelity/ARS Data
Group Separation
U3 Predicting Level of Differentiated Instruction
R2Group R2
Hours Pro Development
Small 0.39 0.01 0.293* (0.28)
Large 2.36 0.215* 0.437* (0.22)
Very Large 5.08 0.401* 0.549* (0.15)
![Page 20: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/20.jpg)
Intent-to-treat (ITT) vs. Fidelity/ARS Index: An Example
• Justice, Mashburn, Pence, & Wiggins (2008) examined:– Language-Focused Curriculum (LFC) in 14
classes;– Classes randomly assigned to LFC and
control;– Core component of LFC is the use of
language stimulation techniques (e.g., open questions, recasts, models); and
– Outcome Growth in expressive language examined (fall to spring)
![Page 21: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/21.jpg)
Justice et al. Continued• Implementation fidelity assessed:
– 3 times using 2 hour observation (45 item check list) 50 min. video sample; and 40 weekly lesson plans.
• Fidelity score = – weighted sum of frequency of the use of 7 language
stimulation techniques (range 0-21);
• Fidelity = score/21; averaged over observations• Results:
– LST teachers average fidelity = 0.57 (range 0.17-0.79)
– Control teachers average fidelity = 0.32 (range 0.17-0.56)
– ANOVA F=11.83, p = .005; d = ARSI = 1.71
![Page 22: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/22.jpg)
Justice et al. Continued
01 02 03
04
(Fall Score) (Gender)+ (SES)+
(Attendance)+
ij oo
ij
Y
r
00 00 01 0( ) jLFC u
00 00 01 0( ) jLST u
Level 1
Level 2 ITT
Level 2 TOT
0,1 Group
Fidelity Score
Both coefficients were not significant and near zero.
![Page 23: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/23.jpg)
Justice, et al. (2008): Some Important Issues
• Few teachers exhibited high levels of LST use (core component of LFC)
• Fidelity overall = 0.45• Justice et al. argue, the large group difference
(ARSI=1.71 for fidelity = 0.57 vs. 0.32) may not have been sufficient because the dosage (0.57) was so far below what is needed to affect language development.
• Other possibilities include:– Reliability of the scaling?– Use of average when trend in observations showed
improvement?– Coverage of central constructs?– Functional form of fidelity-outcome linkage?
![Page 24: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/24.jpg)
Analysis III: Compliance-based Models
![Page 25: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/25.jpg)
ITT and Treatment on the Treated (TOT)
• Average Treatment Effect (ATE) estimate assume full compliance.
• Intent to treat (ITT) estimate ignores compliance. – Common to have “no-shows” (for whatever reasons).
• What is the effect of treatment on those receiving treatment (TOT) – the “shows”?
• Bloom 1984 – proposed an adjustment to the ITT estimate, applies to non-compliance of “no-shows” only: ITT/pT
Where PT is the percent Tx recipients
![Page 26: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/26.jpg)
TOT continued
• Assumes treatment receipt/delivery can be meaningfully dichotomized and there is experimentally induced receipt or non-receipt of treatment.
• Works well when no shows experience no treatment effect,
• Assumes randomization does not influence the outcome unless an individual receives the intended treatment.
• This is plausible when treatments are voluntary, substantial, and without a close substitute.
• In mandatory treatments assumption is less plausible, randomization can affect behavior and outcomes,
![Page 27: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/27.jpg)
Local Average Treatment Effect (LATE)
• LATE (Local Average Treatment Effect):– adjust ITT estimate by T and C treatment receipt
rates (compliers in T and defiers in C)– ITT is adjusted as:
• ITT/(pc-pd)• LATE can approximate causal estimates, if a
series of assumptions hold.• Model can be expressed as • Z D Y (or random assignment treatment
receipt outcome)
![Page 28: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/28.jpg)
Some Key LATE Assumptions
• Defiers do not exist or the average program effect is the same for compliers and defiers (monotonicity assumption).
• The effect of Z (assignment) on Y (outcome) is thru variable D (Tx receipt) or the exclusion restriction (all other paths between Z and Y are excluded).
• Strong covariation between Z and D• Randomization makes the instrument and
predicted D uncorrelated with error (e).• SUTVA (stable unit treatment value assumption)
![Page 29: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/29.jpg)
Analysis Type 4: Mediation Model
![Page 30: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/30.jpg)
Mediation Models
Random Assignment
(Z)
Treatment Receipt
(D)
Outcome
(Y)
This is an extension of LATE and is assumption laden
![Page 31: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/31.jpg)
Key Points and Issues• Fidelity assessment serves two roles:
– Average causal difference between conditions; and– Using fidelity measures to assess the effects of variation in
implementation on outcomes.• Degree of fidelity and Achieved Relative Strength provide fuller
picture of the results• Modeling fidelity depends on:
– Level of implementation fidelity– Outcome variance– Variability in implementation fidelity/ARS l
• Most applications, fidelity is just another Level 2 or 3 variable.• Uncertainty and the need for alternative specifications:
– Measure of fidelity– Index of achieved relative strength– Fidelity-outcome model specification (linear, non-linear)
• Adaptation-fidelity tension
![Page 32: Analyzing Intervention Fidelity and Achieved Relative Strength David S. Cordray Vanderbilt University NCER/IES RCT Training Institute,2010.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649d435503460f94a1f61d/html5/thumbnails/32.jpg)
Questions and Discussion