Building Efficient Comparative Effectiveness Trials ... · • cryptogenic sensory polyneuropathy...
Transcript of Building Efficient Comparative Effectiveness Trials ... · • cryptogenic sensory polyneuropathy...
Building Efficient Comparative Effectiveness Trials through Adaptive Designs, Utility
Functions, and Accrual Rate Optimization: Finding the Sweet Spot
Byron J. Gajewski, PhD
Scott M. Berry, PhD
Mamatha Pasnoor, MD
Mazen Dimachkie, MD
Richard Barohn, MD
Laura Herbelin, BS
1
http://en.wikipedia.org/wiki/Bad_(album)2
Bayesian Adaptive Designs (BAD)
• No longer “a dream for statisticians only”
• Published not only in biostatistical journals but also clinical epidemiology and medical journals
• Save time and money and lean towards more ethical studies
• Scientific contribution to the design, implementation, and analysis of comparative effectiveness clinical trials
• PCORI advocates for their use
3
Comparative Effectiveness • NASCAR• cryptogenic sensory polyneuropathy (CSPN)
– What treatment for pain is the best? Off label and approved drugs used in practice
• We built a BAD with efficiency for finding the best treatment in mind and found three key trial aspects– the Bayesian adaptive design parameters– the utility function for weighing endpoints– the patient accrual rate
• These three developmental parameters are vital for building adaptive, cost-effective comparative effectiveness designs.
4
Comparative Effectiveness • NASCAR• Cryptogenic sensory polyneuropathy (CSPN)
– What treatment for pain is the best? Off label and approved drugs used in practice
• We built a BAD with efficiency for finding the best treatment in mind and found three key trial aspects– the Bayesian adaptive design parameters– the utility function for weighing endpoints– the patient accrual rate
• These three developmental parameters are vital for building adaptive, cost-effective comparative effectiveness designs.
5
Comparative Effectiveness • NASCAR• Cryptogenic sensory polyneuropathy (CSPN)
– What treatment for pain is the best? Off label and approved drugs used in practice
• We built a BAD with efficiency for finding the best treatment in mind and found three key trial aspects– the Bayesian adaptive design parameters– the utility function for weighing endpoints– the patient accrual rate
• These three developmental parameters are vital for building adaptive, cost-effective comparative effectiveness designs.
Non-diabetic
6
Comparative Effectiveness • NASCAR• Cryptogenic sensory polyneuropathy (CSPN)
– What treatment for pain is the best? Off label and approved drugs used in practice
• We built a BAD with efficiency for finding the best treatment in mind and found three key trial aspects– the Bayesian adaptive design parameters– the utility function for weighing endpoints– the patient accrual rate
• These three developmental parameters are vital for building adaptive, cost-effective comparative effectiveness designs
Non-diabetic
7
Comparative Effectiveness • NASCAR• Cryptogenic sensory polyneuropathy (CSPN)
– What treatment for pain is the best? Off label and approved drugs used in practice
• We built a BAD with efficiency for finding the best treatment in mind and found three key trial aspects– the Bayesian adaptive design parameters– the utility function for weighing endpoints– the patient accrual rate
• These three developmental parameters are vital for building adaptive, cost-effective comparative effectiveness designs
Non-diabetic
8
Comparative Effectiveness • NASCAR• Cryptogenic sensory polyneuropathy (CSPN)
– What treatment for pain is the best? Off label and approved drugs used in practice
• We built a BAD with efficiency for finding the best treatment in mind and found three key trial aspects– the Bayesian adaptive design parameters– the utility function for weighing endpoints– the patient accrual rate
• These three developmental parameters are vital for building adaptive, cost-effective comparative effectiveness designs
Non-diabetic
Performance Adaptive Investigation of Neuropathic Pain-Comparison of Treatments in Real-Life Situations (PAIN-CONTRoLS )
9
What has been done on BAD?
• Phase I-III clinical trials– dose finding studies– assessment of safety and efficacy in the
presence of historical prior information. • In many cases these studies have a
functional form that is unique to • Classical pharmaceutical clinical trials
(e.g. control group or dose).
10
What has not been done on BAD?
• A different challenge in comparative effectiveness trials – there is typically no control group– investigating the relative effectiveness – No dose structure to our problem
• We discuss the unique framework of BAD under this setting.
11
What we address here • Combine endpoints with a utility function• Optimize accrual • Ex: PAIN-CONTRoLS
– Endpoints– Models– Simulation
• We find the “sweet spot” balancing – Average number of patients needed – Average length of time to finish the study
12
PAIN-CONTRoLS• Five different drugs (e.g. Lyrica; Cymbalta;
Tramadol; Nortriptyline; Gabapentin)• Multi-site trial (20 sites); accrual about 4-8
patients/week• Nmax=600 (1.5-3.0 years)• Endpoints:
– Efficacy: ½ or better drop in VAS score (baseline to 12 weeks)
– Quit/Dropout: Drop treatment after 12 weeks
13
Combining Endpoints
• Combining two endpoints (Berry et al., 2010)– We detail the building of a utility function here
• Scenario: Drug B > Drug A but higher quit rate– What would that “quit rate” have to be in order for
Drug B to be clinically the same as Drug A?
14
Quit
15
Combining Endpoints
• Utility for Efficiency: 1 for 100% efficacy and utility of 0 for 0% efficacy.
• Utility for quit/ discontinue endpoint we used utility of 0.75 at 0% quit/discontinue with a drop to 0 at 100% quit/discontinue.
• Utility combination U(E,Q)=E+.75-.75Q.
16
Statistical Details
17
Basic analytic examples
• Example 1: one arm• Example 2: two arms
18
Example 1: one arm
• Consider a tolerability endpoint for the PAIN-CONTRoLS study and suppose the endpoint is measured immediately after randomization (Qi=1) or not (Qi=0)– n=85 patients (fixed) – SQ=Σqi
– θ quit rate (unobserved but random)– Δ max. tolerated quit rate (fixed & known)
• Stopping rule: P(θ <Δ |SQ)> γ19
Example 1: one arm
Period 1 (T1)n1
Period 2(T2)n2
Time
20
Example 1: one arm
Period 1 (T1)n1
Period 2(T2) n2
TimeStop if P(θ <Δ |SQ)> γ(Uniform-Binomial)
21
Example 1: one arm
Period 1 (T1)n1
Period 2(T2)n2
TimeStop if P(θ <Δ |SQ)> γ(Uniform-Binomial)
Otherwise move on
22
Example 1: operating characteristics
1. Sampling distribution is Qi|θ0~Bern(θ0), where θ0 is the true quit/discontinue rate.
2. Probability of stopping the trial early at period 1:
11 111
1 0 00 0 1
1 1QQ Q Q
Q
nn SS S n S
QS Q
nnP I d
SS n
,
where I(x>y) is 1 if x>y and 0 otherwise.
a. Expected time (T) of the trial is E(T)=P1T1+(1-P1)T2
b. Expected sample size (N) of the trial is E(N)=P1n1+(1-P1)85.
23
Example 1: operating characteristics
1. Sampling distribution is Qi|θ0~Bern(θ0), where θ0 is the true quit/discontinue rate.
2. Probability of stopping the trial early at period 1:
1
111 1
1 0 00 10
1 1Q QQ Q
Q
n SS n S
Q
nS
QS
nP
nS
I dn S
,
where I(x>y) is 1 if x>y and 0 otherwise.
a. Expected time (T) of the trial is E(T)=P1T1+(1-P1)T2
b. Expected sample size (N) of the trial is E(N)=P1n1+(1-P1)85.
24
Example 1: operating characteristics
1. Sampling distribution is Qi|θ0~Bern(θ0), where θ0 is the true quit/discontinue rate.
2. Probability of stopping the trial early at period 1:
1
111 1
1 0 00 0 1
1 1Q QQ Q
Q
n SS n S
Q
nS
QS
nP
nS
I dn S
,
where I(x>y) is 1 if x>y and 0 otherwise.
a. Expected time (T) of the trial is E(T)=P1T1+(1-P1)T2
b. Expected sample size (N) of the trial is E(N)=P1n1+(1-P1)85.
25
Example 1: operating characteristics
1. Sampling distribution is Qi|θ0~Bern(θ0), where θ0 is the true quit/discontinue rate.
2. Probability of stopping the trial early at period 1:
1
111 1
1 0 00 0 1
1 1Q QQ Q
Q
n SS n S
Q
nS
QS
nP
Sn
I dS n
,
where I(x>y) is 1 if x>y and 0 otherwise.
a. Expected time (T) of the trial is E(T)=P1T1+(1-P1)T2
b. Expected sample size (N) of the trial is E(N)=P1n1+(1-P1)85.
26
Example 1: operating characteristics
1. Sampling distribution is Qi|θ0~Bern(θ0), where θ0 is the true quit/discontinue rate.
2. Probability of stopping the trial early at period 1:
11 111
0 00 0 1
1 1 1QQ Q Q
Q
nn SS S n S
QS Q
nnP I d
SS n
,
where I(x>y) is 1 if x>y and 0 otherwise.
a. Expected time (T) of the trial is E(T)=P1T1+(1-P1)(T1+T2)
b. Expected sample size (N) of the trial is E(N)=P1n1+(1-P1)85.
27
One arm: size and cost (n1+n2=85)
30
40
50
60
70
80
90
25 35 45 55 65 75 85
E(n)
or E
(T) i
n da
ys
n1
θ0=.2 Δ =.3, and γ =.8, n1=30, 35, 40,…,80 T1=T2=28 days. The probability of stopping early varies from 0.4275 for n1=30 and jumps up to 0.7621 for n2=80.
E(N)
E(T)
28
“virtual response”
Example 2: two arms
• Similar notation, but stop if P({θ1 < θ2 | SQ1,SQ2})> γ or if P ({θ1 > θ2 | SQ1,SQ2})>γ
• Operations– Complicated closed form (Kawasaki &
Miyaoka, 2012)– Then using a double sum across SQ1 and SQ2
would allow similar calculations for E(T) and E(N) as done in Example 1
29
Gets more complicated fast
• Five arms and two endpoints• Accrual patterns tend to be random and
staggered; not fixed• Quickly complicate things for closed-form
analytic solutions• Therefore, as advocated by Berry et al.
(2011), we utilize simulations
30
PAIN-CONTRoLS
• Virtual subject response for five arms• Accrual patterns• Design• Adaptive randomization: allocation• Simulation Algorithm
31
Virtual subject response for five arms
• Null case
• Alternative case
0 .3, .3, .3, .3, .3e θ and 0 .2, .2, .2, .2, .2q θ
0 .3, .3, .3, .4, .5e θ and 0 .3, .3, .3, .25, .15q θ
32
Accrual patterns1. mean number of accrued patients per week: ΛT.
2. 1{ }| ~T T T TN N Poisson , where T=1,2,3,…, and 0N =0. The patterns of ΛT
depend on two factors:
a. the number of sites actively enrolling patients into the study and
b. how fast the sites can enroll, which we assume is a constant λ0/2 for each:
0
0
0
0
, 0 <22 , 2 <43 , 4 <6
10 , 20
T
TTT
T
.
33
Design 1. Likelihood: SEjT|njT~Bino(njT ,θe
j) and SQjT|njT~Bino(njT ,θqj).
2. Priors, 2logit ~ 0,100ej N and 2logit ~ 0,100q
j N .
3. Posterior distributions, MCMC.
4. Our stopping criteria:
a. minimum of 200 subjects allocated.
b. Stop the trial if the probability the arm with the maximum utility > 0.90.
c. Utility | 0.75 0.75 |e qjT j EjT j QjTU S S with maximum utility
max, 1 2 3 4 5max , , , ,T T T T T TU U U U U U .
d. The evaluation criteria: probability the arm with the maximum utility > 0.90.
34
Adaptive randomization: allocation
max,*Pr
1jT T jT
jjT
U U Var UV
n
35
Sweet Spot Algorithm (SSA)• Step 0: Set b=0• Step 1: Set b=b+1. • Step 2: Simulate the initial observed data.• Step 3: estimate posterior parameters via simulation
and calculate the stopping rule and the possible next allocation.
• Step 4: repeat steps 2 and 3 after collecting four more weeks of data.
• Step 5: evaluate all of the data after collecting all of the endpoints.
• Step 6: go to step 1 unless b=100, then stop. 36
Results
37
Pmax, N, and T predictive distributions “Alternative Case” (Λ20=8)
0.2 0.4 0.6 0.8 10
50
100
Pmax
Cou
nt
200 300 400 500 6000
20
40
60
n
Cou
nt
40 60 80 100 1200
10
20
30
T
Cou
nt
38
Alternative Case• Success:
– 95% of the trials had early success– 1% late success (trial goes to the maximum
sample size of 600)– 4% of incomplete solutions.
• Sample size: – E(N)=302.2 subjects – 80% of the trials being 362 or smaller.
• Length: – E(T)=61.4 weeks– longest trial taking 100 weeks. 39
Expected size, time, and cost for five arms (effect scenario)
E(N)= 7.4466Λ20 + 241.53
E(T)= 254.82(Λ20)-0.694
0
50
100
150
200
250
300
350
0 2 4 6 8 10 12
E(n)
or E
(T) i
n w
eeks
Λ20
40
Expected size, time, and cost for five arms (effect scenario)
E(N)= 7.4466Λ20 + 241.53
E(T)= 254.82(Λ20)-0.694
0
50
100
150
200
250
300
350
0 2 4 6 8 10 12
E(n)
or E
(T) i
n w
eeks
Λ20
E(Cost)= 7.4466 20Λ +241.53+ 1.25(254.82 20Λ -0.694) Taking derivative w.r.t. 20Λ and solving we get
1/1.694ˆ 1.25*254.82*.694 /7.4466
=7.4
20Λ
41
Expected size, time, and cost for five arms (effect scenario)
E(N)= 7.4466Λ20 + 241.53
E(T)= 254.82(Λ20)-0.694
0
50
100
150
200
250
300
350
0 2 4 6 8 10 12
E(n)
or E
(T) i
n w
eeks
Λ20
350
400
450
500
550
600
0 2 4 6 8 10 12
E(C
)
Λ20
E(Cost)= 7.4466 20Λ +241.53+ 1.25(254.82 20Λ -0.694) Taking derivative w.r.t. 20Λ and solving we get
1/1.694ˆ 1.25*254.82*.694 /7.4466
=7.4
20Λ
42
If we accrue faster will we get less efficacy per unit?
• No!• For the accrual patterns the proportion
times we are successful (i.e. proportion ) is between 0.96 and 1.00
• The margin of error if the true success rate is about0.98 (+/-1.96*sqrt(.98*.02/100)=+/-.0274).
43
Expected size, time, and cost for five arms (null scenario)
E(N) = 0.3863Λ20 + 586.5
E(T) = 584.27(Λ20)-0.879
0
100
200
300
400
500
600
700
0 2 4 6 8 10 12
E(n)
or E
(T) i
n w
eeks
Λ20
44
Discussion: relative to fixed trial• Classical framework: fixed sample size but get
various endpoint efficacy knowledge • We “flip” the approach to clinical trials design
– The effect we learn is fixed– While sample size varies depending on the data– BAD approach is a proxy for the scientific
knowledge
45
Discussion: various extensions of SSA
• Vary the number arms (say 2, 6, or more), • One endpoint instead of two• Change the maximum sample size from 600 to
higher • Change to a minimal efficacy or a futility
stopping rule • The accrual pattern could change
46
Discussion: Accrual• Starts off small and grows (Anisimov, 2011)• Adaptive accrual => accrual prediction models
(e.g. Anisimov & Federov 2007; Gajewski, Simon, and Carlson, 2008; Zhang and Long, 2010; & Anisimov, 2011) => update accrual patterns are in real time
• For example,overpromise and under deliver (e.g. Breau, 2006)
47
Discussion: Generalize• SSA algorithm extensions
– Time to event endpoints– Ordinal or continuous or a mix of the two
endpoints– Dynamic linear models for dose finding
studies – Various types of hierarchical models
48
Discussion
• Sweet spot same for all drugs?– Subjects cost differently by drug
• Generalizability to other Bayesian adaptive clinical trials:– adaptation rule– utility function– accrual should all be parameters considered
for optimizing the design of comparative effectiveness research
49
Benefits of BAD: “B.A. Baracus” trial design
http://www.a-team-inside.com/ba/bosco-b-a-baracus
• Hard work up-frontbut worth it later
• Fit• Efficient• Very good at
getting answers• Bad A**
50
Acknowledgements
• Frontiers: The Heartland Institute for Clinical and Translational Research CTSA UL1TR000001 (Barohn & Aaronson)
• Department of Biostatistics (e.g. matching Frontiers effort) (Mayo)
51
QUESTIONS?
52