Post on 19-Dec-2015
Dynamic Treatment Regimes,STAR*D & Voting
D. Lizotte, E. Laber & S. Murphy
LSU ---- Geaux Tigers!
April 2009
2
Outline
• Dynamic Treatment Regimes
• Constructing Regimes from Data
• A Measure of Confidence: Voting
• STAR*D
3
Dynamic treatment regimes are individually tailored treatments, with treatment type and dosage changing according to patient outcomes. Operationalize clinical practice.
k Stages for one individual
Observation available at jth stage
Action at jth stage (usually a treatment)
4
Goal: Construct decision rules that input information available at each stage and output a recommended decision; these decision rules should lead to a maximal mean Y. Y is a known function of
The dynamic treatment regime is the sequence of two decision rules:
k=2 Stages
5
Action ActionObservations Observations Reward
Stage 1 Stage 2 Stage 1 Stage 2
Deriving the Optimal Dynamic Regime: Move Backwards Through Stages.
You know multivariate distribution
6
Optimal Dynamic Treatment Regime
satisfies
7
Data for Constructing the Dynamic Treatment Regime:
Subject data from sequential, multiple assignment, randomized trials. At each stage subjects are randomized among alternative options.
Aj is a randomized treatment with known randomization probability.
8
Stage 1 Intermediate Stage 2Preference Treatment Outcome Preference Treatment
Bup Continue Remission on Present
Switch R Ven Treatment
Ser MIRT Switch R
+ Bup No NTPAugment R Remission
+ Bus +LI
Augment R +THY
STAR*D
9
STAR*D Analyses
• X1 includes site, preference for future treatment and can include other baseline variables.
• X2 can include measures of symptoms (Qids), side effects, preference for future treatment
• Y is (reverse-coded) the minimum of the time to remission and 30 weeks.
10
Outline
• Dynamic Treatment Regimes
• Constructing Regimes from Data
• A Measure of Confidence: Voting
• STAR*D
11
Regression-based methods for constructing decision rules
•Q-Learning (Watkins, 1989) (a popular method from computer science)
•Optimal nested structural mean model (Murphy, 2003; Robins, 2004)
• The first method is equivalent to an inefficient version of the second method, if we use linear models and each stages’ covariates include the prior stages’ covariates and the actions are centered to have conditional mean zero.
12
There is a regression for each stage.
A Simple Version of Q-Learning –
• Stage 2 regression: Regress Y on to obtain
• Stage 1 regression: Regress on to obtain
13
for patients entering stage 2:
• is the average outcome conditional on patient history (no remission in stage 1; includes past treatment and variables affected by stage 1 treatment).
• is the estimated average outcome assuming the “best” treatment is provided at stage 2 (note max in formula).
• is the dependent variable in the stage 1 regression for patients moving to stage 2
14
Optimal Dynamic Treatment Regime
satisfies
15
A Simple Version of Q-Learning –
• Stage 2 Q function, (Y was dependent variable) yields
• Stage 1 Q function, ( was dependent variable) yields
16
Decision Rules:
17
Outline
• Dynamic Treatment Regimes
• Constructing Regimes from Data
• A Measure of Confidence: Voting
• STAR*D
18
Measures of Confidence
• Classical
– Confidence/Credible intervals and/or p-values concerning the β1, β2.
– Confidence/Credible intervals concerning the average response if is used in future to select the treatments.
19
A Measure of Confidence for use in
Exploratory Data Analysis
• Replication Probability
– Estimate the chance that a future trial would find a particular stage j treatment best for a given sj. The vote for treatment aj* is
20
A Measure of Confidence for use in
Exploratory Data Analysis
Replication Probability
– If stage j treatment aj is binary, coded in {-1,1}, then
21
Bootstrap Voting
Use bootstrap samples to estimate
by
22
The Vote: Intuition
If has a normal distribution with variance matrix then
is
23
Bootstrap Voting
The naïve bootstrap vote estimator
is inconsistent.
24
Bootstrap Voting
A consistent bootstrap vote estimator of
is
where is smooth and
25
Bootstrap Voting
In our simple example
is approximately
26
What does the vote mean?
• is similar to the p-value for the hypothesis in that it converges, as n increases, to 1 or 0 depending on the sign of
• If then the limiting distribution is not uniform; instead converges to a constant.
27
Outline
• Dynamic Treatment Regimes
• Constructing Regimes from Data
• A Measure of Confidence: Voting
• STAR*D
28
Stage 1 Intermediate Stage 2Preference Treatment Outcome Preference Treatment
Bup Continue Remission on Present
Switch R Ven Treatment
Ser MIRT Switch R
+ Bup No NTPAugment R Remission
+ Bus +LI
Augment R +THY
STAR*D
29
STAR*D
Regression formula at stage 2:
30
31
STAR*D
Regression formula at stage 1:
32
STAR*D
Decision Rule for subjects preferring a switch at stage 1
• if offer VEN
• if offer SER
• if offer BUP
33
STAR*D Level 2, Switch
34
35
36
Truth in Advertising:STAR*D
Missing Data + Study Drop-Out
• 1200 subjects begin level 2 (e.g. stage 1)
• 42% study dropout during level 2
• 62% study dropout by 30 weeks.
• Approximately 13% item missingness for important variables observed after the start of the study but prior to dropout.
37
Truth in Advertising:STAR*D
Multiple Imputation within Bootstrap
• 1000 bootstrap samples of the 1200 subjects
• Using the location-scale model we formed 25 imputations per bootstrap sample.
• The stage j Q-function (regression function) for a bootstrap sample is the average of the 25 Q-functions over the 25 imputations.
38
Discussion
• We consider the use of voting to provide a measure of confidence in exploratory data analyses.
• Our method of adapting the bootstrap voting requires a tuning parameter, γ. It is unclear how to best select this tuning parameter.
• We ignored the bias in estimators of stage 1 parameters due to the fact that these parameters are non-regular. The voting method should be combined with bias reduction methods.
39
This seminar can be found at:http://www.stat.lsa.umich.edu/~samurphy/
seminars/LSU2009.ppt
Email me with questions or if you would like a copy!
samurphy@umich.edu