Adaptive Treatment Strategies: Challenges in Data Analysis S.A. Murphy NY State Psychiatric...

Adaptive Treatment Strategies:Challenges in Data Analysis

S.A. MurphyNY State Psychiatric Institute

February, 2009

2

Outline

• What are Adaptive Treatment Strategies?

• Myopic Decision Making

• Constructing Strategies

• Q-Learning & Voting

• Example using CATIE

3

Adaptive Treatment Strategies operationalize multi-stage decision making.

These are individually tailored sequences of interventions, with intervention type and dosage adapted to the individual.

•Generalization from a one-time decision to a sequence of decisions concerning interventions

•Operationalize clinical practice.

Each decision corresponds to a stage of intervention

4

Example of an Adaptive Treatment Strategy

Treatment of depression. Goal is to achieve and maintain remission.

Provide Citalopram for up to 12 weeks gradually increasing dose as required.

If, there is no remission yet either the maximum dose has been provided for two weeks, or 12 weeks have occurred, then

if there has been a 50% improvement in symptoms, augment with Mirtazapine.

else switch treatment to Bupropion.

Else (remission is achieved) maintain on Citalopram and provide web-based disease management.

5

Adaptive Treatment Strategy

• At each stage of intervention– Use tailoring variables (symptoms, adherence,

side effects, social support, family support…),– To select actions/interventions such as

medications, behavioral therapies, augmentations,….

– In order to maximize long term rewards (maximize functionality, symptom-free time).

6

Why use an Adaptive Treatment Strategy?

– High heterogeneity in response to any one intervention

• What works for one person may not work for another

• What works now for a person may not work later

– Improvement often marred by relapse• Remitted or few current symptoms is not the same

as cured.

– Co-occurring disorders/adherence problems are common

7

Outline




• Q-Learning


8

Myopic Decision Making

• In myopic decision making, decision makers use strategies that seek to maximize immediate rewards.

Problems:– Ignore longer term consequences of present actions.– Ignore the range of feasible future actions/interventions– Ignore the fact that immediate responses to present actions

may yield information that pinpoints best future actions

(A strategy tells us how to use the observations to choose the actions/interventions.)

9

Treatment of Schizophrenia

• Myopic action: Offer patients a treatment that reduces schizophrenia symptoms for as many people as possible.

• The result: Some patients are not helped and/or experience abnormal movements of the voluntary muscles (TDs). The class of subsequent medications is greatly reduced.

• The mistake: We should have taken into account the variety of treatments available to those for whom the first treatment is ineffective.

• The message: Use an initial medication that may not have as large a success rate but that will be less likely to cause TDs.

10

Treatment of Opioid Dependence• Myopic action: Choose an intensive multi-component

treatment (methadone + counseling + behavioral contingencies) that immediately reduces opioid use for as many people as possible.

• The result: Behavioral contingencies are burdensome/expensive to implement and many people may not need the contingencies to improve.

• The mistake: We should allow the patient to exhibit poor adherence prior to implementing the behavioral contingencies.

• The message: Use an initial treatment that may not have as large an immediate success rate but will allow us to ascertain whether behavioral contingencies are required.

11

Outline






12

Action ActionObservations Observations Reward

Stage 1 Stage 2 Stage 1 Stage 2

Basic Idea for Constructing an Adaptive Treatment Strategy:

Move Backwards Through Stages.

(Pretend you are “All-Knowing”)

13

2 Stages for each individual

Observations available at jth stage

Action at jth stage

14

2 Stages

History available at each stage

Primary Outcome/Reward:

15

An Adaptive Treatment Strategy is the sequence of decision rules:

A simple decision rule is: given weights β, switch treatment at stage j if

otherwise maintain on current treatment; Sj is a vector summary of the history, Hj.

16

Goal:

Use data to construct decision rules that input information in the history at each stage and output a recommended decision; these decision rules should lead to a maximal mean Y.

In the future we employ the actions recommended by

the decision rules:

17

Idealized Data for Constructing an Adaptive Treatment Strategy:

Data from sequential, multiple assignment, randomized trials in which at each stage subjects are randomized among alternative options.

That is, Aj is a randomized action with known randomization probability.

Binary actions with P[Aj=1]=P[Aj=-1]=.5

18

Outline






19

Regression-based methods for constructing decision rules

•Q-Learning (Watkins, 1989) (a popular method from computer science)

•A-Learning or optimal nested structural mean model (Murphy, 2003; Robins, 2004)

•The first method is an inefficient version of the second method when each stages’ covariates include the prior stages’ covariates and the actions are centered to have conditional mean zero.

20

Action ActionObservations Observations Reward

Stage 1 Stage 2 Stage 1 Stage 2

Basic Idea for Constructing a Regime: Move Backwards Through Stages.

21

(k=2)

Dynamic Programming

22

Approximate for S', S vector summaries of the history,

A Simple Version of Q-Learning –binary actions

• Stage 2 regression: Use least squares with outcome, Y, and covariates to obtain

• Set

• Stage 1 regression: Use least squares with outcome, and covariates to obtain

23

Approximate for S', S vector summaries of the history,

A Simple Version of Q-Learning –binary actions

Stage j decision rule:

Select treatment = 1 if

Otherwise select treatment = -1

24

Voting (exploratory analysis)

•Estimate the chance that if we were to rerun the trial we would see treatment Aj=1 perform better than treatment Aj=-1 for patients with history hj.

That is,

•Use bootstrap

25


•Use bootstrap samples to estimate percentage of the time that treatment A1=1 is favored:

•Natural approach will not work, e.g.

is not necessarily consistent.

• We use an adaptive bootstrap

26


•Use an “adaptive” bootstrap method to estimate percentage of the time that treatment A1=1 is favored:

•Adaptive bootstrap estimator:

27

Outline

• What are Dynamic Treatment Regimes?


• Constructing Regimes



28

Clinical Antipsychotic Trials of Intervention Effectiveness

(Schizophrenia)

• Multi-stage trial of 18 months duration• Relaxed entry criteria • A large number of sites representing a broad

array of clinical settings (state mental health, academic, Veterans’ Affairs, HMOs, managed care)

• Approximately 1500 patients

29

CATIE Randomizations (simplified)

Phase 1

Randomized Treatments OLAN QUET RISP ZIPR PERP Phase 2

Treatment preference Efficacy TolerabilityRandomized Treatments CLOZ OLAN QUET RISP OLAN QUET RISP ZIPR

Phase 3Treatments selected many options by preference

30

Exploratory Analyses

• Reward: Time to Treatment Dropout• Phase 1 analysis:

– Controls: TD, recent exacerbation, site– Tailoring variable: pretreatment PANSS

• Phase 2 analysis: – Controls: TD, recent exacerbation, site– Tailoring variables: “treatment preference,” phase 1

treatment, end of phase 1 PANSS

Constructing Dynamic Treatment Regimes using CATIE

34

Challenges

• It is extremely challenging to provide measures of confidence that possess “good frequentist properties.”

• Clinical Decision Support Systems– We need to be able construct dynamic treatment regimes

that recommend a group of treatment actions when there is no evidence that a particular treatment action is best.

• Even in a randomized trial setting, the most straightforward analyses are subject to confounding bias. Some methods to avoid confounding bias are available.

35

Acknowledgements: This presentation is based on work with many individuals including Eric Laber, Dan Lizotte, John Rush, Scott Stroup, Joelle Pineau, Daniel Almirall and Bibhas Chakraborty,.

Email address: [email protected]

Slides with notes at:

http://www.stat.lsa.umich.edu/~samurphy/

Click on seminars > health science seminars

Adaptive Treatment Strategies: Challenges in Data Analysis S.A. Murphy NY State Psychiatric...

Documents

Transcript of Adaptive Treatment Strategies: Challenges in Data Analysis S.A. Murphy NY State Psychiatric...