Eduardo Bergel’sControlled Clinical Trial

Simulator (CTS) Beta Version 0.21

Used by Dave Sackett (Oct Used by Dave Sackett (Oct ‘04)‘04)

Trout Research & Education Center Trout Research & Education Center at Irish Lake, Canadaat Irish Lake, Canada

Table of Contents

How to How to runrun the CTS the CTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Slide # 4Slide # 4

Effect of sample size on confidence intervalsEffect of sample size on confidence intervals . . . # . . . # 3838

The effects of admitting patients with different The effects of admitting patients with different risks and responsiveness to your RCT . . . . . . . . . risks and responsiveness to your RCT . . . . . . . . . . . . # 53. . . # 53

1. How to run the CTS

You tell the Controlled Trial Simulator

1.1. How manyHow many patients you hope to patients you hope to recruit.recruit.

2.2. Your guess at their risk of Your guess at their risk of eventsevents if if they receive they receive no Rxno Rx or the or the control Rx.control Rx.

3.3. Your hopes for their Your hopes for their responsivenessresponsiveness to to experimental Rx.experimental Rx.

4.4. With what rates of With what rates of cross-overcross-over, , loss-to-loss-to-follow-upfollow-up, and , and compliancecompliance

The CTS will then:

““Run” your trial hundreds or Run” your trial hundreds or thousands of times.thousands of times.

- and tell you, with tables and - and tell you, with tables and graphs, its results, their confidence graphs, its results, their confidence intervals, and the presence and intervals, and the presence and magnitude of bias (e.g., from magnitude of bias (e.g., from cross-overs, low compliance, etc.) cross-overs, low compliance, etc.)

For example, does aspirin reduce strokes after TIA ? We estimated that 21% of patients We estimated that 21% of patients

with TIAs would go on to stroke or with TIAs would go on to stroke or death within 2 years.death within 2 years.

We thought we could reduce this by We thought we could reduce this by one-third with aspirin (to 14%).one-third with aspirin (to 14%).

We thought we could recruit about 600 We thought we could recruit about 600 patients.patients.

For example, does aspirin reduce strokes after TIA ?

We enter these estimates and We enter these estimates and guesses into the RCT Simulator.guesses into the RCT Simulator.

And then tell the RCT Simulator how And then tell the RCT Simulator how many times to run our trial.many times to run our trial.

We thought that 21% would

stroke or die without Rx CER =

21%

We thought that 21% would

stroke or die without Rx CER =

21%We thought that aspirin could reduce these outcomes by 1/3, to 14%

EER = 14%

We thought we could recruit 600

patientsWe thought that

21% would stroke or die

without Rx CER = 21%

We thought that aspirin could reduce these outcomes by 1/3, to 14%

EER = 14%

We thought we could recruit 600

patientsWe thought that

21% would stroke or die

without Rx CER = 21%

We thought that aspirin could reduce these outcomes by 1/3, to 14%

EER = 14%

We wanted our RCT replicated

3000 times

3000 simulations take 5 seconds !

And their results are displayed in a And their results are displayed in a tabletable

CER

CER

EER

CER

EER

300 300

CER

EER

3000 simulatio

ns

300 300

CER

EER

Relative Risk

Reduction

3000 simulatio

ns

300 300

CER

EER

Relative Risk

Reduction

3000 simulatio

ns

Calculates Power !

300 300

CER

EER

Relative Risk

Reduction

3000 simulatio

ns

Calculates Power !

300 300

Can graph results

Graph Options

1.1. Can simply graph the overall RRR Can simply graph the overall RRR and its 95% confidence interval.and its 95% confidence interval.

RRR for aspirin in TIA

X = true RRR that is free of bias

Graphic Options

1.1. Can simply graph the overall RRR Can simply graph the overall RRR and its 95% confidence interval.and its 95% confidence interval.

2.2. Can create a histogram of Can create a histogram of individual RRRs from each individual RRRs from each simulation.simulation.

Histogram of RRR

Graphic Options

1.1. Can simply graph the overall RRR Can simply graph the overall RRR and its 95% confidence interval.and its 95% confidence interval.

2.2. Can create a histogram of Can create a histogram of individual RRRs from each individual RRRs from each simulation.simulation.

3.3. Can plot the individual RRRs from Can plot the individual RRRs from each simulation.each simulation.

Can plot every trial

Can plot every trial

Some RRRs < 0

Can plot every trial

Extreme results must

occur !

Some RRRs < 0

Graphic Options1.1. Can simply graph the overall RRR Can simply graph the overall RRR

and its 95% confidence interval.and its 95% confidence interval.

2.2. Can create a histogram of Can create a histogram of individual RRRs from each individual RRRs from each simulation.simulation.

3.3. Can plot the individual RRRs from Can plot the individual RRRs from each simulation.each simulation.

4.4. Can plot RRR vs. P-values.Can plot RRR vs. P-values.

Can chart the P-values for every trial

Can chart the P-values for every trial

RRR & 95% CI

Can chart the P-values for every trial

P> .05

P<.05

Can chart the P-values for every trial

P> .05

P<.05

Power = 60%

Can chart the P-values for every trial

P<.05

Power = 60%

P almost 0.05

against!

So, that is the basic system

Outputs can also be expressed in all Outputs can also be expressed in all these formats as:these formats as:Relative RisksRelative Risks

Currently under construction:Currently under construction:Absolute Measures (ARR, NNT, etc)Absolute Measures (ARR, NNT, etc)Cluster TrialsCluster Trials

2. The effect of sample size on 95% confidence intervals

Can use it to understand the effect of different sample sizes on the same RRR:

How to do thisHow to do this: : While keeping the treatment effect While keeping the treatment effect constant (say, RRR = 25%), constant (say, RRR = 25%), double, triple and quadruple your double, triple and quadruple your sample size.sample size.

Increasing Sample Size

Can have 4 different (sub) groups

Increasing Sample Size

Increasing Sample Size

Then simulate it 3000 times

Increasing Sample Size

All2500

Increasing Sample Size

250

All2500

Increasing Sample Size

250

1000

All2500

Now you can answer a frequently asked question:

If you want to cut the 95% If you want to cut the 95% confidence interval (CI) on your confidence interval (CI) on your RRR in half, by how much do you RRR in half, by how much do you need to increase your sample size need to increase your sample size (N)?(N)?

You need to remember “the only formula you’ll ever need”

CONFIDENCECONFIDENCE

(recalling that (recalling that narrowernarrower confidence confidence intervals provide intervals provide higherhigher confidence) confidence)

ConfidenceConfidence = =

You need to remember “the only formula you’ll ever need”

CONFIDENCECONFIDENCE

(recalling that (recalling that narrowernarrower confidence confidence intervals provide intervals provide higherhigher confidence) confidence)

ConfidenceConfidence = =

SignalNoise

You need to remember “the only formula you’ll ever need”

CONFIDENCECONFIDENCE

(recalling that (recalling that narrowernarrower confidence confidence intervals provide intervals provide higherhigher confidence) confidence)

ConfidenceConfidence = x = x

√√sample sizesample size

SignalNoise

Increasing Sample Size

250

1000

All2500To cut this

in half

Increasing Sample Size

250

1000

All2500To cut this

in half

You must quadruple

N

3. The effects of admitting patients with different risks and responsiveness to your RCT

Risk and Responsiveness

Who do you Who do you want in your want in your RCT ?RCT ?

ResponsivenessResponsiveness

HighHigh LowLow

RiskRisk

HighHigh

LowLow

Risk and Responsiveness

Who do you Who do you want in your want in your RCT ?RCT ?

ResponsivenessResponsiveness

HighHigh LowLow

RiskRisk

HighHigh IdealIdeal

LowLow

Risk and Responsiveness

Who do you Who do you want in your want in your RCT ?RCT ?

ResponsivenessResponsiveness

HighHigh LowLow

RiskRisk

HighHigh IdealIdeal

LowLow Ever?Ever?

Risk and Responsiveness

Who do you Who do you want in your want in your RCT ?RCT ?

ResponsivenessResponsiveness

HighHigh LowLow

RiskRisk

HighHigh IdealIdeal ??????

LowLow ?????? Ever?Ever?

Risk and Responsiveness

Suppose that . .Suppose that . .ResponsivenessResponsiveness

HighHigh LowLow

RiskRisk(in terms (in terms of of CControl ontrol EEvent vent RRates)ates)

HighHigh

CER= .30CER= .30

LowLow

CER= .10CER= .10

Risk and Responsiveness

And And suppose . . . suppose . . .

ResponsivenessResponsiveness(in terms of (in terms of RRelative elative

RRisk isk RReduction)eduction)

HighHigh

RRR= RRR= 30%30%

LowLow

RRR= RRR= 10%10%

RiskRisk(in terms (in terms of of CControl ontrol EEvent vent RRates)ates)

HighHigh

CER= .30CER= .30

LowLow

CER= .10CER= .10

Risk and Responsiveness

These would These would result in the result in the following following reductions in reductions in events:events:

ResponsivenessResponsiveness(in terms of (in terms of RRelative elative

RRisk isk RReduction)eduction)

HighHigh

RRR= RRR= 30%30%

LowLow

RRR= RRR= 10%10%

RiskRisk(in terms (in terms of of CControl ontrol EEvent vent RRates)ates)

HighHigh

CER= .30CER= .30.30.30

.2.211

LowLow

CER= .10CER= .10

Risk and Responsiveness

These would These would result in the result in the following following reductions in reductions in events:events:

ResponsivenessResponsiveness(in terms of (in terms of RRelative elative

RRisk isk RReduction)eduction)

HighHigh

RRR= RRR= 30%30%

LowLow

RRR= RRR= 10%10%

RiskRisk(in terms (in terms of of CControl ontrol EEvent vent RRates)ates)

HighHigh

CER= .30CER= .30.30.30

.2.211

LowLow

CER= .10CER= .10.10.10

.0.077

Risk and Responsiveness

These would These would result in the result in the following following reductions in reductions in events:events:

ResponsivenessResponsiveness(in terms of (in terms of RRelative elative

RRisk isk RReduction)eduction)

HighHigh

RRR= RRR= 30%30%

LowLow

RRR= RRR= 10%10%

RiskRisk(in terms (in terms of of CControl ontrol EEvent vent RRates)ates)

HighHigh

CER= .30CER= .30.30.30

.2.211

.30.30

.2.277

LowLow

CER= .10CER= .10.10.10

.0.077

Risk and Responsiveness

These would These would result in the result in the following following reductions in reductions in events:events:

ResponsivenessResponsiveness(in terms of (in terms of RRelative elative

RRisk isk RReduction)eduction)

HighHigh

RRR= RRR= 30%30%

LowLow

RRR= RRR= 10%10%

RiskRisk(in terms (in terms of of CControl ontrol EEvent vent RRates)ates)

HighHigh

CER= .30CER= .30.30.30

.2.211

.30.30

.2.277

LowLow

CER= .10CER= .10.10.10

.0.077

.10.10

.0.099

Let’s start with the “ideal” Let’s start with the “ideal”

HighRisk - High Response group HighRisk - High Response group

High risk – high response

High risk – high response

Almost Stat. Sig.

Risk and Responsiveness

SUMMARY:SUMMARY:

Who do you Who do you want in your want in your RCT ?RCT ?

ResponsivenessResponsiveness

HighHigh LowLow

RiskRisk

HighHighIdeal, but often not enough

LowLow

Risk and Responsiveness

SUMMARY:SUMMARY:

Who do you Who do you want in your want in your RCT ?RCT ?

ResponsivenessResponsiveness

HighHigh LowLow

RiskRisk

HighHighIdeal, but often not enough

LowLowWould these pts help ?

Hi Risk-Hi Response + Lo Risk-Hi Response

Lo Risk – Hi Resp

Hi Risk-Hi Response + Lo Risk-Hi Response

Now Stat. Sig !

Risk and Responsiveness

SUMMARY:SUMMARY:

Who do you Who do you want in your want in your RCT ?RCT ?

ResponsivenessResponsiveness

HighHigh LowLow

RiskRisk

HighHighIdeal, but often not enough

LowLowUsually helpful

Risk and Responsiveness

SUMMARY:SUMMARY:

Who do you Who do you want in your want in your RCT ?RCT ?

ResponsivenessResponsiveness

HighHigh LowLow

RiskRisk

HighHighIdeal, but often not enough

Would these pts help ?

LowLowUsually helpful

Hi Risk-Hi Response + Hi Risk-Lo Response

Hi Risk-Hi Response + Hi Risk-Lo Response

Not helpful

Risk and Responsiveness

SUMMARY:SUMMARY:

Who do you Who do you want in your want in your RCT ?RCT ?

ResponsivenessResponsiveness

HighHigh LowLow

RiskRisk

HighHighIdeal, but often not enough

Often not helpful

LowLowUsually helpful

Risk and Responsiveness

SUMMARY:SUMMARY:

Who do you Who do you want in your want in your RCT ?RCT ?

ResponsivenessResponsiveness

HighHigh LowLow

RiskRisk

HighHighIdeal, but often not enough

Often not helpful

LowLowUsually helpful

Would these pts help?

Hi Risk-Hi Response + Lo Risk-Lo Response

Hi Risk-Hi Response + Lo Risk-Lo Response

Makes things much

worse !

Risk and Responsiveness

SUMMARY:SUMMARY:

Who do you Who do you want in your want in your RCT ?RCT ?

ResponsivenessResponsiveness

HighHigh LowLow

RiskRisk

HighHighIdeal, but often not enough

Often not helpful

LowLowUsually helpful

Make things worse!

Taking all comers

If we put all four If we put all four groups of groups of patients into our patients into our trial:trial:

ResponsivenessResponsiveness(in terms of (in terms of RRelative elative

RRisk isk RReduction)eduction)

HighHigh

RRR= RRR= 30%30%

LowLow

RRR= RRR= 10%10%

RiskRisk(in terms (in terms of of CControl ontrol EEvent vent RRates)ates)

HighHigh

CER= .30CER= .30.30.30

.2.211

.30.30

.2.277

LowLow

CER= .10CER= .10.10.10

.0.077

.10.10

.0.099

All patients

All patients

Worse than if we stuck to just half as many high-

response pts !

Risk and Responsiveness

SUMMARY:SUMMARY:

Who do you Who do you want in your want in your RCT ?RCT ?

ResponsivenessResponsiveness

HighHigh LowLow

RiskRisk

HighHighIdeal, but often not enough

Often not helpful

LowLowUsually helpful

Make things worse!

Summer at Irish LakeSummer at Irish Lake