Minimum run-time requirements to reduce Monte Carlo error in stochastic simulations

Volker Foos1, Phil McEwan4, Adam Lloyd2, James Palmer1, Mark Lamotte3 and David Grant2.1. IMS Health, Basel, Switzerland, 2. IMS Health, London, UK, 3. IMS Health, Brussels, Belgium, 4. Center for Health Economics, Swansea University, UK

Mount Hood 2012 – Friday June 8, 2012

The key consideration of PSA is to assess outcome variability that can be attributed to parameter uncertainty

• In HEOR the role of probabilistic sensitivity analysis (PSA) is to propagate the uncertainty of model predictions with respect to the underlying parameter uncertainty.

• However, in Monte Carlo simulation parameter uncertainty coincides with and cannot be distinguished from stochastic uncertainty (1st order variability/random noise/Monte Carlo error [MCE]).

• The minimum criteria for PSA runs should be therefore to reduce 1st order variability to such an extent that a meaningful assessment of parameter uncertainty is possible.

• In Monte Carlo simulation, MCE can be reduced by increasing the number of patients that are processed through the model.

Background

The overall objective was to demonstrate the relation of 1st and 2nd order variability for different degrees of parameter uncertainty

The overall objective of our investigation was to demonstrate therelation of 1st and 2nd order output variability for

1. Increasing number of patients - to identify the number of patients required to allow a meaningful assessment of parameter uncertainty while keeping within reasonable levels of computation time).

2. Two different degrees by which input model parameters were sampled using

• Standard deviations (SD) representing a larger input parameter variability

• Standard errors (SE) representing lower input parameter variability

Objective

We used a well established and validated diabetes simulation model to illustrate the degree of MCE vs. parameter uncertainty

• The IMS CORE diabetes model (CDM), was used to compare the outcome variability for bootstrap simulations with 1,000 repetitions for patient cohorts ranging from 500 to 100,000.

• Model projections were obtained evaluating the cost effectiveness of two hypothetical interventions with differences in clinical effectiveness of 0.5% HbA1c and a 2kg weight change in favor of the treatment- vs. control arm and a 400 Euro difference in treatment costs.

• Each simulation was performed in three ways

1. no parameter sampling

2. parameters were sampled around 5% of their means (intending to represent variability for SE input)

3. parameters were sampled around 25% of their means (intending to represent variability for SD input)

Materials and Methods

The degree of randomness contained in PSA confidence ranges can be assessed by comparing confidence ranges of sampled vs. non sampled analyses

• Output variability was classified as 1st or 2nd order and the amount of MCE assessed by comparing confidence ranges of selected endpoints in sampling vs. non sampling analyses

• The variability of outputs that is attributed to parameter sampling can be assessed by comparing confidence sizes of 1st order simulations vs. 2nd order simulations.

Materials and Methods

• While stochastic models don’t specifically behave in this predictive way our intention was to estimate the run time (number of patients) required to start making predictive assumptions on the parameter uncertainty in the hypothetical absence of 1st order variability.

3

4

5

6

7

8

9

10

11

12

0 0.5 1 1.5 2 2.5 3 3.5

cum

ula

tive

in

icid

ence

(%

)

Deterministic model

Stochastic model

CI without sampling

CI with sampling

CI attributable to parameter uncertainty

=

}

}

The last assessment can be illustrated as follows…

• While confidence ranges decline with increasing number of patients (thereby reducing MCE), the proportion of the confidence range that is attributable to parameter uncertainty remains stable.

• Therefore the number of patient required to reduce 1st order variability below overall outcome variability has to be identified.

• Depending on the selected endpoint, this assessment may be straight forward or not.

Materials and Methods

d_QAL Y no P S A

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

500 1000 2500 5000 10000 25000 50000 100000

d_QAL Y P S A S E

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

500 1000 2500 5000 10000 25000 50000 100000

IC UR no P S A

-300000

-200000

-100000

0

100000

200000

300000

500 1000 2500 5000 10000 25000 50000 100000

IC UR P S A S E

-300000

-200000

-100000

0

100000

200000

300000

500 1000 2500 5000 10000 25000 50000 100000

IC UR P S A S D

-300000

-200000

-100000

0

100000

200000

300000

500 1000 2500 5000 10000 25000 50000 100000

d_QAL Y P S A S D

-0.6

-0.4

-0.2

-0

0.2

0.4

0.6

0.8

500 1000 2500 5000 10000 25000 50000 100000

We selected two different criteria to determine minimum run time requirements

1. When stochastic uncertainty (CI of non PSA analysis) consistently decreased below overall outcome variability (CI of PSA analyses)

2. When stochastic uncertainty (CI of non PSA analysis) decreased below 50% of overall outcome variability (CI of PSA analyses)

• We focused on confidence ranges of the ICER per QALE gained

• The overall criteria was that the prediction of stochastic uncertainty was reliable so the confidence range in non PSA runs decreased with increasing number of patients.

Materials and Methods

Based on the 1st criteria, analyses of the confidence ranges suggested a minimum number of 10,000 patients required to run over 1000 bootstrap repetitions

Results

• Confidence ranges in non PSA runs demonstrated an irregular pattern (increased or maintained stable) with increasing patient number until 10,000 included patients.

• A clearer picture could have been obtained by increasing the number of bootstrap repetitions

• Beyond 10,000 included patients, both confidence ranges for SE and SD based PSA were consistently lower vs. those obtained in non PSA

Incremental costrs per QALY - Width of confidence ranges in "no PSA", PSA SE" and "PSA SD" analyses

0

50000

100000

150000

200000

250000

300000

350000

400000

450000

500 1000 2500 5000 10000 25000 50000 100000

Confidence width

Pat

ien

ts (

N)

Width - No PSA

Width - PSA SE

Width - PSA SD

When minimum run time requirements were defined according to the 50% threshold of stochastic uncertainty contained in overall uncertainty the number of required patient runs increased

Results

• The proportion of stochastic uncertainty contained in overall ICER variability was obtained from the ratio of confidence sizes of the no PSA vs. PSA runs.

• Run time requirements For PSA based on SE input remained at 10,000 included patients.

• When SE inputs were used in PSA the minimum run time requirement increased to 50,000 patients

• The question remains which part of the overall uncertainty is attributable to 1st order and 2nd order.

Proportion of MCE contained in overal ICER variability

86.2%

96.7%

110.3%106.8%

72.6%

53.7%

45.0%

32.2%

86.2%91.8%

79.6% 79.6%

36.6%

12.8%9.4%

5.7%

0

0.2

0.4

0.6

0.8

1

1.2

500 1000 2500 5000 10000 25000 50000 100000

Patients (N)

% M

CE

% of no PSA in PSA SE

% of no PSA in PSA SD

10

A theoretical exercise was conducted to to evaluate the implications two overlaying distributions (random noise and parameter uncertainty) on the joint distribution

• 10000 random draws were generrated in MS Excel with:

• A) mean = 50 +/- 5

• B) mean = 0 +/- 10 to 0.5

• C) Joint distributio was calculated as A+B

• The SD of the joint distributio was compared to SD‘s of distribution B to assess % overestimation

T wo overlaying dis tributions (random nois e and parameter unc ertainty) and the joint dis tribution

0

2

4

6

8

10

12

89% 81% 71% 67% 63% 57% 52% 45% 37% 29% 20% 10%

Deg ree (%) of ra ndom noise within overa ll va ria bility

Sta

nd

ard

Dev

iati

on

A ) V ariability due to parameters amplingB) Random nois e

C) J oint Dis tribution

15%

50%

The proportion of the confidence range attributable to parameter uncertainty was estimated from the relationship between 1st and 2nd order outcome variability obtained from this theoretical exercise

• For SE based PSA the proportion of parameter uncertainty from overall uncertainty was 88.9% (x confidence width)

• For SD based PSA the proportion was 93.3%

R elations hip between proportions of 1s t order and 2nd order variability from overall outc ome variability

96.85%95.41%89.36%

70.93%66.73%

61.87%57.53%

51.27%

44.10%

37.58%

28.36%

19.61%

10.13%

y = -3.0279x 3 + 3.9569x 2 - 2.0979x + 1.3021R 2 = 0.9935

0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%

0.00% 20.00% 40.00% 60.00% 80.00% 100.00% 120.00%

% parameter uncerta inty from overa ll outcome va riability

Per

cen

t 1s

t o

rder

var

iab

ilit

y fr

om

ove

rall

ou

tco

me

vari

abil

ity

% rnd from all

P oly. (% rnd fromall)

% Parameter uncertainty in overall outcome variability:

SE-PSA: 88.9%

SD-PSA: 93.3%

12

Run time requirements to reduce Monte Carlo error are lower whenever the uncertainty of included parameters is increased

• In an ideal world, an infinitely high number of patients should be applied in stochastic models to reduce random noise to zero so parameter uncertainty can be precisely assessed.

• We have shown that even if stochastic uncertainty is not fully eliminated it is possible to predict the expected degree of parameter uncertainty.

• For this, run time has to be increased to the level that allows a reliable interpretation of confidence ranges.

• The higher the underlying parameter uncertainty, the lower are the requirements to reduce 1st stochastic uncertainty in order to assess parameter uncertainty

• In the CORE Diabetes Model, 10,000 and 50,000 patients were required in bootstrap simulations of 1000 repetitive runs to make reliable assumptions on parameter uncertainty when SD and SE were used to represent parameter uncertainty

Conclusion

Foos et al, ADA Philadelphia, 9 June 2012

13

Thank You!

Foos et al, ADA Philadelphia, 9 June 2012