Calibration plots for risk prediction models in the presence of...
Transcript of Calibration plots for risk prediction models in the presence of...
Calibration plots for risk prediction models in the
presence of competing risks
Thomas A Gerds, Thomas H Scheike, Per K Andersen andMichael W Kattan
June 26, 2014
1 / 28
Motivation: patient counseling
Using a statistical model, a database can be queried to obtain atailored prediction for the present patient.
A predicted risk of 17% is called reliable, if it can be expected thatthe event will occur to about 17 out of 100 patients who allreceived a predicted risk of 17%.
A statistical model that predicts the absolute risk of an eventshould be calibrated in the sense that it provides reliable predictionsfor all subjects.
A calibration plot displays how well observed and predicted eventstatus connect on the absolute probability scale.
2 / 28
Calibration plot
Predicted event probability
0 % 25 % 50 % 75 % 100 %
Obs
erve
d ev
ent s
tatu
s
0 %
25 %
50 %
75 %
100 %
Cause−specific Cox regression
Fine−Gray regression
3 / 28
Predicting absolute risks in time-to-event analysis
First pick a time origin at which it is of interest to predict thefuture status of a patient.
Until time t after the time origin three things can happen:
1. the event has occurred
2. a competing event has occurred
3. the patient is alive and event-free.
The patient needs to know the absolute risks of all events (death,disease, recurrence, etc.).
4 / 28
John Klein's data from bone marrow transplant patients
A data frame with 1715 observations1
Transplant
Relapse Death
n= 557
n= 311
The remaining n = 847 patientswere in remission by the end ofthe follow-up period.
We are interested in predictingthe cumulative incidences ofrelapse and death.
1Szydlo, Goldman, Klein et al. Journal of Clinical Oncology, 1997.5 / 28
Observed outcome
Months since transplantation
Cum
ulat
ive
inci
denc
e
0 12 36 60 84
0 %
25 %
50 %
75 %
100 %
Aalen−Johansen estimate
Event
RelapseDeath without relapse
Months since transplantationC
umul
ativ
e in
cide
nce
0 12 36 60 84
0 %
25 %
50 %
75 %
100 %
Kaplan−Meier estimateof censoring probability
Without covariates the marginal Aalen-Johansen estimate is thebest prediction model.
6 / 28
Formula I
Let X be a vector of covariates:
F1(t|X ) = Cumulative incidence of event 1∫ t
0
exp
(−∫ s
0
{λ1(u|X ) + λ2(u|X )}du)
︸ ︷︷ ︸No event of any cause until s
λ1(s|X )︸ ︷︷ ︸Event type 1 at s
ds.
Requires a regression model for the hazard of the competing risksor a regression model for the event-free survival probability.
7 / 28
Formula II
Transformation model
h(F1(t|X )) = β01(t) + β1X1 + · · ·+ βKXK
I h(p) = log(-log(p)) (Fine-Gray model)
I h(p) = log(p/(1-p)) (Logistic model)
I h(p) = log(p) (Log-binomial model)
Requires a regression model for the cumulative probability of beinguncensored: G(t|X) = P(T>t|X)
in what follows: G(t|X)=G0(t).
8 / 28
Interpretation crisis in competing risks
Problems:
I The hazard ratios obtained by cause-speci�c Cox regressionmodels are not directly related to the prediction of thecumulative incidence.
I The absolute values of the regression coe�cients in theFine-Gray model have no direct interpretation.
Proposal: We are interested in regression models for the absoluterisk of relapse in which the regression coe�cients have thefollowing interpretation:
The 5-year risk of relapse changes with a factor exp(β1) for a one
unit change of X1 and given values for the other predictor variables
(X2, ...,XK ).
9 / 28
Interpretation crisis in competing risks
Problems:
I The hazard ratios obtained by cause-speci�c Cox regressionmodels are not directly related to the prediction of thecumulative incidence.
I The absolute values of the regression coe�cients in theFine-Gray model have no direct interpretation.
Proposal: We are interested in regression models for the absoluterisk of relapse in which the regression coe�cients have thefollowing interpretation:
The 5-year risk of relapse changes with a factor exp(β1) for a one
unit change of X1 and given values for the other predictor variables
(X2, ...,XK ).
9 / 28
Absolute risk regression
The regression parameters in the log-binomial model have thedesired interpretation:
F1(t|X ) = exp(β01(t)) exp(β1X1 + · · ·+ βKXK )
A one unit change of the kth covariate:
F1(t|X1, . . . ,Xk = xk , . . . ,XK )
F1(t|X1, . . . ,Xk = (xk + 1), . . . ,XK )= exp{βk(xk − xk + 1)}
= exp(βk).
10 / 28
Bone marrow transplant data: absolute risk of relapse
Factor exp(β) CI.95 P-value
disease:ALL � � �disease:AML 0.86 [0.68;1.08] 0.1982292disease:CML 0.58 [0.44;0.76] 0.0001017karnofsky 1.3 [1.03;1.68] 0.0253975donor:sibling � � �donor:matched 0.72 [0.55;0.95] 0.0222663donor:mismatched 0.27 [0.13;0.57] 0.0006294stage:early � � �stage:intermediate 1.8 [1.37;2.46] < 0.0001stage:advanced 3.1 [2.47;4.02] < 0.0001timedxtx 0.99 [0.98;1] 0.0219938
E.g., The risk of relapse was estimated as 1.8 times higher for disease stage
intermediate compared to disease stage early.
11 / 28
Does this model �t?
Comparison with common alternatives:
I Combination of cause-speci�c Cox regressions (Formula I)
I Fine-Gray regression model (Formula II: di�erent link function)
I Flexible absolute risk regression: allow time-dependentcovariate e�ects βk(t)
Focus: the validity of the model for prediction
I Personalized: re-classi�cation of predicted probabilities
I Calibration plot: distance between predicted expectedprobabilities
I Brier score: mean squared error for predicted probabilities
12 / 28
Does this model �t?
Comparison with common alternatives:
I Combination of cause-speci�c Cox regressions (Formula I)
I Fine-Gray regression model (Formula II: di�erent link function)
I Flexible absolute risk regression: allow time-dependentcovariate e�ects βk(t)
Focus: the validity of the model for prediction
I Personalized: re-classi�cation of predicted probabilities
I Calibration plot: distance between predicted expectedprobabilities
I Brier score: mean squared error for predicted probabilities
12 / 28
Comparison of predicted probabilities
Predicted risk of relapse within 3 year after transplantation
Absolute risk regression
Gra
y−F
ine
regr
essi
on
0 % 25 % 50 %
0 %
25 %
50 %
●
●
●
●●
●
●●
●●
●●●●●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
●
●●●●
●●
●
●●●
●●
●
●●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●●●
●
●
●●●●
●●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●●
●
●
●
●
●●●●
●●●●
●
●
●
●
●●
●●
●
●
●
●●
●●
●
●●
●●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●●
●
●●
●
●●●
●
●
●
●
●
●●
●
●
●
●●●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●●●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●●
●
●●●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●●
●
●●●●
●
●●
●
●●
●●
●
●
●●●
●
●●●
●
●
●●●●
●●
●●
●●●●
●●
●
●
●
●●
●
●
●
●●
●●●
●
●●●●
●
●
●
●●●●●
●●●
●●
●●
●●
●
●
●
●●●
●
●
●●
●
●
●●
●
●
●
●●●●
●●●
●
●
●
●●●●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●●●
●●
●●●
●
●
●
●●
●
●
●
●●
●●
●●
●
●
●
●●
●
●
●●
●●●
●
●
●●
●
●
●●●
●●
●
●
●
●
●●●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●●
●●
●●
●
●
●
●
●●
●
●
●●●
●●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●●
●
●
●
●
●
●●
●
●●●●
●
●
●●●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●●
●
●
●
●
●
●●●
●
●
●
●
●
●●●●
●
●
●
●
●●
●●●
●●
●
●
●●
●
●●●●
●
●●●●
●
●●●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
●
●●●●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●
●●
●
●
●
●
●
●●●
●
●●
●
●●
●
●●●
●
●
●
●●
●●
●
●
●
●
●
●●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●●●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●●●
●
●
●●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●●●
●
●●
●
●●
●
●●●●
●
●●●
●
●
●
●●●●
●
●
●
●
●
●
●
●●●●
●
●
●●
●
●●
●●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●●●●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●●
●
●●
●
●
●
●●●
●
●●●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●●
●●●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●●●
●
●
●●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
Absolute risk regression
Cau
se−
spec
ific
Cox
reg
ress
ion
0 % 25 % 50 %
0 %
25 %
50 %
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●●●
●
●
●●●●
●
●
●
●●●
●●
●
●●
●
●
●
●
●
●●●
●●
●
●●
●
●
●●●●
●● ●●●●●●●●
●
●●
●
●
●● ●●●
●
●
●
●
●●
●
●●
●●
●
●
●●
●
●
●
●
●
●● ●●●
●
●
●
●
●
●
●●
●●
●●
●● ●●
●
●
●
●●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●●●●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●●●●
●
●
●
●
●●●●●●●●●
●●
●
●● ●●
●
●
●
●●
●●
●●●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●● ●●●●●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●●
●
●
●
●●
●
●●
●
●
●●
●●
●
●●
●●
●●●
●
●
●
●●●
●
●
●●●
●●●
●
●
●
●
●●
●
●●●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●●
●●●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●●●● ●●●
●●
●
●
●●
●●
●●
●
● ●●●● ●●●
●
●
●●●● ●●●● ●●●● ●
●
●
●
●
●●
●●
●
●●
●
●
●
● ●●●●
●
●●
●●●●●
●●●
●
●
●●
●●● ●● ●●
●
●
●●●
●
●
●●
●
●
●●●●●●
●● ●
●
● ●●●●●●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●● ●●
●
●
●●
●
●
●●●
●●●●●
●
●
●●●
●
●
●
●●
●●
●
●
●●
●●●
●
●
●●
●●●
●●
●●
●
●
●●●
●●
●●
●
●
●
●
●
●
●●●
●
●
●● ●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●●
●●
●
●●
●
●
●●
●●
●●
●
●
●
●
●●
●
●
●●●
●●
●●●● ●●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●●●
●
●●
●●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●●●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●●
●
●●●●
●●
●
●
●
●●●●
●
●
●
●
●●
●●●
●●●
●
●●
●
●●●●
●
●●●●
●
●●●
●
●
●
●●
●●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●●
●
●●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●●
●
●
●
●●●
●
●
●●●
●
●
●
●
●●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●●
●●
●●
●●
●●
●●
●
●
●●
●
●●
●●
● ●●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●●
●
●
●
●●●
●
●●
●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●●
●●●
●
●●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●●●●●
●
●●●
●
●
●
●●●●
●
●
●
●
●
●
●
●●●●
●
●
●●
●
●●
●●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●
●●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●●●●●
● ●
●
●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●● ●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●
●
●
●
●
●
●
●●●
●
●
●
● ●●
●●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●●
●
● ●●●
●
●
●
●●●
●●
●
●●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●●
●
● ●●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
Absolute risk regression
Tim
e−de
pend
ent e
ffect
s
0 % 25 % 50 %
0 %
25 %
50 %
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●●
●
●
●●●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●●●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
● ●
●●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●●●
● ●●● ●●
●●
●
●
●●●
●
●●●
●
●
●●●
●
●
●
●●
●●
●●
●
●
●
●
●●●●
●●
●
●
●
●
●
●
●●●●
●
●
●
●●●●●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●●●
●
●
●●
●●●
●
●
●
●●●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●●
●●●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●
●●●
●
● ●
●
●
●
●
●●●
●●
●●●
●
●
●●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●●●
●
●
●
●●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●● ●
●●
●●●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●●
●●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●●●
●
●●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
● ●
●
●
●
●●●
●
●
●
●●
●
●●●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●●●●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
● ●●
●
●●
●
●
●
●
●
●
●●●
●
●
●
● ●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
Risk re-classi�cation plots13 / 28
Calibration curve
Ingredients:
I The event status indicator variable:
N(t) = 1{T ≤ t,D = 1}
I The risk prediction model:
r(t|X ) ∈ [0, 1]
I The risk group at p ∈ [0, 1]
Gr (t; p) = {x ∈ Rd : r(t|x) = p}
The calibration curve at time t:
p 7→ C (p, t, r) = E{N(t) | r(t|X ) = p}.= E{N(t) | X ∈ Gr (t; p)}
14 / 28
Calibration curve
Ingredients:
I The event status indicator variable:
N(t) = 1{T ≤ t,D = 1}
I The risk prediction model:
r(t|X ) ∈ [0, 1]
I The risk group at p ∈ [0, 1]
Gr (t; p) = {x ∈ Rd : r(t|x) = p}
The calibration curve at time t:
p 7→ C (p, t, r) = E{N(t) | r(t|X ) = p}.= E{N(t) | X ∈ Gr (t; p)}
14 / 28
Estimation
To obtain the graph we need to estimate the expectation
E{N(t) | X ∈ Gr (t; p)}
Three often encountered practical problems arise:
I Right censoring: if patient i is not followed until time t, thestatus Ni (t) is unknown.
I Continuity: the size of the sets Gr (t; p) may be small and itmay happen that a set includes only a single patient.
I Generalizability: we would like to know if the model will bereliable for new patients, not those in the data set which wasused to specify and estimate the models.
15 / 28
Estimation
To obtain the graph we need to estimate the expectation
E{N(t) | X ∈ Gr (t; p)}
Three often encountered practical problems arise:
I Right censoring: if patient i is not followed until time t, thestatus Ni (t) is unknown.
I Continuity: the size of the sets Gr (t; p) may be small and itmay happen that a set includes only a single patient.
I Generalizability: we would like to know if the model will bereliable for new patients, not those in the data set which wasused to specify and estimate the models.
15 / 28
Estimation approach
I Right censoring: if patient i is not followed until time t, thestatus Ni (t) is unknown:JACKNIFE PSEUDO-VALUES
I Continuity: the size of the sets Gr (t; p) may be small and itmay happen that a set includes only a single patient:NEAREST NEIGHBORHOOD SMOOTHING
I Generalizability: we would like to know if the model will bereliable for new patients, not those in the data set which wasused to specify and estimate the model:BOOTSTRAP-CROSSVALIDATION
16 / 28
Estimation approach
I Right censoring: if patient i is not followed until time t, thestatus Ni (t) is unknown:JACKNIFE PSEUDO-VALUES
I Continuity: the size of the sets Gr (t; p) may be small and itmay happen that a set includes only a single patient:NEAREST NEIGHBORHOOD SMOOTHING
I Generalizability: we would like to know if the model will bereliable for new patients, not those in the data set which wasused to specify and estimate the model:BOOTSTRAP-CROSSVALIDATION
16 / 28
Estimation approach
I Right censoring: if patient i is not followed until time t, thestatus Ni (t) is unknown:JACKNIFE PSEUDO-VALUES
I Continuity: the size of the sets Gr (t; p) may be small and itmay happen that a set includes only a single patient:NEAREST NEIGHBORHOOD SMOOTHING
I Generalizability: we would like to know if the model will bereliable for new patients, not those in the data set which wasused to specify and estimate the model:BOOTSTRAP-CROSSVALIDATION
16 / 28
Estimated calibration curve
Can,B(p, t, r) =1
n
n∑i=1
1
mi
∑b:i∈Vb
Ni(t)Kan(p, rb(t|Xi)) .
17 / 28
Estimated calibration curve
Can,B(p, t, r) =1
n
n∑i=1
1
mi
∑b:i∈Vb
Ni(t)Kan(p, rb(t|Xi)) .
I Ni (t) = jacknife pseudo value for event status at time t based onAalen-Johansen estimate of E(N(t))
I Kan(p,q)= smoothing kernel
I an = bandwidth
I B = number of bootstrap splits: Data = Lb + Vb
I rb = model �tted in learning sample Lb
I mi = the number of splits where patient i is in Vb
I rb(t,Xi ) = prediction for patient in validation sample Vb.
18 / 28
Estimated calibration curve
Can,B(p, t, r) =1
n
n∑i=1
1
mi
∑b:i∈Vb
Ni(t)Kan(p, rb(t|Xi)) .
I Ni (t) = jacknife pseudo value for event status at time t based onAalen-Johansen estimate of E(N(t))
I Kan(p,q) = smoothing kernel
I an = bandwidth
I B = number of bootstrap splits: Data = Lb + Vb
I rb = model �tted in learning sample Lb
I mi = the number of splits where patient i is in Vb
I rb(t,Xi ) = prediction for patient in validation sample Vb.
19 / 28
Estimated calibration curve
Can,B(p, t, r) =1
n
n∑i=1
1
mi
∑b:i∈Vb
Ni(t)Kan(p, rb(t|Xi)) .
I Ni (t) = jacknife pseudo value for event status at time t based onAalen-Johansen estimate of E(N(t))
I Kan(p,q)= smoothing kernel
I an = bandwidth
I B = number of bootstrap splits: Data = Lb + Vb
I rb = model �tted in learning sample Lb
I mi = the number of splits where patient i is in Vb
I rb(t,Xi ) = prediction for patient in validation sample Vb.
20 / 28
E�ect of censoring: 3 months after transplantation
Relapse
Predicted event probability
0 % 25 % 50 % 75 % 100 %
Pse
udo−
obse
rved
eve
nt s
tatu
s
0 %
25 %
50 %
75 %
100 % ● ●●●●● ●●●● ●● ● ●●● ●●●●
●●
●
●●●●
●● ●
●
●
● ●●●
●
●●● ●
●
● ●●●
●● ●
●●● ●
●
● ● ●● ●● ●●
● ● ●●
●●
●
●● ●●● ● ●●● ●
●
●● ●●● ●
●●
●●
●
●
●●● ●●●
●●
●●● ●●● ●●
●
●●● ●●●●● ●
●
●●● ●●●●●
● ●
● ●●● ●●●●●●●
● ●●
●● ●●●●●
●●● ●●
● ●●●
●
● ●●● ●●● ●●
●
● ●
●
● ●●●
●
●
●
●●● ●
● ●
●●●●● ●
●
●
●
● ●●
●
●● ●●● ● ●●●●●
●
●●
●
●
●●
●
●
●● ●●● ●●●●●● ●●● ●●
●
●●●● ●●●
●●
●
●●
●
●
●●●
●●●●●●
●
● ●●●● ●●
●
●
●
●●
● ●
●●● ●●●
●
●●● ●●●
●
●
●
●
●
●
●
●●●
● ●●●●
●
●● ●
●
●●●●
●●
●●● ●● ●●●
●●
●
●●●● ●●
●
●● ●●● ●●●
●
●●●●●● ●●●
●
●● ● ●
●
●●●●● ●●●●
●
●●
●●
●●●●●
●
●
●●
●●
●
●●● ●●●
●
●
●
● ● ●
●
●●●● ●●
●
●
●●
●
●
●●●
●
●●●
●
●
●
● ●
●
●●
●●
● ●●●● ●●●●●
●
●
●
●
●
●
●●
● ●
●
● ●● ●●●●●● ●●● ●● ●●●● ●●● ●●● ●●● ●● ● ●● ● ●●●● ●●● ●●●● ●●● ●●● ●● ●● ●● ●●●● ●●● ●●●●● ●●●● ● ●●●●● ● ●●● ● ●●●●● ●●●●●●●●● ●●●●●● ●●●●● ●●● ●●●●●● ● ●●●●●● ●●●●● ●● ●●●● ●●●●● ●● ● ●● ●● ● ●●●●● ●●●● ●●●● ●●●● ●●●●● ● ●●●●●●●●●●● ●●●●●●● ●●●● ●● ●● ●● ●● ● ●●● ● ●●●● ● ●● ●● ●● ●●● ●● ●●●● ●●●●●●●●● ●●● ● ●● ●● ●●●● ●●● ●● ●●●●● ●●●● ●●●●● ●● ●●● ●●●●●● ●●●●● ● ●●● ●●●●●● ● ●●● ●● ●● ●●●● ●●●● ●●● ● ●● ●●●● ●● ●●●●● ●●● ● ●●●●●●●● ●●●●●●●● ● ●● ●●●●●●● ●●●● ●● ●●●●● ●● ●●●●● ●●●● ●● ●●●●●● ●●● ●●● ●●●● ●● ●● ●●● ●●● ●●●●● ●● ●●●●●●● ●●● ●● ● ●●● ●● ●●●● ●●● ●● ●●● ●● ●● ●●●● ●●●●● ●●●● ● ●●● ●● ●●●● ●●●●●●●●●●●● ●●●● ●●●● ●●● ●●●● ●●● ●●●●● ●●●●●●●●●●●●● ●●●●●● ●●●●● ●●●●● ●●●● ●● ●●● ●●●● ●●●●●●● ●●●●●●●● ●●●●●●● ●● ●●● ●● ● ●●●● ●● ●●●● ●● ●●●● ●●● ●●●●●●● ●●●● ●●●● ●●●●●●●●●●●●●● ●● ●● ●●●● ●●●● ●●●●●●●●●● ●●●● ●●●●●● ●●●●● ●●●●●●●●● ●●●● ●●●●●●● ●● ●●●●●●●● ●●●● ● ●●●●●●●●●●● ●●●●●●●● ●●●●●●●●●●●●● ●● ●● ●●●●●●● ●●●●● ● ●● ●● ●●●● ●●●● ●● ●●●●●● ●●● ●● ●● ●●●●●●●●●● ●●●●●●●●●● ●●● ●●●●● ● ●●●●●●●●●●●● ●●●●●●●●● ●●● ●●●●●● ●●●● ●●●●●●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●●●●● ●● ●●●●●●●● ●●●●●●●●●●●● ●● ●●● ●●● ●● ●●●●● ●●●● ●●●●● ● ●● ●●●●●● ●●●●●●●●●●●● ●●●● ●●●●●●●●●●●●● ●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●● ●●● ●●●●●● ● ●●●●●●●●●● ●●● ●●●●● ●●● ●●●●●●●● ●●● ●●●●●●●●●●●●●●●● ●● ● ●● ●●● ●●●●●●●●● ●●● ●●● ●●●●●●●● ●●●●● ●● ●●●● ●● ●●● ●●●●●● ●●● ●●●●● ●● ●●●●●●● ●●●●●●●●●●●● ●●●●● ●●●●●●●●●●● ●● ●●● ●●● ●●●●●● ● ●●●●● ●●● ●●● ●●●●●● ●●●●● ●●●● ●●●●● ●●●●●●●●●●●● ●●●●●● ●● ●●●●●● ●●●●●●●●●●●●●●●●● ●●●●●●●●●●● ●●● ●●●●●●●●●●● ●● ●●●●● ●●●●●● ●
Death without relapse
Predicted event probability
0 % 25 % 50 % 75 % 100 %
Pse
udo−
obse
rved
eve
nt s
tatu
s
0 %
25 %
50 %
75 %
100 %
● ●●● ●●●●●●● ●●●● ●●● ● ●
●●
●
●●● ●
● ●●
●
●
●●● ●
●
●●●●
●
● ●●●
●● ●
● ●●●
●
●●●● ● ●●●
● ●●●
●●
●
● ● ●● ●●● ●● ●
●
● ●●● ●●
● ●
● ●
●
●
● ●●●●●
●●
● ● ●●● ●● ●
●
● ●● ●● ●● ●●
●
● ●● ● ●● ●●
●●
●● ●●● ●●● ●●
●
●●
●
● ●● ●● ●
●
● ●●●
●
●● ●●
●
●● ●●● ●●● ●
●
● ●
●
●●
●
●
●
●
●
●● ●●
● ●
●●
●
●●●
●
●
●
●● ●
●
●●● ● ●●●● ●● ●
●
●
●
●
●
●●
●
●
●●●● ●●●● ●● ●● ●● ● ●
●
●
● ●●● ●●
●
●
●
●●
●
●
●● ●
●● ●●●●
●
● ●● ● ●● ●
●
●
●
● ●
● ●
● ● ●●● ●
●
●●●● ●●
●
●
●
●
●
●
●
● ●
●
●●●●●
●
●●●
●
●●●●
●●
●● ●● ●● ●
●
● ●
●
●● ●●● ●
●
●●●●●● ●●
●
●● ●●● ●●● ●
●
● ●● ●
●
●●●● ●●●● ●
●
●●
●●
● ●● ●●
●
●
●●
● ●
●
●● ●● ●●
●
●
●
●●●
●
● ● ●●● ●
●
●
●●
●
●
● ● ●
●
●●●
●
●
●
●●
●
●●
●●
●●● ●●● ●●● ●
●
●
●
●
●
●
● ●
●●
● ●●●● ●●● ●● ●● ●●●●● ●●● ●● ●● ●● ● ●● ●●● ●● ● ●●● ● ●●●● ●●●●● ●● ●●● ●● ●●● ●●●● ●●● ●●●● ●●● ●●●● ●●●●●●●●● ● ●●●● ●●● ● ●●●●●● ●● ●● ●●●● ●● ● ●● ●●● ● ●●● ● ●●● ●● ●●●● ●●●● ● ●●● ●● ●● ●● ●●●● ●●● ●● ●●● ●●● ●●●● ●●●● ●●● ●● ● ● ●●●●●●● ●●●●●● ●●●●●● ●●●● ●●●●● ●●● ●● ●●● ● ●● ●●● ●●● ●●● ●●● ●● ● ● ●●●● ● ● ●●●● ●●● ● ●● ●●● ●●● ●●●●● ●● ●●● ●●●●● ●●● ●● ●●●● ●● ●●●●● ●● ●●● ● ●●● ● ●● ● ●● ● ●● ●● ●●● ●● ●●● ● ●● ●●● ●● ●● ●● ●● ●●●● ●● ●●● ●●●●●●●● ● ●●● ●●●●● ●●● ●●● ●●● ●● ●●●● ●●●●● ●●●● ●●● ●● ●● ●● ●● ● ●● ●● ● ●●●● ●●● ●● ●●● ● ●● ●● ●● ●● ● ●●● ●●●●●●●● ●●● ●● ●●●●●● ●● ●● ●●●● ● ●●● ●● ●●● ●●● ●●●● ●●●● ●● ●● ●● ● ●●● ●● ●●● ● ●● ●●● ● ●●● ● ●● ●●● ●●●●●● ●●●● ●●●● ●●●● ●●●●● ●●●● ●● ●● ●● ●●●● ●● ●●● ●●● ● ●● ●● ● ●●● ●●●● ● ●●●● ● ●●● ●●●● ●●● ●● ●● ●● ●●●●●●●●●● ●●● ●●● ●●●● ● ●●●● ● ●● ●● ●●●● ● ● ●● ●●● ●●●●●● ●●●● ● ●●● ● ●●● ●● ●●● ●●●●●● ●●● ●●● ●● ● ●● ● ●● ●●●● ● ●● ●●●●●● ● ●●● ●● ●●●●● ●●●●●●● ● ●●●● ● ●●●● ●● ●●●●● ●●●● ●● ●●●●●● ●●● ● ●●●●● ●●● ●●● ● ●● ●●● ● ● ●●● ●● ● ●●●●● ●●●● ● ●● ●● ●●●● ●● ●● ●● ●●●●●● ●●●●●● ●●●●●● ●●● ●●● ●●●● ●● ●● ●● ●●●● ●● ●●● ●● ●● ●● ●●●● ●● ●●●● ●● ●●● ● ●●●●●● ●●●● ● ●●●● ● ●●●● ●● ●●● ●● ● ●●● ●●●● ●●●●● ●●●● ●●●● ● ●●●● ●●●●● ●● ● ● ● ●● ●●● ●●● ●● ●●● ●● ●●● ●●●●●●●●●● ●● ●●● ●● ●●●●●● ●● ●●● ●●●●● ●●●● ●● ●● ●● ● ●●● ●●● ●●● ●●●●●● ●●●●●● ●●●● ● ●●● ●●● ●●●●●●●● ●●● ●● ●●● ● ●●● ● ● ●●● ●● ●● ● ●●● ●● ●●●●●●●●●● ●● ●●● ●●●●●● ●●● ●●● ●●●●●●● ●●● ●●● ●●●● ●●● ● ●●● ●●● ● ●● ●● ●●● ●●●● ●●●●● ● ●●●●●●● ●●● ●● ● ●●●●●● ●●●● ●●● ●● ●●●● ●● ●●● ●●●● ●● ●● ●● ●●●● ●●● ●● ●●● ●● ●● ●●● ●●● ●●● ●●● ●●●●●●●●● ● ●● ●● ●●●● ●●● ●●●●●● ●●●● ●●● ●●●●● ●●●●● ●●●● ●● ●●● ●●●● ●●●● ● ●● ●●● ●●●● ●●●●● ● ●●● ●● ●● ●● ●●● ●●●●● ●● ●
Figure: Risks predicted by two independent absolute risk regression
models, one for relapse and one for death without relapse.
21 / 28
E�ect of censoring: 1 year after transplantation
Relapse
Predicted event probability
0 % 25 % 50 % 75 % 100 %
Pse
udo−
obse
rved
eve
nt s
tatu
s
0 %
25 %
50 %
75 %
100 % ● ●●●●● ● ●●● ●● ● ●●● ●●●●
●●
●
● ●●●
●● ●
●
●
● ●●●
●
●●● ●
●
● ● ●●
●● ●
●● ● ●
●
● ● ●● ●● ● ●
● ● ●●
●●
●
●● ●●● ● ●●● ●
●
●● ●●● ●
●●
●●
●
●
●●● ●●●
●●
●●● ●●● ●●
●
● ●● ●● ●●● ●
●
● ●● ●● ●●●
● ●
● ●●● ●●● ●●●
●
● ●
●
● ● ●●●●
●
●●● ●
●
● ●●●
●
● ●●● ●●● ●●
●
● ●
●
● ●
●
●
●
●
●
●●● ●
● ●
●●
●
●● ●
●
●
●
● ●●
●
●● ●●● ● ●●●●●
●
●
●
●
●
●●
●
●
●● ●●● ●●●●● ● ●●● ●●
●
●
●●● ●●●
●
●
●
● ●
●
●
● ●●
●● ●●● ●
●
● ●● ●● ●●
●
●
●
●●
● ●
●●● ●● ●
●
●●● ●●●
●
●
●
●
●
●
●
●●
●
● ●●●●
●
● ● ●
●
●● ●●
● ●
●●● ●● ●●
●
● ●
●
●●●● ●●
●
●● ●●● ●●●
●
●●●●●● ●●●
●
●● ● ●
●
● ●●●● ●●●●
●
● ●
●●
●●●●●
●
●
●●
●●
●
●●● ●●●
●
●
●
● ● ●
●
●●●● ●●
●
●
●●
●
●
●●●
●
●●●
●
●
●
● ●
●
● ●
●●
● ●●●● ●●● ●●
●
●
●
●
●
●
●●
● ●
●
● ●
● ●
● ●●● ●
●
●● ●
● ●●
●● ●●●
●
●● ●
●
●
●
● ●
●
● ●
●
●●
●
●
●
● ● ●●
●
●
●
● ●●
●
●
●
●●
●
● ●●●● ●●● ●
●
●●
● ●●●
●
● ●
●
●
●●
● ●
●●
● ●
●
●●
●
●●
●
●● ●●
●● ●
●●● ●
● ●
●●●● ●
●
●
●
● ●
●
●●
●
●
●●
●●
●
●
●●●● ●
●
●
●●
●
●●
●
●●
●●
●
●
●
●
● ●
●
●● ●●
●●
●
●
●●
●● ●● ●
●
●●●
●
● ● ●●
● ●
●●●
●
●
●●
●
●
●
●
●● ●
●
● ●
●
●
● ●
●
●
●
●● ● ●
●●
●
●
●
●● ●
●
● ●
●
●
●
●●
●
●
● ●●
●●
● ●●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
● ●●
●
●
● ●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
● ●
● ●●
● ●●
● ●
●
●
● ●
● ●
●
●●●●
●●
●● ●●● ●
●
● ●●●● ●
●
●
● ●●●
●
●
●
●
●●●
● ●
●●●
●
●● ●●
●
●● ●
●●
●●
●● ●
●
●
●
●
●
●●● ●●●●● ●●
●
●
●
●
● ●
●
●
●
●
● ●● ●●
●●
●●
● ●●●
●
● ●●
●
●
●●
●●
●
●
●
● ●
●
●●●
●
●
●
●
●
●●●
● ●●
●
●● ●
●
●
● ●● ●●
●
● ●
●
● ●● ●● ● ●● ●● ●●●● ● ●● ●● ●●●● ● ●●● ●● ●● ●● ●●●●●● ● ●● ● ●● ●●●● ●●●● ●● ● ●● ●● ●● ● ●●●●● ● ●● ● ●● ●●● ●● ●● ●● ●●●● ●●●●● ●● ●●● ●●● ● ●● ●●● ●●●● ●●●● ●●● ● ●●●● ●●● ●●●●● ●● ●● ●● ● ●● ● ●●●● ●● ●●●● ●● ●●●● ●●● ●●● ●●●● ●●● ● ●● ●● ●●● ●●● ●●●●● ●●● ●● ●● ●● ●● ●●● ● ●● ● ●●●●●●● ●●●● ●●● ●●● ● ●●●● ● ●●●●● ●● ● ●●● ● ●●●● ●● ● ●● ●●●●●●●● ●● ●● ● ●●● ●●●●● ●●● ●●●● ●●●● ●●● ●● ●●●●● ●●● ●● ●● ●●●● ● ●● ●●● ●● ● ●● ●● ●●●● ●●●● ●● ●●●● ●● ●●● ●● ●● ●●●● ●●●●●● ●●●● ●●●●●● ●●● ● ●● ●● ● ●● ●●● ●●● ●●● ● ●●● ●● ●●●● ●● ● ●● ●●●● ●●●● ●●●●● ●●●●●●● ●● ● ●●●●●● ●● ● ●● ●● ●●●●● ●●●● ●● ●●● ●●●●● ●●●●●●●●●● ●● ●● ●●● ●●● ●● ●●●●● ●● ●● ●●●●● ● ●● ●●●● ●● ●●● ●● ●●● ●●● ● ●●●● ●● ●● ●● ●●● ●● ●● ●● ●● ●● ●●●●●● ●●●●●● ●●● ●●●●●● ●●●● ●●● ●●● ●●● ● ●●●● ● ●●●●● ●●● ●●● ●● ●●● ●●●●●●●● ●●● ●● ●●●●●●● ●●● ● ●●● ●● ● ●● ●● ● ●●●● ●●●●● ●●● ●●● ●●●● ● ●●● ●●●● ● ●● ●●●● ●● ●●● ●● ●● ●● ●●● ●●●●● ●● ●●●● ● ●● ●●●● ●●●● ● ●● ● ●● ●●● ● ●●●●●● ●●●● ●● ●●● ●●● ●●●●●● ● ●●● ● ● ●●● ●●● ●●● ●●● ●●●●● ●● ●● ●●● ●● ●●● ●● ● ●●●● ●● ●●● ●●● ●● ● ●●●●● ●● ● ●●●● ●●●● ●●● ●●● ●●●●● ● ●●●●● ●●● ●●● ●● ●●●●●● ●● ●●●●● ●●●●● ● ●
Death without relapse
Predicted event probability
0 % 25 % 50 % 75 % 100 %
Pse
udo−
obse
rved
eve
nt s
tatu
s
0 %
25 %
50 %
75 %
100 %
● ●●● ●●●●●●● ●●●● ●●● ● ●
●●
●
●●● ●
● ●●
●
●
●●● ●
●
●● ●●
●
● ●●●
●● ●
● ●●●
●
●●●● ● ●●●
● ●●●
● ●
●
● ● ●● ●●● ●● ●
●
● ●●● ●●
● ●
● ●
●
●
● ●●●●●
●●
● ● ●●● ●● ●
●
● ●● ●● ●● ●●
●
● ●● ● ●● ●●
●●
●● ●●● ●●● ●●
●
●●
●
● ●● ●● ●
●
● ●●●
●
●● ●●
●
●● ●●● ● ●● ●
●
● ●
●
●●
●
●
●
●
●
●● ●●
● ●
●●
●
●●●
●
●
●
●● ●
●
● ●● ● ●●●● ●● ●
●
●
●
●
●
●●
●
●
●●●● ●●●● ●● ●● ●● ● ●
●
●
● ●●● ●●
●
●
●
●●
●
●
●● ●
●● ● ●●●
●
● ●● ● ●● ●
●
●
●
● ●
● ●
● ● ● ●● ●
●
● ●●● ●●
●
●
●
●
●
●
●
● ●
●
●● ●●●
●
●●●
●
●●●●
●●
● ● ●● ●● ●
●
● ●
●
●● ●●● ●
●
●●●●●● ●●
●
●● ●●● ●●● ●
●
● ●● ●
●
●●●● ●●●● ●
●
●●
●●
● ●● ● ●
●
●
●●
● ●
●
●● ●● ●●
●
●
●
●●●
●
● ● ●●● ●
●
●
●●
●
●
● ● ●
●
●●●
●
●
●
●●
●
●●
●●
●●● ●●● ●●● ●
●
●
●
●
●
●
● ●
●●
●
●●
●●
●●● ●●
●
● ●●
●●●
● ●● ●●
●
● ●●
●
●
●
●●
●
●●
●
●●
●
●
●
●●● ●
●
●
●
● ●●
●
●
●
●●
●
●● ●●●● ●●●
●
●●
● ●●●
●
●●
●
●
●●
●●
●●
●●
●
●●
●
● ●
●
● ● ●●
●● ●
● ●● ●
● ●
●●● ●●
●
●
●
●●
●
● ●
●
●
● ●
●●
●
●
●●●● ●
●
●
● ●
●
●●
●
● ●
● ●
●
●
●
●
● ●
●
● ●● ●
●●
●
●
● ●
●●● ●●
●
● ●●
●
●● ● ●
●●
●●●
●
●
●●
●
●
●
●
●●●
●
●●
●
●
● ●
●
●
●
●● ●●
● ●
●
●
●
● ● ●
●
●●
●
●
●
● ●
●
●
●● ●
●●
● ● ●
●
●
●
● ●
●
●●
●
●
●
●
● ●
●
●
●
● ●●
●
●
●●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
● ●
●
●●
●●
●
●
●
●
●
●
●
●●
● ● ●
●● ●
●●
●
●
● ●
●●
●
● ●●
●
●●
●
●● ● ●●
●
●● ●
● ●●
●
●
●● ●
●
●
●
●
●●●
●
●●
●
●●●
●
● ● ●
●●
●●
●●● ●
●● ●
●●
●●
●●
● ●●●● ●●●●
●
●
●●
● ●●
●
●
● ●● ●●
●
●
●
●● ●●
● ●
●●●
●
●
●
●●
●
●●
●
●●
●
● ●●
●
● ● ●●●
●
●●●●●●● ●●
●
●●
●
●
●
●●
●
●● ●● ●●●● ● ● ●● ●● ●●● ●●● ●●●● ●●●● ●● ●● ●● ● ●●● ●● ●●● ● ●● ●●● ● ●●● ● ●● ●●● ●● ●●●● ●● ● ● ●●● ● ●●●● ●●●●● ●●●● ●● ●● ●● ●●●● ●● ●●● ●●● ● ●● ●● ● ●●● ●●●● ● ●●●● ● ●●● ●●●● ●●● ●● ●● ●● ●●● ●● ●●●●● ●●● ●●● ●●●● ● ●●●● ● ●● ●● ●●●● ● ● ●● ●●● ●●●●●● ●●●● ● ●●● ● ●●● ●● ●●● ●●●●●● ●●● ●●● ●● ● ●● ● ●● ●●●● ● ●● ●●●● ●● ● ●●● ●● ●●●●● ●● ●●●● ● ● ●●●● ● ●●●● ●● ●●●●● ●●●● ●● ●●● ●●● ●●● ● ●● ● ●● ●●● ●●● ● ●● ●●● ● ● ●●● ●● ● ●●●●● ●●●● ● ●● ●● ●● ●● ●● ●● ●● ●●●●●● ●● ● ●●● ●● ●●●● ●●● ●●● ●●● ● ●● ●● ●● ●●●● ●● ●●● ●● ●● ●● ●●●● ●● ●●●● ●● ●●● ● ●●●●●● ●●●● ● ●●●● ● ●●●● ●● ●●● ●● ● ●●● ●●●● ●●●● ● ●●●● ●● ●● ● ●●●● ●●●●● ●● ● ● ● ●● ●●● ●●● ●● ●●● ●● ●●● ●●●●●●● ●●● ●● ●●● ●● ●●●● ●● ●● ●●● ●●●●● ●●●● ●● ●● ●● ● ●●● ●●● ●●● ● ●●●●● ●●●●●● ●●●● ● ●●● ●●● ●●●● ● ●● ● ●●● ●● ●●● ● ●●● ● ● ●●● ●● ●● ● ●●● ●● ●● ●●● ●●●●● ●● ●●● ●●● ●●● ● ●● ●●● ●●●●●●● ●●● ●●● ●●●● ●●● ● ●●● ●●● ● ●● ●● ●●● ●●●● ● ●●●● ● ●●●●●●● ●● ● ●● ● ●●●●●● ●●●● ●●● ●● ●●●● ●● ●● ● ●●●● ●● ●● ●● ●●●● ●●● ●● ●●● ●● ●● ●●● ●●● ●●● ●●● ●● ●●●● ●●● ● ●● ●● ●●●● ●●● ●●●●●● ●●●● ●●● ●●●●● ●●●●● ●●●● ●● ●●● ●●●● ● ●●● ● ●● ●●● ●●●● ●●●●● ● ●●● ●● ●● ●● ●●● ●●● ●● ●● ●
Figure: Risks predicted by two independent absolute risk regression
models, one for relapse and one for death without relapse.
22 / 28
E�ect of censoring: 3 years after transplantation
Relapse
Predicted event probability
0 % 25 % 50 % 75 % 100 %
Pse
udo−
obse
rved
eve
nt s
tatu
s
0 %
25 %
50 %
75 %
100 % ● ●●● ●● ● ●● ● ●● ● ●●● ●●●●
●●
●
● ●●●
●● ●
●
●
● ●● ●
●
●●● ●
●
● ● ●●
●● ●
●● ● ●
●
● ● ●● ●● ● ●
● ● ●●
●●
●
●● ●●● ● ●●● ●
●
●● ●●● ●
●●
●●
●
●
●●● ●●●
●●
●●● ●●● ●●
●
● ●● ●● ●●● ●
●
● ●● ●● ●●●
● ●
● ●●● ●●● ●●●
●
● ●
●
● ● ●●●●
●
●●● ●
●
● ●●●
●
● ●●● ●●● ●●
●
● ●
●
● ●
●
●
●
●
●
●●● ●
● ●
●●
●
●● ●
●
●
●
● ●●
●
●● ●●● ● ●●●●●
●
●
●
●
●
●●
●
●
●● ●●● ●●●●● ● ●●● ●●
●
●
●● ● ●●●
●
●
●
● ●
●
●
● ●●
●● ●●● ●
●
● ●● ●● ●●
●
●
●
●●
● ●
●●● ●● ●
●
●●● ●●●
●
●
●
●
●
●
●
●●
●
● ●●●●
●
● ● ●
●
●● ●●
● ●
●●● ●● ●●
●
● ●
●
●●● ● ●●
●
●● ●●● ●●●
●
●●●●●● ●●●
●
●● ● ●
●
● ●●●● ●●●●
●
● ●
●●
●● ●●●
●
●
●●
●●
●
●●● ●●●
●
●
●
● ● ●
●
● ●●● ●●
●
●
●●
●
●
●●●
●
●●●
●
●
●
● ●
●
● ●
●●
● ●●●● ●●● ●●
●
●
●
●
●
●
●●
● ●
●
● ●
● ●
● ●●● ●
●
●● ●
● ●●
●● ●●●
●
●● ●
●
●
●
● ●
●
● ●
●
●●
●
●
●
● ● ●●
●
●
●
● ●●
●
●
●
●●
●
● ●●●● ●● ● ●
●
●●
● ●●●
●
● ●
●
●
●●
● ●
●●
● ●
●
●●
●
●●
●
●● ●●
●● ●
●●● ●
● ●
●●●● ●
●
●
●
● ●
●
●●
●
●
●●
●●
●
●
●● ●● ●
●
●
●●
●
●●
●
●●
●●
●
●
●
●
● ●
●
●● ●●
●●
●
●
●●
●● ●● ●
●
●●●
●
● ● ●●
● ●
●●●
●
●
●●
●
●
●
●
●● ●
●
● ●
●
●
● ●
●
●
●
●● ● ●
●●
●
●
●
●● ●
●
● ●
●
●
●
●●
●
●
● ●●
●●
● ●●
●
●
●
● ●
●
●●
●
●
●
●
●●
●
●
●
● ●●
●
●
● ●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
● ●
● ●●
● ● ●
● ●
●
●
● ●
● ●
●
●●●
●
●●
●
● ●●● ●
●
● ●●
●● ●
●
●
● ●●
●
●
●
●
●
●●
●
● ●
●
●●
●
●
● ●●
●
●
● ●
●●
●●
●● ●
●
●
●
●
●
●
●● ●● ●●● ●●
●
●
●
●
● ●
●
●
●
●
● ●● ●
●
●
●
●●
● ●
●●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
● ●●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
● ●
●● ●
●
●
●●●
● ●
●
●
● ●●
●●
●
●
●● ●
●
●
● ●
●
●
●●
●
●● ● ●
●
●
●
● ●●
●● ●
●●
●●
●
●
●●●
●●●
● ●●●
●
●●
●
● ●●●
●●● ●● ●● ●●
●
●●● ●●
●● ●
●
● ●
●
● ●
●
●● ●● ●
●
● ●●● ●
●●●● ●●● ● ●
●●● ●
●
●
●●
●●● ●
●
●● ●● ●●
●
●
●
●●● ●● ●●● ● ●●
●
● ●● ●●●●●
●
●
● ●●●
●
●
●
●
●●●
●
●● ●●
● ●
●●●●●
●● ●●
●● ●● ●● ●●● ● ●● ● ●●●
●
●●
● ●
●●● ●●● ●●● ● ●●●● ● ●
●●
●● ●● ●
●
●● ●
●●
●
●
●● ● ●●●
● ●
●●●●
●
●
●
●
● ●
●●● ●●●●● ●●● ●
●
●● ●●●● ●●● ●● ●
●
●●● ●●● ●● ●● ●●●●
●
●● ●●● ●● ● ●● ●● ●●●● ●●●● ●●
●
●●● ●● ●●● ●● ●● ●●●● ● ●●●●● ●●●● ●●●●●● ●●● ●
●
● ●● ●
●
● ●●● ●●● ●●● ● ●●● ●● ●●●● ●● ● ●● ●●●● ●●●● ●
●
●●● ●●●●●●● ●● ● ●●●●●● ●● ● ●● ●● ●●●●● ●●●● ●● ●●● ●●●●● ●●●●●●●●●● ●● ●● ●●● ●●● ●● ●●●●● ●● ●● ●●●●● ● ●● ●●●● ● ● ●●● ●● ●●● ●●● ● ●●●● ●● ●● ●● ●●● ●● ●● ●● ●● ●● ●●●●●● ●●●●●● ● ●● ●●●●●● ●●●● ●●● ●●● ●●● ● ●●●● ● ●●●●● ●●● ● ●● ●● ●●● ●●●● ●●●● ●●● ●● ●●●●●●● ●●● ● ●●● ●● ● ●● ●● ● ●●●● ●●●●● ●●● ●●● ●●●● ● ●●● ●● ●● ● ●● ●●●● ●● ●●● ●● ●● ●● ●●● ●●●●● ●● ●●●● ● ●● ●● ●● ●●●● ● ●● ● ●● ●●● ● ●●● ●●● ●●●● ●● ●●● ●●● ●●●●●● ● ●●● ● ● ●●● ●●● ●●● ●●● ●●●●● ●● ●● ●●● ●● ●●● ●● ● ●●●● ●● ●●● ●●● ●● ● ●●●●● ●● ● ●● ●● ●●●● ●●● ●●● ●●●●● ● ●●●●● ●●● ●●● ●● ●●● ●●● ●● ●● ●●● ● ●●●● ● ●
Death without relapse
Predicted event probability
0 % 25 % 50 % 75 % 100 %
Pse
udo−
obse
rved
eve
nt s
tatu
s
0 %
25 %
50 %
75 %
100 %
● ●●● ●●●●●●● ●●●● ●●● ● ●
●●
●
●●● ●
● ●●
●
●
●●● ●
●
●● ●●
●
● ●●●
●● ●
● ●●●
●
●●●● ● ●●●
● ●●●
● ●
●
● ● ●● ●●● ●● ●
●
● ●●● ●●
● ●
● ●
●
●
● ●●●●●
●●
● ● ●●● ●● ●
●
● ●● ●● ●● ●●
●
● ●● ● ●● ●●
●●
●● ●●● ●●● ●●
●
●●
●
● ●● ●● ●
●
● ●●●
●
●● ●●
●
●● ●●● ● ●● ●
●
● ●
●
●●
●
●
●
●
●
●● ●●
● ●
●●
●
●●●
●
●
●
●● ●
●
● ●● ● ●●●● ●● ●
●
●
●
●
●
●●
●
●
●●●● ●●●● ●● ●● ●● ● ●
●
●
● ●●● ●●
●
●
●
●●
●
●
●● ●
●● ● ●●●
●
● ●● ● ●● ●
●
●
●
● ●
● ●
● ● ● ●● ●
●
● ●●● ●●
●
●
●
●
●
●
●
● ●
●
●● ●●●
●
●●●
●
●●●●
●●
● ● ●● ●● ●
●
● ●
●
●● ●●● ●
●
●●●●●● ●●
●
●● ●●● ●●● ●
●
● ●● ●
●
●●●● ●●●● ●
●
●●
●●
● ●● ● ●
●
●
●●
● ●
●
●● ●● ●●
●
●
●
●●●
●
● ● ●●● ●
●
●
●●
●
●
● ● ●
●
●●●
●
●
●
●●
●
●●
●●
●●● ●●● ●●● ●
●
●
●
●
●
●
● ●
●●
●
●●
●●
●●● ●●
●
● ●●
●●●
● ●● ●●
●
● ●●
●
●
●
●●
●
●●
●
●●
●
●
●
●●● ●
●
●
●
● ●●
●
●
●
●●
●
●● ●●●● ●●●
●
●●
● ●●●
●
●●
●
●
●●
●●
●●
●●
●
●●
●
● ●
●
● ● ●●
●● ●
● ●● ●
● ●
●●● ●●
●
●
●
●●
●
● ●
●
●
● ●
●●
●
●
●●●● ●
●
●
● ●
●
●●
●
● ●
● ●
●
●
●
●
● ●
●
● ●● ●
●●
●
●
● ●
●●● ●●
●
● ●●
●
●● ● ●
●●
●●●
●
●
●●
●
●
●
●
●●●
●
●●
●
●
● ●
●
●
●
●● ●●
● ●
●
●
●
● ● ●
●
●●
●
●
●
● ●
●
●
●● ●
●●
● ● ●
●
●
●
● ●
●
●●
●
●
●
●
● ●
●
●
●
● ●●
●
●
●●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
● ●
●
●●
●●
●
●
●
●
●
●
●
●●
● ● ●
●● ●
●●
●
●
● ●
●●
●
● ●●
●
●●
●
●● ● ●●
●
●● ●
● ●●
●
●
●● ●
●
●
●
●
●
● ●
●
●●
●
●●
●
●
● ● ●
●
●
●●
●●
● ●
●● ●
●
●
●
●
●
●
● ●●●● ●●●●
●
●
●
●
● ●
●
●
●
●
●● ●●
●
●
●
●●●●
● ●
●●●
●
●
●
●●
●
●●
●
●●
●
●●●
●
●●
●●
●
●
●●●●●
●● ●●
●
●●
●
●
●
●●
●
●●
●●
●
●●●● ● ●●
●● ●●
● ●
●
●
●●●
● ●
●●
● ●●
●
●●●●
●●●
●● ●●●
●
●●
●●●● ●●
● ●
●
●
●
●
● ●
●
●
●●
● ●
● ●●
●
●●
● ●
●●
● ●
●●●● ●●●●
●● ●●
●
● ●●●● ●●
●●●
●
●● ● ●●
●●
● ●●
● ●
●
●● ● ●●●
●
● ●●● ●●●● ●●●●
● ●● ●●
●
●● ●● ●●●●● ●●● ●●● ●●●● ● ●
●●
● ● ●● ●
●
●●●●
●
● ●● ●●● ●●●●●● ●
●
●● ●
●
●● ● ●●● ●● ●●● ●●●●●● ●●● ●●● ●● ● ●● ● ●● ●
●
●● ● ●● ●
●
●● ●● ● ●●● ●● ●●●●● ●●
●
●●● ● ● ●●●● ● ●●●● ●● ●●●●● ●●●● ●● ●●● ●●● ●●● ● ●● ● ●● ●●● ●●● ● ●● ●●● ● ● ●●● ●● ● ●
●
●●● ●●●● ● ●● ●● ●● ●● ●●
●
● ●● ●●●●●● ●● ● ●●● ●● ●●●● ●●●
●
●● ●●● ● ●● ●● ●● ●
●
●● ●● ●●● ●● ●● ●●
●
●●● ●● ●●●● ●● ●●● ● ●●●●●● ●●●● ● ●●●● ● ●●●● ●● ●●● ●● ● ●●● ●●●● ●●●● ● ●●●● ●● ●● ● ●●●● ●●●●● ●● ● ● ● ●● ●●● ●●● ●● ●●● ●● ●●● ●●●●●●● ●●● ●● ●●● ●● ●●●● ●● ●● ●●● ●●●●● ●●●● ●● ●● ●● ● ●●● ●●● ●●● ● ●●●●● ●●●●●● ●●●● ● ●●● ●●● ●●●● ● ●● ● ●●● ●● ●● ● ● ●●● ● ● ●●● ●● ●● ● ●●● ●● ●● ●●● ●●●●● ●● ●●● ●●● ●●● ● ●● ●●● ●●●●●●● ●●● ●●● ●●●● ●●● ● ●●● ●●● ● ●● ●● ●●● ●●●● ● ●●●● ● ●●●●●●● ● ● ● ●● ● ●●●●●● ●●●● ●●● ●● ●●●● ●● ●● ● ●●●● ●● ●● ●● ●●●● ●●● ●● ●●● ●● ●● ●●● ●●● ●●● ●●● ●● ●●●● ●●● ● ●● ●● ●●●● ●●● ●●●●●● ●●●● ●●● ●●●●● ●●●●● ●●●● ●● ●●● ●●●● ● ●●● ● ●● ●●● ●●●● ●●●●● ● ●●● ●● ●● ●● ●●● ●●● ●● ●● ●
Figure: Risks predicted by two independent absolute risk regression
models, one for relapse and one for death without relapse.
23 / 28
E�ect of bandwidth: event= relapse, t=36 months
Calibration in the largebandwidth=1
Predicted event probability
0 % 25 % 50 % 75 % 100 %
Pse
udo−
obse
rved
eve
nt s
tatu
s
0 %
25 %
50 %
75 %
100 % ● ●●●●● ● ●●● ●● ● ●●● ●●●●
●●
●
● ●●●
●● ●
●
●
● ●●●
●
●●● ●
●
● ● ●●
●● ●
●● ● ●
●
● ● ●● ●● ● ●
● ● ●●
●●
●
●● ●●● ● ●●● ●
●
●● ●●● ●
●●
●●
●
●
●●● ●●●
●●
●●● ●●● ●●
●
● ●● ●● ●●● ●
●
● ●● ●● ●●●
● ●
● ●●● ●●● ●●●
●
● ●
●
● ● ●●●●
●
●●● ●
●
● ●●●
●
● ●●● ●●● ●●
●
● ●
●
● ●
●
●
●
●
●
●●● ●
● ●
●●
●
●● ●
●
●
●
● ●●
●
●● ●●● ● ●●●●●
●
●
●
●
●
●●
●
●
●● ●●● ●●●●● ● ●●● ●●
●
●
●●● ●●●
●
●
●
● ●
●
●
● ●●
●● ●●● ●
●
● ●● ●● ●●
●
●
●
●●
● ●
●●● ●● ●
●
●●● ●●●
●
●
●
●
●
●
●
●●
●
● ●●●●
●
● ● ●
●
●● ●●
● ●
●●● ●● ●●
●
● ●
●
●●●● ●●
●
●● ●●● ●●●
●
●●●●●● ●●●
●
●● ● ●
●
● ●●●● ●●●●
●
● ●
●●
●●●●●
●
●
●●
●●
●
●●● ●●●
●
●
●
● ● ●
●
●●●● ●●
●
●
●●
●
●
●●●
●
●●●
●
●
●
● ●
●
● ●
●●
● ●●●● ●●● ●●
●
●
●
●
●
●
●●
● ●
●
● ●
● ●
● ●●● ●
●
●● ●
● ●●
●● ●●●
●
●● ●
●
●
●
● ●
●
● ●
●
●●
●
●
●
● ● ●●
●
●
●
● ●●
●
●
●
●●
●
● ●●●● ●●● ●
●
●●
● ●●●
●
● ●
●
●
●●
● ●
●●
● ●
●
●●
●
●●
●
●● ●●
●● ●
●●● ●
● ●
●●●● ●
●
●
●
● ●
●
●●
●
●
●●
●●
●
●
●●●● ●
●
●
●●
●
●●
●
●●
●●
●
●
●
●
● ●
●
●● ●●
●●
●
●
●●
●● ●● ●
●
●●●
●
● ● ●●
● ●
●●●
●
●
●●
●
●
●
●
●● ●
●
● ●
●
●
● ●
●
●
●
●● ● ●
●●
●
●
●
●● ●
●
● ●
●
●
●
●●
●
●
● ●●
●●
● ●●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
● ●●
●
●
● ●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
● ●
● ●●
● ●●
● ●
●
●
● ●
● ●
●
●●●
●
●●
●
● ●●● ●
●
● ●●
●● ●
●
●
● ●●
●
●
●
●
●
●●
●
● ●
●
●●
●
●
● ●●
●
●
● ●
●●
●●
●● ●
●
●
●
●
●
●
●● ●●●●● ●●
●
●
●
●
● ●
●
●
●
●
● ●● ●
●
●
●
●●
● ●
●●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
● ●●
●
●● ●
●
●
●●
●●
●
●
●●
●
● ●●
●
● ●
●● ●
●
●
●●●● ●
●
●● ●●●●
●
●
●● ●●
●
● ●
●
●
●●
●
●● ● ●●
●
●
● ●●
●● ●
●●
●●
●
●
●●●
●●●
● ●●●
●
●●
●
● ●●●
●●● ●● ●● ●●
●
●●● ●●
●●●
●
● ●
●
● ●
●
●● ●● ●
●
● ●●●●
●●●● ●●● ● ●
●●● ●
●
●
●●
●●● ●
●
●● ●● ●●
●
●
●
●●● ●● ●●●● ●●
●
●●● ●●●●●
●
●
●●●●
●
●
●
●
●●●
●
●● ●●
● ●
●●●●●●● ●
●●● ●● ●● ●●● ● ●● ● ●●●
●
●●
● ●
●●● ●●● ●●● ● ●●●● ● ●
●●
●● ●● ●
●
●● ●
●●
●
●
●● ● ●● ●
●●
●●●●
●
●
●
●
● ●
●●● ●●●●● ●●● ●
●
●● ●●●● ●●● ●● ●
●
●●● ●●● ●● ●● ●●●●
●
●● ●●● ●● ● ●● ●● ●●●● ●●●● ●●
●
●●● ●● ●●● ●● ●● ●●●● ●●●●●● ●●●● ●●●●●● ●●● ●
●
● ●● ●
●
● ●●● ●●● ●●● ● ●●● ●● ●●●● ●● ● ●● ●●●● ●●●● ●
●
●●● ●●●●●●● ●● ● ●●●●●● ●● ● ●● ●● ●●●●● ●●●● ●● ●●● ●●●●● ●●●●●●●●●● ●● ●● ●●● ●●● ●● ●●●●● ●● ●● ●●●●● ● ●● ●●●● ●● ●●● ●● ●●● ●●● ● ●●●● ●● ●● ●● ●●● ●● ●● ●● ●● ●● ●●●●●● ●●●●●● ●●● ●●●●●● ●●●● ●●● ●●● ●●● ● ●●●● ● ●●●●● ●●● ●●● ●● ●●● ●●●●●●●● ●●● ●● ●●●●●●● ●●● ● ●●● ●● ● ●● ●● ● ●●●● ●●●●● ●●● ●●● ●●●● ● ●●● ●●●● ● ●● ●●●● ●● ●●● ●● ●● ●● ●●● ●●●●● ●● ●●●● ● ●● ●●●● ●●●● ● ●● ● ●● ●●● ● ●●●●●● ●●●● ●● ●●● ●●● ●●●●●● ● ●●● ● ● ●●● ●●● ●●● ●●● ●●●●● ●● ●● ●●● ●● ●●● ●● ● ●●●● ●● ●●● ●●● ●● ●●●●●● ●● ● ●●●● ●●●● ●●● ●●● ●●●●● ● ●●●●● ●●● ●●● ●● ●●●●●● ●● ●●●●● ●●●●● ● ●
●
Localized calibrationbandwidth=0
Predicted event probability
0 % 25 % 50 % 75 % 100 %
Pse
udo−
obse
rved
eve
nt s
tatu
s
0 %
25 %
50 %
75 %
100 % ● ●●●●● ● ●●● ●● ● ●●● ●●●●
●●
●
● ●●●
●● ●
●
●
● ●●●
●
●●● ●
●
● ● ●●
●● ●
●● ● ●
●
● ● ●● ●● ● ●
● ● ●●
●●
●
●● ●●● ● ●●● ●
●
●● ●●● ●
●●
●●
●
●
●●● ●●●
●●
●●● ●●● ●●
●
● ●● ●● ●●● ●
●
● ●● ●● ●●●
● ●
● ●●● ●●● ●●●
●
● ●
●
● ● ●●●●
●
●●● ●
●
● ●●●
●
● ●●● ●●● ●●
●
● ●
●
● ●
●
●
●
●
●
●●● ●
● ●
●●
●
●● ●
●
●
●
● ●●
●
●● ●●● ● ●●●●●
●
●
●
●
●
●●
●
●
●● ●●● ●●●●● ● ●●● ●●
●
●
●●● ●●●
●
●
●
● ●
●
●
● ●●
●● ●●● ●
●
● ●● ●● ●●
●
●
●
●●
● ●
●●● ●● ●
●
●●● ●●●
●
●
●
●
●
●
●
●●
●
● ●●●●
●
● ● ●
●
●● ●●
● ●
●●● ●● ●●
●
● ●
●
●●●● ●●
●
●● ●●● ●●●
●
●●●●●● ●●●
●
●● ● ●
●
● ●●●● ●●●●
●
● ●
●●
●●●●●
●
●
●●
●●
●
●●● ●●●
●
●
●
● ● ●
●
●●●● ●●
●
●
●●
●
●
●●●
●
●●●
●
●
●
● ●
●
● ●
●●
● ●●●● ●●● ●●
●
●
●
●
●
●
●●
● ●
●
● ●
● ●
● ●●● ●
●
●● ●
● ●●
●● ●●●
●
●● ●
●
●
●
● ●
●
● ●
●
●●
●
●
●
● ● ●●
●
●
●
● ●●
●
●
●
●●
●
● ●●●● ●●● ●
●
●●
● ●●●
●
● ●
●
●
●●
● ●
●●
● ●
●
●●
●
●●
●
●● ●●
●● ●
●●● ●
● ●
●●●● ●
●
●
●
● ●
●
●●
●
●
●●
●●
●
●
●●●● ●
●
●
●●
●
●●
●
●●
●●
●
●
●
●
● ●
●
●● ●●
●●
●
●
●●
●● ●● ●
●
●●●
●
● ● ●●
● ●
●●●
●
●
●●
●
●
●
●
●● ●
●
● ●
●
●
● ●
●
●
●
●● ● ●
●●
●
●
●
●● ●
●
● ●
●
●
●
●●
●
●
● ●●
●●
● ●●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
● ●●
●
●
● ●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
● ●
● ●●
● ●●
● ●
●
●
● ●
● ●
●
●●●
●
●●
●
● ●●● ●
●
● ●●
●● ●
●
●
● ●●
●
●
●
●
●
●●
●
● ●
●
●●
●
●
● ●●
●
●
● ●
●●
●●
●● ●
●
●
●
●
●
●
●● ●●●●● ●●
●
●
●
●
● ●
●
●
●
●
● ●● ●
●
●
●
●●
● ●
●●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
● ●●
●
●● ●
●
●
●●
●●
●
●
●●
●
● ●●
●
● ●
●● ●
●
●
●●●● ●
●
●● ●●●●
●
●
●● ●●
●
● ●
●
●
●●
●
●● ● ●●
●
●
● ●●
●● ●
●●
●●
●
●
●●●
●●●
● ●●●
●
●●
●
● ●●●
●●● ●● ●● ●●
●
●●● ●●
●●●
●
● ●
●
● ●
●
●● ●● ●
●
● ●●●●
●●●● ●●● ● ●
●●● ●
●
●
●●
●●● ●
●
●● ●● ●●
●
●
●
●●● ●● ●●●● ●●
●
●●● ●●●●●
●
●
●●●●
●
●
●
●
●●●
●
●● ●●
● ●
●●●●●●● ●
●●● ●● ●● ●●● ● ●● ● ●●●
●
●●
● ●
●●● ●●● ●●● ● ●●●● ● ●
●●
●● ●● ●
●
●● ●
●●
●
●
●● ● ●● ●
●●
●●●●
●
●
●
●
● ●
●●● ●●●●● ●●● ●
●
●● ●●●● ●●● ●● ●
●
●●● ●●● ●● ●● ●●●●
●
●● ●●● ●● ● ●● ●● ●●●● ●●●● ●●
●
●●● ●● ●●● ●● ●● ●●●● ●●●●●● ●●●● ●●●●●● ●●● ●
●
● ●● ●
●
● ●●● ●●● ●●● ● ●●● ●● ●●●● ●● ● ●● ●●●● ●●●● ●
●
●●● ●●●●●●● ●● ● ●●●●●● ●● ● ●● ●● ●●●●● ●●●● ●● ●●● ●●●●● ●●●●●●●●●● ●● ●● ●●● ●●● ●● ●●●●● ●● ●● ●●●●● ● ●● ●●●● ●● ●●● ●● ●●● ●●● ● ●●●● ●● ●● ●● ●●● ●● ●● ●● ●● ●● ●●●●●● ●●●●●● ●●● ●●●●●● ●●●● ●●● ●●● ●●● ● ●●●● ● ●●●●● ●●● ●●● ●● ●●● ●●●●●●●● ●●● ●● ●●●●●●● ●●● ● ●●● ●● ● ●● ●● ● ●●●● ●●●●● ●●● ●●● ●●●● ● ●●● ●●●● ● ●● ●●●● ●● ●●● ●● ●● ●● ●●● ●●●●● ●● ●●●● ● ●● ●●●● ●●●● ● ●● ● ●● ●●● ● ●●●●●● ●●●● ●● ●●● ●●● ●●●●●● ● ●●● ● ● ●●● ●●● ●●● ●●● ●●●●● ●● ●● ●●● ●● ●●● ●● ● ●●●● ●● ●●● ●●● ●● ●●●●●● ●● ● ●●●● ●●●● ●●● ●●● ●●●●● ● ●●●●● ●●● ●●● ●● ●●●●●● ●● ●●●●● ●●●●● ● ●
Kernel smootherautomatically selected
bandwidth=0.044
Predicted event probability
0 % 25 % 50 % 75 % 100 %
Pse
udo−
obse
rved
eve
nt s
tatu
s
0 %
25 %
50 %
75 %
100 % ● ●●●●● ● ●●● ●● ● ●●● ●●●●
●●
●
● ●●●
●● ●
●
●
● ●●●
●
●●● ●
●
● ● ●●
●● ●
●● ● ●
●
● ● ●● ●● ● ●
● ● ●●
●●
●
●● ●●● ● ●●● ●
●
●● ●●● ●
●●
●●
●
●
●●● ●●●
●●
●●● ●●● ●●
●
● ●● ●● ●●● ●
●
● ●● ●● ●●●
● ●
● ●●● ●●● ●●●
●
● ●
●
● ● ●●●●
●
●●● ●
●
● ●●●
●
● ●●● ●●● ●●
●
● ●
●
● ●
●
●
●
●
●
●●● ●
● ●
●●
●
●● ●
●
●
●
● ●●
●
●● ●●● ● ●●●●●
●
●
●
●
●
●●
●
●
●● ●●● ●●●●● ● ●●● ●●
●
●
●●● ●●●
●
●
●
● ●
●
●
● ●●
●● ●●● ●
●
● ●● ●● ●●
●
●
●
●●
● ●
●●● ●● ●
●
●●● ●●●
●
●
●
●
●
●
●
●●
●
● ●●●●
●
● ● ●
●
●● ●●
● ●
●●● ●● ●●
●
● ●
●
●●●● ●●
●
●● ●●● ●●●
●
●●●●●● ●●●
●
●● ● ●
●
● ●●●● ●●●●
●
● ●
●●
●●●●●
●
●
●●
●●
●
●●● ●●●
●
●
●
● ● ●
●
●●●● ●●
●
●
●●
●
●
●●●
●
●●●
●
●
●
● ●
●
● ●
●●
● ●●●● ●●● ●●
●
●
●
●
●
●
●●
● ●
●
● ●
● ●
● ●●● ●
●
●● ●
● ●●
●● ●●●
●
●● ●
●
●
●
● ●
●
● ●
●
●●
●
●
●
● ● ●●
●
●
●
● ●●
●
●
●
●●
●
● ●●●● ●●● ●
●
●●
● ●●●
●
● ●
●
●
●●
● ●
●●
● ●
●
●●
●
●●
●
●● ●●
●● ●
●●● ●
● ●
●●●● ●
●
●
●
● ●
●
●●
●
●
●●
●●
●
●
●●●● ●
●
●
●●
●
●●
●
●●
●●
●
●
●
●
● ●
●
●● ●●
●●
●
●
●●
●● ●● ●
●
●●●
●
● ● ●●
● ●
●●●
●
●
●●
●
●
●
●
●● ●
●
● ●
●
●
● ●
●
●
●
●● ● ●
●●
●
●
●
●● ●
●
● ●
●
●
●
●●
●
●
● ●●
●●
● ●●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
● ●●
●
●
● ●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
● ●
● ●●
● ●●
● ●
●
●
● ●
● ●
●
●●●
●
●●
●
● ●●● ●
●
● ●●
●● ●
●
●
● ●●
●
●
●
●
●
●●
●
● ●
●
●●
●
●
● ●●
●
●
● ●
●●
●●
●● ●
●
●
●
●
●
●
●● ●●●●● ●●
●
●
●
●
● ●
●
●
●
●
● ●● ●
●
●
●
●●
● ●
●●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
● ●●
●
●● ●
●
●
●●
●●
●
●
●●
●
● ●●
●
● ●
●● ●
●
●
●●●● ●
●
●● ●●●●
●
●
●● ●●
●
● ●
●
●
●●
●
●● ● ●●
●
●
● ●●
●● ●
●●
●●
●
●
●●●
●●●
● ●●●
●
●●
●
● ●●●
●●● ●● ●● ●●
●
●●● ●●
●●●
●
● ●
●
● ●
●
●● ●● ●
●
● ●●●●
●●●● ●●● ● ●
●●● ●
●
●
●●
●●● ●
●
●● ●● ●●
●
●
●
●●● ●● ●●●● ●●
●
●●● ●●●●●
●
●
●●●●
●
●
●
●
●●●
●
●● ●●
● ●
●●●●●●● ●
●●● ●● ●● ●●● ● ●● ● ●●●
●
●●
● ●
●●● ●●● ●●● ● ●●●● ● ●
●●
●● ●● ●
●
●● ●
●●
●
●
●● ● ●● ●
●●
●●●●
●
●
●
●
● ●
●●● ●●●●● ●●● ●
●
●● ●●●● ●●● ●● ●
●
●●● ●●● ●● ●● ●●●●
●
●● ●●● ●● ● ●● ●● ●●●● ●●●● ●●
●
●●● ●● ●●● ●● ●● ●●●● ●●●●●● ●●●● ●●●●●● ●●● ●
●
● ●● ●
●
● ●●● ●●● ●●● ● ●●● ●● ●●●● ●● ● ●● ●●●● ●●●● ●
●
●●● ●●●●●●● ●● ● ●●●●●● ●● ● ●● ●● ●●●●● ●●●● ●● ●●● ●●●●● ●●●●●●●●●● ●● ●● ●●● ●●● ●● ●●●●● ●● ●● ●●●●● ● ●● ●●●● ●● ●●● ●● ●●● ●●● ● ●●●● ●● ●● ●● ●●● ●● ●● ●● ●● ●● ●●●●●● ●●●●●● ●●● ●●●●●● ●●●● ●●● ●●● ●●● ● ●●●● ● ●●●●● ●●● ●●● ●● ●●● ●●●●●●●● ●●● ●● ●●●●●●● ●●● ● ●●● ●● ● ●● ●● ● ●●●● ●●●●● ●●● ●●● ●●●● ● ●●● ●●●● ● ●● ●●●● ●● ●●● ●● ●● ●● ●●● ●●●●● ●● ●●●● ● ●● ●●●● ●●●● ● ●● ● ●● ●●● ● ●●●●●● ●●●● ●● ●●● ●●● ●●●●●● ● ●●● ● ● ●●● ●●● ●●● ●●● ●●●●● ●● ●● ●●● ●● ●●● ●● ● ●●●● ●● ●●● ●●● ●● ●●●●●● ●● ● ●●●● ●●●● ●●● ●●● ●●●●● ● ●●●●● ●●● ●●● ●● ●●●●●● ●● ●●●●● ●●●●● ● ●
Kernel smootherbandwidth=0.1
Predicted event probability
0 % 25 % 50 % 75 % 100 %
Pse
udo−
obse
rved
eve
nt s
tatu
s
0 %
25 %
50 %
75 %
100 % ● ●●●●● ● ●●● ●● ● ●●● ●●●●
●●
●
● ●●●
●● ●
●
●
● ●●●
●
●●● ●
●
● ● ●●
●● ●
●● ● ●
●
● ● ●● ●● ● ●
● ● ●●
●●
●
●● ●●● ● ●●● ●
●
●● ●●● ●
●●
●●
●
●
●●● ●●●
●●
●●● ●●● ●●
●
● ●● ●● ●●● ●
●
● ●● ●● ●●●
● ●
● ●●● ●●● ●●●
●
● ●
●
● ● ●●●●
●
●●● ●
●
● ●●●
●
● ●●● ●●● ●●
●
● ●
●
● ●
●
●
●
●
●
●●● ●
● ●
●●
●
●● ●
●
●
●
● ●●
●
●● ●●● ● ●●●●●
●
●
●
●
●
●●
●
●
●● ●●● ●●●●● ● ●●● ●●
●
●
●●● ●●●
●
●
●
● ●
●
●
● ●●
●● ●●● ●
●
● ●● ●● ●●
●
●
●
●●
● ●
●●● ●● ●
●
●●● ●●●
●
●
●
●
●
●
●
●●
●
● ●●●●
●
● ● ●
●
●● ●●
● ●
●●● ●● ●●
●
● ●
●
●●●● ●●
●
●● ●●● ●●●
●
●●●●●● ●●●
●
●● ● ●
●
● ●●●● ●●●●
●
● ●
●●
●●●●●
●
●
●●
●●
●
●●● ●●●
●
●
●
● ● ●
●
●●●● ●●
●
●
●●
●
●
●●●
●
●●●
●
●
●
● ●
●
● ●
●●
● ●●●● ●●● ●●
●
●
●
●
●
●
●●
● ●
●
● ●
● ●
● ●●● ●
●
●● ●
● ●●
●● ●●●
●
●● ●
●
●
●
● ●
●
● ●
●
●●
●
●
●
● ● ●●
●
●
●
● ●●
●
●
●
●●
●
● ●●●● ●●● ●
●
●●
● ●●●
●
● ●
●
●
●●
● ●
●●
● ●
●
●●
●
●●
●
●● ●●
●● ●
●●● ●
● ●
●●●● ●
●
●
●
● ●
●
●●
●
●
●●
●●
●
●
●●●● ●
●
●
●●
●
●●
●
●●
●●
●
●
●
●
● ●
●
●● ●●
●●
●
●
●●
●● ●● ●
●
●●●
●
● ● ●●
● ●
●●●
●
●
●●
●
●
●
●
●● ●
●
● ●
●
●
● ●
●
●
●
●● ● ●
●●
●
●
●
●● ●
●
● ●
●
●
●
●●
●
●
● ●●
●●
● ●●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
● ●●
●
●
● ●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
● ●
● ●●
● ●●
● ●
●
●
● ●
● ●
●
●●●
●
●●
●
● ●●● ●
●
● ●●
●● ●
●
●
● ●●
●
●
●
●
●
●●
●
● ●
●
●●
●
●
● ●●
●
●
● ●
●●
●●
●● ●
●
●
●
●
●
●
●● ●●●●● ●●
●
●
●
●
● ●
●
●
●
●
● ●● ●
●
●
●
●●
● ●
●●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
● ●●
●
●● ●
●
●
●●
●●
●
●
●●
●
● ●●
●
● ●
●● ●
●
●
●●●● ●
●
●● ●●●●
●
●
●● ●●
●
● ●
●
●
●●
●
●● ● ●●
●
●
● ●●
●● ●
●●
●●
●
●
●●●
●●●
● ●●●
●
●●
●
● ●●●
●●● ●● ●● ●●
●
●●● ●●
●●●
●
● ●
●
● ●
●
●● ●● ●
●
● ●●●●
●●●● ●●● ● ●
●●● ●
●
●
●●
●●● ●
●
●● ●● ●●
●
●
●
●●● ●● ●●●● ●●
●
●●● ●●●●●
●
●
●●●●
●
●
●
●
●●●
●
●● ●●
● ●
●●●●●●● ●
●●● ●● ●● ●●● ● ●● ● ●●●
●
●●
● ●
●●● ●●● ●●● ● ●●●● ● ●
●●
●● ●● ●
●
●● ●
●●
●
●
●● ● ●● ●
●●
●●●●
●
●
●
●
● ●
●●● ●●●●● ●●● ●
●
●● ●●●● ●●● ●● ●
●
●●● ●●● ●● ●● ●●●●
●
●● ●●● ●● ● ●● ●● ●●●● ●●●● ●●
●
●●● ●● ●●● ●● ●● ●●●● ●●●●●● ●●●● ●●●●●● ●●● ●
●
● ●● ●
●
● ●●● ●●● ●●● ● ●●● ●● ●●●● ●● ● ●● ●●●● ●●●● ●
●
●●● ●●●●●●● ●● ● ●●●●●● ●● ● ●● ●● ●●●●● ●●●● ●● ●●● ●●●●● ●●●●●●●●●● ●● ●● ●●● ●●● ●● ●●●●● ●● ●● ●●●●● ● ●● ●●●● ●● ●●● ●● ●●● ●●● ● ●●●● ●● ●● ●● ●●● ●● ●● ●● ●● ●● ●●●●●● ●●●●●● ●●● ●●●●●● ●●●● ●●● ●●● ●●● ● ●●●● ● ●●●●● ●●● ●●● ●● ●●● ●●●●●●●● ●●● ●● ●●●●●●● ●●● ● ●●● ●● ● ●● ●● ● ●●●● ●●●●● ●●● ●●● ●●●● ● ●●● ●●●● ● ●● ●●●● ●● ●●● ●● ●● ●● ●●● ●●●●● ●● ●●●● ● ●● ●●●● ●●●● ● ●● ● ●● ●●● ● ●●●●●● ●●●● ●● ●●● ●●● ●●●●●● ● ●●● ● ● ●●● ●●● ●●● ●●● ●●●●● ●● ●● ●●● ●● ●●● ●● ● ●●●● ●● ●●● ●●● ●● ●●●●●● ●● ● ●●●● ●●●● ●●● ●●● ●●●●● ● ●●●●● ●●● ●●● ●● ●●●●●● ●● ●●●●● ●●●●● ● ●
24 / 28
E�ect of cross-validation
Predicted event probability
0 % 25 % 50 % 75 % 100 %
Pse
udo−
obse
rved
eve
nt s
tatu
s
0 %
25 %
50 %
75 %
100 % 1000 bootstrap cross−validation stepsSame data used twice
25 / 28
Comparison of models
Relapse (t=36 months)Same data twice
Predicted event probability
0 % 25 % 50 % 75 % 100 %
Pse
udo−
obse
rved
eve
nt s
tatu
s
0 %
25 %
50 %
75 %
100 %Absolute risk regression
Cause−specific Cox
Fine−Gray
Bootstrap cross−validationB=1000
Predicted event probability
0 % 25 % 50 % 75 % 100 %P
seud
o−ob
serv
ed e
vent
sta
tus
0 %
25 %
50 %
75 %
100 %Absolute risk regressionCause−specific CoxFine−Gray
26 / 28
Summary of calibration: Brier score
BS(t, r) = E{N(t)− r(t|X )}2
Apparent performance (same data twice)
time Reference riskRegression CauseSpeci�cCox FGR timevar
3 5.2 4.9 4.8 4.9 4.712 12.1 10.5 10.4 10.3 10.336 15.2 13.2 13.2 13.2 13.1
Crossvalidation performance (B=1000)
time Reference riskRegression CauseSpeci�cCox FGR timevar
3 5.2 5 4.9 4.9 512 12.1 10.7 10.6 10.6 10.736 15.3 13.5 13.5 13.5 13.5
I The lower the betterI The null model ignores the covariatesI Conclusion: All models are better than reference, but otherwise comparable
27 / 28
Summary and discussion
I The transformation model with log-link yields absolute riskratios adjusted for confounders.
I A calibration plot is a graphical tool to investigate thereliability of a prediction model.
I It can be estimated in the presence of competing risks andright censored data based on
I external validation dataI cross-validation
I The scatterplot of pseudo-values indicates the distribution ofthe predicted risks and the level of censoring.
I Estimating a calibration plot is as hard as estimating a densityand the choice of independent bandwidth allows the user tomanipulate the calibration plot.
28 / 28