© Nuffield Trust

Evaluation methods – where can predictive risk models help?

Adam Steventon

Nuffield Trust

8 July 2013

© Nuffield Trust

The problem with observational studies

Eligible patients

All patients Intervention patients

n 54,990 556

% aged 85+ 21.6 46.2

Prior emergency admissions 0.5 1.4

Number chronic conditions 1.0 1.1

Predictive risk score 22.3 33.6

Intervention

patients

Source: Steventon et al (2012)

© Nuffield Trust

Solutions, 1) before-after study

© Nuffield Trust

Solutions, 2) regression adjustment

Y = f(age, number of chronic conditions,

prior emergency admissions,

intervention status)

© Nuffield Trust

Eligible patients

Intervention

patients

Matched

controls

All patients Matched controls

Intervention patients

n 54,990 556 556

% aged 85+ 21.6 46.2 46.2

Prior emergency admissions

0.5 1.4 1.4

Number chronic conditions

1.0 1.3 1.1

Predictive risk score 22.3 33.5 33.6

Solutions, 3) Matched controls

Source: Steventon et al (2012)

© Nuffield Trust

How to select matched controls

Propensity score (Rosenbaum and Rubin 1983)

- Predictive risk of receiving the intervention

Prognostic score (Hansen 2008)

- Predictive risk of experiencing the outcome (e.g. emergency

hospitalisation), in the absence of the intervention

Genetic matching (Sekhon and Grieve 2012)

- computer-intensive search algorithm

© Nuffield Trust

Advantages / disadvantages

Disadvantage – only allows for observed variables

But

Matching as ‘data pre-processing’ – reduces dependence of estimated

intervention effects on regression model specification

Intuitive?

Good for routine monitoring – once controls found, data can be updated

© Nuffield Trust

Overcoming regression to the mean using a control group

Start of intervention

© Nuffield Trust

Overcoming regression to the mean using a control group

Start of intervention

© Nuffield Trust

Overcoming regression to the mean using a control group

Start of intervention

© Nuffield Trust

Overcoming regression to the mean using a control group

Start of intervention

© Nuffield Trust

Solutions, 4) regression discontinuityW

inn

ing th

e n

ext

ele

ctio

n

Fraction of votes awarded to Democrats in the previous election

Source: Lee and Lemieux (2009)

© Nuffield Trust

What is being done at the moment?Telehealth studies in Pubmed, 2006-2012

Descriptive Before afterDose

responseControlled All

Number of studies 3 24 1 16 44

Median number of patients in telemonitored group (range)

45(40 to 851)

35 (7 to 17,025)

246102

(19 to 1,767)*45

(7 to 17,025)*

Endpoints

Mortality 2 - 1 3 6

Hospital use (or costs) 3 6 - 12 21

Clinical (e.g. HbA1c) - 17 - 4 21

Patient reported outcomes (e.g. quality of life)

1 10 - 3 14

Source: Steventon, Krief and Grieve (work in progress)

© Nuffield Trust

References

Lee DS, Lemieux T. Regression discontinuity designs in economics. 2009.

Available from: http://www.nber.org/papers/w14723.pdf?new_window=1

Sekhon JS, Grieve RD. A matching method for improving covariate balance in

cost-effectiveness analyses. Health economics 2012;21:695–714.

Rosenbaum P, Rubin D. The central role of the propensity score in observational

studies for causal effects. Biometrika 1983;70:41–55.

Hansen BB. The prognostic analogue of the propensity score. Biometrika

2008;95:481–8.

Steventon A, Bardsley M, Billings J, Georghiou T, Lewis GH. The role of

matched controls in building an evidence base for hospital-avoidance schemes:

a retrospective evaluation. Health services research 2012;47:1679–98.

© Nuffield Trust8 July 2013

www.nuffieldtrust.org.uk

Sign-up for our newsletter:www.nuffieldtrust.org.uk/newsletter

Follow us on Twitter: http://twitter.com/NuffieldTrust

© Nuffield Trust

adam.steventon@nuffieldtrust.org.uk