1

Patient Journey Record (PaJR)Online Prediction System

Jing Su, Lucy Hederman, Atieh Zarabzadeh, Dee Grady, Carmel Martin, Kevin Smith, Carl Vogel, Enda Madden, Brendan Madden

Trinity College Dublin, UCC, PHC Research, GroupNos

HISI 2011

2

Avoidable hospitalisations

• PaJR is a telephone service that targets avoidable

hospitalisations

• Most hospital admissions

• are in older, sicker people with multiple diseases

and conditions

• are unpredictable in the short term with current

systems

HISI 2011

3

$5,000

$1

$10

$100

$1,000

Early Alerts of deterioration helps prevent the patient entering expensive treatment

Self care Complicated Complex Hospital

$/day

HISI 2011

4

The PaJR ServiceMichael Anon

HISI 2011

5

The PaJR SurveyMichael Anon

HISI 2011

6

The PaJR approach

• Use lay callers• Ask the questions that predict

hospitalisation• Not disease specific

• Intervene early• Alert accurately …

• ~ 50% reduction in hospital admissions in pilot sites

HISI 2011

7

PaJR Prediction Service• Predicts patient

deterioration based on record of call. – Self-rated health, taking

meds,…– Brief text entries

• Uses a predictive model learned from examples of calls leading to unplanned events.

Michael Anon

HISI 2011

8

Machine Learning

ML: automatically induce from the examples a model that accurately

predicts new cases.

Ok A&E Hosp ??Ok A&E

...

HISI 2011

9

Simple predictive model: a decision tree

A

B B

A

CK=y K=n K=y

K=n K=y

K=y

- decision node

- leaf node (K means UnplannedEvent)

noyes

<n ≥n ≥p<p

yes no

A sample decision tree on UnplannedEvent

Eg: A = “her sister”; B = AvgWordLength; C = takingMeds

HISI 2011

10

Machine learning

• ML Tools such as Weka or Timbl provide algorithms to produce prediction models from examples (“training data”).

• The examples must be presented to ML tool as a collection of features.

• Expertise and skill is needed to identify / derive / represent features of examples that might predict the outcome.

HISI 2011

11

PaJR’s Current ML

• Predicts unplanned events, urgent unplanned events, self rated health.

• Uses decision trees.• Weights false negatives 500 more costly

than false positives– A missed deterioration is bad.– An inappropraite alert to a carer to call a

patient is OK.

HISI 2011

12

Accuracy

• Predicting urgent unplanned events (UUE)

• Training data– 1621 phone calls– 27 urgent unplanned

events

• False Negatives cost 500 times FPs

True False

Negative(OK)

1091 4

Positive(UUE)

23 453

• Fewer than 1/3 of the calls are incorrectly prioritised

• Under 1/6 of the calls that should be prioritised are not.

HISI 2011

13

Error Analysis

• False positive rate is worth further investigation:– ML predicts an urgent unplanned event.– No urgent unplanned event occurred.– But is that because the PaJR caller intervened (with advice,

referral, comfort, …) and averted the event ML predicted?

• Analysing FP cases, we found evidence of some intervention in a small number of cases.– Further work needed.

• More significantly, UUEs were rarely (6/27) ‘anticipated’ by lay callers (they didn’t intervene), whereas ML predicted 23 of them.

HISI 2011

14

Challenges

• Data– ML requires lots of examples of each outcome.

Thanks to PaJR the number of unplanned events among the users is declining.

• Features– We have lots of data for each case but it takes time

and skill to identify features predictive of deterioration.

• Prediction Engine Pipeline– The management of multiple cases, multiple models,

etc.

HISI 2011

15

Benefits of Machine Learning

• Compared with static rule-based alerts– ML allows identification of features that emerge as predictive of

deterioration.– ML uses evidence from data on real patients.– ML can be easily transferred to new settings and new services– ML adapts over time

• Compared with experienced callers without ML– ML allows high accuracy, high volume at low cost. – ML will identify features across callers, across time, etc.– ML has perfect memory – callers go on leave, move on.

HISI 2011

16

Thank you

Any questions?

Lucy Hederman hederman@tcd.ie

HISI 2011

17

HIDE????Machine Learning Pipeline

Database

Existing surveys

New survey

Training data

Test data

Querier Language Parser

Decision Tree

Predicting Unplanned

Events / SRH

CSV

Qualitative feedback

Hours

Moments (< 1 minute)

Parsed Feature

s

Off-line training and online predicting!

Update Decision Tree model at regular intervals

ML Algorithm

HISI 2011