www.england.nhs.uk

Patient safety data or patient safety intelligence?

Dr Frances HealeyRGN, RMN, PhDHead of Patient Safety Insight NHS England

6 July 2015

www.england.nhs.uk

1st rule of #statisticsclub

Qualitative data are at least equally important, and probably much more important, than quantitative data…..

2

….. but it’s quantitative data that have the pitfalls & perils, so that is my focus today

www.england.nhs.uk 3

www.england.nhs.uk 4

www.england.nhs.uk 5

Dammit Jim, I’m a doctor, not a rocket

scientist

Dammit Frances, I’m a nurse, not statistician

www.england.nhs.uk

2nd rule of #statisticsclubYou are probably much less logical than you think you are

6

www.england.nhs.uk

http://blogs.bmj.com/bmj/2014/05/09/tara-lamont-on-failing-well-archie-cochranes-legacy/

@TaraJLamont

Archie Cochrane

77

www.england.nhs.uk 8www.england.nhs.uk

“The results at that stage showed a slight numerical advantage for those who had been treated at home. It was of course completely insignificant statistically.

“I rather wickedly compiled two reports, one reversing the numbers of deaths on the two sides of the trial. As we were going into committee, in the anteroom, I showed some cardiologists the results……..

www.england.nhs.uk 9

“……they were vociferous in their abuse: `Archie’, they said, `we always thought you were unethical. You must stop the trial at once…’

“I let them have their say for some time and then apologised and gave them the true results, challenging them to say, as vehemently, that coronary care units should be stopped immediately.

“There was dead silence and I felt rather sick because they were, after all, my medical colleagues.”

Professor Archibald Cochrane & Max Blythe One Man's Medicine (1989) p.211

www.england.nhs.uk

Cognitive dissonance

10

• We have a strong need for our personal beliefs and our personal actions to chime

• The discomfort we feel when they don’t is ‘cognitive dissonance’

http://britishgeriatricssociety.wordpress.com/2013/05/16/all-down-to-numbers/

• If we believe we are part of effective, motivated, caring teams, it is very hard to also simultaneously believe:o We haven’t achieved real

improvements in safety o We might be less safe than

peers

www.england.nhs.uk 11

http://www.health.org.uk/multimedia/slideshow/hard-data-soft-intelligence/

Data (mis)used for reassurance

www.england.nhs.uk

Which is not to say we shouldn’t use data for motivation

“The consistent delivery of well-executed safe care under typically difficult circumstances tends to go unrecognised"

www.england.nhs.uk 13

www.england.nhs.uk

3rd rule of #statisticsclubOne size does not fit all - there is no such thing as a good indicator, or a good data source, just one that is good in particular situation for a particular purpose

14

Oct - Dec 2003

Jan -

Mar 2004

Apr -

Jun 2004

Jul - Sep 2004

Oct - Dec 2004

Jan -

Mar 2005

Apr -

Jun 2005

Jul - Sep 2005

Oct - Dec 2005

Jan -

Mar 2006

Apr -

Jun 2006

Jul - Sep 2006

Oct - Dec 2006

Jan -

Mar 2007

Apr -

Jun 2007

Jul - Sep 2007

Oct - Dec 2007

Jan -

Mar 2008

Apr -

Jun 2008

Jul - Sep 2008

Oct - Dec 2008

Jan -

Mar 2009

Apr -

Jun 2009

Jul - Sep 2009

Oct - Dec 2009

Jan -

Mar 2010

Apr -

Jun 2010

Jul - Sep 2010

Oct - Dec 2010

Jan -

Mar 2011

Apr -

Jun 2011

Jul - Sep 2011

Oct - Dec 2011

Jan -

Mar 2012

Apr -

Jun 2012

Jul - Sep 2012

Oct - Dec 2012

Jan -

Mar 2013

Apr -

Jun 2013

Jul - Sep 2013

Oct - Dec 2013

Jan -

Mar 2014

Apr -

Jun 2014

Jul - Sep 2014

Oct - Dec 2014

0

50000

100000

150000

200000

250000

300000

350000

400000

450000

500000

Incid

ents

Sub

mitt

ed

www.england.nhs.uk 15

CULTURE indicators

STRUCTURE indicators

PROCESS indicators

planning process indicators

delivery process indicators

OUTCOME indicatorsAre we

safe today?

Types of indicator

www.england.nhs.uk

Our high tech interactive facility

16

1.2.3.4.5.

www.england.nhs.uk

What type of indicator?

17

1. CULTURE indicator

2. STRUCTURE indicator

3. PROCESS - planning process indicator

4. PROCESS - delivery process indicator

5. OUTCOME indicator

97% of patients who need a pressure reliving mattress received it within four hours

www.england.nhs.uk

What type of indicator?

18

1. CULTURE indicator

2. STRUCTURE indicator

3. PROCESS - planning process indicator

4. PROCESS - delivery process indicator

5. OUTCOME indicator

86% of nurses agree that most pressure ulcers can be prevented

www.england.nhs.uk

What type of indicator?

19

1. CULTURE indicator

2. STRUCTURE indicator

3. PROCESS - planning process indicator

4. PROCESS - delivery process indicator

5. OUTCOME indicator

We have 42 pressure relieving mattresses per 100 beds

www.england.nhs.uk

4th rule of #statisticsclubWe don’t do structural measurement nearly often enough

20

30%

9%

26%

35%

on all wards

on most wards

on one or some wards

not on any wards

“This [MH unit for older people] has no physio input. Balance and strength assessments never get done”

“We cannot put walking frames within reach as there is no room left once you have a chair beside the bed”

Royal College of Physicians 2012 Report of the 2011 inpatient falls pilot audit www.rcplondon.ac.uk

Weekend access to mobility aids for new patients

www.england.nhs.uk

Purposes of safety measurement

21

One often used model:measurement for researchmeasurement for judgement measurement for improvement

Alternative less prone to misunderstanding measurement to understand prioritiesmeasurement to see how we compare to othersmeasurement to see if we’re getting better (or worse)

www.england.nhs.uk

0-4

5-9

10-1

4

15-1

9

20-2

4

25-2

9

30-3

4

35-3

9

40-4

4

45-4

9

50-5

4

55-5

9

60-6

4

65-6

9

70-7

4

75-7

9

80-8

4

85-8

9

90-9

4

95-9

9

100+

0%

2%

4%

6%

8%

10%

12%

14%

16%

18%

Breakdown by age of falls in acute clusters

Age group

% of all repor-ted acute falls

00 (12 AM

- Midnight)

01 (1

AM)

02 (2

AM)

03 (3

AM)

04 (4

AM)

05 (5

AM)

06 (6

AM)

07 (7

AM)

08 (8

AM)

09 (9

AM)

10 (10 AM

)

11 (11 AM

)

12 (12 PM

- Midday)

13 (1

PM)

14 (2

PM)

15 (3

PM)

16 (4

PM)

17 (5

PM)

18 (6

PM)

19 (7

PM)

20 (8

PM)

21 (9

PM)

22 (10 PM

)

23 (11 PM

)

0%

1%

2%

3%

4%

5%

6%

Falls incidents by hour of occurrence, for acute clusters

Hour

% of all reported acute falls

Understanding priority areas

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Acute hospitals Community hospitals Mental health units

Location of incident

Per c

ent o

f sam

ple

Apparently unwitnessed by staff

Witnessed by staff

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Acute hospitals Community hospitals Mental health units

Location of incident

Per c

ent o

f sam

ple

Apparently unwitnessed by staff

Witnessed by staff

0%

5%

10%

15%

20%

25%

30%

35%

Acute hospitals Community hospitals Mental health units

Whilst walking

From beds

Circumstances unclear

From chairs

From toilet or commode

Other

0%

5%

10%

15%

20%

25%

30%

35%

Acute hospitals Community hospitals Mental health units

Whilst walking

From beds

Circumstances unclear

From chairs

From toilet or commode

Other

www.england.nhs.uk

5th rule of #statisticsclubMeasurement to see how we compare to others: when it comes to comparing outcomes, case mix really matters

23

(but case mix shouldn't be a problem for well-designed process measures)

www.england.nhs.uk

Safety outcomes & case mix

0%

5%

10%

15%

20%

25%

0-4 5-9 10-14 15-19 20-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65-69 70-74 75-79 80-84 85-89 90-94 95-99 100+

Age of patient (years)

Pe

r c

en

t

Per cent of totalbed daysPer cent of totalfalls

0%

5%

10%

15%

20%

25%

0-4 5-9 10-14 15-19 20-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65-69 70-74 75-79 80-84 85-89 90-94 95-99 100+

Age of patient (years)

Pe

r c

en

t

Per cent of totalbed daysPer cent of totalfalls

85 years +

Deandra S et al. Arch Gerontol Geriatr 56 (2013) 407–415

NPSA Slips trips and falls in hospital data update NPSA 2010

Risk factors for hospital falls Odds Ratio

History of falls 2.85 (1.14–7.15)

Cognitive impairment 1.52 (1.18–1.94)

www.england.nhs.uk

Older people are not evenly distributed

25

Therefore unadjusted higher/lower rates of falls compared to other trusts with very different age profiles are highly unlikely to be useful indicators of relative safety

All ages falls rate

0.01.02.03.04.05.06.07.08.09.0

RETIREM

ENT TO

WN B

RETIREM

ENT TO

WN D

RETIREM

ENT TO

WN A

RETIREM

ENT TO

WN C

URBAN TEACHIN

G B

URBAN TEACHIN

G C

URBAN TEACHIN

G D

URBAN TEACHIN

G A

falls

per

1,0

00 b

ed d

ays

www.england.nhs.uk

50-fold differences between wards

26Royal College of Physicians 2011 The FallSafe Quality Improvement project: report for the Health Foundation

Therefore higher/lower rates of falls compared to other wards in the same trust highly unlikely to be useful local indicators of safety

www.england.nhs.uk

Comparable and non-comparable safety indicators

www.england.nhs.uk 28

http://britishgeriatricssociety.wordpress.com/2013/12/19/fallsafe-are-culture-clashes-good-for-us/

0

5000

10000

15000

20000

25000

6th rule of #statisticsclub

Numbers don’t know if they are in research study or a QI project

www.england.nhs.uk

7th rule of #statisticsclubWhen it comes to measuring if we have got better (or worse) size matters

(and we usually have plentiful processes to measure but far fewer outcomes)

www.england.nhs.uk 30

www.england.nhs.uk

Sample safety outcome indicators: scaled to ward-level*

IF these safety outcomes were distributed evenly across acute wards, an average ward would have around:

• One case of c difficile per year

• One MRSA bloodstream infection per decade

• One new pressure ulcer per quarter

• One fall with minor injury per month

• One fall with hip fracture every five years

* Approximations based on c. 5,000 acute/rehabilitation hospital wards in England, PHE trust attributed/trust-assigned HCAI data, NRLS reported falls, assumption that acute ‘new’ p ulcer prevalence as measured by ST represents about 4 x acute p ulcer incidence

www.england.nhs.uk

What scale and time would give you a reasonable chance of being able to distinguish a 25% improvement from natural variation ?

32

For falls with injury

1. One ward two years

2. Ten wards two years

3. One medium sized hospital two years

4. Five hospitals two years

5. Fifty hospitals two years

www.england.nhs.uk

What scale and time would give you a reasonable chance of being able to distinguish a 25% improvement from natural variation ?

33

For hospital-associated MRSA?

1. One ward two years

2. Ten wards two years

3. One medium sized hospital two years

4. Five hospitals two years

5. Fifty hospitals two years

www.england.nhs.uk 34

Collaboration is not just good for learning,

it increases sample size

www.england.nhs.uk

8th rule of #statisticsclubYour data don’t have to be perfect to be good enough – but you do need to know how imperfect they are

35

www.england.nhs.uk

0

2

4

6

8

10

12

14

16

60% certain last fall was

reported

77% certain last fall was

reported

0

2

4

6

8

10

12

14

16

0

2

4

6

8

10

12

14

16

www.england.nhs.uk

9th rule of #statisticsclubIf it looks too good to be true, it probably is!

37

Jan

Feb

Mar

Apr

May Jun

Aug Oct

Nov Jan

Oct

Dec

Aug

Sep Oct

Nov

Dec Jan

Feb

Mar

Apr

May Jun

Jul

Aug

Sep Oct

Nov

Dec

2008 2009 2010 2011

0

10

20

30

40

50

60

70

80

0%

1%

2%

3%

4%

5%

6%

7%

8%

9%

10%

Falls

www.england.nhs.uk

10th rule of #statisticsclubDon’t try to measure too much

38

www.england.nhs.uk

www.england.nhs.uk

www.england.nhs.uk

www.england.nhs.uk

85%

15%

time spent improvingtime spent measuring

www.england.nhs.uk

Thank you for listening

frances.healey@nhs.net

@FrancesHealey

Please tweetfurther #statisticsclub rules of your own