Applications of risk estimationHeather Dawe – Clinical Indicator Programme Manager

Who we are

Established in 2005,The NHS Information Centre is the centralauthoritative source of health and social care information, for England acting as a ‘hub’ for high-quality, national and local, comparative data for all ‘secondary uses’

Our role

Quality & StandardsEnsuring the right information quality, governance and standards in data and data collections

Delivering solutionsWork collaboratively alongside customers and partners to best utilise our information to tailor make solutions

AccessImproving access to and interpretation of data through better presentation and reportingEnsuring fair and equal access to the information

Data and Data Collection

Key Data Sources

Sources of health and social care data managed by the IC include:

• Hospital Episode Statistics (HES)

• General Practice Extraction Service (GPES)

• Electronic Staff Record (ESR – NHS workforce data)

• Clinical Audit

Secondary uses of operational data

These are all examples of the secondary use of data derived from operational management systems

• The benefits to using this kind of data is that it relieves burden on the frontline and it is more likely to be of robust quality

• Together these data can form a holistic view of the English health and social care system

• Much potential for research and development

• Key programme for the IC: Establishment of an ‘honest broker’ and ‘safe haven’ capable of managing the authorised disclosure of information to users

What are the information challenges facing the NHS?

Source: Independent research 2008

What do we know? • The demand for the effective use of clinical indicators within the NHS is growing • There is significant risk that the plethora of different methodologies currently in use by (all kinds of) suppliers serve to diminish the value of this use in the NHS • The NHS IC has a role to ensure these methodologies become standardised for the NHS and available for all to access, use and comment upon

• And there will be comments!!!

Clinical Indicators

Clinical indicators and risk

Clinical indicators such as the HSMR are risk-adjusted outcome measures

The risk adjustment takes into account process factors such as patient casemix (age, gender, co-morbidity levels) to estimate risk of individual patient mortality

Typically logistic regression is used to provide a national baseline model, which factors in casemix and other significant variables to provide the expected number of mortalities for an NHS provider

These expected mortalities are then compared to the observed mortalities for the provider, and the clinical indicator is derived

Visualising indicator variation and change

Monitoring quality and performance cross-sectionally and over time

Clinical indicators can be used to monitor and compare outcomes, both cross-sectionally and over time

Cross-sectional comparison: comparing outcomes across NHS providers

Comparison over time: monitoring outcomes from individual providers over time

Statistical process control (SPC) techniques are utilised to visualise and test differences both cross-sectionally and over time

Monitoring cross-sectionally

Monitoring over time

SMRs over time

40

50

60

70

80

90

100

110

120

130

1 2 3 4 5 6 7 8 9 10 11 12 13

Year

SMR

Utilisation of prior risk estimation to predict outcomes

Monitoring outcomes by comparing to a baseline

Example of deriving a prior risk model to predict future outcomes

Comparison of expected outcomes (priors) to observed

‘Today’s posterior is tomorrow’s prior’ – how and when should we recalibrate our baseline models to incorporate change into the baseline model?

Clearly a link between prior-based estimation and actual events – a link between previously observed, expected and observed events – and in the real world is that link the need to monitor and regulate?

When is monitoring using risk estimationa risk in itself?

Can it provide a perverse incentive?

The majority of healthcare treatment is delivered following robust and well-used standards and processes

As these standards and processes should deliver the same quality of care across the board, and have minimal variation in outcome, it is viable to monitor and compare

What about high risk treatment? What about innovative pioneering treatment that is unproven, carries potential for great benefit but also carries more risk?

It is important that any risk based estimation and measures of outcomes either incorporates this higher risk or estimates these events separately, across peer groups

There are known unknowns...

How do we know what to monitor, to estimate the risk of and then compare expected to observed events?

Shipman, Bristol and Mid-Staffordshire - key examples of where post-event reviews have stated the importance of monitoring to help reduce the risk of repeat occurrences

Monitoring mortality rates, emergency readmission rates, incidence of hospital superbugs are all important and useful examples of utilising outcome measures as a proxy for quality of care

What else is there? How do we know how to look for it? Clinical engagement is fundamental and so is the utilisation of modern business intelligence techniques

Forming a business intelligence function

Data and systems

Business knowledge

Analysis and

statistics

Data mining strategy

Developing a data mining strategy

Clinical indicators developed within the NHS IC are currently principally based on secondary care data (HES)

Combining the key datasets available to us and looking holistically for unusual patterns and trends will help to identify ‘known unknowns’

Risk estimation and prediction forms a key way in which outcomes can be monitored and unusual patterns of behaviour identified

Strategy will be implemented efficiently and effectively, working in partnership with numerous stakeholders, utilising business intelligence

In summary

Risk estimation and comparing predicted to actual events forms an integral part of the generation of clinical indicators

Extending our work to look across the system is an essential element of our developing data mining strategy

This strategy will utilise existing and emerging methods used to both visualise and estimate outcomes in healthcare, and will be delivered in partnership with others

We need to learn and work with experts to ensure the strategy is developed and delivered effectively. Any ideas???