Reviewing Methods Used in

Patient Safety Research:

Advantages and Disadvantages

Dr Jeanette Jackson

(j.jackson@abdn.ac.uk)

This SPSRN work is funded by

Introduction

• Effective management of patient safety in

healthcare requires:

1. an understanding of the causes of adverse events and related outcomes

2. a capacity to measure adverse events and their causes as well as related outcomes at different levels (individual, unit, organization,

industry, national, international)

• Measurement of industry safety status is achieved

by a range of methods based on key performance

indicators for risk factors and safety events as well

as leading indicators for safety (including causes

like cultural factors)

Objectives

1. To review possible methods for measuring patient

safety

2. To evaluate available methods for measuring

patient safety

Methods Measuring

Patient SafetyMethods can be classified based on different criteria:

1. Related to the source of data (Runciman et al., 2008): e.g., research

data from previous studies, routine surveillance data, quality

assurance and risk management data

2. Human factor and ergonomic methods used in system design and evaluation (Carayon, 2007): e.g., cognitive work analysis, incident

analysis, work systems and process analysis, usability evaluation,

assessing safety culture and climate

3. WHO report on methods measuring patient safety (Jeffs, Law & Baker, 2008): retrospective, prospective and concurrent states of

patient safety

Methods Measuring

Patient SafetyRetrospective Methods:

• Medical Records, Incident Reporting Systems, Claims and

Complaints, Staff Questionnaires, Interviews

• Exploring adverse event, errors and near misses

• Awareness about patient safety issues

• Inform improvement programmes

Methods Measuring

Patient SafetyProspective Methods:

• Prospective Analysis Tools, Simulations, Safety Culture

Assessment

• Identifying sources of potential error based on knowledge about

error rates

Methods Measuring

Patient SafetyConcurrent Methods:

• Direct Observations, Video Techniques, Checklists and Audits

• Systems that update information once they receive data

• Monitoring and controlling patient care processes

• Estimates of error rates

• Reaction to error as it occurs

Retrospective Methods

Medical Records:

• Indicators of the cause, nature and incidence of harm

• Most frequently used for researching medical error

• Using ‘triggers’, e.g., high risk medication or laboratory values

• Adverse events studies: 3-17%

Medical Records

Advantages:

• Highlighting particular incidents / areas that require further

understanding / training

Disadvantages:

• Missing incidents that are not reported• Not everything documented, e.g., confounding factors or human

factors

• Time consuming, staff costs (training for screening and reviewing)

Retrospective Methods

Incident Reporting Systems:

• Different systems implemented in different health care systems and

settings

• Reporting adverse outcomes for patients, errors and near misses

(e.g., adverse drug events, problems with medical devices, safety of blood products)

• Information on causes of adverse events, provide warnings, point to

important problems, raise awareness and enhance safety

Incident Reporting Systems

Advantages:

• Systems can be implemented across multiple sites

• Minimal costs for volume of data (?)

• Can highlight effectiveness of existing policies / procedures

Disadvantages:

• Depending on coding systems used as well as the ability of using

these systems reliably• What and how to report (high hazard events, willingness to report)

• Number of reported incidents (lack of resources, varying quality)

• Ensuring that feedback (e.g., newsletter, meetings) is given and

actions are taken - effectiveness of learning from incident data

unclear (lack of documentation)• Lack of clarity in roles and responsibilities

• A large amount of time and resources involved

Retrospective Methods

Claims and Complaints:

• Claims reflect a patients’ demand for compensation (general safety

issues) whereas complaints reflect a patients’ subjective

impressions regarding their healthcare (specific incidents)

• Incidence data, experiences with intervention programmes, starting point for reviews and patient safety activities

• Identifying potential problem areas or clinical issues such as

organisational and management issues, teamwork skills, individual

clinicians behaviours

Claims and Complaints

‘When I first started out in practice,

I lied awake at night worrying about my patients.

Now I lie awake worrying about their lawyers.’

(Howard Fischer, MD)

Advantages:

• Themes can be picked up from a large number of claims / complaints

• Looking into 1 case in depth can provide important information for learning

Disadvantages:

• Subjective perspective

• Underreporting by elderly people

• Expensive

• One major event might influence the whole picture

Detecting Adverse Events

(adapted from P. Hebert)

Method AE / 1000 admissions

1. Incident reports 5

2. Retrospective chart reviews 30

3. Stimulated voluntary reports 30

4. Automated flags 55

5. Daily chart reviews 85

6. Automated flag and daily review 130

7. Trigger tool 400Jha et al. (1998). Identifying Adverse Drug Events: Development of a Computer-based Monitor and Comparison with

Chart Review and Stimulated Voluntary Report. Journal of the American Medical Informatics Association, 5, 305-315.

O’Neill et al. (1993). Physician Reporting Compared with Medical -Record Review to Identify Adverse Medical Events.

Annals of Internal Medicine, 119(5), 370-376.

Prospective Methods

Prospective Analysis Tools:

• Proactively identifying, prioritising and mitigating patient safety risk

• Wide range of methods such as Failure Modes Effects and Critical

Analysis, Health Care Failure Mode and Effect Analysis, Hazard

Analysis

• Use at local level, discovered information is not shared throughout

organisations, not identifying combinations of events leading to

incidents

• Probabilistic risk assessment (mixture of process analysis

techniques and decision making processes) to balance and prioritise

between competing goals as well as to identify combinations of

events leading to potential harm

Prospective Analysis Tools

Advantages:

• Uncovering unrecognised system level problems

• Boosting staff communication and morale

• Process of scoring failure modes regarding their probability of

occurrence and severity of consequences (group census)

Disadvantages:

• Logistic challenge

• Complex methodology involved

• Process of scoring failure modes probability of occurrence and

severity of consequences (depending on setting, people)

Prospective Methods

Simulations:

• Training of performance skills in a scenario with the same realistic

problems and demands as in real life settings

• Learning opportunity for healthcare workers and teams about

consequences of their actions

• Identify problem conditions / actions to improve healthcare delivery

• Teach and test specific technical (e.g., performing a new procedure)

and non-technical (e.g., communication) skills

Simulations

Advantages:

• Can identify additional risks that may not have been considered

• Control of the situation and problem area

Disadvantages:

• Challenge to create realistic teams with real life skills mix (by

experience and profession)

• Expensive (staff involvement, equipment, …)• Transfer from simulation to real world

Concurrent Methods

Direct Observations:

• Data collection on errors, adverse events, near misses, team

performance, decision making and organisational culture

• Information related to the analysis of specific types of procedures

(e.g., surgery)

• Structured observations using rating systems, e.g., surgeons’ non-

technical skills, to rate skills and provide feedback during a

postoperative debrief (Yule et al., 2006)

• Range of behaviour rating tools available for individual and team

assessment in acute medicine (see Flin & Mitchell, 2009)

Direct Observations

Advantages:

• Can highlight particular factors of concern

• Can influence local behaviours / policies

Disadvantages:

• Observer as a distraction / interruption

• Observer training

• Time consuming (data collection and analysis)

Concurrent Methods

Checklists and Audits:

• Identifying active errors in the environment such as delay in care,

equipment failure, information transfer, non-compliance with hospital

policy, …

• Checklists are context specific (e.g., ICU versus general ward)

• Safety audits can be performed using checklists, e.g., during and

after morning rounds to identify unlabeled medication at the bedside,

missing ID bands or inappropriate pulse oximeter alarm setting

Checklists and Audits

Advantages:

• Public charts to display ongoing results

• Influence of culture (motivation to get it right)

Disadvantages:

• Outcome measure identification (who decides what to display?)

• Lots of pressure to ward staff

• Lack of compliance

Selection of Methods

• Appropriate selection of methods depends on the question

being addressed, the resources available, and on contextual

factors

Framing Patient Safety Research

Multilevel Framework of Patient Safety Research (Jackson & Flin, in prep):

Organizational

Structure

Unit

Management

Worker

BehavioursOutcomes

Individual

Differences

• Based the causal chain and different levels of analysis (i.e., individual,

team, unit, and organisational) proposed by industrial and patient safety

models

• Applies within an organisation even though external factors such as

government and regulators responsibilities exist outside an organisation

Any Questions?

Dr Jeanette Jackson

(j.jackson@abdn.ac.uk)

This SPSRN work is funded by