Evaluation of computerised decision support in
eMed systems
Melissa Baysari
With Wu Yi Zheng, David Lowenstein, Anmol Sandhu, Ric Day, Johanna Westbrook, Rosemary Burke, Eliza Kenny & Meredith Makeham
Computerised decision support
2
Can mean different things to different people
Computerised alerts
Order sentences
Reference material
Drop down lists
Notes or instructions
Calculators
Etc
AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
Alert effectiveness
3 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
Literature tells us that alerts can result in substantial changes in
prescribing behaviour
BUT
Most studies evaluate an alert for a specific condition or problem
e.g. alerts designed to reduce the use of contraindicated drugs in
patients with renal failure drop in proportion of patients receiving a
contraindicated medication from 89% to 47%
Less evidence for the effectiveness of basic decision support
alerts within eMM systems
e.g. few studies showing that DDI alerts lead to reductions in DDIs
JAMIA 2005 12:269-74
Alert fatigue
4 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
A consequence of too many alerts being presented
Main barrier to prescriber acceptance of computerised alerts
A significant problem for hospitals because it
results in user frustration & annoyance
leads to prescribers learning to ignore all alerts, even those that present
useful & sometimes safety critical information
Alert fatigue affects most doctors in most organisations
most alerts are overridden
Effective alerting
5 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
Many strategies have been proposed for reducing number of alerts (and
minimising alert fatigue), such as:
- Customising alerts for clinicians
- Increasing alert specificity
- Presenting only high-level (severe) alerts to clinicians
- Improving alert design
These strategies sound simple, but are very difficult to implement
Effective warning design From the human factors literature (process industries etc)
Alert content
7 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
Message
length
Short messages are easier and faster to process.
As a general rule, do not include more than 5-6 bits of
information
Abbreviations Avoid using, unless they are understood even by the most
inexperienced users
Procedures Break procedures up into short, sequential steps, with each
step presented on a separate line
Symbols Should be used wherever possible to avoid the need to read
text.
Only familiar or standard symbols should be used
Wording Affects comprehension and speed of reading.
If possible, use familiar words, active sentences, and positive
statements
Warning signs
8 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
These convey information about a potential hazard or risk
Include 4 main components:
• A signal word, larger than the rest of the alert text
• A description of the hazard
• A description of the consequence of the hazard
• A description of the behaviour needed to avoid the hazard
Examples
9 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
Messages should be structured in short, concise statements using active
verbs
WARNING
Surface is slippery when wet
May cause you to fall
Please use hand-rail
WARNING This document failed to save
You will lose all changes
To save, click OK
ALERT
Patient is allergic to penicillin
Continuing with the order may
cause anaphylactic shock
Click here to cancel order
Click here to continue order
Layout of the message
10 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
People read warnings in an unstructured way – they scan to identify
objects or words of interest
If large amounts of information need to be displayed, group into smaller
units
Short messages could be centred but longer messages (across multiple
lines) should be displayed on numbered lines and left justified
Good vs. bad warnings
11 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
This is an emergency. To make an emergency
call, press the red button. Wait for emergency
services to answer and then speak clearly into
the speaker.
EMERGENCY
To make an emergency call:
1. Press the red button
2. Wait for emergency services to answer
3. Speak clearly into speaker
Appearance of text
12 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
• If message is more than 3 words in length, use mixed upper and lower
case
• Short messages (ALERT, WARNING) can be written in capital letters
• Minimise use of italics, they are harder to read
• Colour combinations that have good contrast are easier to read
• Use colours that are expected (e.g. red for warning)
Phansalkar’s review and HF tool
Human factors principles for design and implementation of medication
safety alerts
2010 review of the literature
14 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
Aim: to summarise human factors research on computer-based alerting
systems from a range of industries
They identified 11 HF principles
Alarm philosophy Colour
False alarms Learnability and confusability
Placement Textual information
Visibility Habituation
Prioritization Mental models
Proximity of task components
How to measure HF compliance?
15 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
I-MeDeSA - Instrument for evaluating human factors principles in
medication related decision support alerts
Tool measures compliance to 9 HF principles
Composed of 26 items with binary scoring (0 or 1)
Development and validation (for DDI alerts):
• Items were created for the quantifiable HF principles from the review
• 3 HF experts reviewed, modified and eliminated items
• 3 reviewers tested the items on their EMR systems
• 2 reviewers evaluated the same DDI alerts and IRR was assessed (k=0.76)
• Validity assessed – correlated performance of DDI alerts on I-MeDeSA with
age of the system
Example items
16 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
HF principle Item
Placement Is the alert linked with the medication order by
appropriate timing? (i.e. a DDI alert appears as soon as a
drug is chosen)
Visibility Is the area where the alert is located distinguishable from
the rest of the screen?
This might be achieved through the use of a different
background colour, a border, highlighting, bold
characters, occupying the majority of the screen etc
Applications of I-MeDeSA
17 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
Used to assess HF compliance of DDI alerts in 14 EHRs in the US
Used to assess DDI alerts in a large hospital in Korea
Not been used in Australia
Our study
Aims
19 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
1. To compare DDI alert interfaces in 8 electronic systems currently
used in Australia, in terms of their compliance with HF principles
1. To identify any potential problems with using I-MeDeSA
1. To determine whether alerts which are more compliant with HF
principles are also viewed more favourably by doctors and
pharmacists
Method – Part 1
20 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
Systems evaluated:
• To generate DDI alerts for evaluation, major DDIs were prescribed in the
training/test versions of each system
• 3 reviewers independently evaluated each DDI alert, then met to reach a
consensus on a score for each system
• A screen-shot of 1 alert (allopurinal + azathioprine) was taken from each
system for Part 2
Cerner PowerChart TrakCare
MedChart Medical Director
iPharmacy Best Practice
MOSAIQ FRED
Modified scoring
21 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
• During piloting, we found that scoring ‘yes’ for a number of I-MeDeSA
items was dependent on scoring ‘yes’ for the preceding item
• Is the prioritization of alerts indicated by appropriate colour?
• Does the alert use prioritization with colours other than red and green
to take into consideration users who may be colour blind?
• We also found that some items only applied to systems with multiple
levels of alerts in place
• Ten items were excluded to create a modified scoring system
Part 2 – user survey (n = 45)
22 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
• Which electronic prescribing system or medication dispensing
system are you currently using or have you used in the past?
• How long have you been using electronic prescribing or
electronic medication dispensing systems?
• On average, how many DDI alerts do you experience in a day?
• Please review the 9 DDI alerts below and rank the alert
interfaces from best to worst (1=best, 9=worst) using the table
below. Please also tell us what you like or dislike about the alerts
(e.g. the alert text was short).
23 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
24 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
25 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
Compliant alert
26 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
Results Data collection and analysis is ongoing
Main lessons learnt
28 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
• HF compliance was poor across the 8 systems
• Large variation in the design of DDI alert interfaces
• Non significant correlation between I-MeDeSA assessment
scores and user preferences
BUT
• Our ‘compliant’ alert was most preferred – this suggests that
the HF principles in I-MeDeSA are sound
• Lots of problems with I-MeDeSA including subjective items,
dependent items, arbitrary scoring (weights) assigned to each
HF principle
Where to now?
29 AUSTRALIAN INSTITUTE OF HEALTH INNOVATION
FACULTY OF MEDICINE AND HEALTH SCIENCES
• Finalise survey data collection and analysis
• Explore better ways to assess alert design:
• Working with our colleagues in Lille who have developed an
evidence-based framework linking alert usability principles to
usability flaws and usage problems
Top Related