The Effectiveness of Checklists versus Bar-codes Towards ... · iii Acknowledgments I would...
179
The Effectiveness of Checklists versus Bar-codes Towards Detecting Medication Planning and Execution Errors by Emily Rose A thesis submitted in conformity with the requirements for the degree of Master of Health Science in Clinical Engineering Institute of Biomaterials and Biomedical Engineering University of Toronto © Copyright by Emily Rose 2012
Transcript of The Effectiveness of Checklists versus Bar-codes Towards ... · iii Acknowledgments I would...
The Effectiveness of Checklists versus Bar-codes Towards Detecting Medication Planning and Execution
Errors
by
Emily Rose
A thesis submitted in conformity with the requirements for the degree of Master of Health Science in Clinical Engineering
Institute of Biomaterials and Biomedical Engineering University of Toronto
© Copyright by Emily Rose 2012
ii
The Effectiveness of Checklists versus Bar-codes Towards
Detecting Medication Planning and Execution Errors
Emily Rose
Master of Health Science in Clinical Engineering
Institute of Biomaterials and Biomedical Engineering
University of Toronto
2012
Abstract
The primary objective of this research was to evaluate the effectiveness of a checklist, compared
to a smart pump and bar-code verification system, at detecting different categories of errors in
intravenous medication administration. To address this objective, a medication administration
safety checklist was first developed in an iterative user-centered design process. The resulting
checklist design was then used in a high-fidelity simulation experiment comparing the
effectiveness of interventions towards two classifications of error: execution and planning errors.
Results showed the checklist provided no additional benefit for error detection over the control
condition of current nursing practice. Relative to the checklist group, the smart pump and bar-
coding intervention demonstrated increased effectiveness at detecting planning errors. Results of
this work will this work will help guide the selection, implementation and design of appropriate
interventions for error mitigation in medication administration.
iii
Acknowledgments
I would foremost like to extend my deepest appreciation to my supervisors, Dr. Joseph Cafazzo
and Dr. Patricia Trbovich, who have provided me with outstanding guidance, support and
mentorship. I could not have wished for a better experience and am constantly inspired by their
dedication and brilliance.
Secondly, I would like to thank the members of my committee, Dr. Kim Vicente, Dr. Linda
McGillis-Hall, for their insight and expertise. Thank you for your enthusiasm and encouragement
for this work.
I am also very grateful to the members of the Centre for Global eHealth Innovation community
for their help throughout this process. In particular, Stefano Gelmi and Michael Lipton, for
helping with the set-up and troubleshooting; thanks to Tara McCurdie, Wayne Ho, Alison
Bisson, Aarti Mathur, Mark Fan, Sonia Pinkney, Melissa Griffin, Chris Colvin for lending their
expertise; thanks also to the students who helped facilitate the study sessions and manage the
observation room cameras: Christopher Flewwelling, Lata Grover, Sarah Greenberg, and Rossini
Yue. I am indebted to you all for your hard work and tremendous help in completing this project.
Finally, I would like to acknowledge the incredible support and enthusiasm for this project from
Anne Vandeursen and the nurses from the GIM units at TGH and TWH hospital. Anne, thank
you for being my clinical guide and for all the introductions you provided to smooth the way.
Lastly, to all my friends and family, thank you for helping me work through the stressful
moments!
iv
Table of Contents
Acknowledgments iii
Table of Contents iv
List of Tables vii
List of Figures ix
List of Abbreviations xi
List of Appendices xii
1 Introduction 1
1.1 Chapter Outline 2
2 Background 4
2.1 Human Factors in Healthcare 4
2.2 Intravenous Medication Errors 5
2.3 Human Performance and Error 7
2.4 Hierarchy of Effectiveness 9
2.5 Alternative Models for Classification of Remedial Actions 12
2.6 Evaluation of Effectiveness 14
2.7 Application of Checklists in Healthcare 21
2.8 Guidelines for Checklist Development and Design 22
2.9 Smart Pumps and Bar-code Technology 23
3 Objective and Hypothesis 25
3.1 Gap in Knowledge 25
3.2 Hypotheses 25
3.3 Specific Aims 26
4 Development of an IV Medication Administration Safety Checklist 27
4.1 Checking It Twice 27
4.2 Framework for Development of Medical Checklists 28
v
4.3 Initial Checklist Design 30
4.4 Usability Testing Methodology 33
4.4.1 Sampling and Recruitment of Participants 33
4.4.2 Location 33
4.4.3 Procedure 34
4.4.4 Data Collection 35
4.4.5 Data Analysis 36
4.5 Results: First Design Iteration 36
4.5.1 Checklist Completion 36
4.5.2 Qualitative Analysis by Workflow 38
4.5.3 Post-Test Questionnaire 40
4.6 Usability Issues and Checklist Redesign 41
4.7 Results: Second Design Iteration 46
4.7.1 Checklist Completion 46
4.7.2 Post-Test Questionnaire 48
4.8 Discussion 50
4.9 Limitations 51
4.10 Summary 52
5 Checklist versus Bar-code Smart Pump (High-Fidelity Simulation) 53
5.1 Objectives 53
5.2 Methodology 53
5.2.1 Experimental Design 53
5.2.2 Location 54
5.2.3 Sampling and Recruitment of Participants 54
5.2.4 Experimental Set-up 55
5.2.5 Procedure 65
5.2.6 Statement of Specific Hypotheses 67
5.2.7 Data Collection 69
5.2.8 Data Analysis 69
5.3 Results 73
5.3.1 Demographics Questionnaire 74
5.3.2 Error Detection Rates 75
vi
5.3.3 Medication Delivery Failures 80
5.3.4 Post-test Questionnaire 82
5.4 Discussion 85
5.4.1 Error Detection Rates 85
5.4.2 Confirmation or Rejection of Specific Hypotheses 89
5.4.3 Medication Delivery Failures 93
5.5 Study Limitations 94
5.6 Summary 97
6 Conclusions 98
6.1 Relevance to the Field 99
6.2 Future Work 99
References 101
Appendices 105
vii
List of Tables
Table 1: Description of the Hierarchy of Effectiveness ................................................................ 10
Table 2: Review of studies concerning strategy of interventions in medication errors ................ 15
Table 3: List of Nielsen usability heuristics61
............................................................................... 29
Table 4: Definitions of success and failure for checklist completion ........................................... 36
Table 5: Comparative analysis of post-test survey results between design iterations .................. 48
Table 6: Task, Planted Errors and Error type ............................................................................... 64
Table 7: Statements of specific hypotheses .................................................................................. 67
Table 8: Pass and fail criteria for error detection .......................................................................... 70
Table 9: Demographic survey results for each intervention group ............................................... 74
Table 10: Confirmation of rejection of specific hypotheses. ........................................................ 89
Table 11: Sample of data collection log ..................................................................................... 144
Table 12: Test statistics for comparison between groups ........................................................... 152
Table 13: Descriptive statistics of mean error detection ............................................................. 154
Table 14: Between group and within-subjects significant effects .............................................. 155
Table 15: Marginal mean across condition ................................................................................. 155
Table 16: One way ANOVA of two-way interaction between groups by error type ................. 155
Table 17: Marginal mean across error type ................................................................................ 156
Table 18: One way ANOVA of two-way interaction between groups by condition .................. 156
Table 19: Pair-wise comparisons ................................................................................................ 156
viii
Table 20: Statistics for within subject comparison of individual planted errors by group ......... 156
Table 21: Descriptive statistics of medication failures ............................................................... 157
Table 22: First usability test post-test questionnaire results ....................................................... 159
Table 23: Second usability test post-test questionnaire results ................................................... 160
Table 24: Checklist group post-test questionnaire results .......................................................... 162
Table 25: Smart pump group post-test questionnaire results ...................................................... 163
Table 26: Post-test survey results comparative analysis between checklist and smart pump groups
..................................................................................................................................................... 165
ix
List of Figures
Figure 1: Classification of medication errors by the psychological classification system36
........... 8
Figure 2: Hierarchy of Effectiveness proposed by ISMP16
.......................................................... 10
Figure 3: The hierarchy and description of tools for remedial action from the TGRA report.38
.. 13
Figure 4: Initial IV medication administration checklist design ................................................... 32
Figure 5: Total number of failures by checklist section from 25 checklist uses ........................... 37
Figure 6: Activity diagram of nursing medication administration workflow. .............................. 39
Figure 7: New design of IV medication administration safety checklist ...................................... 45
Figure 8: Total number of failures by checklist section for 35 checklist uses .............................. 46
Figure 9: Comparison of total failures between usability tests ..................................................... 47
Figure 10: Visual representation of experimental design ............................................................. 54
Figure 11: Bed layout in simulated GIM unit with COW cart in the center of the floor .............. 56
Figure 12: Image of patient bed with IV pole location and end table of supplies ........................ 56
Figure 13: Close-up of mock patient displaying wristband and IV access ................................... 57
Figure 14: Alaris system by cardinal health with two pump modules .......................................... 59
Figure 15: Example of clinical advisory in drug library ............................................................... 60
Figure 16: Guardrails drug library screen ..................................................................................... 61
Figure 17: Patient wristband with bar-code .................................................................................. 62
Figure 18: Sample medication label with bar-code ...................................................................... 63
Figure 19: Mean error detection to visualize three-way interaction ............................................. 76
x
Figure 20: Differences between condition and group after averaging across the two error types
(execution and planning)............................................................................................................... 77
Figure 21: Differences between error type and group after averaging across the two conditions
(baseline and intervention) ............................................................................................................ 78
Figure 22: Error detection rate by planted error between baseline and intervention in the checklist
group (N=24) ................................................................................................................................ 79
Figure 23: Percentage of nurses that detected each error type in the smart pump group (N=24) 80
Figure 24: Number of medication delivery failures by type for the checklist group (N=24) ....... 81
Figure 25: Mean number of medication delivery failures by type for the smart pump group
(N=24) ........................................................................................................................................... 82
Figure 26: Differences between condition and error type after averaging across the two groups
(checklist and smart pump) ......................................................................................................... 153
Figure 27: Differences in mean error detection rates in main effect of error type ..................... 154
Figure 28: Differences in mean error detection rates in main effect of condition ...................... 154
xi
List of Abbreviations
AIP – Ambulatory Infusion Pump
BCMA – Bar-code Medication Administration
CPOE – Computerized Physician Order Entry
eMAR – Electronic Medication Administration Record
GIM – General Internal Medicine
HoE – Hierarchy of Effectiveness
IOM – Institute of Medicine
ISMP – Institute for Safe Medication Practices
IV – Intravenous
LPN – Licensed Practical Nurse
MOE – Medication Order Entry
OR – Operating Room
RN – Registered Nurse
TGH – Toronto General Hospital
TWH – Toronto Western Hospital
UHN – University Health Network
VTBI – Volume To Be Infused
WHO – World Health Organization
xii
List of Appendices
Appendix A: UHN Form for Independent Double Check of Ambulatory Infusion Pumps .................... 105
Appendix B: Research Ethics Board Approval .................................................................................. 106
Appendix C: Consent Forms ............................................................................................................ 108
Appendix D: Experimental Protocol and Script ................................................................................ 114
Appendix E: Sample Physician Paper Medication Order ................................................................... 141
Appendix F: Sample Physician Electronic Order ............................................................................... 142
Appendix G: Sample Investigator Data Collection Sheet .................................................................. 144
Appendix H: Demographic Questionnaire ....................................................................................... 146
Appendix I: Post-Test Questionnaires ............................................................................................. 148
Appendix J: Statistics for Participant Demographics ........................................................................ 152
Appendix K: AdditionalStatistics for Error Detection Rates .............................................................. 153
Appendix L: Usability Test Post-Test Survey Results ........................................................................ 159
Appendix I: High-Fidelity Simulation Post-Test Survey Results ......................................................... 162
1
1 Introduction
Annually, between 44,000 to 98,000 deaths in the United States are attributable to preventable
medical errors estimated by the Institute of Medicine (IOM) in the 1999 report, To Err is Human:
Building a Safer Healthcare System.1
Specifically, medication errors were accountable for 7,000
deaths with a preventable adverse drug event occurring in an estimated 2% of all hospital
admissions.1
Similarly, Baker et al. (2004) conducted a related study of adverse medical events in
Canadian hospitals. The findings of this study reported that 7.5% of admissions to acute care
hospitals experienced an adverse event, of which 36.9% were deemed highly preventable.2
Drug-
and fluid- related events were identified as the second most common type of adverse event next to
surgical procedures.2
In a more recent report from the IOM, Aspden et al. (2007) estimated at least
1.5 million people are affected annually by adverse medication events with at least one error
occurring per patient per day.3
In light of the reported prevalence of preventable medical errors in healthcare and its adverse
effect on patient outcomes, the need for error interventions is apparent. Studies have investigated
the implementation and effect on error reduction of various interventions, such as training,
checklists, smart pumps and bar-code medication administration systems.4-15
However, these
studies examine the utility of only a single strategy and fail to compare the relative effectiveness
between different mitigation strategies. They also fail to distinguish the effectiveness of the
interventions at preventing different types of errors, particularly rule- or knowledge-based
mistakes (e.g. giving penicillin without knowing the patient is allergic) and execution failures,
either action-based slips (e.g. misreading the drug label) or memory lapses (e.g. forgetting to start
an infusion or unclamp tubing).
A theoretical ranking of the potency of a particular solution to mitigate errors has arranged
interventions into a hierarchy of effectiveness. This hierarchy has been referenced in the “grey”
literature16,17
as a ranking of the most effective error prevention tools but lacks a comprehensive
study validating the appropriateness of this ranking. The hierarchy states that design oriented error
prevention tools, which alter the system, are more powerful than people oriented strategies, which
still rely on human vigilance. Experimental evidence for the hierarchy of effectiveness is difficult
2
to establish due to tendencies to report only successes and not negative results in published
literature.
Therefore, the objective of this research is to directly compare the effectiveness of a human
oriented checklist with bar-code smart pump technology and to assess the effectiveness of each
strategy in relation to the type of error. Errors are categorized into planning failures (rule- and
knowledge-based mistakes) and execution failures (action-based slips and memory lapses) based
on the psychological classification. Evidence of the relative effectiveness of a behavioral oriented
checklist intervention to smart pump and bar-code technology will aid in selecting and
implementing appropriate error prevention tools to minimize adverse events and improve patient
safety. Further, this study establishes which error prevention tools are most effective at detecting
planning versus execution failures and the extent to which they can be relied upon to successfully
mitigate errors.
1.1 Chapter Outline
Chapter 2 presents the relevant background for this study covering the application of human
factors to healthcare, the instance of IV medication errors and the relationship to human
performance and error, the hierarchy of effectiveness and other models for error mitigation
strategies. Following this generic presentation of error and remedial actions, a review of specific
strategies applied to healthcare is described. In particular, the application, design and development
of checklists in healthcare are examined in addition to smart pumps and bar-code technology.
Upon establishing the current state of the field, Chapter 3 presents the gaps in knowledge
followed but the main hypotheses and specific aims of this study.
Chapter 4 then discusses the development of an IV medication administration safety checklist. A
closer analysis of previous work in this area is presented prior to outlining the objectives, approach
and methodology. The framework for developing the checklist is provided before presenting the
results and iterative designs of the checklist after usability testing.
3
The main high-fidelity simulation experiment comparing the checklist versus bar-code smart pump
system in detecting IV medication errors is presented in Chapter 5. This chapter includes the
objectives, methodology, results, discussion and limitations of the study.
Finally, Chapter 6 presents the final conclusions of this research and its contributions to the fields
of medication administration and patient safety and directions for future work.
4
2 Background
2.1 Human Factors in Healthcare
Human factors engineering takes into account human capabilities and limitations in the design of
systems, tools, processes and devices to minimize error.18
Medical errors and accidents can occur
despite safe practice and the diverse nature of these errors attributes them to a multitude of
individual and systemic contributory factors and causes.19
Anderson et al. define human factors
engineering as the discipline that studies the capabilities and limitations of humans and the design
of devices and systems for improved performance.18
Medical errors may lead to an adverse event or iatrogenic injury in which the patient is harmed as
a result of the error. Baker et al. define adverse events as inadvertent injuries or complications
leading to death or disability upon discharge, or requiring extended hospital stays as a result of the
medical care received.2 Similarly, iatrogenic injuries defined as injuries or disabilities resulting
from the medical treatment rather than from the patient’s underlying disease condition.2
A study by
Vincent et al. discusses failures leading to iatrogenic injury, in which patients are erroneously
injured by medical treatment. 20
Iatrogenic injury occurs in 4% of hospital admissions and requires
a human factors understanding to enhance safety and reduce clinical risk.20
Similarly, Baker et al.
have reported as many as 16% of hospital patients in Australia are injured as a result of their
treatment.2 According to Baker et al., identification and reporting of errors is needed to reduce the
potential for them to lead to adverse events or iatrogenic injury.2 This requires non-punitive
policies to diminish blame which currently discourages errors from being reported.2
O’Connor et
al. report a degree of error occurs in 5-15% of all hospital admissions, with 45% of errors
occurring in the operating theatre.21
This error rate is attributed to the high-risk environment
created by staffing limitations, high turnover rates, and site and side-specific surgical procedures.21
Further error prevention challenges in healthcare identified by Anderson et al. include delayed
feedback to healthcare workers, high cognitive workload, and poor ergonomic design.18
In
addition, a review by Scanlon et al. that evaluates a human factors approach to understanding
failures in a critical care setting, recognizes the physical factors contributing to human
performance to include time-pressure, volume and intensity of information, and frequent
interruptions.22
Human factors principles can thus be applied in healthcare to mitigate errors and
improve the design of the clinical environment, workflow and policies to enhance patient safety.
5
Easty et al. demonstrated the application and functional role of human factors engineering applied
to the healthcare setting and evaluated its effectiveness at improving errors in healthcare
delivery.23
Carayon et al. also emphasized the need for increased collaboration between health
sciences, human factors and systems engineering in identifying methods for analyzing, preventing
and mitigating medical errors.24
Due to the complexity of the interactions between clinicians and
technology in healthcare, application of human factors strategies to prevent technology-induced
errors is needed.25
As these studies indicate, adverse events in healthcare are complex in nature and require a better
understanding of the types and causes of errors and an investigation of the effectiveness of error
prevention strategies to guide the design and subsequent implementation of interventions to reduce
adverse events.
2.2 Intravenous Medication Errors
Medication administration has been recognized as a significant source of errors in healthcare and a
gross threat to patient safety.3, 26-32
While errors occur in the ordering and prescription stages of the
medication pathway, medication administration errors are of concern as many of them are
considered unlikely to be intercepted.26
In particular, errors due to intravenous (IV) medication
administration are considered the most serious and likely to result in direct patient harm.27
In
contrast to other medication errors, IV medication errors have been associated with a significantly
higher rate of deaths.28
The seriousness of IV medication errors is due to a combination of factors
which include: the immediate physiological availability of drugs administered intravenously, the
frequent IV delivery of high-alert medications with a narrow therapeutic range, the complexity of
the task and the complex technologies involved in delivery.26, 27
Westbrook et al. observed at least one clinical error in nearly 70% of all IV medications
administered, of which 25% were determined to be serious errors with the potential for permanent
harm to the patient.29
Similarly, Husch et al. found 67% of IV infusions to be associated with an
administration error26
while Taxis et al. reported overall error rates of 48% in a study at a German
hospital.30
The literature investigating IV medication errors includes the following types of
errors:27, 29
Omission (patients do not receive ordered dose)
6
Wrong dose (the amount administered is different from the amount ordered)
Incorrect programming (infusion devices delivering incorrect drug amounts)
Wrong drug (a different medication is infused from medication listed on the order)
Wrong rate (medication is delivered at a different rate than what was ordered)29
Wrong time (delay of rate or medication change26
or in administration of a dose)
Incorrect preparation of drug (e.g. wrong diluents/mix)
Drug incompatibility
Wrong patient
Wrong administration technique
Wrong route
Of these errors, wrong administration technique and wrong route are considered the most
dangerous by Hicks et al. as they more often result in severe outcomes.27
Other studies report
errors in the administration rate of bolus to be the most common error with serious implications.29,
31, 32
In an American study, Hicks et al. used the national medication error reporting program
(MEDMARX) to investigate IV-related medication errors by volume, type, cause and outcome.27
In addition to errors classified as performance deficits, protocol deviations and inaccurate or
omitted transcription, three factors were identified that expose patients to potential IV medication
errors.27
These factors were drug shortages, tubing interconnectivity and calculation errors. Drug
shortages led to substitutions being made for unavailable drugs, which were later determined by
pharmacy not to be bioequivalent. Tubing interconnectivity exposed the possibility of switching
peripheral IV and epidural lines or delivering oral medication via IV infusion.27
Finally, mistakes
in calculations led to wrong rates, doses or concentrations given often resulting in overdoses.27
A study by Fahimi et al. considered the effect of shift time on the rate of IV errors and found the
9:00am shift to have the highest rate of errors.32
This shift was identified as the peak time for
distractions, thereby increasing nursing workload and hindering the ability of nurses to focus on
their medication administration tasks.32
While Fahimi et al.32
found no correlation between nurse
age, experience or other demographics, Westbrook et al. found experience to be related to error
rates.29
Using multivariable logistic regression, Westbrook et al. found error rates and severity to
decrease with clinical experience.29
Specifically, Westbrook et al. reported the risk of error to
7
decrease by 11% for each year of experience up to six years, after which there was no change with
additional years of experience.29
While much of the literature focuses on the frequency, severity and type of IV medication errors,
some studies attempt to identify the causes of these errors using human error theory.27, 29, 31, 32
Understanding the cause of IV errors from a psychological and system’s perspective is recognized
as a crucial component for reducing these errors and selecting appropriate mitigation strategies.
2.3 Human Performance and Error
Human error is generalized by Reason as all instances in which a planned sequence of mental or
physical events does not result in the intended outcome.33
Spencer further identifies errors as
frequently occurring by-products of normal thinking and not character defects to be treated by
discipline and education.34
Three potential classification systems exist to help understand medication error.35
These
classification systems include:35
contextual, which gives a specific time, place, medication and persons involved
modal, which describes the way the error occurred
and psychological, which explains how the events occurred in terms of mistakes, slips and
lapses.
The psychological classification of errors is useful to understand why the error occurred and
potentially determine the best strategy to manage the error. By the psychological classification
system presented by Ferner and Aronson (Figure 1) errors are categorized into execution failures
(slips and lapses) and planning failures (mistakes).36
Slips are attributable to attention failures
whereas lapses relate to internal failures of memory rather than observable actions.33
Errors which
occur despite actions going as planned are defined as mistakes where the executed plan did not
achieve the intended outcome.33
8
Figure 1: Classification of medication errors by the psychological classification system36
This classification is an adaptation of Rasmussen’s conceptual framework for human performance
which is structured into three basic error types: skill-based errors, rule-based mistakes and
knowledge-based mistakes.33
These levels of performance are ranked according to the degree of
familiarity with the task or environment.33
The skill-based level of human performance involves
little thought and is generally related to preprogrammed instructions.33
In the context of
medication administration, a nurse neglecting to start an IV infusion due to a distraction would
constitute an execution failure, or more specifically, an action-based slip. Mistakes, either rule-
based or knowledge-based, are associated with slightly more complex tasks.33
According to
Reason, rule-based mistakes relate to problems which occur despite training or experience due to
an error in applying a rule.33
This may involve the misclassification of a situation or incorrect
recall of a procedure.36
Finally, knowledge-based mistakes are identified in a situation requiring
on-the-spot solutions with errors occurring due to a bias or mindset that dictates an inappropriate
solution for the given situation.33
Knowledge-based mistakes generally occur in novel situations of
limited familiarity and rely on conscious, analytical processes and abstract knowledge.33
Failure to
recognize an inappropriately prescribed dosage during medication administration is an example of
a knowledge-based mistake.
Human Errors
Actions intended but not performed
Planning Failures
Knowledge-based mistakes
Rule-based mistakes
Good rules misapplied
Bad rule or failure to apply a good rule
Execution Failures
Action-based slips
Memory-based lapses
9
Based on the skill-rule-knowledge framework, expectation of errors may relate to the familiarity
and complexity of the task. Mechanistic tasks, which compare two tangible sources of information
and require minimal stored knowledge or integration of information, may be prone to action-based
slips or memory lapses. Abstract tasks requiring integration of knowledge from multiple sources
and analytical thought processes for interpretation and evaluation of a situation relate to the rule-
and knowledge-based level of error.
Ferner and Aronson use the psychological classification of medication errors to determine the best
strategies to manage medication errors.36
Knowledge-based mistakes are suggested to be best
managed by teaching and computerized decision support while rule-based mistakes, such as
confusion over the correct intravenous administration rate, may require modifications to protocols
and guidelines.36
Training and checking procedures are suggested to be effective for action-based
slips, such as misreading the wrong dose on the label. Finally, memory-based lapses, for example,
missed medication delivery, may be addressed by reducing distractions and patient load. This
assessment of medication errors suggests that error prevention strategies should thus consider the
type of error to determine the most effective intervention.
2.4 Hierarchy of Effectiveness
The Hierarchy of Effectiveness, proposed by the Institute of Safe Medication Practices (ISMP) is a
ranking of the potency of a particular solution to mitigate errors.16
As seen in Figure 2, the
hierarchy is divided into 6 categories ranked from most effective (top) to least effective (base) in
mitigating errors. These tools to address failure modes were developed internally by ISMP through
expert opinion and observations during hospital consults.37
They range from basic education and
training at the bottom of the hierarchy to more invasive forcing functions at the top. The hierarchy
thus serves as a tool to aid in the design and evaluation of interventions to reduce errors.16, 38
10
Figure 2: Hierarchy of Effectiveness proposed by ISMP16
The overall concept of the hierarchy attempts to address errors by improving the system rather
than focusing on a punitive human approach to prevent errors. According to the hierarchy,
interventions, which address human behavior, such as training and policies, act in a reactive
manner. They do not address underlying systemic or environmental issues leading to error and so
by the hierarchy are less effective.16
Those interventions which address the technology and attempt
to eliminate the error at a systems level, such as automation and forcing functions, are by contrast
more proactive and thus more effective using the hierarchy schema.16
The various levels of the
hierarchy are described below in Table 1.
Table 1: Description of the Hierarchy of Effectiveness
Level of Hierarchy Advantages Disadvantages
Forcing Functions
May be physical attributes
of the system, like the
mechanical inability of oral
syringes to fit into IV
catheters, or a constraint
such as a withholding step
whereby the next stage in
Effectively eliminates
human error by
eradicating the
possibility of making a
particular error.16
Built into the device or
system and are thus
‘people-independent’.
May be more expensive and
resource intensive to
implement.
11
the process may not be
completed until the
previous step is executed.16
Automation and
Computerization
Bar-coded medication
administration (BCMA)
systems and smart infusion
technology are examples of
automation capable of
detecting misidentification
of patients or drugs and
may include warnings of
incorrect drug
information.13, 39-40
Attempts to fix the
system rather than
relying on human
vigilance.17
Reduces the tasks
required by a human
and reliance on human
capacity and memory
as it may include tasks
such as performing
calculations or
providing reminders.16,
17, 38
Introduction of new errors
and changes in behavior may
result. (As described by
Cochran et al., users created
workarounds to built-in
checks and found ways to
override key warnings for
incorrect drug
information.40
)
Standardization and
Simplification
Standard protocols for
chemotherapy
administration seek to
streamline the workflow
and increase compliance
with policies for good
practice.41
Increases familiarity of
a system, device or
process.
Critical checks become
routine and are less
likely to be omitted by
a slip or memory
lapse.41
Continues to require human
vigilance for implementation
rather than addressing the
technological system or
workflow.17
Reminders, Checklists
and Double checks
May include visible
signage, checklists for
specific tasks or double
checking procedures, either
dependent or independent.
Independent double checks
are often used for
verification tasks where a
second nurse will double
check the patient ID and
pump programming before
an infusion is given.
May reduce error based
on the lower
probability of two
people performing the
same error.
Reduces reliance on
internal memory to
recall protocols and
tasks.
May improve
compliance to standard
practices and aid in
standardizing a
procedure or workflow.
Reactive rather than
proactive; it does not prevent
human error from occurring
but rather is meant to detect
error.17
Dependent on proper
implementation and may be
difficult to maintain with
changes in organizational
leadership.42
Design and content of the
checklist may dictate the
type of error towards which
it is effective and thus may
provide only a solution for
specific, predictable errors.
12
Rules and Policies
Policy may require nurses
to check two elements of
patient ID (name and MRN)
before administering
medication.
May encourage
adherence to
standardized practices
and support other
effective error
intervention
strategies.42
Does not fix the physical
system.17
Requires proper
implementation by the
organizational structure for
enforcement.
Susceptible to changes in
leadership within an
organization.38
Education and Training
Specific training on a new
device, simulation training
to teach skills and practice
clinical techniques, or
presentations to emphasize
a particular aspect of patient
care are some examples of
educational interventions.
May improve
knowledge to reduce
mistakes due to
insufficient education
to successfully
complete a given task.
Relies purely on human
performance and ability and
does not address the latent
failures in the system.17
Sustainability is questionable
if situations are not regularly
experienced (i.e. training on
a particular infusion pump
may reduce errors initially
however; over time it may be
ineffective if the device is
not consistently used).
Unlikely to alter rate of slips
and lapses as they have been
shown to be experienced by
experts even more than
novices.43
2.5 Alternative Models for Classification of Remedial Actions
The ISMP presents the hierarchy as an accepted ranking of the relative effectiveness of error
mitigation strategies regardless of the type of error. Thomadsen and Lin, in the Taxonomic
Guidance for Remedial Actions (TGRA) report, present a similar classification of remedial actions
for adverse events (shown in Figure 3).38
This report classifies errors based on research by
Norman, Rasmussen and Van Der Schaaf and presents remedial actions for different classifications
of errors. The taxonomy of remedial actions presents a ranking similar to the ISMP hierarchy of
effectiveness but also presents the PRISMA tool and Eindhoven classification model for risk
management.
13
Figure 3: The hierarchy and description of tools for remedial action from the TGRA
report.38
PRISMA is a risk management tool developed at the Eindhoven University of Technology in the
Netherlands by Van Der Schaaf. It consists of three components: causal tree incident descriptions,
Eindhoven classification model (ECM) for system failure and classification/action matrix.44
This
tool classifies root causes and contributing factors and translates them into proposals for
preventative action. Causal tree incident descriptions identify the critical activities leading up to an
incident and present the root causes at the bottom of the tree.44
The ECM then classifies the
identified root causes of the adverse events based on three main categories of system failure:
technical, organizational and human.44
Technical failure refers to failures in the design,
construction, or materials of equipment while organizational failures consider management
•sound and visual control, cleaning, isolation, design
1. Fix environmental problems
• interlocks, barriers, computer entry with feedback
2. Forcing functions & constraints
•bar-codes, computer entry, computer feedback, automatic monitoring
3. Automation & computerization
•check-off forms, establishing/clarifying protocol, alarms, labels and signs, reducing similarities
4. Protocols, standards & information
• independent review, redundant measures, operational checks, comparison with standards, increased monitoring, status check, acceptance test
5. Independent verification & redundancy
•External or internal audit, priority, establish/clarify communication, staffing, scheduling, mandatory pauses, repair, preventative maintenance, establish/perform quality control assurance
6. Rules & policies
• initial training, experience, instrustion
7. Education & information
14
priorities, culture of the workplace and protocols.44
The human failure category considers skill-,
rule- and knowledge-based failures.44
Following the classification of the identified root causes, the classification/action matrix is used to
link failure modes to specific classes of action. The five classes of action proposed by Van Der
Schaaf include: equipment (redesign of hardware, software or interfaces), procedures (completing
or improving protocols), information and communication (improving access to information and
communication structure), training (retraining to improve skills) and motivation (applying
principles for positive behavior modification).44
Actions in the matrix are linked to errors based on
their expected effectiveness. Considering the human failures, equipment improvements are
suggested to be effective for skill-based errors while training is expected to be effective for human
rule-based errors.44
Improving access to information and the structure of communication is
expected to be effective for human knowledge-based errors.44
Relating the PRISMA model, specifically the classification/action matrix component, to the
hierarchy of effectiveness, components from the human-oriented levels of the hierarchy are
suggested to be effective for rule- and knowledge-based errors while more technological solutions
or improvements are needed to address skill-based errors. This model thus differs from the theory
of the hierarchy, which ranks strategies regardless of the error they are meant to address. Thus, the
investigation of the effectiveness of strategies from both the human and technological levels of the
hierarchy to various types of human error may discern which guidance for interventions is most
appropriate.
2.6 Evaluation of Effectiveness
A survey of publications on patient safety and error interventions related to medication
administration was conducted to investigate evidence relating to the effectiveness of different error
mitigation strategies. A systematic search of the literature was done through ISI Web of
Knowledge, Scopus and MEDLINE databases. Various combinations of the following keywords
were used: medication administration, adverse drug events, interventions, and human factors. The
results of the keyword searches were screened by title and abstract for relevance according to
predetermined criteria. Criteria for studies selected for review included those, (a) published in
English in the last 10 years and (b) containing experimental or observational studies on
15
interventions to reduce medication administration errors. Based on the predetermined criteria, a
total of 10 studies were selected. Table 2 summarizes the type of intervention, methodology, and
main findings of the relevant studies.
Table 2: Review of studies concerning strategy of interventions in medication errors
Author and
Intervention Methodology Types of Errors Results
1. Schneider
et al.
(2006)4
Education
and
Training
Evaluated the effectiveness of
an interactive CD-ROM
program with a group of 30 RNs
by direct observation of error
rates pre- and post-intervention.
Deviations from
safe practices
(e.g. borrowing a
dose from another
patients’ supply).
Preparation and
administration
errors (e.g.
incorrect time,
dose or
technique).
Deviations from
prescribed
therapy (e.g.
failure to check
drug, dose, rate,
form or route
against prescribed
order).
Results showed a
decrease in safe
administration practice
errors after CD-ROM
training however; no
changes in preparation
and administration errors
or deviations from
prescribed therapy were
seen. Based on the
improvement in safe
practice errors, the
intervention was useful
in addressing rule-based
mistakes where there was
a failure to apply a good
rule however; it was
ineffective at addressing
execution failures (e.g.
wrong time, or dose).
2. Trivalle et
al. (2010)5
Education
and
Training
The impact of an educational
intervention (a teaching unit
providing oral and written
advice on prescription habits) on
the rate of adverse drug events
(ADEs) in 16 geriatric centers
was investigated. ADEs were
recorded by review of patient
charts 2 weeks prior to the
intervention and for 2 weeks
post-intervention with 1 week in
between for the educational
phase.
The specific errors
leading to the
ADEs were
identified as:
High dose
Double therapy
Under dose
Inappropriate
drug
Drug-drug
interactions
Previous same
adverse drug
reaction
Following the 1-week
educational program,
14% fewer adverse drug
events were observed in
the intervention group
compared to the control
group. ADEs were
identified by signs or
symptoms most
frequently related to an
ADE. The results did not
indicate which types of
errors were addressed by
the intervention so it
remains unclear which
16
Unidentified
aspects may have led to
the reduction in ADEs.
3. Blank et
al. (2011)6
Education
and
Training
Pre and post outcomes in a 3-
month single group study after
an educational intervention on
“Preventing Medication and IV
Administration Errors” were
compared. Outcome measures
were identified by chart reviews
and voluntary error reports. Pre-
and post-intervention tests and
surveys were also used to assess
knowledge of safe practices.
Medication errors
related to the ‘5
rights’ but also
included:
Omission of
medications and
IVs that were
ordered.
Medications
given and IV
lines started with
no written order.
Documentation
errors.
While testing knowledge
improved, this result was
not seen to translate to
clinical practice, as there
was no change in total
medication errors post-
intervention. Education
was therefore unable to
address medication errors
in practice, which may
have occurred due to
slips or lapses. This
result is consistent with
the human error theory in
which execution errors
(slips or lapses) are
known to occur
regardless of experience
and observed as often
with experts as with
novices.34
4. Ford et al.
(2010)7
Education
and
Training
The effectiveness of traditional
didactic lectures to a simulation-
based training session using
case scenarios and a patient
simulator was evaluated. Nurses
were required to discover and
remedy errors, and practice
administration techniques.
Medication administration
errors were observed for 24
nurses at baseline and twice
post-intervention. Knowledge
was also assessed by quizzes.
Medication errors
included:
Incorrect
preparation of
drug
Expired drug
Improper dose or
quantity
Mislabeling
Omission
Prescribing
Unauthorized/wro
ng drug
Wrong
administration
technique
Wrong dosage
form
Wrong route
Wrong time
The study did not
analyze changes in
individual error types
however; overall error
rates decreased following
the simulation training
and were sustained in the
final observation 8-12
weeks post-intervention.
Conversely, for the
didactic lecture group,
error rates increased in
the final post-
intervention. Knowledge
assessed by quiz scores
improved after both types
of educational sessions.
These results seem to
imply that education will
only have an effect on
practice (i.e. error rates)
17
if there is an extra
component in the
training, such as
simulation in this case.
5. Mills et al.
(2008)8
Education,
reminders
and forcing
functions
Root cause analysis reports from
Veterans Affairs hospitals were
used to determine actions taken
to reduce adverse drug events
and their effectiveness. Actions
included updating equipment,
changes to drug storage,
training/education, improving
clinical care at the bedside and
the use of alerts and forcing
function in the medication order
entry process.
Errors leading to
ADEs included:
Wrong dose Wrong drug Wrong patient Failure to give
medication Insufficient
monitoring and evaluation of patient on med
Wrong route Allergic reaction Wrong time Prescription of
wrong medication
Reductions in adverse
drug events were
correlated to
improvements in
equipment and the use of
alerts and forcing
functions in the
medication order entry
process. Training was
associated with worse
outcomes. These results
appear to suggest again
that training does not
affect practical outcomes.
6. Relihan et
al. (2010)9
Training,
reminders
and
checklists
Pre- and post-intervention
(behavior modification and staff
education; checklists; visible
symbols in the form of a red
vest; and signage) observation
of nurses in acute medical
admissions unit was compared
to measure the rate of
interruptions.
This study did not
directly measure
error rates but
focused on the
reduction of
interruptions and
distractions known
to correlate to error
rates.
The rate of interruptions
decreased to 43% of the
original level but could
not be correlated to a
specific intervention.
While the study provided
a breakdown of the type
of interruption, it did not
identify the direct impact
of individual
interventions on the
volume of errors.
7. White et
al.
(2010)10
Checklists
and double-
checks
Error detection rates between
two different checklists were
compared for four categories of
medication administration errors
in a high-fidelity simulation
study of medication
administration in an outpatient
chemotherapy unit.
Errors were
classified as:
Pump
programming
errors
Patient ID errors
Drug label and
order mismatch
Clinical decision
The revised checklist
designed with explicit
step-by-step instructions
for detecting specific
errors was more effective
at detecting errors in
verification tasks.
Neither version of the
checklist identified
clinical decision errors.
18
errors Thus, checklists were
found to be more
effective for error
detection tasks than for
clinical decision-making
tasks.
8. Trbovich
et al.
(2010)11
Automation
and
computerize-
tion
A high-fidelity simulated
inpatient unit was used to test
detection rates of planted
medication errors between
different infusion pump types.
The study compared traditional
pumps to smart pumps and
smart pumps with barcode
capabilities.
Planted errors
included:
Wrong drug
Wrong patient
Wrong dose
Pump
programming
errors
Bag mis-
alignment for
secondary
infusions
Tubing
arrangement
errors
Failure to
unclamp tubing
Error rates of “wrong
drug” did not vary with
pump type. Patient ID
errors were remedied
more significantly with
the barcode pump than
both the traditional and
smart pump. Hard limit
wrong dose error
remediation was higher
for smart and barcode
pumps than traditional
pumps. However, soft,
changeable limits in
smart pumps had no
significant effect in
preventing dosing errors.
Thus, higher degrees of
automation and
computerization were
found to increase error
detection.
9. Rothschild
et al.
(2005)12
Automation
and
computerizat
-ion
Adverse events using infusion
pumps with and without clinical
decision support for 744 cardiac
surgery admissions were
compared. Decision support
during medication
administration consisted of
alerts, reminders and drug rate
limits on the infusion pump.
Errors from stages
of administration,
monitoring,
ordering and filling
were assessed.
Specific types of
errors included:
Wrong dose
Wrong rate
Wrong
concentration
Allergies
Omitted
medication
Wrong drug
No measurable impact on
the serious medication
error rate was found,
hypothesized to be due to
poor compliance (saw
violations with 25%
bypassing the drug
library and 7.7% of
medications administered
without physician
orders). Thus, this study
reported no benefit from
higher degrees of
automation and
computerization on the
rate of serious
19
medication errors.
10. Helmons
et al.
(2009)13
Automation
and
computerizat-
ion
Six indicators of medication
administration accuracy and
nine types of errors were
compared pre- and post-
implementation of BCMA
across 4 different patient care
areas (two medical-surgical
units, one medical ICU and one
surgical ICU).
The nine types of
medication
administration
errors studied
included:
Unauthorized
drug
Wrong dose
Wrong form
Wrong route
Wrong technique
Extra dose
Omission
Wrong time
Drug not
available
Medication errors
decreased by 58% when
wrong time errors were
excluded in the medical
surgical units. Charting
was improved in the
ICUs but otherwise no
change in total
medication errors was
observed.
Implementation of a
BCMA system had
limited affect in an ICU
but was effective in
reducing other errors,
primarily drug not
available and omission
errors.
11. DeYoung
et al.
(2009)14
Automation
and
computerizat
-ion
Direct observation of
medication administration in an
ICU was conducted to compare
error rates before and after
implementation of BCMA.
Medication
administration
errors considered in
this study included:
Wrong time
Omission
Wrong drug
Wrong dose
Wrong route
Drug without an
order
Documentation
error
Medication error rate was
reduced by 56%
following
implementation of
BCMA, but the benefit
was due to reduction in
wrong time errors. No
significant difference
was found for any other
error types. Thus, the
BCMA system alone was
not effective at reducing
most types of errors.
12. Morriss et
al.
(2009)15
Automation
and
computerizat
-ion
An observational study of
medication errors in a neonatal
intensive care unit was
conducted. Daily patient
medical records were used to
identify medication errors and
potential or preventable adverse
drug events.
Errors included:
Omitted dose
Wrong dose
ordered or given
Wrong time
Wrong rate
Wrong technique
Transcription
error
Duplicate orders
Implementation of a
barcode medication
administration system
reduced the risk of
preventable ADEs by
47%. The BCMA system
also detected more
errors, primarily wrong-
time errors than before
implementation. Thus,
this automated
20
intervention was
effective in detecting
errors and reducing the
rate of preventable
ADEs.
Based on the literature, current interventions directed towards addressing adverse drug events and
medication administration errors include educational programs and training, checklists, smart
infusion technology and bar-code medication administration systems.4-15
Policies regarding alerts,
reminders and equipment improvements have also been investigated.8,9
Other strategies such as
computerized physician order entry systems aim to address prescription and dispensing errors but
may also have some effect on administration errors.44
Studies investigating training and educational programs show improvements in knowledge but
lack coherence in effecting change in clinical practice.6,7
The effectiveness was also found to
change based on the type of training and type of errors.4, 7
Studies by Schneider et al., Trivalle et
al., Blank et al., and Ford et al., showed either no changes in error rates or only slight
improvement after implementation of the educational intervention compared to pre-intervention
error rates.4-7
Mills et al. however, reported training to be negatively correlated to improved
outcomes.8
The effectiveness of supposedly more powerful strategies is similarly inconclusive. White et al.
reported the effectiveness of checklists towards medication administration errors to vary based on
the design and the type of error.10
Similarly, Trbovich et al. reported variations in error detection
between smart infusion pumps, barcode pumps and traditional pumps based on the type of error
and the rigorousness of constraints.11
Similar findings by Rothschild et al. demonstrated no
differences in medication error rates between smart pumps with clinical decision support and those
without, perhaps due to violations which bypass barriers.12
As these studies are generally limited to a single level of intervention, a more thorough
comparison of error detection rates between interventions for different categories of errors is still
needed.
21
2.7 Application of Checklists in Healthcare
Human-oriented strategies such as training are often favored in practice as they are less resource-
intensive and challenging to implement than technological interventions.38
The use of independent
redundancies in the form of checklists and double checks are also used extensively in the aviation
industry to increase compliance to standards and reduce cognitive demands.36
The success of this
strategy in aviation has created interest in its application to the healthcare field.
In healthcare, checklists have been applied to improve adherence to evidence-based best practices
in various clinical areas including mechanical ventilation in the ICU, catheter insertions and
surgery.45
Infection rates due to central venous catheter insertions prompted the development of a
five-item checklist to reduce catheter-related bloodstream infections in the ICU.46-47
The checklist
items included reminders for hand hygiene prior to insertion, sterilization of the site with
chlorhexidine, sterile draping of the patient, donning of sterile mask, gown and gloves and
application of sterile dressing.46
Each item on the checklist is part of the established best practice
guidelines, so deviations from the standard of care would be classified as either a rule-based
mistake, due to the failure to properly apply a good rule, or as an execution failure due to a slip or
memory lapse. Pronovost et al. compared infection rates before, during and 18 months after the
implementation of a central line checklist implemented in conjunction with other process
improvements and demonstrated a reduction of up to 66%.47
The surgical safety checklist was also introduced to reduce the risk of iatrogenic injury from
surgery. This checklist has become a highly promoted tool for improved patient safety as part of
the World Health Organization’s (WHO) initiative for safer surgery.48
A study by Haynes et al.
implemented the checklist in eight globally distributed hospitals to access the rates of morbidity
and mortality from surgery.49
The surgical checklists focuses on six surgical safety policies,
implemented to varying extents at each hospital site. The study reported a 0.7% decline in
mortality rates and a 4% decline in inpatient complications.49
It is unclear however, if the checklist
improved patient safety by detecting errors or rather introduced evidence-based standards to all
hospital sites.
Based on the application of checklists for central lines and surgery, it is uncertain whether this
strategy is directly correlated to reduction of adverse events and improved patient safety. A study
22
by Pape et al. used a checklist to support a “focused protocol” during medication administration to
minimize errors due to interruptions.41
This checklist focused on items to remind nurses to stay on
task to ensure the correct unit dose was given to the correct patient.41
The checklist was used as a
tool to minimize interruptions and so was not assessed for efficacy in detecting errors. The study
by White et al. investigating the design of a checklist for an independent double checks on
medication administration errors demonstrated the effectiveness of a checklist in detecting some
pump programming, patient ID and drug order/label mismatch errors.10
These errors may be
classified as execution failures due to either action-based slips or memory lapses.10
This study
found however, that neither version was able to detect clinical decision errors which may be
classified as planning failures, specifically, knowledge-based mistakes. Investigation of the
appropriateness of a checklist relative to alternative technological interventions to prevent
medication administration errors is thus of interest.
2.8 Guidelines for Checklist Development and Design
As demonstrated in the study by White et al., the design and validation of checklists are critical to
their effectiveness.10
Guidelines for designing an effective checklist for clinical practice originate
from the aviation industry and human factors engineering.50
These guidelines have been adapted in
an effort to provide standardized methodology for the development and design of checklists in
medicine.45, 51
Principles for the creation of a checklist include the following:10, 45, 50-52
Customize checklist to the environment in which it is to be used through an iterative user-
centered design process using observational techniques to determine needs and workflow.
Where possible, list critical items at the beginning of a task checklist.
Keep the list short with explicit instructions for predictable high-risk errors.
Break general or abstract errors into more specific steps with check mark for each step.
Put items in sequence to correspond to the tools used or workflow of the task, using the
same terms and language as the equipment.
Group items corresponding to similar systems or tasks (based on the principle that related
items are grouped together in memory50
) and physically separate groups of checklist items.
Test usability of the checklist with a small group of end-users for validation.
In addition to the above recommendations, Verdaasdonk et al. include requirements for the
23
physical and print characteristics of paper checklists for surgery and describe steps for
implementation.52
While no literature has attributed the use of checklists towards causing adverse
events, Hales et al. caution that excessive use or poor design of checklists could add unnecessary
complexity and infringe on clinical judgement.45
Thus, important consideration for the application
and design of a checklist is necessary to avoid possible negative consequences and ensure its
efficacy.
2.9 Smart Pumps and Bar-code Technology
While checklist interventions fall into the lower range of the Hierarchy of Effectiveness, other
interventions to address medication errors may be found at the technical levels of automation and
computerization. Technological improvements to address IV infusion errors consist of smart
infusion pumps with dose error reduction software and bar-code readers. Smart infusion pumps
with specific drug libraries and built in dosing limits were designed to minimize programming
errors over general infusion devices designed to improve the accuracy of IV infusions.11
Smart
pumps have medication safety software to provide clinical advisories and an additional layer of
protection from keystroke pump programming errors.11
While smart pumps have the potential to
remedy wrong dose errors outside defined limits, they still do not include measures to prevent
other types of medication errors (i.e. wrong drug, wrong patient, wrong time). The integration of
bar-code readers with smart infusion pumps was thus introduced to address these other error
types.11
Conflicting reports exist as to the efficacy of smart pumps, bar-code medication administration
(BCMA) systems and the integration of BCMA with smart infusion pumps. While IV medication
errors have been found to be detectable by smart pumps with clinical decision support, due to poor
compliance (i.e. overrides and workarounds), they have limited impact on the rate of serious
medication errors.11,12, 53
McAlearney et al. observed these workarounds for nurses using smart
infusion pumps with decision support.54
Challenges noted were related to the drug library
composition, tubing (e.g. difficulty removing tubing and false alarms for occlusions due to little
dents in the tubing from the clamps), and IV volumes (i.e. IV solution labels do not account for
manufacturers overfill or reconstituted medications such that there is often residual volume at the
end of the infusion).54
To overcome the IV volume issue, nurses reported programming extra
volume to be administered to ensure no residual volume was left in the bag for reconstituted
24
medications.54
Nurses also reported manually programming the pumps and disabling the smart
software features when IV medications were not found in the drug library, medications were
received in concentrations different from those in the library, or when IV medications were outside
the defined dose or rate limits.54
Differences in the remediation of wrong dose errors were also
reported to depend on whether hard or soft limits were engaged.11
Based on these observations,
smart pumps have the potential to be effective error remediation strategies but require measures to
reduce challenges associated with their use in order to minimize workarounds.
A similar analysis of BCMA systems also provided mixed outcomes with some studies13,
reporting
a reduction of errors, while others found BCMA systems to have a minimal impact on safety.14, 55-
57 The limited efficacy in error prevention is again attributed to workarounds and the introduction
of new errors.55, 58
Miller et al. investigated the high-alert medication alerts types and reviewed the
workarounds used by pharmacy and nursing.58
17% of scanned medications were reported to issue
an error alert, of which 77% of clinician overrides were not documented to indicate the reason for
the override.58
An average of three workarounds per administration were observed.58
These
included failure to scan medications/armband by nurse and removing dosage from unit-dose
packaging before scanning.58
Cochran et al. further identified failure modes associated with
BCMA systems to include: mislabeling of medication with incorrect bar-code, lack of bar-code,
inability to scan, override of warning, bar-code not scanned, workarounds (scanning patient ID
from chart rather than wristband or scanning medication at nurses’ station rather than at bedside),
wrong patient and unavailable system.40
The mixed outcomes of these technologies in mitigating errors suggest the need for a fully
integrated system59
and further improvement to address challenges in use and integration of the
technology into the workflow. Thus, despite the intentions of these technologies to address
medication errors, it is of interest to assess their efficacy in relation to a supposedly less effective
intervention, specifically, the checklist.
25
3 Objective and Hypothesis
3.1 Gap in Knowledge
Based on the literature reviewed in Chapter 2, there is an apparent lack of evidence for the utility
of checklists in the IV medication administration process for the prevention of errors. Furthermore,
the relative effectiveness of a checklist intervention has not been compared to other error
prevention solutions, namely, smart pumps and bar-code medication administration systems.
Finally, little work has been done to differentiate the performance of these interventions based on
the category of error. Thus, this study aimed to address the following questions:
1. Do checklist and smart infusion pumps with bar-code capabilities differ in their
effectiveness at mitigating errors in IV medication administration?
2. Does the relative effectiveness of checklists versus bar-code smart pumps depend on the
classification of error (either execution or planning failures)?
3.2 Hypotheses
1. As mitigation strategies used in reducing adverse events are hypothesized to vary in their
effectiveness, interventions directed towards changing human behavior which still rely on
human vigilance, will be less effective at mitigating errors than those which intervene at a
system’s level. Checklists, ranked at the lower end of the hierarchy, are hypothesized to be
significantly less effective overall than the computerized and automated solution of smart
infusion pumps and bar-code verification. This is hypothesized due to the degree to which
they continue to rely on human vigilance to detect errors and the reactive nature of checklists.
2. The effectiveness of the interventions is also hypothesized to vary based on the type of error
but it is expected that the hierarchy will be maintained for both planning and execution
failures. Checklists may be effective at detecting execution errors (action-based slips or lapses)
during mechanistic tasks that require comparison of information from one tangible source to
another.10
However, the automated nature of bar-code verification is expected to be even more
effective than the checklist reminder for these execution errors arising from mechanistic tasks.
Checklists are hypothesized to have little effectiveness towards catching planning failures,
specifically, knowledge-based errors, arising from more cognitively challenging abstract
tasks.10
Knowledge-based mistakes may arise for tasks which require clinicians to compare
26
tangible information from a physical source to their own abstract knowledge or to integrate
different types of information from multiple sources. In these scenarios, it is suspected that the
lower level checklist intervention will not be as effective as the more automated and
computerized smart pump bar-code intervention, which is able to integrate information. As a
result, these technological tools which are able to store and integrate information will be more
useful in providing clinical decision support and thus will be more effective at mitigating
abstract errors than human oriented checklists.
3.3 Specific Aims
The specific aims of the proposed research are to:
1. Develop a checklist for the IV medication administration process through an iterative design
and usability testing process.
2. Compare the detection of planted errors in various medication administration scenarios
between two types of interventions, a paper-based checklist versus a smart infusion pump and
bar-code verification system, in a high-fidelity simulation of a general inpatient unit.
3. Characterize the effectiveness of these interventions at mitigating errors as a function of
planning versus execution failures.
27
4 Development of an IV Medication Administration Safety Checklist
As discussed, checklists have become prevalent in healthcare as a low-cost safety measure
following the development of a safe surgical checklist and use of a checklist to prevent central
venous catheter line infections.46-48
The studies of these checklists examined their appropriateness
only in a limited, short-term manner. The implementation and measured effectiveness of these
checklists accompanied changes in other practice measures and their effectiveness was not
assessed over long-term use. In the medication administration process, the appropriateness of a
checklist has yet to be investigated altogether, in addition to the effectiveness of a checklist
intervention on directly mitigating human error compared to more technological interventions.
Both the literature and current practice, in the institution where this research was conducted,
lacked an existing checklist specifically for the IV medication administration process. Therefore, a
checklist was created based on safe medication practices and current protocol for independent
double-checking within the hospital. The protocol outlines the many rights of medication
administration (right drug, concentration, route, dose, rate, time and patient) and dictates that
mental checks must be done prior to each administration. In the case of high-risk medications, an
independent double-check is also conducted by a second RN. A checklist used to complete the
independent double checking procedure for ambulatory infusion pumps (AIP) in chemotherapy
was investigated and modified in a previous study by White et al.10
This form (provided in
Appendix A) was used in this study as a guide from which to develop the checklist intended for
use by a single nurse during routine IV medication administration.
4.1 Checking It Twice
The 2010 study by White et al. compared two different designs of a checklist for independent
double-checking of AIPs to determine which components were effective in mitigating bedside
medication errors.10
The current checklist used in the chemotherapy unit was compared to a new
version designed after careful observations of workflow, the way in which nurses used the current
checklist, and the specific risks associated with the medication administration process. The new
version of the checklist a) reordered the sequence of items to mirror the task sequence, b) added a
specific patient armband check, and c) included a reminder intended to address clinical decision
errors. Medication error detection rates were compared between the old and new versions of the
28
checklist and also between individual types of errors. The results clearly demonstrated that general
reminders (where the checklist either did not specify the exact information to verify or simply
reminded the user to apply the appropriate clinical knowledge) were much less effective than
explicit steps. This was evident both between error types and checklist versions. For example, the
addition of a general reminder for clinical decision considerations had no impact on the detection
of clinical decision errors as neither version of the checklist demonstrated effectiveness in
detecting these errors.10
Alternatively, the addition of a specific mechanistic patient identification
check significantly increased the patient identification error detection rate between versions.10
Thus, this study illustrated that checklist design may have a general impact on its efficacy towards
detecting errors and specified which components of the design were most effective. The findings
also suggest that checklists may be limited in their ability to mitigate certain types of errors. The
initial work in this study provided a good foundation from which to adapt a checklist for the
broader administration of IV medications. This study helped ensure that the design parameters and
content for the IV medication safety checklist developed for this thesis were appropriate.
4.2 Framework for Development of Medical Checklists
Hales et al. conducted a review of the process for developing medical checklists and identified
several criteria for consideration: context, content, structure, images and usability.45
Context
referred to the environment and process in which the checklist would be used and becomes
relevant for purposes of implementation. Guidelines for content suggested that the checklist should
incorporate evidence-based best practices and hospital policies and procedures while the structure
criteria required that the checklist be presented in a sequence that mirrors the workflow. The image
criterion encompassed the requirements for readability, clarity and consistency of language and
colours (e.g. red should be used only in situations that reflect the same meaning as when the colour
red is used in the hospital environment). Finally, criteria for usability required that the checklist be
validated in the clinical environment it is to be used, include critical points but still allow the user
to exercise judgment, and not be so tedious as to disrupt patient care.
In addition to recommendations by Hales et al., White et al. provided additional principles for
developing checklists specifically related to detecting errors. White et al suggested, after
determining high risk or high probability errors, to include specific items related to each
predictable error with detailed information of what to check.10
For more abstract or general errors,
29
it was recommended that it be divided into multiple smaller, more specific steps and to maintain a
checklist of manageable length, errors deemed low risk or probability should be left off.10
The
recommendation by Hales et al. and White et al. were thus considered in the design of the initial
checklist.
Besides recommendations specific to the design of medical checklists, basic usability principles
were also applied. A well-established set of 10 usability heuristics for good interface design was
developed by Neilsen.61
These heuristics used to assess usability of user interfaces are listed with
Nielsen’s description in Table 3 below. A heuristics evaluation is often conducted in which the
interface is assessed for its compliance or violation of each principle. The severity of each
violation is ranked to determine the critical issues for redesign. The set of heuristics for good user
interface design was thus applied to the IV medication administration checklist.
Table 3: List of Nielsen usability heuristics61
Heuristic Description
1. Visibility of system status The system should provide appropriate and timely feedback
to inform users about what is going on.
2. Match between system and the
real world
Language, including words, phrases and concepts, should not
be in system-oriented terms but rather be familiar to the user.
The system should follow real-world conventions presenting
information in a natural and logical order.
3. User control and freedom
A clear “emergency exit” should be provided to allow users
to leave an unwanted state through a direct path in case users
mistakenly choose an undesired system function. The system
should support undo and redo.
4. Consistency and standards
The system should follow platform conventions and users
should not have to wonder whether different words,
situations, or actions mean the same thing.
5. Error prevention
In addition to good error messages, the system should be
designed to prevent errors by either eliminating error-prone
conditions or checking for them and presenting users with a
confirmation option before committing to an action.
6. Recognition rather than recall Objects, action and option should be visible to minimize the
user's memory load. The user should not have to remember
information from one part of the system to another.
30
Instructions for use should be visible or easily accessible
when needed.
7. Flexibility and efficiency of
use
The system should accommodate both experienced and
novice users with the inclusion of accelerators to speed up
the interaction for an expert user, allowing users to tailor
frequent actions.
8. Aesthetic and minimalist
design
Irrelevant or rarely needed information should not be
included as it diminishes the relative visibility of relevant
information.
9. Help users recognize,
diagnose, and recover from
errors
Error messages should be expressed in plain language (no
codes), precisely indicate the problem, and constructively
suggest a solution.
10. Help and documentation
Help and documentation should be available if needed and
should be easy to search, specific to the user’s task, listed in
concrete steps to be carried out and not be too large.
4.3 Initial Checklist Design
Based on the framework for design and existing checklists, an initial design of the IV medication
administration checklist was established. This initial design shown in Figure 4 was divided into
four sections to reflect different aspects of the medication administration process. The first section
addressed the initial checks of the medication label against the physicians order. These checks
typically occur before moving to the patients’ bedside as they served as preliminary verifications
to ensure the correct drug, dose and volume was retrieved from pharmacy for the correct patient.
Individual checks are explicitly included for each item that nurses are required to verify for safe
administration.
Following the initial check of the medication label against the physician’s order, the second
section of the checklist follows the nurse to the bedside to complete the patient ID verification. A
second explicit patient ID check is included to remind the user to verify the patient information on
the armband to the medication label before hanging the medication and programming the pump. A
reminder to program the pump according to the medication label is also included. Prior to
connecting the IV to the patient and starting the infusion, the checklist includes an abstract step
indicating for the user to pause and consider the appropriateness of all other planning, non-
mechanistic aspects of the order before proceeding. This phase attempts to address potential
31
considerations not explicitly indicated on the checklist such as allergies, drug compatibilities, and
clinical appropriateness of the physician’s order.
The final section of the checklist reminds the user to complete a double check of the pump
programming parameters. It was intended that the user would transcribe values from the pump
screen onto the appropriate box on of the checklist and then cross reference them to the
information on the medication label. This step was to ensure that all pump infusion parameters
matched those on the medication label and physician’s order and that no inadvertent errors were
made in entering the pump parameters. A final explicit check was added to remind the user to
ensure all clamps were open and that the infusion had started before leaving the bedside.
The checklist was laid out such that each item included a checkbox for the user to physically mark.
This was intended to help track which stage had been completed and also to force the user to
interact with the checklist. For the purpose of direct observation, this also assisted the observer in
tracking the steps of the user and ensured that they were acknowledging the checklist.
32
Figure 4: Initial IV medication administration checklist design
33
4.4 Usability Testing Methodology
Testing with end users was conducted as part of an iterative design process in order to minimize
usability issues and ensure that the design of the checklist would not negatively impact its
effectiveness. Details of the usability tests are subsequently discussed in the following section.
4.4.1 Sampling and Recruitment of Participants
Five nurses were recruited for each round of usability testing. Nurses were recruited for the study
by a sign-up sheet for interested individuals and announcements during morning nurse huddles on
the GIM units at TGH and TWH. Research Ethics Board approval (Appendix B) from both the
University Health Network (Reference # 11-0758-AE) and the University of Toronto (Reference
#27148) were obtained prior to the study being undertaken.
All 10 nurses recruited for the two usability tests were staff nurses from GIM units and all reported
working 12 hour shifts, both day and night. The majority of participants reported their work status
as full-time (7/10; 70%), with the remainder either part-time (1/10; 10%) or casual (2/20; 20%).
The majority of participants were female (8/10; 80%), and ages ranged 18-29 years (6/10; 60%),
30-39 (3/10; 30%) and 50-64 (1/10; 10%). Most nurses (9/10; 90%) had less than 10 years of
experience, (1-4 years (4/10; 40%), 5-9 years (5/10; 50%)), with the final participant (1/10; 10%)
having more than 30 years of experience as an RN. All participants had less than 10 years of
experience working in their current clinical unit ranging from less than one year (1/10; 10%), 1-4
years (4/10; 40%) and 5-9 years (5/10; 50%). Finally, most nurses reported programming infusion
pumps at least 3 times a day (8/10; 80%) while the remaining (2/10; 20%) reported programming
pumps fewer than 2 times a day.
4.4.2 Location
The usability testing was conducted at the Global Center for eHealth Innovation in the simulation
and usability labs. This space was selected as it allowed for the realistic reproducibility of a true
clinical setting and participants could be observed using the checklist under conditions that
mimicked their real work environment. Audio and video recording of the scenarios could be
collected unobtrusively through both the one-way glass and overhead ceiling mounted cameras.
34
4.4.3 Procedure
Each participant was introduced to the study and the lab, given a consent form, asked to complete
a brief background questionnaire, provided with training on using the checklist, asked to complete
the various infusion tasks using the checklist, and then participated in a short debrief to gain
feedback regarding his/her experience with the checklist. Upon completion, nurses were asked to
complete a final questionnaire relating to their experience. The following section provides further
details on this process.
4.4.3.1 Introduction, consent, and background questionnaire
When the participant first arrived, s/he was greeted by the test facilitator and given the consent
form (provided in Appendix C). The participant had 15 minutes to review and sign the form and
ask questions. To ensure that they felt at ease, the participant was given a brief orientation to the
lab, and was introduced to the confederate nurse. At this point the facilitator emphasized that the
nature of the study is to evaluate their interactions with the checklist, and not to evaluate their
individual performance. Also, they were reminded that the study is a simulation and does not
involve any real patients or drugs. They were then given a demographic questionnaire (Appendix
H) to collect information on their nursing background.
4.4.3.2 Training
The participant was trained on how to incorporate the checklist into the medication administration
workflow. This training consisted of a review of each section of the checklist, and then an
opportunity to read through the checklist and ask any questions about the process. Participants
were instructed to use the checklist concurrently as they completed each medication administration
task and not to treat it as documentation to retrospectively complete at the conclusion of the tasks.
4.4.3.3 Usability Test
Participants were briefed by a confederate nurse on each patients’ medical history, shown the
physician orders for IV infusions on the MOE/MAR, and asked to administer the orders
accordingly. The orders required the nurse participant to set up and program an infusion while
using the checklist during critical steps. The confederate nurse guided the participant through the
scenarios and also provided distractions as in a real clinical environment. The scenarios used in the
35
usability test were a subset of scenarios taken from the main study protocol provided in Appendix
D.
Behind a one-way mirror, the test facilitator recorded any issues in completing the tasks, using the
checklist, and any other observations made. The test facilitator was able to communicate with the
confederate nurse using a microphone and wireless radio. However, direct interaction between the
test facilitator and the participant was minimized in order to gain insights into the participants’
thought processes and true behaviors. In the event that a participant was unsure or confused during
a task, they were asked to ”think-aloud” or communicate with the confederate nurse for direction
when needed.
4.4.3.4 Post-test Questionnaire and Debrief
After completing all the scenarios, the participant was interviewed by the study facilitator in a
semi-structured format. The participant was encouraged to discuss their experience and challenges
with using the checklist as well as their perceptions of its effectiveness and impact on nursing
practice and patient safety.
A final questionnaire (see Appendix I) was administered to the participant, which addressed
his/her experience with the checklist. Following the questionnaire an informal debriefing occurred
in order to clarify participant comments or address concerns or questions arising from the study.
Once the questionnaire was completed and all questions answered, the participant was thanked and
compensated $100 for his/her time.
4.4.4 Data Collection
Data was collected through direct observation and post-test review of audio and video recordings
of each session. Data collection templates were created in Microsoft Excel to provide consistency
between participants. Observations were collected on the participants’ successful completion of
the initial verification checks, bedside check, pause, and pump verification checks. Any
challenges, confusion or omissions were noted as well as observations on the integration of the
checklist into the workflow.
36
4.4.5 Data Analysis
Data collected through direct observation was mostly qualitative. Descriptive statistics were done
on quantitative data collected for completion success of each section of the checklist. Quantitative
data obtained from the post-test questionnaire was also analyzed in SPSS (v. 20.0). Quantitative
results were obtained by using a Likert scale (1-5) measuring either positive or negative responses
to the given statements. The Mann-Whitney U test was performed to determine any significant
differences in responses between the first iteration of usability testing and the second iteration.
This test was appropriate as it is a non-parametric test used for comparing differences in ordinal
data between two independent groups. Qualitative comments provided by participants were also
collected and summarized.
4.5 Results: First Design Iteration
4.5.1 Checklist Completion
The first usability test consisted of 5 scenarios requiring the completion of the entire checklist (a
subset of scenarios used in the second phase of the study, see Appendix D). For each scenario
observations of failed completions were recorded for each of the 4 sections of the checklist: 1)
initial check of medication label to physician’s order, 2) bedside check and pump programming, 3)
stop, and 4) double check of infusion pump and medication label (pump verification). Successful
completion of these sections required that the participant check the boxes in each section and
complete the related tasks. Examples of unsuccessful completion include failure to use the
checklist and check boxes or completion of checks after starting the infusion. Definitions for
success and failure of each section are outlined further in Table 4 below.
Table 4: Definitions of success and failure for checklist completion
Success Failure
1. Initial checks Completes check boxes while at
the COW, comparing the
medication label to the
physician’s order.
Does not complete checks or
completes checks retrospectively,
after starting the infusion.
2. Bedside check Completes wristband check and
programs the pump, checking
Does not complete checks or
completes checks retrospectively,
37
boxes for each step. after starting the infusion.
3. Stop Pauses and checks box after
completing bedside checks and
pump programming, prior to
starting infusion.
Does not complete checks or
completes checks retrospectively,
without completing the action.
4. Pump
Verification
Enters values from the pump
screen into the correct box and
compares written values against
the medication label. Double
checks against physician’s order
and checks clamps.
Does not complete checks or
completes checks, enters values
into the wrong fields or enters
values without referring to the
pump (i.e. from memory or
reading from medication label).
Based on the definitions in Table 4, the numbers of failures for each checklist section were
recorded for each participant. Figure 5 shows the total number of completion failures amongst all
participants for each checklist section out of a total of 25 checklists requiring completion.
Figure 5: Total number of failures by checklist section from 25 checklist uses
The initial and bedside checks were completed most successfully with the fewest failures. Failures
in completing the initial check consisted of omissions in using the checklist (16%; 4/25) and
retrospective completion at the conclusion of the infusion setup (16%; 4/25). Participants
completing the checklist at the conclusion of the task treated the checklist as an extra form of
documentation but disregarded its intention to serve as reminders during the medication
8 10
20 22
0
5
10
15
20
25
Initial Checks Bedside Check Stop PumpVerification
To
tal
Nu
mb
e r
of
Fa
ilu
res
Checklist Section
38
administration process. Similarly, bedside check failures were also due to omissions in using the
checklist (16%; 4/25) and retrospective completion at the conclusion of the infusion setup (24%;
6/25). The ‘stop’ step of the checklist was poorly completed with 20 (80%) observed failures over
all participants. Only 4 instances (16%) of successful completion were observed over all 25
checklists used. This step was frequently skipped altogether (24%; 6/25) or checked off
retrospectively at the end of the infusion setup, for sake of checklist completion, without any
observation of the action being completed (56%; 14/25). The final pump verification and double
check section had the poorest observed compliance with only 2 (8%) instances of correct
completion were observed. Of the 22 failures (88%) in this section, most participants made
multiple mistakes to contribute to the overall failure. Half of these failures (11) were due to
complete omission of this section. For those instances where an unsuccessful attempt was made to
complete this section, it was observed that participants entered values from memory or copied
values from the medication label, without referring to the pump screen 9 out of 25 times, entered
parameters into the wrong field twice and checked boxes without any observed verification action
a total of 5 out of 25 times.
4.5.2 Qualitative Analysis by Workflow
Integration of the checklist into the IV medication administration workflow was an important
consideration for usability and efficiency. Based on direct observation and post-test feedback from
the end users the typical workflow observed in the simulation was mapped out (Figure 6). This
workflow was useful in determining elements of the initial checklist design that were poorly
placed and provided guidance for the appropriate order of steps.
39
Medication Administration Workflow
Bedside:Primary Infusion
Other LocationBedside:
Secondary Infusion
Ph
ase
Open electronic medication order for patient
Review medications
Pick-up medication from pharmacy
Verify that all patient and drug information on medication
labels matches physician order (i.e. check patient name, MRN,
DOB, drug name, concentration, dose, route,
volume, time).
Is all the information
correct?
Bring medication and COW to bedside
Yes
Correct error.
No
Verify patient name, MRN, DOB on wristband against medication label
and with patient
Hang primary bag and prime line
Turn on infusion pump and load cassette
Program infusion parameters
Swab IV ports and connect line to patient site
Ensure all clamps are open
Start infusion and check to ensure it is running
Is there a secondary
infusion to set up?
No
Lower primary bag
Yes
Hang secondary bag above primary, swab ports and connect to primary line above the pump
Verify patient name, MRN, DOB on wristband against medication label
and with patient
Program secondary infusion parameters
Ensure secondary clamp is open
Run infusion
Figure 6: Activity diagram of nursing medication administration workflow.
40
4.5.3 Post-Test Questionnaire
Quantitative results of the post-test questionnaire provided an indication for the ease of use,
perceived efficiency and overall impressions of the initial checklist design. Responses were
collected for only 4 of the 5 participants due to time restraints with one of the participants.
Responses to statements relating to ease of use were generally between borderline and positive.
The lowest response in this category (3.25) referred to the statement that it was easy to follow
along with the steps. The poor response to this statement was likely due to the confusion and
inability to follow the steps in the pump verification section. Additionally, responses relating to
efficiency were also between borderline and agreement. The overall impression of the checklist
was slightly negative (2.75) when asked if participants would want to use the checklist on their
unit. Based on the qualitative comments included below in this section, the negative overall
impression of the checklist was likely due to the perceived increase in workload and time required
to complete the checklist.
In addition to ease of use, efficiency and overall impressions, the post-test questionnaire asked
participants to rank the perceived effectiveness of the checklist at addressing different types of
errors. Generally, the checklist was perceived to have a positive likelihood of detecting all errors
(wrong drug, concentration, patient, dose and rate). Detection of wrong patient and wrong dose
were ranked the most positively, each with a mean ranking of 4.25 on the Likert scale. It appeared
that nurses felt the inclusion of specific reminders to verify each of these details of the medication
order would help detect any wrong patient or wrong drug, concentration, dose or rate.
Qualitative responses of the post-test questionnaire are summarized below:
Several nurses indicated they like the visual reminder of the STOP sign which provided a
chance to review their work and think things through.
A few nurses felt the reminders would be especially useful for newer staff or nursing
students and like the reminders to complete the initial checks of the medications and
physician order.
Several nurses disliked the perceived repetition and felt there was too much information.
Most nurses indicated they found the areas for intermittent and continuous pump
parameters unclear and confusing.
41
Most nurses felt the checklist added too much to their workload and would not be feasible
on a busy ward with a full patient load.
A few nurses suggested that physically completing the checklist may not be time-efficient
but that it may be useful at the bedside for viewing purposes, to serve as a ‘mental’
checklist.
One nurse felt the checklist would reduce IV medication errors if used but that it needed to
be made less of a chore for the staff to encourage its use.
4.6 Usability Issues and Checklist Redesign
Following the first round of testing, usability issues were identified and the design of the checklist
was altered in an attempt to address the critical issues. Descriptions of the identified issues are
subsequently provided along with the accompanying design changes. The second iteration of the
checklist design is presented in Figure 7.
1. Too much content
The strongest user feedback stemmed from concerns regarding the tedious nature of the checklist.
Qualitative comments by users repeatedly touched on the amount of information and content on
the checklist. It was felt that there were too many steps, too much reading and too much
information to view and that overall it would not be practical for many nurses. Several users
mentioned that they did not feel it would be time-efficient and would be too much paperwork to
complete the entire form, especially if the workload was high.
Observations of the completion rates for the checklist also support this notion, as very few
instances of wholly completed checklists were observed. The retrospective completion of the
checklist for certain sections, in a sense playing ‘catch-up’, suggests that the content may have
been too overwhelming to complete alongside the task. Mismatch between the checklist step and
workflow may also have been a contributory factor.
Design Changes: In order to reduce the perceived workload of the checklist and minimize the
amount of information, section headings were removed. These headings did not contribute to the
understanding of the section content, and violated the usability heuristic for simplicity. In addition,
the step to ‘program pump according to medication label’ was removed. This step was removed
42
based on guidelines by White el al. stating to include only those steps related to high-risk errors in
order to manage the length of the checklist.10
As this step did not address a specific error, it was
deemed unnecessary.
Finally, to improve consistency in the layout, the initial patient ID check was included in the list of
items to compare between the medication label and physicians’ order. In this way, the first section
of the checklist presents a list of all the items to verify when comparing the medication label to the
physicians order. This rearrangement also reduced the content so that the user could more quickly
read each item to be verified.
2. Misplacement of ‘Stop’
From direct observation, it appeared that very few participants complied with the ‘Stop’ step in the
checklist. As only 4 instances were observed where the participant both completed the action and
filled out the checklist, it seems this step was often omitted. However, based on user feedback, this
step was well liked. Users appreciated this step as a reminder to think, question and review the
work they were doing. While the step was valued, with all users agreeing that it was important to
be included on the checklist, the step was misplaced. Placing the ‘Stop’ following pump
programming was observed to be too late in the workflow. Once the user was engaged in
programming the pump, they were no longer considering the details of the order, but were rather
focused on the physical task of setting up the infusion. This was apparent in observations as users
often set the checklist aside once moving to the bedside (after the armband check) and did not
resume completion until after the infusion had been started. The remainder of the checklist was
thus completed retrospectively.
User feedback also indicated that the ‘Stop’ would be more appropriate prior to moving to the
bedside while the user is still looking at the physicians’ order. It is following the verification tasks
of the medication label and the order that the nurse is considering the clinical appropriateness of
the order. Once the user has moved to the bedside, they do not often stop for clinical
considerations.
Design Changes: The ‘Stop’ step for clinical considerations was moved to immediately follow the
verification steps between the medication label and physicians’ order, prior to moving to the
43
bedside. Having this step before moving to the bedside encourages users to pause while they still
have convenient access to patient and order information and additional clinical decision support
tools (online drug guide, formulary etc.).
3. Pump verification and final double checks
The most severe usability issue was centered on the final pump verification and double check
steps. This section was very poorly completed, with most users abandoning the checklist
altogether upon reaching this point. Users found the directions unclear and were unsure of the
purpose of the section and what was required. Several users felt this section was additional work to
the fluid balance sheets they complete already, recording the medication and volume. It was not
clear that the purpose of this section was to double-check the pump programming. This confusion
was observed as it translated into incorrect completion of this section. Rather than transcribing the
values read from the pump screen, most users completed this section from memory after leaving
the bedside or by copying values from the medication label. This misunderstanding may be due to
a poor match between the checklist layout and the actual pump screen and violation of usability
heuristic #2 (Table 3).
The division of continuous and intermittent infusion parameters also made nurses unsure what
information to transcribe from the pump. It was observed in a few instances that information was
entered into the wrong fields. For intermittent infusions, typically infused over a shorter duration,
the absence of a rate field caused confusion. Even for intermittent infusions, nurses are
accustomed to calculating and inputting the rate or using the volume over time programming
option on the pump. The pump screen then displays the rate rather than the duration. Additionally,
the language of continuous and intermittent infusions was a source of confusion for some nurses.
This did not match the language of the pump, which refers to infusions as either primary or
secondary.
Design Changes: In order to simplify the final section of the checklist and improve the relationship
between the actual pump screen and checklist, an image of the pump screen was taken to replace
the columns for copying values. This image provides a field to input the rate and volume to be
infused, identical to the layout on the pump. In this way, it becomes more obvious to the user in
which fields to transcribe information from the pump. Rather than being required to locate a
44
parameter on the pump screen and then determine which field to insert it on the checklist, the
checklist now matches exactly to the pump screen layout. Checkboxes to indicate either a primary
or secondary infusion were also included just at it appears on the actual pump. This solution also
ensured the language on the checklist was consistent with the pump and simplified the fields. The
steps for double-checking the pump parameters to the medication label and physicians’ order were
then included as individual steps, consistent with the layout of the rest of the checklist. The final
check to ensure all clamps are open and the infusion has started remains as the last item on the
checklist.
45
Figure 7: New design of IV medication administration safety checklist
46
4.7 Results: Second Design Iteration
4.7.1 Checklist Completion
The second iteration of usability testing involved a new group of five nurses and consisted of 7
scenarios (subset of scenarios used in second phase, see Appendix D), which required completion
of the entire checklist. Observations of completion failures were again recorded for the four
sections of the checklist: 1) initial check of medication label to physician’s order, 2) stop, 3)
bedside armband check, and 4) double check of infusion pump and medication label (pump
verification). Definitions for success and failure for each section remained the same as in the first
usability test (Table 4).
Figure 8: Total number of failures by checklist section for 35 checklist uses
Completion of the initial checks had 5 out of 35 observed instances of failures (14%). These
failures in completing the initial check consisted of omissions in using the checklist (6%; 2/35)
and retrospective completion at the conclusion of the infusion setup (9%; 3/35). The bedside
check, now reduced to a single reminder to complete the armband check, was observed to be
completed successfully in all instances. Compliance with the ‘Stop’ step showed only 2 observed
instances (6%) where the participant failed to pause and complete the step on the checklist. Only 4
failures (11%) were observed for the final pump verification section following the design changes.
Of these, one instance was observed where participant recorded the value from memory without
5
0
2
4
0
1
2
3
4
5
6
Initial Checks Bedside Check Stop PumpVerification
To
tal
nu
mb
er
of
fail
res
Checklist Section
47
referring to the pump while 3 instances (9%) were observed where the participant transcribed the
values from the medication order prior to programming the pump. In the later failure mode, the
participant used the checklist to calculate the rate and then entered the values from the checklist
into the pump.
4.7.1.1 Comparison between Usability Tests
Between group comparisons of total failures were measured using a 4 (checklist section: 1. initial
check, 2. bedside check, 3. stop and 4. pump verification) x 2 (usability test: 1, 2) mixed factors
ANOVA with repeated measures in the first factor.
Figure 9: Comparison of total failures between usability tests
The results are shown in Figure 9. This analysis showed significance for both main effects of
usability tests [F(1, 8) = 19.056, p < .01] and checklist section [F(3, 24) = 4.963, p < .01]. A
significant effect was also found for the two-way interaction between usability test and checklist
section [F(3, 24) = 4.610, p < .05]. A follow-up independent t-test showed significant difference
between usability tests for the total number of ‘stop’ section completion failures t(8) = 4.129, p <
.01. In the first usability test, 20 failures were observed whereas only 2 were observed in the
second test following the checklist redesign. A significant difference was also found in the total
number of pump verification completion failures t(8) = 5.091, p < .01 with 22 observed failures in
the first iteration of testing compared to only 4 in the second round.
0
5
10
15
20
25
Initial Checks Bedside Check Stop PumpVerification
To
tal
Nu
mb
er
of
Fa
ilu
res
Checklist Section
Usability Test 1 Usability Test 2
* *
48
4.7.2 Post-Test Questionnaire
The same post-test questionnaire used in the first usability test was again given to provide an
indication for the ease of use, perceived efficiency and overall impressions of the new checklist
design. As in the first round of usability testing, most statements relating to ease of use were
viewed positively. The lowest mean ranking (3.40) was given to the statement regarding the
meaningfulness of reminders. The responses to efficiency were also generally positive. When
asked if nurses would want to use the checklist on their unit, the response was between
disagreement and borderline at only 2.80 on the five-point scale. With regards to the perceived
effectiveness at detecting errors, mean responses ranged from 3.20 for likelihood of detecting the
wrong concentration to 4.00 for detecting the wrong patient.
The quantitative responses to the post-test questionnaire were compared to the responses obtained
in the first round of usability testing with the original checklist design (Table 5). While responses
were generally more positive for the second design, this difference was statistically insignificant
for all but one statement. Users’ response to the statement, “it was easy to navigate the items and
follow along with steps” showed a statistically significant difference between the first and second
design iteration (U = 1.50, p < 0.05). A significantly more positive mean response (4.40 vs. 3.25)
was observed with the revised checklist design. Further breakdown of responses to the post-test
questionnaire are provided in Appendix L.
Table 5: Comparative analysis of post-test survey results between design iterations
Ease of use: Test
Group N
Mean
Ranking
Std.
Dev.
Mann
Whitney
U Test
Statistic
p-value
(Asymp. 2-
tailed)
Significant
Difference?
1. Overall, the checklist
was easy to use
1
2
4
5
3.75
4.20
0.50
0.45 6.00 .180 No
2. The checklist used
familiar, easy-to-
understand language
1
2
4
5
4.00
4.20
0.82
0.45 8.50 .661 No
3. It was easy to navigate
the items and follow
along with the steps
1
2
4
5
3.25
4.40
0.50
0.55 1.50 .026 Yes
4. The checklist was easy
to read
1
2
4
5
4.00
4.40
0.00
0.55 6.00 .176 No
5. It was easy to check 1 4 3.50 0.58 6.00 .273 No
49
pump programming
parameters (e.g. dose,
volume, etc.)
2 5 4.00 0.71
6. It was easy to correct
mistakes
1
2
4
5
3.50
4.20
1.00
0.45 6.00 .180 No
7. Reminders and
messages were
meaningful
1
2
4
5
4.50
3.40
0.58
1.14 4.00 .126 No
Efficiency: Test
Group N
Mean
Ranking
Std.
Dev.
Mann
Whitney
U Test
Statistic
p-value
(Asymp. 2-
tailed)
Significance?
1. The time required to
complete the checklist
was appropriate
1
2
4
5
3.75
4.20
1.26
0.45 8.50 .661 No
2. The number of
additional steps was
acceptable
1
2
4
5
3.50
3.60
0.58
1.14 9.00 .794 No
Overall impression: Test
Group N
Mean
Ranking
Std.
Dev.
Mann
Whitney
U Test
Statistic
p-value
(Asymp. 2-
tailed)
Significance?
1. The checklist meets all
my needs for IV
infusions
1
2
4
5
3.25
4.00
0.96
0.71 5.50 .225 No
2. I want to use this
checklist on my unit
1
2
4
5
2.75
2.80
0.96
1.10 9.00 .796 No
Likelihood of detecting: Test
Group N
Mean
Ranking
Std.
Dev.
Mann
Whitney
U Test
Statistic
p-value
(Asymp. 2-
tailed)
Significance?
1. Wrong drug 1
2
4
5
3.50
3.80
1.29
0.84 8.50 .702 No
2. Wrong concentration of
drug
1
2
4
5
3.75
3.20
0.96
1.48 8.00 .606 No
3. Wrong patient receiving
IV fluids
1
2
4
5
4.25
4.00
0.96
0.71 8.00 .600 No
4. Wrong dose delivered
to patient
1
2
4
5
4.25
3.60
0.96
1.14 6.50 .373 No
5. Wrong rate delivered to
patient
1
2
4
5
3.75
3.40
0.96
1.34 8.50 .705 No
50
In addition to the quantitative aspects of the post-test survey, the qualitative responses are
summarized below:
Most nurses liked that the checklist was easy to use, easy to read and was not too lengthy.
A few nurses felt the checklist added steps to the medication administration process,
making the process longer and more complex. They felt this made more room for error and
that it could be an added distraction.
A few nurses also felt the checklist would not be used on a real clinical unit as it added one
more piece of paper to the workloads of nurses.
4.8 Discussion
The second iteration of usability testing with a new group of end users revealed fewer usability
issues. Despite involving more scenarios, the second usability test revealed significantly fewer
instances of failed completion on the ‘stop’ and pump verification sections than the first test. This
result, in conjunction with the qualitative feedback and observations suggested that the second
design iteration resolved the major usability issues discovered in the first usability test. The
amount of information on the checklist was generally deemed appropriate by nurses following the
design change. They felt all the steps included were necessary and should not be removed.
It was also qualitatively observed that nurses had fewer issues integrating the checklist into the
workflow. They tended to complete the ‘Stop’ step correctly and paused to consider the order as
indicated on the checklist. The general workflow observed saw nurses complete the initial
verification checks while at the COW station. Nurses then paused to consider the order (e.g. asked
a clarification question or consulted the IV formulary) and completed the ‘stop’ check. Upon
moving to the bedside, nurses completed the armband check before setting aside the checklist.
Nurses then preceded to set-up the infusion and program the pump as usual. Before starting (or
immediately after running, but before leaving the patient’s bedside) nurses returned to the
checklist and completed the verification of the pump parameters and final double checks. This
workflow was observed to be fairly consistent between users and a few even indicated that they
felt the checklist helped to standardize their workflow.
The confusion regarding the pump verification and final double checks observed in the first
usability study was resolved with the second design iteration. The more accurate mapping of the
51
checklist to the pump screen prompted nurses to look at the pump and transcribe the values as
directed. In the earlier design version, it was observed that nurses tended to input the values from
memory or by reading the medication label, without referencing the pump. After the design
change, the observed failures involved a single participant writing in values prior to programming
the pump. The participant then used the written values on the checklist as a reference to program
the pump. However, this same participant was also observed successfully completing this section
in other instances. In addition, standardization of the final double checks to remain consistent with
the overall layout of the checklist was also observed to encourage users to compare the proper
sources of information (i.e. pump settings to medication label and physician order).
While no major usability issues were identified with the second design, nurses still had critiques
with the use of a paper-based checklist. The main critiques were issues of space availability and
efficiency. Users indicated that in a true busy ward environment completing extra documentation
with a typical patient load would not be welcomed by staff. This identifies another inherent
limitation in the routine use of checklist. In addition, it was stated that delivering the checklist in a
paper form to be filled could introduce challenges due to limitations in space and writing surfaces,
particularly impractical for systems that have moved away from paper documentation. It was
suggested that this checklist would be better as a visual reminder at the bedside, however in this
form there would be no active use of the checklist, rendering it unreliable. It was also proposed
that the checklist could somehow be integrated into the MOE/MAR which would require this
system to be available in multiple locations on the ward for convenient access. While valid
concerns for implementation, these comments were not related to the specific design of the
checklist, but rather the delivery and perceived burden of having to use a checklist. The solution to
these concerns was considered out of scope for this study, therefore, as no severe usability issues
were identified in the second iteration of design and testing, the second version of the checklist
was used in the subsequent phase of this study.
4.9 Limitations
The major limitation in the checklist development phase was the potential modification of
workflow due to the conditions of the simulation. The simulation attempted to provide a high-
fidelity replication of the real clinical environment. However, discrepancies in the physical space
and the task requirements may have altered the typical nursing workflow. The workflow presented
52
in Figure 6 is a more simplified variation of the IV medication administration process. In the
controlled simulation lab environment, supplies and equipment were readily available and nurses
did not have to contend with as many added complications or major distractions. The availability
of the COW allowed the user unlimited access to the MOE/MAR during administration. On a real
clinical unit, nurses are often required to share one COW amongst multiple rooms. They do not
always have the space or resources to have access to the MOE/MAR directly at the bedside. In
addition, infection control policies often require patient isolation in which case nurses are refrained
from bringing any shared equipment into the patient room. In these cases on the GIM floor, nurses
may perform the medication order verification checks against the medication label outside of the
room. Once in the room, the nurse then compares the patient armband information to the label but
no longer has access to readily verify the physician order. The simulated environment provided
idealized, controlled conditions as it allowed space to maneuver the COW directly to the bedside
for convenience while programming the pump and setting up the IV infusion. The results were
thus also idealized and it is expected that in a real clinical environment, the performance of the
checklist would be likely poorer than stated. The checklist developed was thus generalized for the
ideal medication administration process and may require additional evaluation and modification to
reflect the practice on a specific clinical unit prior to implementation.
4.10 Summary
The first phase of this work aimed to develop a checklist for the IV medication administration
process intended to be used by a single user. The initial design of the checklist was guided by the
safe medication practices at the study site and an existing form for conducting an independent
double-check of ambulatory infusion pumps in chemotherapy. Two iterations of design and
usability testing were conducted. The redesigned checklist following the first usability test yielded
no major usability issues in a second test. This rigorous user-centered design process was
undertaken to ensure that the design of the checklist would enable the best performance to be
expected from this type of intervention. Thus, in subsequent testing, the relative effectiveness of
the checklist can be assured to be as a result of the nature of the intervention, rather than poor
design or usability.
53
5 Checklist versus Bar-code Smart Pump (High-Fidelity Simulation)
5.1 Objectives
Following the design and usability validation of the IV medication administration safety checklist
in Chapter 4, the effectiveness of this intervention was compared to that of a bar-code smart
infusion pump system in a high-fidelity simulation experiment. In a simulated general internal
medicine unit setting, participants were asked to perform IV medication administration in a
baseline condition with no intervention and in an intervention condition using either the checklist
or a smart pump and bar-coding system. Participants were observed throughout this process to
view their interactions with the intervention and their ability to detect planted errors. Both
qualitative and quantitative data were collected to determine the error detection rate when using
one of the experimental group interventions compared to the baseline control condition in which
no interventions was used. It follows that the overall objective of this phase was to determine
which intervention was most effective at detecting planted errors and whether the detection rates
depended on the classification of the error.
5.2 Methodology
5.2.1 Experimental Design
A high fidelity simulation experiment was conducted to test IV medication administration
performance by nurses during fabricated scenarios with planted errors using either a paper-based
checklist (Figure 7 from Chapter 4) adapted from the current practice of independent double
checks for ambulatory infusion pumps10
or a smart pump and bar-coding system which simulated
closed-loop medication administration. Error detection rates between the checklist and bar-code
smart pump system were compared using a between-subject design with parameters (e.g.
experimental setup, tasks, and planted errors) kept the same for both groups. A within-subject
baseline control condition was included with the use of equivalent scenarios of tasks and planted
errors. The experimental design is visually represented in Figure 10. The sequence of planted
errors within the scenarios (as well as the order of intervention versus control condition within a
group) was counterbalanced to minimize any order effects during the experiment.
54
Figure 10: Visual representation of experimental design
5.2.2 Location
The high-fidelity experiment was conducted at the Global Center for eHealth Innovation in the
simulation and usability labs. This space was selected as it allowed for realistic reproducibility of a
true clinical nursing environment and participants could complete scenarios uninterrupted while
being observed behind one-way glass. Audio and video recording of the scenarios was also
collected unobtrusively through both the one-way glass and overhead ceiling mounted cameras.
5.2.3 Sampling and Recruitment of Participants
A similar study assessing the effectiveness of interventions to prevent interruptions in nursing
workflow during medication administration determined that a sample size of 24 clinicians per
group ensured an 80% probability of statistical significance in the difference between groups at a
significance level α=0.05 and effect size of 0.75.60
Thus, 24 nurses per group were recruited.
Participants were recruited from a pool of General Internal Medicine ward nurses from two sites of
a large academic teaching hospital in Toronto, Canada.
Nurses were recruited for the study by visits to the specific wards during nursing team huddles and
sign-up sheets posted on the general medicine wards for interested individuals. Research Ethics
Board approval from both the hospital facility (Reference #11-0758-AE) and the University of
Toronto (Reference #27148) was received prior to the study being undertaken (Appendix B).
Participants were assigned to groups in a non-randomized manner based on their date of
55
availability as the checklist intervention group was conducted first, prior to the smart pump and
bar-code verification intervention group. Participant demographics were collected in addition to
the incorporation of a within-subject control condition to ensure that comparisons could be made
between the two intervention groups.
Participant demographics collected by survey prior to the start of the study revealed that all
participants practiced in a GIM unit and worked a mix of day and night shifts. Forty-six females
and two males participated in the study ranging in age from 18-29 years (26/48; 55%), 30-39 years
(10/48; 21%), 40-49 years (8/48; 17%) and 50-64 years (4/48; 8%). Most nurses reported working
12 hour shifts (44/48; 92%) while a few reported working shorter shifts of less than 12 hours
(4/48; 8%). Forty-three nurses identified themselves as a staff nurse while five selected the other
category describing their role as patient care coordinator, nurse educator, clinical support nurse or
graduate nurse. Nurses reported their work status as either full-time (39/48; 81%), part-time (6/48;
13%) or casual (3/48; 6%).
At the time of the study, 15% (7/48) reported working as an RN for less than 1 year, 38% (18/48)
for 1-4 years, 29% for 5-9 years (14/48), 4% (2/48) for 10-19 years, 8% (4/48) for 20-29 years and
6% (3/48) for over 30 years. Most nurses had worked in the current GIM unit less than 1 year
(7/48; 15%), 1-4 years (21/48; 44%), and 5-9 years (14/48; 29%). A few nurses had worked 10-19
years (3/48; 6%) and more than 20 years (3/48; 6%).
All nurses were familiar with the Grasby 3000 large volume infusion pump and used it to deliver
IV medications in their unit. The frequency of infusion pump use varied between less than once a
day (8/48; 17%), 1-2 times per day (17/48; 35%), 3-5 times per day (17/48; 35%) and more than 5
times per day (6/48; 13%).
5.2.4 Experimental Set-up
5.2.4.1 Simulated Environment
The simulated GIM unit consisted of a three patient bed layout. Beds were arranged to replicate
the GIM wards with curtains separating each patient area. Figures 11, 12 and 13 show the general
lab set-up. End tables were located at the end of each bed and provided gloves, hand sanitizer,
alcohol swabs and medication administration pads. A laptop placed on a rolling cart was used to
56
simulate a computer-on-wheels that could be brought to each patients’ bedside and a binder was
provided that included both the hospital standard and restricted nursing IV drug list. To recreate
realistic acoustic conditions, a soundtrack of a busy hospital unit was played on the overhead
speakers throughout the simulation.
Figure 11: Bed layout in simulated GIM unit with COW cart in the center of the floor
Figure 12: Image of patient bed with IV pole location and end table of supplies
57
Figure 13: Close-up of mock patient displaying wristband and IV access
5.2.4.1.1 Experiment Props
All props involved in the simulation were replicated from those used in real clinical environments.
Medication bags, syringes, IV tubing sets, medication orders (both paper and electronic), labels
and trays were identical to those used in clinical practice. All medication protocols used in the
experiment provided therapeutic information based on consultation from a hospital pharmacist on
the GIM unit as well as a GIM nurse educator. Real medications used in the clinical setting were
substituted with water using simple food coloring to mimic the realistic appearance of colored
drugs.
5.2.4.1.2 Interventions
The IV medication administration safety checklist (designed and validated through usability
testing in Chapter 4) or a smart pump and bar-code system was used in the experimental
condition. Both the checklist and smart pump and bar-code intervention were unfamiliar to the
nurses prior to participating in the study. No intervention was used for the baseline control
condition. Nurses performed IV medication administration tasks in equivalent scenarios to those
during the intervention half of the experiment using traditional Grasby 3000 large volume infusion
pumps.
58
1. IV Medication Administration Safety Checklist:
Blank copies of the paper-based checklist were provided on a clipboard to be completed for each
drug administration. The final version of the checklist (Figure 7) developed after two rounds of
usability testing in the first phase, was used in the simulation experiment. The Grasby 3000 large
volume infusion pump was used with this intervention to reflect the traditional infusion pumps
used in the GIM units.
2. Smart Pump and Bar-code System:
The Alaris System by Cardinal Health (Figure 14) was chosen as the smart pump for this study
as it provided drug libraries with a dose error reduction system (termed ‘Guardrails’) and clinical
advisories (Figure 15). Initially, this system was also chosen as it could be interfaced with a bar-
code verification module. However, due to limitations in the network configuration of the pumps
this aspect was not used.
The drug library was configured for this study to mimic the realistic capabilities of the ‘smart’
pump features. Unique ‘Guardrails Drug’ and ‘Guardrails IV Fluid’ libraries (Figure 16) were
created for the GIM clinical care area. Guardrails Editor Software©
Version 9.4 software was
used to configure this drug library within the Master library. The ability to create unique libraries
for each clinical care area from the master list accommodates the differences in approved drugs
and concentrations across clinical care areas. The specific drugs and concentrations used in this
study, along with a number of common drugs and standard concentrations (for increased
realism), were set-up in the drug library. The parameters for each drug (concentrations, hard and
soft dose rate and duration limits, and clinical advisories) were configured based on the hospital
standard and restricted IV drug formularies as well as input from GIM staff, including a
pharmacist and clinical nurse educator. The parameters and clinical appropriateness of the drugs
and planted errors used in each scenario were also verified against the hospital IV formularies
and with GIM staff. Ensuring that the drug library accurately reflected the guidelines in the
hospital IV formularies made certain that any violations were within the scope of knowledge and
expectations for their nursing staff.
59
Figure 14: Alaris system by cardinal health with two pump modules
60
Figure 15: Example of clinical advisory in drug library
61
Figure 16: Guardrails drug library screen
Due to network configuration requirements for the bar-code module from the Alaris System, an
alternative bar-code verification solution was created. Bar-codes included on the patients’
wristband (Figure 17) and drug labels (Figure 18) were scanned using a 2D barcode reader
application (RedLaser Version 3.2.0) on an iPhone 4 mobile device by Apple. The scanning
application displayed to the user the information contained within the bar-code. In the event of a
planted error, the bar-code included a built-in error message to indicate the source of the error
(e.g. ***Error: Drug ID Mismatch***). The language of the error message was replicated from
the Alaris system when using the Alaris auto ID (bar-code) module. The alternative bar-code
solution was intended to mimic an ideal closed-loop medication administration system in which
62
the CPOE and smart pump are integrated. From the user’s perspective, the bar-code verification
system could compare the details of the drug label, patient armband and electronic physicians
order to ensure all information matched.
Figure 17: Patient wristband with bar-code
63
Figure 18: Sample medication label with bar-code
5.2.4.2 Actors and Mock Patients
Three patient mannequins were used in beds to represent a realistic patient load for a nurse in a
true clinical setting. An actor played the role of the charge nurse for the unit. The charge nurse
actor facilitated the testing sessions by introducing the participant to the ward, providing patient
histories and guiding the participants through the scenarios. The charge nurse actor also brought
out medications to the participant from the pharmacy and provided the liaison between the
participant, pharmacy and the physician when the need arose. If the participant detected a planted
error, the nurse actor was required to perform the necessary actions and also communicate with the
observer via wireless microphone throughout the simulation.
In addition, the nurse actor was also required to create interruptions during the scenarios at precise
moments with a high degree of repeatability between participants. The interruptions were timed to
coincide with a verification task in which the participant was completing a check comparing some
aspect of the physician order, drug label or patient wristband. These distractions were included to
create a realistic busy work environment typical of a real nursing ward. As tasks performed by the
64
nurse actor were required to be consistent between experiments and scenarios, detailed scripts and
training were provided prior to the start of the simulation.
5.2.4.3 Test Scenarios and Planted Errors
The participants performed various tasks of IV medication administration requiring different
degrees of cognitive function. Basic mechanistic tasks included checking the patient ID (i.e.,
verifying that the patient ID on the patient wristband matches the patient ID on the drug label and
physician order) and drug label/order verification (i.e., the information on the drug label matches
the information on the drug order). Slightly more complex integrated tasks were classified as those
involving pump programming and abstract higher-level tasks related to clinical decision-making
(e.g., recognizing an order for an inappropriate dose). The ability to successfully complete these
tasks and identify planted errors related to each category was measured.
Participants completed 9 scenarios for each condition (experimental using either checklist or bar-
code smart pump intervention and baseline with no intervention) for a total of 18 scenarios. The 9
scenarios contained a mix of planted errors and no planted errors. Situations with no planted errors
were included to minimize awareness of the participant to the planted errors. The planted errors,
related to the rights of safe medication administration and adapted from earlier studies by White et
al.10
and Trbovich et al.11
, reflected both planning mistakes (either rule-based or knowledge-
based) and execution errors (either action-based slips or memory lapses) associated with the given
tasks. Specifically, the planning errors included: drug/fluid incompatibility, inappropriate dose,
inappropriate dose rate and inappropriate duration. Execution errors included: wrong patient,
wrong drug, and wrong dose. In addition, the abilities to correctly program and start infusion
pumps, set-up drug and fluid bags, and ensure all clamps were open were also included as
measures of successful IV medication administration. A description of each error is provided in
Table 6 below.
Table 6: Task, Planted Errors and Error type
Error Type Task Planted Errors
Execution
failures
(action-based
Patient ID
verification
Wrong patient: mismatch, patient information
(name, DOB, MRN) on armband does not
match the physician order and the drug label
65
slips and
memory lapses)
Drug label/order
verification
Wrong drug: mismatch, drug name on label
does not match to the physician order
Wrong dose: mismatch, dose on the drug label
does not match to the physician order
Pump Programming No planted error
Open clamps No planted error
Planning
failures
(rule- and
knowledge-
based mistakes)
Clinical decision
Incompatible drug/fluid: secondary drug on
physician order is incompatible with primary
fluid line
Inappropriate dose: prescription error, dose
provided on the physician order is clinically
inappropriate as it is outside the allowable safe
limits for a drug
Inappropriate dose rate: prescription error,
infusion rate on physician order is outside of
safe dose limits (hard limit)
Inappropriate duration: prescription error,
duration on physician order is outside of
standard limit (soft limit)
5.2.5 Procedure
The same procedure was applied to each participant and included an introduction to the study and
the lab, a brief background questionnaire, training on either using the checklist or on the bar-code
smart pump system(depending on the experimental group), IV medication administration tasks in
the simulated clinical environment, and finally a short debrief and post-test questionnaire. A
detailed summary of the procedure is outlined in this section while the full protocol is provided in
Appendix D. The sequence of planted errors was counterbalanced to minimize learning and order
effects so that eight different orders of the script were used. The order of condition (baseline or
intervention) was also counterbalanced between participants to again minimize learning effects
and the possibility of heightened awareness towards errors contributing to the effect.
66
5.2.5.1 Introduction, Consent and Demographic Questionnaire
Upon arrival, the participant was greeted by the test facilitator and given a brief overview of the
study and introduction to the simulation lab. At this point the facilitator emphasized that the nature
of the study is to evaluate the workflow and the participants’ interaction with the technology but
not to evaluate their individual performance. Also, they were reminded that the study is a
simulation and does not involve any real patients or drugs. Participants were then given a consent
form (see Appendix C) to review and sign and asked to complete a background questionnaire (see
Appendix H). Each session was scheduled to last around three hours. The sessions were
conducted in two halves with participants completing one half using an intervention and the other
with no intervention (baseline control). Each participant was given the opportunity for a brief
break in between parts.
5.2.5.2 Participant Training
Participants were trained on the method of IV medication administration relating to their
experimental group. This training consisted of a demonstration, and then an opportunity to practice
or review the steps and ask questions about the process. Participants were instructed to use the
mitigation strategies (either the checklist or bar-code smart pump) they were trained on during the
experimental phase of the session. The training was provided immediately preceding the
experimental part of the session. As the order of experimental and control conditions were
counterbalanced between participants, some participants received training at the start of the session
while others received training after completing the baseline control and prior to beginning the
second half of the session using the experimental intervention.
5.2.5.3 Post-Experiment Debriefing and Questionnaire
Following completion of all of the tasks in each of the baseline and experimental conditions, the
participant was interviewed by the study facilitator in a semi-structured format. The participant
was encouraged to discuss their experience and challenges with the tasks or any aspect of the
intervention they used (checklist or bar-code smart pump) as well as their perceptions of its
effectiveness and impact on nursing practice.
67
At the conclusion of the study, a final questionnaire (see Appendix I) was administered to the
participant, which compared his/her experience with the interventions. The participant was also
requested to keep the details of the scenarios confidential and not disclose them to nursing
colleagues to minimize bias and awareness of planted errors in future participants. Participants
were compensated $225 for their participation.
5.2.6 Statement of Specific Hypotheses
In addition to evaluating the relative effectiveness of the interventions, this study also allowed for
comparison of each intervention to the current practice observed in the baseline condition. This
analysis was conducted to better understand the action of each intervention towards mitigating a
particular planted error. Thus, statements of specific hypotheses were made for each planted error
type and medication delivery failures and are listed in Table 7 below. Table 7: Statements of
specific hypotheses
Error Type Error Hypotheses of effectiveness
Execution
(verification,
mechanistic
tasks) 1. Wrong patient
(planted error)
Checklist: The percentage of nurses who detect the wrong patient
ID error will be significantly higher with the use of the checklist
compared to the baseline condition.
Bar-code and Smart Pump System: The percentage of nurses who
detect the wrong patient ID error will be significantly higher with
the use of the bar-code system compared to the baseline condition.
2. Wrong drug
(planted error)
Checklist: The percentage of nurses who detect the wrong drug ID
error will be significantly higher with the use of the checklist
compared to the baseline condition.
Bar-code and Smart Pump System: The percentage of nurses who
detect the wrong drug ID error will be significantly higher with the
use of the bar-code system compared to the baseline condition.
3. Wrong dose
(planted error)
Checklist: The percentage of nurses who detect the wrong dose
error will be significantly higher with the use of the checklist
compared to the baseline condition.
Bar-code and Smart Pump System: The percentage of nurses who
detect the wrong dose error will be significantly higher with the
use of the bar-code and smart pump system compared to the
baseline condition.
68
4. Medication
delivery error
(no planted
error)
Checklist: The percentage of nurses who fail to correctly delivery
medication (i.e. forget to check armband, forget to open clamps,
make programming keystroke error, forget to run infusion) will be
significantly lower with the use of the checklist compared to the
baseline condition.
Bar-code and Smart Pump System: The percentage of nurses who
fail to correctly delivery medication (i.e. forget to check armband,
forget to open clamps, make programming keystroke error, forget
to run infusion) will be significantly lower with the use of the bar-
code and smart pump system compared to the baseline condition.
Planning
(abstract,
clinical
decision
tasks)
5. Inappropriate
dose (planted
error)
Checklist: The percentage of nurses who detect the inappropriate
dose with the use of the checklist will be the same compared to the
baseline condition.
Bar-code and Smart Pump System: The percentage of nurses who
detect the inappropriate dose error will be significantly higher with
the smart pump compared to the baseline condition.
6. Inappropriate
rate (planted
error)
Checklist: The percentage of nurses who detect the inappropriate
rate with the use of the checklist will be the same compared to the
baseline condition.
Bar-code and Smart Pump System: The percentage of nurses who
detect the inappropriate rate error will be significantly higher with
the smart pump compared to the baseline condition.
7. Inappropriate
duration
(planted error)
Checklist: The percentage of nurses who detect the inappropriate
duration with the use of the checklist will be the same compared to
the baseline condition.
Bar-code and Smart Pump System: The percentage of nurses who
detect the inappropriate duration error will be significantly higher
with the smart pump compared to the baseline condition.
8. Primary fluid
and drug
incompatibilit
y (planted
error)
Checklist: The percentage of nurses who detect the wrong patient
ID error will be significantly higher with the use of the checklist
compared to the baseline condition.
Bar-code and Smart Pump System: The percentage of nurses who
detect the wrong patient ID error will be significantly higher with
the use of the checklist compared to the baseline condition.
69
5.2.7 Data Collection
Data was collected via direct observation by an observer from an observation room located behind
a one-way glass window. A data collection template (Appendix G) was created to ensure that the
same details were collected for each participant. The data collection log listed each of the tasks to
be completed for a given scenario and a pass/fail evaluation. Time stamping integrated into the
template also allowed the observer to record events with accurate timing for later comparison to
video or for further task time analysis. A pilot was conducted of each intervention group to ensure
that the data collection was comprehensive and matched tasks in each scenario.
Audio and video recording was also done of each testing session to provide the opportunity to
review sessions at a later date and allow for further analysis. The ceiling mounted cameras were
manipulated throughout the session in order to track the participant as they moved through the
tasks and ensure essential details (e.g. pump programming) were captured. Discrete
communication with the charge nurse via wireless microphone and radio also allowed the observer
to verify any missed actions by the participant.
5.2.8 Data Analysis
Quantitative data collected from this study was analyzed in Microsoft Excel and SPSS (v.20.0). A
statistical significance level of alpha = 0.05 was used for all tests. The main analysis was
conducted on error detection rates between the checklist and bar-code smart pump groups by
condition (baseline or intervention) and by error type (execution or planning). Additional analysis
of individual planted error types and medication delivery failures was conducted to compare the
intervention condition to the baseline for each experimental group.
Further, quantitative analysis of the demographic questionnaire was conducted to ensure
equivalence between populations of participants. Finally, the post-test questionnaire results were
analyzed to assess differences in the usability and perceived effectiveness of the two interventions.
Details regarding each type of data analysis are subsequently presented.
5.2.8.1 Demographic Questionnaire
A chi-squared test of homogeneity was used to establish whether or not there were any
demographic differences between participants in the two experimental groups for each question of
70
the demographic survey (Appendix H). The chi-squared test was appropriate as it is used for
categorical data from two independent populations (between-group). For those tests in which the
chi-squared test was invalid (i.e. 20% of cells are below the expected cell count), a Fisher’s Exact
test was done as it is more accurate for small samples.
5.2.8.2 Error Detection Rates
As errors were intentionally planted in the simulations, the error detection rate was based on the
number of participants who acknowledged the error while completing each task. All error
detection was valued as either a pass or fail. The criteria for success and failure of each planted
error are detailed in Table 8.
Table 8: Pass and fail criteria for error detection
Success Failure
1. Wrong patient Detected wrong patient error
prior to running infusion
Did not notice wrong patient
before the infusion was started
2. Wrong drug Detected wrong drug error prior
to running infusion
Did not notice wrong drug before
the infusion was started
3. Wrong dose Detected wrong dose error prior
to running infusion
Did not notice wrong dose before
the infusion was started
4. Incompatibility Detected incompatibility between
primary fluid and secondary drug
prior to running secondary
infusion
Did not detect incompatibility
between primary fluid and
secondary drug prior to starting
secondary infusion
5. Inappropriate
dose
Nurses passed if they:
a) detected the clinically
inappropriate dose error
without aid from the pump
and alerted the confederate
nurse to verify with the
physician
b) detected the error after
entering the drug library and
asked to verify with the
physician
Did not detect inappropriate dose
and proceeded to administer the
medication, programming a basic
infusion outside of the drug
library if using a smart pump
71
6. Inappropriate
rate (hard limit)
Nurses passed if they:
1) detected the clinically
inappropriate rate error
without aid from the pump
and either asked to verify
with the physician or chose to
program a rate within the
acceptable range
2) detected the error after hitting
a hard limit and either asked
to verify with the physician
or reprogrammed within an
acceptable range
Nurses failed if they:
a) did not detect the
inappropriate rate and
proceeded to program the
pump at the rate given
b) hit a hard limit and chose to
override limit by
programming it as a basic
infusion
c) reprogrammed the infusion
outside the soft limit range
d) needed instructions from the
confederate nurse to
understand that a limit was
reached
7. Inappropriate
duration (soft
limit)
Nurses passed if they:
3) detected the clinically
inappropriate duration error
without aid from the pump
and either asked to verify
with the physician or chose to
program a duration within the
acceptable range
4) detected the error after hitting
a soft limit and either asked
to verify with the physician
or reprogrammed the duration
within an acceptable range
Nurses failed if they:
e) did not detect the
inappropriate duration and
proceeded to program the
infusion over the duration
given
f) hit a soft limit and chose to
override limit by continuing
with the existing
programming parameters
g) reprogrammed as a basic the
infusion outside the soft limit
range after limit was hit
h) needed instructions from the
confederate nurse to
understand that a limit was
reached
5.2.8.2.1 Comparative Analysis
The main analysis was conducted using a 2 (task-type: 1. execution, 2. planning) x 2 (condition: 1.
Baseline, 2. Intervention) x 2 (intervention group: 1. checklist, 2: bar-code smart pump) mixed-
factorial ANOVA with repeated measures. This analysis tested for significance of between-group
and within-group effects. While the data collected for each error type is dichotomous, individual
72
error types were combined and grouped into their classification of either execution (N=3) or
planning (N=3) to transform the dependent variable data into continuous data. For significant two-
way interactions, follow-up pair-wise comparisons were conducted to elucidate the source of
significance.
5.2.8.2.2 Error Rates for Individual Planted Errors
To further investigate the effectiveness of each intervention towards an individual planted error
and address the specific hypotheses for each individual planted error, the McNemar test was used
to compare differences between baseline and intervention within each experimental group. The
McNemar test was appropriate as it is used for assessing significant differences between two
dependent dichotomous variables.
5.2.8.3 Medication Delivery Failures
In addition to planted errors, medication delivery failures were also analyzed between baseline and
intervention conditions for each experimental group. These failures all fall into the category of
execution errors arising from action-based slips or lapses. The following failures were evaluated:
Failure to open all clamps: considered a failure if the nurses started the infusion without
opening all clamps on the IV lines. A nurse recovered from the error by immediately
returning to open the clamp without any instruction from the confederate nurse before
moving on to the next patient.
Failure to run infusion: considered a failure if the nurse declared the task finished and
moved onto the next patient without starting the infusion. A nurse recovered from the error
by immediately returning to start the infusion without any instruction from the confederate
nurse before moving on to the next patient.
Failure to check patient armband: considered a failure if the nurse set up an infusion
without verifying patient identification information by checking the patient armband. A
nurse recovered from this error if they successfully remedied the oversight without
receiving any instruction from the confederate nurse and did so before the infusion was
connected to the patient.
73
Failure to correctly program pump: considered a failure if the nurse made any mistakes in
programming. This included transcription errors (keystroke errors) in which the wrong
value is entered into the pump, or entering parameters into the wrong field (e.g. entering
the VTBI in the rate field and vice versa). A nurse could recover from these errors if they
remedied the mistake without any instruction from the confederate nurse prior to starting
the infusion. Calculation errors were also included in this category although calculations
were not required in any scenario. If the nurse chose to manually calculate the rate for an
intermittent infusion (where infusion durations are given on the order) rather than use the
pump feature, any errors in calculating the correct rate were considered failures if the user
programmed the pump with the erroneous rate. As nurses typically program intermittent
(secondary) infusions within the hospital formulary guidelines, some leniency was allowed
in cases where the programmed duration was not identical to the physicians order but was
still within the guidelines (e.g. programmed morphine to run over 15 minutes instead of 10
minutes).
The t-test for two dependent samples was used to analyze differences in the mean number of
failures for each medication delivery failure between the baseline and intervention conditions
within each experimental group. As multiple failures could occur for each participant, the data is
continuous so this test was appropriate to compare means between two dependent samples.
5.2.8.4 Post-experiment Questionnaire
Quantitative data was obtained from the post-experiment questionnaire for responses measured
using the 5-point Likert scale (Strongly Disagree – Disagree – Borderline – Agree – Strongly
Agree). Descriptive statistics were done to obtain mean rankings of responses to each statement. A
Mann-Whitney U test was performed on each question to determine any significant differences in
responses between experimental groups. This test was appropriate as it is a non-parametric test
used for comparing differences in ordinal data between two independent groups.
5.3 Results
A number of statistical tests were conducted to determine significance between the checklist and
smart pump experimental groups. Comparative analyses were performed between groups for the
following categories of data:
74
Demographic questionnaire data
Error detection rates
Post-test questionnaire data
In addition, within group analyses between baseline and intervention condition were conducted for
individual planted error rates and medication delivery failures.
5.3.1 Demographics Questionnaire
All nurses were recruited from GIM units within the study hospitals. No statistical difference was
observed in the gender, role, work status, length of shift, overall nursing experience, nursing
experience in current GIM unit, or daily frequency of programming infusion pumps between the
two groups.
A significant difference was found in age [2 (2, N = 48) = 7.549, p < 0.05] distributions between
the two participant populations. Further details of the statistical analysis are given in Appendix J.
The difference in age between the two groups showed an overall trend of younger participants in
the smart pump and bar-code intervention group (Table 9). However, as age was the only
significant difference between groups, the lack of significance amongst all other characteristics
still suggested a comparison between the two groups was valid. Additionally, differences between
participant populations were controlled in this study by the inclusion of a within-subject baseline
condition such that each participant served as their own control. Differences in error detection
between intervention groups can thus be attributed to the effectiveness of the interventions, rather
than any discrepancies between participants.
Table 9: Demographic survey results for each intervention group
Checklist Group
Smart Pump and
Bar-code Group
Age Range
18-29 years old
30-39 years old
40-49 years old
50-64 years old
>65 years old
46% (11/24)
13% (3/24)
25% (6/24)
17% (4/24)
0% (0/24)
63% (15/24)
29% (7/24)
8% (2/24)
0% (0/24)
0% (0/24)
Sex Male
Female
8% (2/24)
92% (22/24)
0% (0/24)
100% (24/24)
75
Role in Hospital Staff Nurse
Nurse Manage
Advanced Practice Nurse
Other
88% (21/24)
0% (0/24)
0% (0/24)
13% (3/24)
92% (22/24)
0% (0/24)
0% (0/24)
8% (2/24)
Clinical Unit GIM 100% (24/24) 100% (24/24)
Shift Times Both day and night 100% (24/24) 100% (24/24)
Length of Shift 12 hours
Less than 12 hours
92% (22/24)
8% (2/24)
92% (22/24)
8% (2/24)
Work Status Full-time
Part-time
Casual
71% (17/24)
25% (6/24)
4% (1/24)
92% (22/24)
0% (0/24)
8% (2/24)
Experience as an RN Less than 1 years
1 to 4 years
5 to 9 years
10 to 19 years
More than 20 years
13% (3/24)
29% (7/24)
25% (6/24)
4% (1/24)
29% (7/24)
17% (4/24)
46% (11/24)
33% (8/24)
4% (1/24)
0% (0/24)
Experience working
in current clinical
unit
Less than 1 years
1 to 4 years
5 to 9 years
10 to 19 years
More than 20 years
13% (3/24)
33% (8/24)
29% (7/24)
13% (3/24)
13% (3/24)
17% (4/25)
54% (13/24)
29% (7/24)
0% (0/24)
0% (0/24)
Frequency of
programming
infusion pumps
Less than once a day
1 to 2 times a day
3 to 5 times a day
More than 5 times a day
13% (3/24)
38% (9/24)
38% (9/24)
13% (3/24)
21% (5/24)
33% (8/24)
33% (8/24)
13% (3/24)
5.3.2 Error Detection Rates
5.3.2.1 Comparative Analysis
The error detection rate was compared between intervention groups by condition (baseline or
intervention) and by error type (execution or planning). A 2 (error type: execution or planning) x 2
(condition: baseline or intervention) x 2 (intervention group: checklist or smart pump and bar-
coding) mixed-factorial ANOVA with repeated measures for the first two factors was conducted to
determine significant effects.
76
The three way interaction between error type, condition and intervention group was not found to
be significant, likely due to the lower error detection rates in the baseline condition for both
intervention groups, as was expected. Figure 19 shows the two-way simple interactions of the
estimated error detection means for the checklist group (left) and smart pump and bar-coding
group (right), the difference between which constitutes the three-way interaction. The two-way
interactions in each group shows the trend expected based on the overall hypothesis stated in
Chapter 3. The graph of the checklist intervention group shows similar means between the
baseline and intervention for planning errors however, the checklist intervention appears to be
greater in detecting execution errors. For the smart pump group, the mean error detection appears
higher in the intervention condition than the baseline for both error types.
Figure 19: Mean error detection to visualize three-way interaction
While the three-way interaction was not significant, several two-way interactions were found to be
significant. A significant two-way interaction was observed between condition (either baseline or
intervention) and intervention group, F (1, 46) = 8.745, p < .05. Figure 20 shows the differences in
marginal mean error detection rates between intervention groups by condition. This figure
indicates there to be no difference in error detection for the baseline condition between
intervention groups.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Execution Planning
Me
an
Err
or
De
tect
ion
Ra
te
Checklist Group
Baseline
Intervention
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Execution Planning
Me
an
Err
or
De
tect
ion
Ra
te
Smart Pump Group
Baseline
Intervention
77
Figure 20: Differences between condition and group after averaging across the two error
types (execution and planning)
Nurses detected significantly more errors overall when using the smart pump and bar-code
intervention (M: 79%; SE: 3%) compared to current practice in the baseline condition (M: 43%;
SE: 6%), [t (23) = -5.865, p < .05]. Conversely, there was no significant difference in error
detection between the baseline (M: 44%; SE: 5%) and checklist intervention (M: 52%; SE: 5%).
While the smart pump and bar-code intervention nearly doubled the performance compared to the
current practice, the checklist showed an insignificant improvement of less than 10%. The error
detection rates between interventions, 79% (SE: 3%) for the smart pump and bar-coding versus
only 52% (SE: 5%) with the checklist, were significantly different [F (1, 46) = 13.866, p < .05];
this was largely due to improved planning error detection using the smart pump and bar-code
system as described further in Figure 21.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Checklist Smart Pump
Ma
rgin
al
Me
an
Err
or
De
tect
ion
Ra
te
Baseline
Intervention
78
Figure 21: Differences between error type and group after averaging across the two
conditions (baseline and intervention)
A significant two-way interaction was also observed between error type and intervention group, F
(1, 46) = 8.223, p <.05, shown in Figure 21. In the checklist intervention group, nurses detected
significantly fewer planning errors (M: 37%; SE: 5%) compared to execution errors (M: 59%; SE:
45) [t (23) = 4.067, p < .05]. Conversely, in the smart pump and bar-coding intervention group,
error detection did not vary between execution (M: 61%; SE: 5%) and planning error types (M:
61%; SE: 5%). Comparison between intervention groups revealed that only detection of planning
errors was significantly different between groups [F (1, 46) = 9.370, p <.05] while no difference in
execution errors was found between groups. Nurses detected significantly more planning errors
from the smart pump and bar-code group at 61% (SE: 5%) compared to the checklist group at only
37% (SE: 5%). Additional details of the comparative analysis are available in Appendix K.
5.3.2.2 Within-Group Detection of Planted Errors
A separate analysis was conducted on error detection rates for each individual planted error type
within each intervention group. The McNemar test was used to compare rates between the baseline
condition and intervention condition within each group. No significant difference was observed
between the baseline condition and intervention condition for any individual planted error type in
the group using the checklist intervention (Figure 22).
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Checklist Smart Pump
Ma
rgin
al
Me
an
Err
or
De
tect
ion
Ra
te
Execution Errors
Planning Errors
* *
79
Figure 22: Error detection rate by planted error between baseline and intervention in the
checklist group (N=24)
For the bar-code smart pump group shown in Figure 23, a significant difference was observed for
the wrong dose execution error with an error detection rate of 46% in the baseline condition and
92% when using the bar-code and smart pump system (p < .05). A significant difference in error
detection rate between baseline and intervention condition was also observed for inappropriate
rate, inappropriate duration and incompatibility errors. The inappropriate rate was detected by only
29% of nurses in the baseline condition compared to 100% of nurses when using the smart pump
intervention. Inappropriate duration was detected by 42% of nurses in the baseline compared to
88% when using the smart pump. Finally, the incompatibility error was detected by only 38% of
nurses in the baseline condition versus 83% when using the smart pump with clinical advisories.
79%
21%
58%
42%
25%
33%
46%
79%
38%
79%
42%
33% 33%
50%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
% o
f n
urs
es
wh
o d
ete
cte
d p
lan
ted
err
or Baseline Checklist Intervention
Execution Errors Planning Errors
80
Figure 23: Percentage of nurses that detected each error type in the smart pump group
(N=24)
5.3.3 Medication Delivery Failures
Each participant completed 9 scenarios per condition (baseline and intervention) which involved
setting up a total of 8 primary infusions and 5 secondary infusions. Excluding failures due to
planted errors, medication delivery failures were recorded for closed clamps, missed armband
checks, programming errors and failures to start the infusion for each infusion.
In the checklist group, the only significant difference found between the baseline and intervention
condition was in the number of missed armband checks, t (23) = 2.782, p < .05. A total of 23
(mean: 0.96 per nurse) armband checks were missed prior to administration for the baseline
condition compared to only 2 (mean: 0.04 per nurse) missed armband checks when using the
checklist intervention (Figure 24). Thus the checklist reminder did improve compliance for
remembering to check the patient identification prior to administration.
58%
33%
46%
38%
50%
29%
42%
75%
63%
92%
83%
58%
100%
88%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
% o
f n
urs
es
wh
o d
ete
cte
d p
lan
ted
err
or
Baseline Bar Code Smart Pump Intervention
Execution Errors Planning Errors
81
Figure 24: Number of medication delivery failures by type for the checklist group (N=24)
Figure 25 shows the mean number of failures by failure type between the baseline and
intervention condition for the bar-code smart pump group. Of the four medication delivery type
failures, a significant difference was seen in the frequency of missed armband checks, [t (23) =
2.108, p < .05] and the number of programming errors, [t (23) = 2.901, p < .05] between the
baseline and intervention conditions. In the baseline condition, 32 armband checks (mean: 1.33 per
nurse) were neglected whereas in the barcode scanning condition significantly fewer, only 6
failures to check the patient armband (mean: 0.25 per nurse), were observed. A total of 16
programming errors (mean: 0.67 per nurse) were recorded in the baseline condition using the
Grasby pumps, whereas significantly fewer, only 1 programming error (mean: 0.04 per nurse), was
observed when using the Alaris smart pump.
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
Missed armbandcheck
Programmingerrors
Closed clamps Failure to start
Av
era
ge
Nu
mb
er
of
Fa
ilu
res
Medication Delivery Failure Type
Baseline
Checklist Intervention
82
Figure 25: Mean number of medication delivery failures by type for the smart pump group
(N=24)
5.3.4 Post-test Questionnaire
5.3.4.1 Checklist Group
Quantitative post-test questionnaires responses measured on the five-point Likert scale were
compared between the two participant groups. For the group using the checklist intervention, all
statements relating to ease of use were positive with an average ranking over all questions relating
to usability of 4.49. Questions related to perceived efficiency of the checklist were less positive
with a mean ranking of 3.90. Similarly, the response to statements assessing the overall impression
of the checklist had a mean ranking of only 3.81.
In addition to ease of use, efficiency and overall impressions, the post-test questionnaire asked
participants to rank the perceived effectiveness of the intervention at addressing different types of
errors. The checklist was perceived to be most likely to detect wrong drug and wrong patient errors
with mean rankings of 4.13 and 4.17, respectively. Detection of wrong concentration of drug and
wrong rate had the lowest mean rankings of 3.83 and 3.89, respectively.
Qualitative responses of the post-test questionnaire for the checklist are summarized below:
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
Missed armbandcheck
Programming errors Closed clamps Failure to start
Av
era
ge
Nu
mb
er
of
Fa
ilu
res
Medication Delivery Failure Type
Baseline
Smart Pump Intervention
83
Many nurses felt that the checklist was straightforward, easy to use and followed a logical
order of steps that matched the workflow on the unit.
Overall, nurses felt it was helpful for double checking to ensure safe medication
administration practice.
Most nurses liked that it had reminders for steps they felt could be forgotten if workload
was high. They liked the specific reminders for the ‘rights’ of medication administration
and to check clamps.
Several nurses specified that they liked the visual ‘Stop’ reminder to think again about the
details of the order and medication.
Several nurses felt some items on the checklist were repetitive, particularly the double
checks following pump programming.
A few nurses also disliked the amount of time it took to complete and felt it was too time
consuming to complete on a busy unit. In the clinical setting, this could result in a
deterioration of performance over time.
Many nurses felt a paper-based checklist might not be feasible on the unit stating
limitations in space, the need for a pen and the overall increase in paperwork to complete a
checklist for each medication administration instance. They felt it might be best as a posted
reminder in the medication room and at the bedside. This feedback was consistent with
remarks from the initial development of the checklist in Chapter 4, concurring the
opinions of the separate group of nurses to use the checklist during the usability study.
A few nurses were concerned that the checklist might detract from a nurse’s critical
thinking ability. They questioned the added value of the checklist, feeling that with
increasing familiarity, there may be a tendency to mark boxes without actually checking.
5.3.4.2 Smart Pump Group
Responses the post-test questionnaire by the smart pump group revealed a positive overall mean
ranking of 4.23 over all statements relating to ease of use. With regards to efficiency, the mean
ranking was 3.73 while the mean ranking of overall impression was only 3.58.
84
For questions regarding the perceived likelihood of detecting errors, participants gave the highest
likelihood for detecting wrong drug errors with a mean ranking of 4.17. The likelihoods of
detecting wrong concentration, wrong patient, wrong dose and wrong rate were all ranked highly.
Qualitative responses for the bar-code smart pump system are summarized below:
Nurses liked that the bar-code system provided another form of double checking in
addition to their own checks and any independent double checks by a colleague. They felt
it provided more accurate patient and drug verification and minimized the likelihood of
human error. A few nurses even felt the bar-code verification was more time efficient, an
opposite reaction from the group of nurses using the checklist.
Some nurses however, felt the bar-code scanning took more time with some difficulties in
reading the bar-codes. While they felt it was an additional check to their own verification,
some were uncomfortable relying on the technology to complete the checks. While most
nurses reported to be comfortable using the technology, they were concerned that the
technology was a distraction with the potential to cause them to miss a check.
Most nurses liked the hard and soft dose rate limits on the pump and felt it provided
another measure of safety, especially for less familiar drugs. The pump advisories and
limits were felt to be the most critical features in preventing errors by most nurses.
Most nurses found the system easy to use and felt the prompts and error messages were
useful and meaningful.
A few nurses were worried that the smart pump minimized their critical thinking and were
concerned that they would become too reliant on the pump to detect errors. They were
concerned that the overall increase in automation removed more responsibility from the
RN and requirements for critical thinking skills. They felt this could hypothetically lead to
unsafe patient care if too much reliance was placed on the technology.
5.3.4.3 Comparative Analysis between Groups
The Mann Whitney U test was conducted on each post-test question for comparative analysis
between the checklist and smart pump groups. Significant differences were found between groups
for three questions in the survey, all relating to ease of use.
85
The statement, “overall, the [intervention] was easy to use”, was significantly different between
groups [U = 153.00, p < .01] with the checklist having a significantly higher mean ranking of 4.63
compared to only 4.04 for the smart pump group. The checklist group also had a significantly
higher ranking of 4.63, compared to 3.92 for the smart pump group, in response to the statement:
“it was easy to navigate items and follow along with the steps” [U = 129.00, p < .01]. The mean
ranking of responses to the statement “it was easy to correct mistakes” was also statistically
significant, U = 154.00, p < .01. The checklist group had a higher mean ranking of 4.46 versus
only 3.75 for the smart pump group.
5.4 Discussion
The overall aim of this study was to evaluate the effectiveness of a checklist intervention
compared to a bar-code smart pump solution in the detection of two classifications of errors. The
following section examines the results of this study to gain a better understanding for the relative
effectiveness of these interventions at mitigating errors.
5.4.1 Error Detection Rates
A comprehensive analysis of error detection rates between groups was conducted to elucidate the
effects of error type and intervention on error detection. This analysis was intended to address the
main research questions posed in this study in Chapter 3. Outlined below are the primary
conclusions drawn from the results:
1. Overall, the bar-code and smart pump intervention was significantly more effective at
detecting errors than the checklist.
2. Additionally, in comparison of the effectiveness to current practice, only the bar-code smart
pump intervention was more effective overall. Using the checklist intervention did not
significantly increase overall error detection compared to current practice.
3. In particular, the smart pump was significantly better at detecting planning errors than the
checklist. The checklist was particularly poor at detecting planning errors compared to
execution errors. The detection rate of execution errors was not significantly different between
checklist and smart pump and bar-code intervention groups.
86
The first primary hypothesis stated in Section 3.2 is confirmed by conclusions #1 and #2. These
conclusions were derived from the results of the two-way interaction effect between condition and
intervention group. The two-way interaction revealed differences in overall error detection rates
between each intervention group and to the within-group baseline. As desired, the baseline control
detection rate was found to be the same between intervention groups. The consistency in the
control between groups therefore attributes the significant interaction to differences in the
intervention condition. The baseline condition represented the control state in which no
intervention beyond the current practice was used to help nurses detect errors. It was expected that
some error detection would still occur in this condition due to nursing vigilance, critical thinking
skills and the application of current education and policies for safe medication practices (i.e.
mental checks and training on the ‘rights’ of medication administration).
In the checklist group, no significant difference in error detection rate was observed between the
checklist intervention condition (M: 52%; SE: 5%) and the baseline control (M: 44%; SE: 5%)As
the checklist intervention belongs to the lower level of the hierarchy of effectiveness, this result
provides evidence in support of the Hierarchy of Effectiveness. Lower level interventions to
mitigate the risk of medication errors are intrinsic in the baseline condition as nurses receive
training for safe medication practices and follow policies to ensure the ‘rights’ of medication
administration. Therefore, based on the results, the checklist intervention did not add any
additional power to the current ability of nurses to detect errors. While no additional benefit was
found to overall error detection rates, further investigation of effectiveness in the real clinical
environment should be conducted prior to disregarding the use of checklists altogether.
Conversely, the smart pump intervention was significantly more effective at detecting errors (M:
79%; SE: 3%) than the baseline control condition (M: 43%; SE: 6%), as expected. This result
agrees with a previous investigation of smart pumps by Trbovich et al. which also reported smart
pumps to detect more errors than traditional infusion pumps.11
This rate was also significantly
higher than the error detection rate when using the checklist intervention (M: 52%; SE: 5%). These
observations confirm the first hypothesis and provide support for the Hierarchy of Effectiveness.
The bar-code smart pump system lies on the higher level of the Hierarchy of Effectiveness for
interventions as it involves aspects of automation, computerization and forcing functions (in the
case of the hard limit). The bar-code smart pump solution intervenes at a system’s level and
87
minimizes the reliance on human vigilance. Alternatively, the checklist intervention, which lies on
the lower end of the Hierarchy, is a reactive intervention directed towards changing human
behavior. It continues to rely on human vigilance to detect errors and is thus susceptible to human
error and biases (e.g. confirmation bias). Based on this result, in the application of the medication
administration process, this relationship between human level interventions and system level
interventions appears to be maintained. Conclusion #3 addresses the second research question
posed in Chapter 3 regarding effectiveness of interventions by classification of error. Execution
failures in this study were related to simple, mechanistic tasks requiring the comparison of one
source of tangible information to another whereas, planning errors arose from more cognitively
challenging clinical decision tasks which required the application and integration of abstract
knowledge from multiple sources to the given situation. The detection of planning errors was
expected to be more challenging than execution errors and by extension, hypothesized to require a
more technological solution.
The significant interaction between error type and intervention group revealed the detection of
planning errors to be significantly greater in the smart pump and bar-code intervention group (M:
61%; SE: 5%) compared to the checklist group (M: 37%; SE: 5%). This result was expected due to
the nature of the error detection tasks and previous results by White et al. which found errors that
fell into the classification of planning failures, unlikely to be detected by checking procedures.10
Furthermore, Ferner and Aronson suggested knowledge-based mistakes (which fall under the
category of planning errors) to be best managed by computerized decision support.36
Thus, the
clinical decision support provided by the smart pump through the drug library dose error reduction
system and clinical advisories was expected to be effective at mitigating planning errors. No
significance was observed in the detection rate of execution errors between the checklist and smart
pump groups. The lack of significance for execution errors between the two groups is likely due to
the high rate of execution error detection in the baseline condition.
The relative ineffectiveness of checklists observed in this study can be generalized to other
applications of checklists in healthcare. Though only detection of errors in medication
administration were evaluated in this study, regardless of the application, checklists were
demonstrated to be a poor intervention, particularly for the detection of planning errors which
require application and integration of knowledge from multiple sources. Previous studies reporting
88
successful results from checklist interventions failed to rigorously demonstrate a direct effect as a
result of the checklist alone. In the case of the WHO safe surgical checklist evaluated in 8 global
sites, all surgical safety policies included as steps on the checklist were not in place at all locations
prior to the trial.49
Thus, the positive result of this study cannot be differentiated between the use
of the checklist or the corresponding introduction of all surgical safety measures (such as use of
pulse oximetry, or routine administration of prophylactic antibiotics in the OR) to the sites.
Similarly, positive reduction in infection rates from central venous catheter insertions were
reported when applying a 5-step checklist to this procedure.46,47
However, these studies
implemented five interventions simultaneously such that the overall result was attributed to the
overall combination of interventions rather than the use of the checklist alone.
While rigorous optimization of the checklist was undertaken, no such optimization of the smart
pump and bar-code system was conducted. Despite this lack of optimization, the smart pump and
bar-code system was still shown to be significantly better in overall error detection than both the
checklist and current practice. Improvements to achieve a closed-loop medication administration
system in which the smart pump, bar-coding and MOE/MAR are fully integrated could further
improve the performance of this system. This study found no significance in the detection of
wrong patient and wrong drug execution errors when using the smart pump and bar-coding
compared to the current practice. Because the bar-code verification system was not integrated with
the pump, nurses forgot to scan the patient wristband altogether in a few instances. No forcing
function required nurses to scan the patient ID prior to programming the pump by the system used
in this study. Additionally, in the event that the bar-code verification was used, the system still
relied on vigilance from the nurse in acknowledging that the automatic verification detected an
error. Again, no forcing function prevented nurses from proceeding with the infusion once an error
was detected. Thus, a fully integrated system in which bar-code verification is required prior to
pump programming could minimize these modes of failure. Optimization of the smart pump and
bar-coding system is thus expected to further improve the error detection performance relative to
the checklist and current practice.
While the bar-coding aspect of the technological intervention was directed towards addressing
execution errors by automating verification tasks, features of the smart pump (drug library with
dose error reduction system and clinical advisories) provided the clinical decision support to
89
improve the detection of planning errors. When using the smart pump, 100% of inappropriate rate
errors were detected compared to only 29% in current practice. This result was due to the hard
dose rate limits in the drug library which prevented the nurse from continuing with the infusion at
the erroneously high rate. Similarly, 88% of inappropriate duration planning errors were detected
with the smart pump due to the soft limit warning while only 42% were detected using the
traditional pump. In addition to hard and soft limits, selection of standard doses from the drug
library and clinical advisories contributed to the detection of wrong dose and incompatibility
errors, respectively. Based on the features identified to contribute to the effectiveness of the smart
pump, the results of this study may be generalized across all smart pump models in which these
features are incorporated and properly implemented. Though still more effective than lower-level
interventions, differences in effectiveness between smart pump models may create sublevels
within the particular automation and computerization level of the Hierarchy of Effectiveness.
However, failure to implement a comprehensive drug library with hard and soft dose limits and
clinical advisories may diminish the effectiveness of the smart pump intervention.
5.4.2 Confirmation or Rejection of Specific Hypotheses
To further understand the relationship between the error type and intervention, a discussion of
error detection rates for each individual planted error from the two classification groups is
presented. Statements of specific hypotheses outlined in Section 5.2.6 on how each individual
intervention would perform in comparison to the current practice for a specific planted error are
addressed in this section. Rejection or confirmation for each specific hypothesis is given in Table
10.
Table 10: Confirmation of rejection of specific hypotheses.
Specific Hypothesis Intervention Group
#1. Wrong patient Checklist
Smart Pump and Bar-code
Rejected
Rejected
#2. Wrong drug Checklist
Smart Pump and Bar-code
Rejected
Rejected
#3. Wrong dose Checklist
Smart Pump and Bar-code
Rejected
Confirmed
90
#4. Inappropriate Dose Checklist
Smart Pump and Bar-code
Confirmed
Rejected
#5. Inappropriate Rate Checklist
Smart Pump and Bar-code
Confirmed
Confirmed
#6. Inappropriate Duration Checklist
Smart Pump and Bar-code
Confirmed
Confirmed
#7. Incompatibility Checklist
Smart Pump and Bar-code
Confirmed
Confirmed
Wrong patient, wrong drug and wrong dose errors fall under the classification of execution errors.
In the case of the wrong patient error, a discrepancy in the patient identification information
between the patient in the bed and the patient on the order and medication label is present. As part
of the safe medication administration practice, nurses are required to verify patient name, medical
record number (MRN) and date of birth. This task involves matching information first between the
medication label and the physician’s order and then also to the information on the patient
wristband. Similarly, the associated tasks for wrong drug and wrong dose errors involve
comparing information on the medication label to the physician’s order. The mechanistic task of
comparing two sources of tangible information is susceptible to action-based slips or memory
lapses. Forgetting to check the patient’s armband, drug name or dose on the medication label, or
not recognizing discrepancies between information sources due to distractions (attention failures)
or confirmation bias are all possible reasons for failures to detect these execution errors.
While checking procedures are suggested by Ferner and Aronson to be effective for managing
action-based slips,36
the lack of difference between the current practice and checklist intervention
suggests that nurses did not receive any added advantage from using the checklist. It was expected
that the checklist, which includes specific reminders to check the patient ID, drug name and dose,
would increase the percentage of nurses who detected the errors compared to the current practice.
However, current established practices for checking patient identification were already well
practiced by participants in the baseline condition.
There was a similar lack of significance between the baseline and bar-code smart pump
intervention for wrong patient and wrong drug errors. This was surprising as it was expected the
automatic verification of bar-code scanning would reduce any errors due to confirmation bias or
91
attention lapses while completing the check. In review of the literature, the analyses of
implemented bar-code medication administration systems have shown mixed results, with some
finding minimal impact on safety.14, 55-57
. These studies cite workarounds and the introduction of
new errors as the limiting factor.55, 58
. In this study, failures to detect the error were as a result of
omission in use of the system (i.e. forgetting to scan the wristband) or failure to acknowledge the
error message by the system. While nurses were alerted to the error, it was often observed that they
would discount the alert and continue to deliver the medication if they could not identify the
source of the error independently from the technology. This observation suggests that the error
messages provided by the system may not have been sufficient for effective error detection.
Additionally, while some participants continued with the medication administration if they were
convinced there was no discrepancy, this practice likely would not be permitted on a true clinical
unit. While policy could dictate how these errors are to be handled on the nursing ward, the nature
of the system still allows for workarounds and relies on compliance as was observed in previous
studies.55, 58
As previously discussed, further optimization of the smart pump and bar-code system
to create a fully integrated closed-loop medication administration process could improve the
performance in detection of these execution errors.
Unlike wrong patient and wrong drug errors, a significantly higher percentage of nurses detected
the wrong dose error with the bar-code smart pump system than with no intervention. This was
likely due to a combination of features from both the automated verification by the bar-code
system and the smart pump drug library. Nurses were instructed to program infusions through the
drug library which requires the user to select both the drug and dose to be infused. Several
standard doses for each medication were included in the library based on the hospital IV
formulary. In the event that the nurse could not locate the desired dose in the drug library, a double
check of the medication label and order was often completed. This feature thus provided some
clinical decision support as it presents standard doses for medications. By critically considering
this information in comparison to the dose on the medication label, potential discrepancies may be
determined.
Contrasting errors related to verification tasks discussed previously, inappropriate dose, rate and
duration errors fall into the category of planning failures as they arise from knowledge-based
failures. The nurse must recognize that some aspect of the medication order is clinically
92
inappropriate (e.g. dose is ten times too high due to transcription error by physician). This requires
the application of abstract clinical knowledge and integration of information from multiple sources
(e.g. IV formulary, nursing experience). While explicit reminders to verify the correct information
between the medication label and physician’s order are included on the checklist, considering the
clinical appropriateness of the order is a more cognitively challenging task. Only a general
reminder to stop and apply knowledge to verify the logic of the order is included on the checklist.
As was discovered by White et al.10
, this step on the checklist failed to be effective in the detection
of all planning errors (inappropriate dose, inappropriate rate, and inappropriate duration) over the
baseline rate.
In the smart pump condition no significance was found in the detection of inappropriate dose
errors. While the pump provides a measure of clinical decision support by requiring users to select
a dose in the drug library, this was not sufficient to be significantly more effective. While the
absence of the erroneous dose from the drug library may have prompted nurses to reconsider the
order, it failed to lead them to discover the error. This is likely due to the fact that no discrepancy
was found between the dose on the medication label and the physician’s order which convinced
the nurse that the medication dose was correct. If the dose could not be found in the drug library,
nurses setup the infusion as a basic infusion, bypassing the drug library. This work-around may
have limited the ability of the pump to mitigate inappropriate dose errors.
Alternatively, the inappropriate rate error was significantly higher with the smart pump compared
to the baseline condition. The detection rate of this error was 100% in the smart pump condition
due to the hard limit alert by the smart pump. When a dose rate exceeds a built in hard limit
(configured in the drug library by pharmacy), an alert is given in which the user cannot continue
with the infusion. The hard limit is an effective forcing function, preventing the user from
continuing without correcting the error. The only option is to reprogram the infusion with
parameters below the hard dose limit. Upon hitting a hard limit, nurses either consulted the IV
formulary and reprogrammed the rate to within the appropriate range or asked to verify with a
physician. While no nurses were observed using a workaround by programming the infusion
outside of the drug library, this was identified in multiple studies as a major limitation in the
impact of smart pumps on serious medication errors.11, 12, 53, 54
93
Similarly, a greater percentage of nurses detected the inappropriate duration when using the smart
pump compared to the baseline condition. The soft limit alert on the pump warned users that the
infusion parameters violated a soft limit. This soft limit warning however allows users to override
and continue with the infusion without reprogramming. Even with this option, the soft limit was
still significantly more effective at detecting the inappropriate duration error than no intervention.
It was observed that most nurses chose to comply with the warning and reprogrammed the pump
after consulting the formulary or notifying the physician. However, some nurses did choose to
override the soft limit and continue with the infusion despite the warning. By allowing an override
of the soft limit warning, poor compliance has the potential to limit the effectiveness of this
feature.12
The final planted error involved incompatibility between the primary IV fluid and a secondary
medication. This error was not classified as either execution or planning and so was not considered
in the main analysis. Checking compatibility between the primary and secondary infusions
required either referencing a compatibility chart or consulting the IV drug formulary. This process
may also require an additional element of critical thinking and clinical knowledge. For example,
the antibiotic ceftriaxone should not be infused with any calcium containing solutions, thus it
should not be infused with lactated Ringer’s IV fluid (which contains calcium). While nurses may
be aware that this antibiotic cannot be infused with calcium, the additional knowledge and
identification that lactated Ringer’s contains calcium is required. For this reason, this error was not
included in the either category of error. While no difference was found between the checklist and
baseline condition, the percentage of nurses who detected the incompatibility error with the smart
pump and bar-coding was significantly greater compared to the baseline. This was due to the
clinical advisory feature in the smart pump drug library. Clinical advisories provided medication
specific alerts, integrating significant information included in the IV formulary for a given
medication into the smart pump. This feature provided the needed clinical decision support and
also served as a last minute reminder for specific considerations prior to infusing the selected
medication.
5.4.3 Medication Delivery Failures
In addition to planted errors, four types of medication delivery failures were compared between
baseline and intervention conditions. These failures included forgetting to check the patient
94
armband, failure to open all clamps failure to start infusion and any pump programming errors and
were all considered falling within the classification of execution errors. It was hypothesized that
medication delivery failures would be reduced in both intervention groups compared to the
baseline.
In the checklist group, a significant difference was observed only in the number of missed
armband checks. The checklist had no observed significant effect on programming errors or
failures to open clamps and start infusions compared to the baseline. The reduction in missed
armband checks suggests that the checklist reminder was useful in prompting nurses to complete
the task. In comparison, no difference was found in the other delivery failures for which the
checklist included a specific check. Programming errors observed included entry of parameters
into the wrong fields (i.e. entering the rate into the field for volume to be infused), keystroke errors
(e.g. entering 1000 instead of 100) or programming the secondary infusion incorrectly (i.e.
neglecting to select secondary and overwriting the settings for the primary infusion). The check to
verify the pump settings did not result in a significant difference in the occurrence of pump
programming errors. However, due to the small instance of total observed failures, a true
evaluation of the checklist on these errors is difficult.
Similar to the checklist intervention, the bar-code smart pump intervention also showed
significance in the number of missed armband checks. Fewer missed armband checks were
observed with the intervention. This was likely due in part to the introduction of new technology
as nurses were specifically directed to use the bar-code verification system. Additionally, a
significant decrease was observed in the number of programming errors. Users made fewer errors
in programming the smart pump than the traditional infusion pump. As nurses were unfamiliar
with the smart pump, this result is possibly due to increased attentiveness in programming as the
task was still unfamiliar.
5.5 Study Limitations
The major limitations of this study are described below:
1. Simulation of clinical environment: This study used a simulated clinical environment in
which to evaluate the effectiveness of the interventions. While high-fidelity of the
95
simulation with the true clinical environment was desired, several discrepancies were still
present. Consistent scripted interruptions by the confederate nurse were included to divert
attention from the task at hand and a sound track of ambient hospital noise was played.
However, these measures did not provide the same level of background activity and
disruptions as the typical ward. As such, nurses could focus more acutely on their work and
were perhaps more vigilant in their medication administration tasks than on the ward. The
additional pressure of being observed in their task could also have led to higher vigilance,
leading nurses to detect more errors. The overall rate of error detection thus may have been
greater than the rate of detection in the true clinical setting. For this reason, differences in
error detection from the baseline condition to illustrate the effectiveness of the
interventions may have been reduced.
2. Awareness for planted errors: In order to explore the impact of different classifications of
error, an unrealistically high number of errors were included in this study. Planted errors
were present in 7 of the 9 scenarios for each condition (14 out of 18 scenarios for each
participant). The inclusion of two scenarios with no planted errors was intended to help
mask the overall presence of errors. However, over the course of the experiment, it is still
possible that increased awareness for errors developed. To minimize this learning effect,
the orders of scenarios and conditions (baseline or intervention first) were counterbalanced.
Despite these efforts, a longer study with more scenarios to discretely mask the inclusion
of planted errors may be more effective in decreasing the vigilance and overall awareness
for errors.
In addition to the high frequency of planted errors, both the training provided and the
nature of either the checklist intervention or bar-code smart pump system may have
provided some indication to participants on the true purposes of the study. The true goals
of the study were not revealed to the participant until completion of the experiment and it
was emphasized that the study assessed all aspects of the medication administration
workflow (including hand washing, patient care etc.) in an effort to minimize this
awareness.
96
3. Modification of behavior: While nurses were instructed to complete the medication
administration process in the same manner as on the ward, the simulated nature of the
experiment introduced possible changes in typical behavior. Due to restrictions in time and
resources, certain tasks were completed for the participant, which were inconsistent with
reality. Each medication bag was already spiked with the IV set and the lines pre-primed.
Flushing of IV access points was also not required by nurses. The omission of these tasks
in the simulation may have contributed to assumptions that the study was concerned only
with the infusion setup. It was also observed that the use of hand sanitization, gloves and
alcohol swabs for IV access points was inconsistent by most nurses. Due to the non-real
conditions, it may thus also have been assumed that certain errors were a product of the
simulation and not relevant to the study. In a real clinical setting, it is expected that any
errors detected in the medication order would be corrected, whereas it is possible that some
participants did not react to errors in the same manner as they would on a clinical unit.
4. Novelty of interventions: Prior to the study, nurses were unfamiliar with the checklist and
bar-code smart pump interventions used. Training was provided in order to introduce
participants to the proper use of the checklist or bar-code smart pump system. Due to the
novelty of a new tool, nurses may have been more compliant and attentive to the correct
and intended use of the intervention. In the case of the checklist, users may have made a
more heedful effort to comply with the instructions for the purpose of the study. In
addition, the controlled nature of the simulation and unintended influences from the
training provided may have encouraged users to comply with the demonstrated
instructions. For example, limited instances of overriding the hard limit on the pump or
programming an infusion outside of the drug library may have been observed due to the
user uncertainty with the device. With increased familiarity, it is possible that users may be
more inclined to explore deviations from the training workflow as was observed by
Rothschild et al.12
As such, the results of this study are limited to the initial effectiveness
level of the interventions. Thus, there is the potential that the results obtained in this study
may not be maintained with increasing familiarity to the interventions.
97
5.6 Summary
The effectiveness of a paper-based checklist at detecting IV medication administration errors was
compared to a smart pump and bar-coding system in a high-fidelity simulation study. The rate of
error detection for each intervention type was compared to a baseline control condition. The
comparison also included the additional factor of error type, either execution or planning errors.
Results revealed that the checklist intervention provided no significant benefit to nurses in
detecting planning errors compared to the current practice using the traditional Grasby pump.
However, the bar-code smart pump system was observed to be significantly more effective in
detecting errors compared to the current practice. In particular, it was noted that a greater
percentage of nurses detected planning errors in the smart pump group than the checklist group.
These results thus illustrate the limited effectiveness of a checklist for error detection in the
medication administration process.
98
6 Conclusions
Medication administration errors have been identified as a significant cause of adverse events in
health care. As such, an investigation of strategies to prevent these errors and an understanding of
their effectiveness is needed. The primary objective of this study was therefore to experimentally
compare the effectiveness a human oriented intervention to a design oriented intervention in the
detection of different categories of error. Specifically, the present work investigated the use of a
medication administration safety checklist in detecting execution and planning errors compared to
a smart pump and bar-code verification system. The first phase of this work required the
development of a checklist specific to the IV medication administration process. The checklist was
designed based on policies for safe medication practices and existing forms for independent double
checks of ambulatory infusion pumps in chemotherapy. Two cycles of usability testing were
conducted to improve the design and minimize aspects which could negatively impact its
effectiveness.
The effectiveness of the resulting checklist was subsequently tested, in the second phase of this
work, in a high-fidelity simulation experiment. This experiment compared the detection rates of
planted errors (classified into either execution or planning failures) between current nursing
practice (control) and either the checklist intervention or a smart pump and bar-code verification
system. Results showed there to be no significant improvement in the percentage of nurses who
detected errors when using the checklist intervention compared to current practice with no
intervention. Thus, the checklist appeared to provide no additional benefit for error detection in
this setting. Alternatively, results indicated that the bar-code smart pump system was significantly
more effective at detecting errors than current practice with a traditional infusion pump.
Specifically, a greater percentage of nurses detected planning errors in the smart pump group
compared to the checklist group.
The results of this work provide insight into the utility of a checklist in the medication
administration process and it overall effectiveness relative to current practice and a higher-level
intervention. The implications of this research in the fields of patient safety and guidance for
future directions are discussed further below.
99
6.1 Relevance to the Field
The present work contributes to the fields of patient safety and medication administration practice
in the evaluation of error mitigation strategies. Results of this work provide relevant considerations
in the selection, implementation and design of appropriate interventions for medication
administration errors. In particular, these results prompt caution in the widespread implementation
of checklists as a sound error prevention strategy. As this study demonstrated, the independent
effectiveness of a checklist strategy in the medication administration process was not found to add
benefit for error detection over current practice. This result should be applied in the field of patient
safety to ensure that checklists are not the only strategy relied upon for error intervention. In
addition, although implementation of smart pumps requires more overall resources and planning,
this work reinforces evidence for the value of smart pumps in error detection. The overall
distinctions in error detection rates between execution and planning errors suggested that planning
errors require a more automated solution with clinical decision support and the ability to integrate
information. These errors are unlikely to be addressed by checking procedures. Thus these results
are useful in consideration of strategies for mitigation of a particular error type and provide
guidance for the selection of interventions in nursing practice.
6.2 Future Work
Based on the results of this study, recommendations for future work are suggested to address both
the limitations of this study and expand the field of research. One avenue for additional work is
investigation of the interventions presented in this study in a real clinical environment. While the
present work evaluated the effectiveness in a controlled simulation environment, additional factors
may alter this result in the clinical setting. Challenges in implementation, changes in compliance
over time, pressures of time and workload, and the introduction of new potential errors are all
interesting possibilities that may have a significant impact on the effectiveness of the
interventions. Some elements of this proposed investigation have been done in the evaluation of
smart pump technology and bar-code medication administration systems. However, the
complications for the effectiveness of a medication administration checklist on the clinical ward
have not been investigated. While the present work resulted in no overall significance when using
the checklist, the controlled environment and high detection rate of errors in the baseline condition
may have underestimated its value. In a more variable clinical setting with a higher workload and
100
more distractions, error detection performance with no intervention may be less than that observed
in the simulated setting. In this setting, (or with more power by increasing the number of planted
errors), checklists may show some improvement in performance for detecting execution errors as
the reminders may be useful for maintaining a standardized workflow and coping with
interruptions.
Secondly, the design of a more effective intervention based on the potential workarounds and
failure modes identified for the current solutions may be explored. While a paper-based checklist
was used in this study, it was suggested by users that the checklist somehow be integrated into the
MOE/MAR, assuming that it is available in multiple locations for convenient access. Alternatives
in the delivery mode of the checklist may borrow from elements on the higher level of the
hierarchy of effectiveness. This possible intervention could create a pseudo-forcing function for
each check, requiring users to confirm the completion of each check prior to completing the
administration sign-off. In combination with better integration of the MOE/MAR to the smart
pump and bar-code verification, a more effective intervention could be developed. Similarly,
improvements in the design of the smart pump and bar-coding system could also further improve
the performance of the intervention. Creating a closed-loop medication administration process by
fully integrating the bar-code verification, smart pump and MOE/MAR could improve the
effectiveness of the system.
Finally, while this work examined a checklist in the medication administration process, the results
of this study may encourage further investigation of checklists applied to other processes in
healthcare for a more thorough evaluation of their generalized effectiveness.
101
References
1. Kohn LT, Corrigan JM, Donaldson MS, eds. To Err is Human: Building a Safer Healthcare
System. Washington, D.C.: Institute of Medicine. National Academy Press; 1999.
2. Baker GR, Norton PG, Flintoft V, et al. The Canadian Adverse Events Study: the incidence
of adverse events among hospital patients in Canada. CMAJ. 2004;170(11):1678-1686.
3. Aspden P, Wolcott J, Bootman JL, Cronenwett LR, eds. Preventing Medication Errors.
Washington, D.C.: Institute of Medicine. National Academy Press; 2007.
4. Schneider PJ, Pedersen CA, Montanya KR, Curran CR, Harpe SE, Bohenek W, Perratto B,
Swaim TJ, Wellman Ke. Improving the safety of medication administration using an
interactive CD-ROM program. Am J Health-Syst Pharm. 2006;63:59-64.
5. Trivalle C, Cartier T, Verny C, Mathieu AM, Davrinche P, Agostini H, Becquemont L,
Demolis P. Identifying and preventing adverse drug events in elderly hospitalized patients:
a randomized trial of a program to reduce adverse drug effects. JNHA: Clinical trials and
aging. 2009;14(1):57-61.
6. Blank F, Tobin J, Macomber S, Jaouen M, Dinoia M, Visintainer P. A “back to basics”
approach to reduce ED medication errors. J Emerg Nurs. 2011;37(2):141-7.
7. Ford DG, Seybert AL, Smithburger PL, Kobulinsky LR, Samosky JT, Kane-Gill SL.
Impact of simulation-based learning on medication error rates in critically ill patients.
Intensive Care Med. 2010;36:1526-1531.
8. Mills PD, Neily J, Kinney LM, Bagain J, Weeks WB. Effective interventions and
implementation strategies to reduce adverse drug events in the Veterans Affairs (VA)
system. Qual Saf Health Care. 2008;17:37-46.
9. Relihan E, O’Brien V, O’Hara S, Silke B. The impact of a set of interventions to reduce
interruptions and distractions to nurses during medication administration. Qual Saf Health
Care. 2010;19:1-6.
10. White RE, Trbovich PL, Easty AC, Savage P, Trip K, Hyland S. Checking it twice: an
evaluation of checklists for detecting medication errors at the bedside using a
chemotherapy model. Qual Saf Health Care. 2010;19:562-567.
11. Trbovich PL, Pinkney S, Cafazzo JA, Easty AC. The impact of traditional and smart pump
infusion technology on nurse medication administration performance in a simulated
inpatient unit. Qual Saf Health Care. 2010;19:430-434.
12. Rothschild JM, Keohane CA, Cook FE, Orav JE, Burdick E, Thompson S, Hayes J, Bates
DW. A controlled trial of smart infusion pumps to improve medication safety in critically
ill patients. Crit Care Med. 2005;33(3):533-540.
13. Helmons PJ, Wargel PN, Daniels CE. Effect of bar-code-assisted medication
administration on medication administration errors and accuracy in multiple patient care
areas. Am J Health-Syst Pharm. 2009;66:1202-1210.
14. DeYoung JL, VanderKooi ME, Barletta JF. Effect of bar-code-assisted medication
administration on medication error rates in an adult medical intensive care unit. Am J
Health-Syst Pharm. 2009;66: 1110-5.
15. Morriss FH, Abramowitz PW, Nelson SP, Milavetz G, Michael SL, Gordon SN,
Pendergast JF, Cook EF. Preventable adverse drug events in a neonatal intensive care unit:
a prospective cohort study. J Pediatr. 2009;154:363-8.
102
16. Medication Error Prevention Toolbox. ISMP Medication Safety Alert. [Online] Available:
http://www.ismp.org/print.asp. 1999; 4. Accessed September 20, 2010.
17. Greenall J, Senders JW. Medication Safety Alerts: Root Cause Analysis, Learning from
Adverse Events and Neat Misses. Canadian Journal of Hospital Pharmacy. 2006; 59(1):34-
36.
18. Anderson J, Gosbee LL, Bessesen M, Williams L. Using human factors engineering to
improve the effectiveness of infection prevention and control. Crit Care Med. 2010;38(8
Suppl):S269-81.
19. Brady AM, Redmond R, Curtis E, Fleming S, Keenan P, Malone AM, Sheerin F. Adverse
events in health care: a literature review. J Nurs Manag. 2009 ;17(2):155-64.
20. Vincent C, Knox E. Clinical risk modification, quality, and patient safety:
interrelationships, problems, and future potential. Best Pract Benchmarking Healthc.
1997;2(6):221-6.
21. O'Connor T, Papanikolaou V, Keogh I. Safe surgery, the human factors approach. Surgeon.
2010 ;8(2):93-5.
22. Scanlon MC, Karsh BT. Value of human factors to medication and patient safety in the
intensive care unit. Crit Care Med. 2010;38(6 Suppl):S90-6.
23. Easty, A.C., Cafazzo, J.A., Chagpar, A. Improving Safety in Healthcare through the
Establishment of a Healthcare Human Factors Team. Diagnostic and Therapeutic
Instrumentation, Clinical Engineering p 324-7, 2009.
24. Carayon P, Wood KE. Patient safety - the role of human factors and systems engineering.
Studies in Health Technology & Informatics. 2010; 153:23-46.
25. Borycki E, Kushniruk A, Brender J. Theories, models and frameworks for diagnosing
technology-induced error. Stud Health Technol Inform. 2010;160:714-8.
26. Husch M, Sullivan C, Rooney D, Barnard C, Fotis M, Clarke J, Noskin G. Insights from
the sharp end of intravenous medication errors: implications for infusion pump technology.
Qual Saf Health Care. 2005;14:80-86.
27. Hicks RW, Becker SC. An overview of intravenous-related medication administration
errors as reported to MEDMARX, a national medication error-reporting program. Journal
of Infusion Nursing. 2006;29(1):20-27.
28. Phillips J, Beam S, Brinker A. Retrospective analysis of mortalities associated with
medication errors. Am J Health Syst Pharm. 2001;58:1835-41.
29. Westbrook J. Rob E, Woods A, Parry D. Errors in the administration of intravenous
medications in hospital and the role of correct procedures and nurse experience. BMJ Qual
Saf. 2011: 1-8.
30. Taxis K, Barber N. Causes of intravenous medication errors – observation of nurses in a
German hospital. J Public Health. 2004;12:132-138.
31. Taxis K, Barber N. Causes of intravenous medication errors: an ethnographic study. Qual
Saf Health Care. 2003;12:343-348.
32. Fahimi F, Ariapanah P, Faizi M, Shafaghi B, Namdar R, Ardakani MT. Errors in
preparation and administration of intravenous medications in the intensive care unit of a
teaching hospital: an observational study. Australian Critical Care. 2008;21:110-116.
33. Reason J. Understanding adverse events: human factors. Qual Health Care. 1995;4:80-89.
34. Spencer FC. Human error in hospitals and industrial accidents: current concepts. J Am Coll
Surg. 2000 ;191(4):410-8.
103
35. Johnson M, Young H. The application of Aronson’s taxonomy to medication errors in
nursing. J Nurs Care Qual. 2011;26(2): 128-135.
36. Ferner RE, Aronson JK. Clarification of terminology in medication errors. Drug Safety.
2006;29(11):1011-1022.
37. Dabliz R. “ISMP Error Prevention Toolbox”. [Online]. Available email: [email protected].
June 13, 2011.
38. Thomadsen B, Lin SW. Taxonomic Guidance for Remedial Actions. Advances in Patient
Safety: From Research to Implementation. 2005; Volume 2: Concepts and Methodology.
[Online] Available: http://www.ncbi.nlm.nih.gov/books/bv.fcgi?rid=aps.section.
39. Poon EG, Keohane CA, Bane A, et al. Impact of Barcode Medication Administration
Technology on How Nurses Spend Their Time Providing Care. J Nurs Adm.
2008;38(12):541-549
40. Cochran GL, Jones KJ, Brockman J, Skinner A, Hicks RW. Errors prevented by and
associated with bar-code medication administration systems. Joint Commission Journal on
Quality and Patient Safety. 2007;33(5):293-301.
41. Pape TM, Guerra DM, Muzquiz M, Bryant JB, Ingram M, Schranner B, Alcala A, Sharp J,
Bishop D, Carreno E, Welker J. Innovative approaches to reducing nurses’ distractions
during medication administration. The Journal of Continuing Education in Nursing.
2005;36(3): 108-116.
42. Conley DM, Singer SJ, Edmondson L, Berry WR, Gawande AA. Effective surgical safety
checklist implementation. J Am Coll Surg. 2011;In press: 1-7.
43. Armitage G. Human error theory: relevance to nurse management. Journal of Nursing
Management. 2009;17:193-202.
44. Schaaf TW van der. PRISMA: a risk management tool based on incident analysis.
Eindhoven University of Technology, Eindhoven, The Netherlands. 1996.
45. Hales B, Terblanche M, Fowler R, Sibbald W. Development of medical checklists for
improved quality of patient care. International Journal for Quality in Health Care.
2008;20(1): 22-30.
46. Berenholtz AM, Pronovost PJ, Lipsett PA, Hobson D, Earsing K et al. Eliminating
catheter-related bloodstream infections in the intensive care unit. Crit Care Med.
2004;32(10): 2014-2020.
47. Pronovost P, Needham D, Berenholtz A, et al. An intervention to decrease catheter-related
bloodstream infections in the ICU. New England Journal of Medicine. 2006;355:2725-32.
48. WHO. “WHO surgical safety checklist and implementation manual”. 2011. [Online].
Available: http://www.who.int/patientsafety/safesurgery/ss_checklist/en/index.html.
[Accessed: March 31, 2011].
49. Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat A-S, Dellinger EP, et al. A
surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J
Med. 2009;360(5):491-9.
50. Degani A, Wiener EL. Cockpit checklists: concepts, design and use. Human Factors.
1993;35(2):28-43.
51. Winters BD, Gurses AP, Lehmann H, Sexton JB, Rapersad CJ, Pronovost PJ. Clinical
review: checklists – translating evidence into practice. Critical Care. 2009;13(6):210.
52. Verdaasdonk EG, Stassen PL, Widhiasmara PP, Dankelman J. Requirements for the design
and implementation of checklists for surgical processes. Surg Endosc. 2009;23:715-726.
104
53. Elias BL, Moss JA. Smart pump technology. Computers, Informatics, Nursing.
2011;29(3):184-190.
54. McAlearney AS, Vrontos J, Schneider PJ, Curran CR, Czerwinski BS, Pedersen CA.
Strateic work-arounds to accommodate new technology: the case of smart pumps in
hospital care. J Patient Safety. 2007;3:75-81.
55. Sakowski J, Newman JM, Dozier K. Severity of medication administration errors detected
by a bar-code medication administration system. Am J Health-Syst Pharm;65:1661-1666.
56. Patterson ES. Cook RI, Render ML. Improving patient safety by identifying side effects
from introducing bar-coding in medication administration. J Am Med Inform Assoc.
2002;9:540-553.
57. Snyder ML, Carter A, Jenkins K, Fantz CR. Patient misidentifications caused by errors in
standard bar-code technology. Clinical chemistry. 2010;56(10):1554-1560.
58. Miller DF, Fortier CR, Garrison KL. Bar-code medication administration technology:
characterization of high-alert medication triggers and clinician workaround. Ann
Pharmacother. 2011;45:162-8.
59. Prusch AE, Suess TM, Paoletti RD, Olin ST, Watts SD. Integrating technology to improve
medication administration. Am J Health-Syst Pharm. 2011;68:835-42.
60. Prakash, V. Interventions to Mitigate the Effects of Interruptions During High-Risk
Medication Administration. Master’s Thesis. IBBME, University of Toronto. 2010.
61. Zhang J, Johnson TR, Patel VL, Paige DL, Kubose T. Using usability heuristics to
evaluated patient safety of medical devices. Journal of Biomedical Information.
2003;36(1):23-30.
62. Nielsen J. Usability Engineering. San Diego: Academic Press; 1993.
105
Appendix A: UHN Form for Independent Double Check of Ambulatory Infusion Pumps
106
Appendix B: Research Ethics Board Approval
1.
107
2.
108
Appendix C: Consent Forms
The Effectiveness of Checklists vs. Bar-Codes toward Detecting Medication Planning and Execution Errors
Part 1: Medication Administration Safety Checklist Usability Study
Investigators Dr. Joseph Cafazzo
Dr. Patricia Trbovich
You are being asked to take part in a research study. Before agreeing to participate in this study, it is important that you read and understand the following explanation of the proposed study procedures. The following information describes the purpose, procedures, benefits, discomforts, risks and precautions associated with this study. It also describes your right to refuse to participate or withdraw from the study at any time. In order to decide whether you wish to participate in this research study, you should understand enough about its risks and benefits to be able to make an informed decision. This is known as the informed consent process. Please ask the study staff to explain any words you don’t understand before signing this consent form. Make sure all your questions have been answered to your satisfaction before signing this document.
Background and Purpose
You have been asked to take part in this research study because researchers at the University Health Network (UHN) with the Healthcare Human Factors Group are investigating strategies to reduce errors in IV medication administration. The researchers are interested in refining the design of a medication administration safety checklist to improve patient safety. Your participation helps us ensure the checklist developed is acceptable to nurses with optimized effectiveness. The feedback and usability evaluated will be used to make improvements to the design and use of the checklist to ensure it is easy to use and fits into the workflow. Approximately 5 nurses will be involved in the study.
Procedures
If you agree to participate in the study, you will be asked to complete a series of clinical tasks in a simulated environment. You will be in a laboratory facility with clinical equipment and scenarios but no real patients or patient care. You will be asked to perform several medication administration tasks to a simulated patient mannequin. You may be asked to provide feedback about proposed design solutions. The session will last no more than one hour and will be videotaped for later analysis.
Background questionnaire: You will be asked to fill out a background questionnaire to collect demographic information and years of experience working as a nurse.
Training: You will be trained on the using the checklist to reduce errors in medication administration. This training will consist of a demonstration (by the study facilitator) on how to carry out the task with the checklist followed by an opportunity to practice. You will be instructed to use the checklist during the study.
Usability Testing: When the trial starts, there will be mannequins as mock patients in the unit. Also, there will be a confederate nurse played by an actor. The confederate nurse will be informed of the study proceeding and will be present in the lab supporting you throughout the simulation. The confederate nurse will not be observed for the purposes of the study. This nurse will (1) update you on the patients who are in the unit at the time and (2) ask you to perform some medication administration tasks.
Questionnaire and Debrief:
109
After completing the usability testing, you will have a semi-structured interview with the test facilitator. Questions regarding your experience, the accuracy of the simulation environment, and ease of task completion may be raised in the interview. You will then be asked to complete a questionnaire about your perceptions of the checklist and your task performance. Following the questionnaire an informal debriefing may occur in order to clarify your comments or address concerns or questions you may have about the study.
Risks
There are no medical risks if you take part in this study, but being in this study may make you feel uncomfortable. You may refuse to answer any question or ask to stop the observations at any time if there is any discomfort.
Please remember that only work environmental factors are being evaluated and not you. However, if you feel anxious and/or uncomfortable, please bring your concerns to the researcher’s attention immediately. Your participation will have no impact on your employment.
Benefits
You may or may not receive direct benefit from being in this study. Information learned from this study may help us to understand how to reduce errors and improve the design of equipment and processes in hospitals across Canada to better support your work practices.
Confidentiality
All information obtained during the study will be held in strict confidence. You will be identified with a subject number only. No names or identifying information will be used in any publication or presentations. No information identifying you will be transferred outside the investigators in this study. If the videos from the research are shown outside the research team, your face will be blurred and all identifying information made anonymous.
The University Health Network Research Ethics Board (a group of people who oversee the ethical conduct of research studies) may look at the study records for auditing purposes.
The information will be kept for a minimum of two years, a maximum of seven years, after the completion of the study and then destroyed by shredding of paper or erasing of digital information. Any personal identifiable information is stored and protected on servers with adequate security measures. All information that allows correlation between unique identifiers with participant’s personal data and all data collected during the study will be kept secure in a locked filing cabinet.
Participation
Your participation in this study is voluntary. You can choose not to participate or you may withdraw at any time without reason. Whether you choose to participate or not has no impact on your employment. You may
also refuse to answer any question you do not want to answer.
Reimbursement
You will be compensated for your time during this study at one and one-half times the current hourly rate for senior nurses.
Questions About the Study
If you have any question about this study please contact the Principal investigator Dr. Joseph Cafazzo at 416-340-3634.
110
If you have any questions about your rights as a research participant, please call the Chair of the University Health Network Research Ethics Board at (416) 581-7849. They are not involved with the research project in any way and calling them will not affect your participation in the study.
Consent
I have had the opportunity to discuss this study and my questions have been answered to my satisfaction. I consent to take part in the study with the understanding I may withdraw at any time. I have received a signed copy of this consent form. I voluntarily participate in this study.
Use of Videos
I allow video clips in which my face is obscured (i.e., blurred) to be shared for education and presentation purposes: Yes, video clips may be shared No, video clips may NOT be shared
Participant’s Name (Please Print) Participant’s Signature Date
(You will be given a signed copy of this consent form)
I confirm that I have explained the nature and purpose of the study to the participant named above. I have answered all questions.
Name of person obtaining consent Signature Date
111
The Effectiveness of Checklists vs. Bar-Codes towards Detecting Medication Planning and Execution Errors
Part 2: Medication Administration Safety Study
Investigators Dr. Joseph Cafazzo Dr. Patricia Trbovich You are being asked to take part in a research study. Before agreeing to participate in this study, it is important that you read and understand the following explanation of the proposed study procedures. The following information describes the purpose, procedures, benefits, discomforts, risks and precautions associated with this study. It also describes your right to refuse to participate or withdraw from the study at any time. In order to decide whether you wish to participate in this research study, you should understand enough about its risks and benefits to be able to make an informed decision. This is known as the informed consent process. Please ask the study staff to explain any words you don’t understand before signing this consent form. Make sure all your questions have been answered to your satisfaction before signing this document. Background and Purpose You have been asked to take part in this research study because researchers at the University Health Network (UHN) with the Healthcare Human Factors Group are investigating strategies to improve IV medication administration. The researchers are interested in determining the most effective interventions to improve patient safety. Your participation helps us ensure the strategies developed are acceptable to nurses with optimized effectiveness. In addition to a potential impact on the equipment and policies surrounding medication administration at the UHN, the results will be shared with the international health care community with the aim of improving patient safety. Approximately 48 nurses will be involved in the study. Procedures If you agree to participate in the study, you will be asked to complete a series of clinical tasks in a simulated environment. You will be in a laboratory facility with clinical equipment and scenarios but no real patients or patient care. You will be asked to perform several medication administration tasks to a simulated patient mannequin. You may be asked to provide feedback about proposed design solutions. The session will last approximately three hours and will be videotaped for later analysis. Background questionnaire: You will be asked to fill out a background questionnaire to collect demographic information and years of experience working as a nurse. Training: You will be trained on the strategy to improve medication administration. This training will consist of a demonstration (by the study facilitator) on how to carry out the task with the strategy followed by an opportunity to practice. You will be instructed to use the tool you have been trained on during the experimental trial. Experimental Trial: When the trial starts, there will be mannequins to simulate patients in the unit. Also, there will be a confederate nurse played by either a nurse or an actor playing the role of a nurse. The confederate nurse will be informed of the study proceeding and will be present in the lab supporting you throughout the simulation. The confederate nurse will not be observed for the purposes of the study. This nurse will (1) update you on the patients who are in the unit at the time and (2) ask you to perform some medication administration tasks. Questionnaire and Debrief: After completing the usability testing, you will have a semi-structured interview with the test facilitator. Questions regarding your experience, the accuracy of the simulation environment, and ease of task completion may be raised in the interview. You will then be asked to complete a questionnaire about your perceptions of your task performance. Following the questionnaire an informal debriefing may occur in order to clarify your comments or address concerns or questions you may have about the study.
112
Risks There are no medical risks if you take part in this study, but being in this study may make you feel uncomfortable. You may refuse to answer any question or ask to stop the observations at any time if there is any discomfort. Please remember that only work environmental factors are being evaluated and not you. However, if you feel anxious and/or uncomfortable, please bring your concerns to the researcher’s attention immediately. Your participation will have no impact on your employment. Benefits You may or may not receive direct benefit from being in this study. Information learned from this study may help us to understand how to reduce errors and improve the design of equipment and processes in hospitals across Canada to better support your work practices. Confidentiality All information obtained during the study will be held in strict confidence. You will be identified with a subject number only. No names or identifying information will be used in any publication or presentations. No information identifying you will be transferred outside the investigators in this study. If the videos from the research are shown outside the research team, your face will be blurred and all identifying information made anonymous. The University Health Network Research Ethics Board (a group of people who oversee the ethical conduct of research studies) may look at the study records for auditing purposes. The information will be kept for a minimum of two years, a maximum of seven years, after the completion of the study and then destroyed by shredding of paper or erasing of digital information. Any personal identifiable information is stored and protected on servers with adequate security measures. All information that allows correlation between unique identifiers with participant’s personal data and all data collected during the study will be kept secure in a locked filing cabinet. Participation Your participation in this study is voluntary. You can choose not to participate or you may withdraw at any time without reason. Whether you choose to participate or not has no impact on your employment. You may also refuse to answer any question you do not want to answer. Reimbursement You will be compensated for your time during this study at one and one-half times the current hourly rate for senior nurses. Questions About the Study If you have any question about this study please contact the Principal investigator Dr. Joseph Cafazzo at 416-340-3634. If you have any questions about your rights as a research participant, please call the Chair of the University Health Network Research Ethics Board at (416) 581-7849. They are not involved with the research project in any way and calling them will not affect your participation in the study. Consent I have had the opportunity to discuss this study and my questions have been answered to my satisfaction. I consent to take part in the study with the understanding I may withdraw at any time. I have received a signed copy of this consent form. I voluntarily participate in this study. Use of Videos I allow video clips in which my face is obscured (i.e., blurred) to be shared for education and presentation purposes:
113
Yes, video clips may be shared No, video clips may NOT be shared
Participant’s Name (Please Print) Participant’s Signature Date (You will be given a signed copy of this consent form) I confirm that I have explained the nature and purpose of the study to the participant named above. I have answered all questions.
Name of person obtaining consent Signature Date
114
Appendix D: Experimental Protocol and Script
General Roles of Team
Study Coordinator (Data Collector) Meet and greet participant Introduce participant to the study and lab Obtain consent Provide Questionnaire 1 - Demographics Conduct training (Checklist or smart pump) Manipulate cameras and start/stop recordings Start soundtrack for ambient hospital environment Provide guidance to confederate nurse through walkie-talkie if necessary Observe experimental sessions and record on data collection log Debrief participants following each part of the study session Provide Questionnaire 2
Confederate Nurse Guide participants through scenarios, providing instructions for each task Relay issues or questions to study coordinator via walkie-talkie Set up planted errors, change patient beds, take down drugs and reprogram pumps as scripted for scenario transitions Interrupt participants as indicated in the script Act as liaison to pharmacy and physician when required Set-up of labs between protocol
1 Protocol Setup
1.1 Items to give participant before coming to the center Directions to the lab and contact information in case of any difficulties Time and date of scheduled study session Reminder that the experiment will last approximately 3 hrs Reminder that they will not be paid on the spot but rather though cheque Brief explanation of study and participant requirements Copy of consent form to review prior to participating Come to the session in clothes that are most comfortable (work attire or street clothes)
1.2 Scenario Setup MOE/MAR Set-up
*Bring up mock MOE/MAR for either Part A or Part B, depending which part is to be done first:
Part A: Part B:
Patient Drug Orders Patient Drug Orders
1. Mrs. Katherine Tuer 1. Haloperidol 2. D5W 3. N-Acetylcysteine
1. Mr. Peter Wilson a. D5W b. Phenytoin c. Piperacillin-tazobactam
2. Mrs. Penelope Sillian a. Ringers Lactate b. Ceftriaxone c. Octreotide
2. Mr. Steven Campbell a. Ringers Lactate b. Diphenhydramine c. Vancomycin d. Pantoprazole
3. Mr. Kevin Piu a. Normal Saline 3. Mr. Xian Chan a. Normal Saline
115
b. Morphine c. Potassium Phosphate
b. Hydromorphone c. Dimenhydrinate
4. Mr. Albert Scalp a. Amphotericin B Liposomal
b. Normal Saline c. Magnesium Sulphate d. Dimenhydrinate
4. Mrs. Agnes Taylor a. D5W b. Ticarcillin c. Sodium Bicarbonate
General Ward Set-up
Laptop charged with mock MOE/MAR opened Paper, pen, pencil, calculator on MOE/MAR cart Formulary binder on front table Clipboard with checklists on MOE/MAR cart (If checklist part) Infusion pumps by each bedside (either Grasby or Alaris depending which part) Dim lights to the second highest level Bring out med bins for the first scenario and put on cart Change IV collection bags Laptop and extension cord (plug in laptop) Make sure tubing is primed and IV bags are full Garbage can, alcohol wipes and hand sanitizer at each bed
Nurse Participant
Wireless microphone Lab coat
Nurse Actor
Wireless microphone Walkie Talkie Scrubs and ID badge
Training Station Set-up
Table Chair for participant Printed consent forms Cheque requisitions on table Payment receipts printed Laptop with Survey Monkey questionnaires Have paper questionnaires in case internet goes down Either Checklist or Alaris Pump and iPhone (barcode reader) for training depending on group Primary and secondary lines primed with fluid bags attached for training Drinks/snacks
Control Room Set-up
Easel pad (write participant name, ID and pump rotation for each experiment) Laptop for data collection Synch time on laptops with camera recording time Put do not disturb signs on lab and control room doors Walkie Talkie
116
1.3 Protocol A: Patient Setup
1.3.1 Mrs. Katherine Tuer (bed 1)
Bed 1
Pump 1 Pump 2
Inappropriate dose: hard
limit
Wrong Dose: label
mismatch
Patient Name: Mrs. Katherine Tuer
Patient Set-up:
Wig Access points: 2 separate peripheral IV lines Wristband
Equipment Set-up:
2 single channel Grasby pumps (or dual channel Alaris if smart pump group) Adjust pole and pump height for participant Piggyback hook Bin of IV medications
o D5W: maintenance 1000mL bag attached to primary line Infuse at 100mL/h Ensure bag full and line primed
o Haloperidol: piggyback (inappropriate dose on order – hard limit) 100mL bag attached to secondary line 50mg/100mL, infuse over 30 min (inappropriate dose) Ensure bag full and line primed
o N-acetylcysteine (wrong dose – mismatch) 1,000mL bag attached to primary line 0.7g/1,000mL, infuse at 62.5mL/h (wrong dose – mismatch) Ensure bag full and line primed
In med room:
o Haloperidol (inappropriate dose on order – hard limit) 100mL bag attached to secondary line 5mg/100mL, infuse over 30 min (correct dose ) Ensure bag full and line primed Have paper physician order for correct dose
o N-acetylcysteine (correct dose) 1,000mL bag attached to primary line 7g/1,000mL, infuse at 62.5mL/h Ensure bag full and line primed
1.3.2 Mrs. Penelope Sillian (bed 2)
Bed 2
Pump 1 Pump 2
Inappropriate dose rate: hard
limit
Maintenance and piggyback (incompatible)
117
Patient Name: Mrs. Penelope Sillian
Patient set-up:
Wig Access points: 2 separate peripheral IV lines Wristband
Equipment Set-up:
2 single channel Grasby pumps (or dual channel Alaris if smart pump group) Adjust pole and pump height for participant Piggyback hook Bin of IV medications
o Ringers Lactate: maintenance 1,000mL bag attached to primary line Infuse at 100mL/h Ensure bag full and line primed
o Ceftriaxone: piggyback (incompatible with primary fluid) 50mL bag attached to secondary line 1g/50mL, infuse over 15 min Ensure bag full and line primed
o Octreotide (Inappropriate dose rate – hard limit) 250mL bag attached to primary line 500mcg/250ml, infuse at 500mcg/hr Ensure bag full and line primed Have paper physician order available with corrected dose rate
1.3.3 Kevin Piu (bed 3)
Bed 3
Pump 1 Pump 2
Maintenance and piggyback
(wrong patient)
Inappropriate duration – soft limit
Patient Name: Mr. Kevin Piu
Patient set-up:
Access points: 2 separate peripheral IV lines Wristband for Mr. Kevin Siu on mannequin (wrong patient) Wristband for Mr. Kevin Piu under bed covers
Equipment Set-up:
2 single channel Grasby pumps (or dual channel Alaris if smart pump group) Adjust pole and pump height for participant Piggyback hook Bin with IV medications
o Normal Saline: maintenance 1,000mL bag with primary line Infuse at 100mL/h Ensure bag full and line primed
o Morphine: piggyback 50mL bag with secondary line 2mg/50mL, infuse over 10 min Ensure bag full and line primed
o Potassium Phosphate (inappropriate duration) 415mL (500mL bag) with primary line 15mmol/415mL, infuse over 1 hr Ensure bag is full and line primed
118
Have paper physician order available with corrected duration
1.3.4 Albert Scalp (bed 4)
Bed 4
Pump 1 Pump 2
No error Wrong drug:
mismatch
Patient Name: Mr. Albert Scalp
Patient set-up:
Access points: 2 separate peripheral IV lines Wristband
Equipment Set-up:
2 single channel Grasby pumps (or dual channel Alaris if smart pump group) Adjust pole and pump height for participant Piggyback hook Bin with IV medications
o Normal Saline: maintenance 1000mL NS bag with secondary line Infuse at 100mL/h Ensure bag full and line primed
o Magnesium Sulphate: piggyback 100mL bag with secondary line 1g/100mL, infuse over 1h Ensure bag full and line primed
o Dimenhydrinate: piggyback 50mL bag with secondary line 25mg/50mL, infuse over 15 min Ensure bag full and line primed
o Amphotericin B (Wrong drug) 250mL bag with primary line 425mg/250mL, infuse over 3hrs Ensure bag full and line primed
In med room:
o Amphotericin B Liposomal (correct drug) 250mL bag with primary line 425mg/250mL, infuse over 3hrs Ensure bag full and line primed
1.4 Protocol B: Patient Setup
1.4.1 Peter Wilson (bed 1)
Bed 1
Pump 1 Pump 2
Maintenance and piggyback
(incompatibility) No error
Patient Name: Mr. Peter Wilson
Patient Set-up:
119
Access points: 2 separate peripheral IV lines Wristband
Equipment Set-up:
2 single channel Grasby pumps (or dual channel Alaris if smart pump group) Adjust pole and pump height for participant Piggyback hook Bin with IV medications
o D5W: maintenance 1,000mL bag with primary line Infuse at 100mL/h Ensure bag full and line primed
o Phenytoin: piggyback (incompatible with primary fluid) 100mL NS bag with secondary line 300mg/100mL, infuse over 60 min Ensure bag full and line primed
o Piperacillin-tazobactam: piggyback 50mL bag with secondary line 4.5g/50mL, infuse over 30 min Ensure bag full and line primed
1.4.2 Steven Campbell (bed 2)
Bed 2
Pump 1 Pump 2
Inappropriate duration – soft
limit
Wrong dose label
mismatch
Patient Name: Mr. Steven Campbell
Patient set-up:
Access points: 2 separate peripheral IV lines Wristband
Equipment Set-up:
2 single channel Grasby pumps (or dual channel Alaris if smart pump group) Adjust pole and pump height for participant Piggyback hook Bin with IV medications
o Ringers Lactate: maintenance 1000mL bag with primary line Infuse at 50mL/h Ensure bag full and line primed
o Vancomycin (Inappropriate duration – soft limit) 250mL bag with secondary line 1g/250mL, infuse over 30min Ensure bag full and line primed Have paper physician order available with corrected duration
o Diphenhydramine: piggyback (no error) 50mL bag with secondary line 25mg/50mL, infuse over 15min Ensure bag full and line primed
o Pantoprazole (wrong dose mismatch) 100mL bag with primary line 4.0mg/100mL, infuse at 20mL/h (wrong dose) Ensure bag full and line primed
120
In med room
o Pantoprazole (correct dose) 100mL bag with primary tubing 40mg/100mL, infuse at 20mL/h Ensure bag full and line primed
1.4.3 Xian Chan (bed 3)
Bed 3
Pump 1 Pump 2
Maintenance and piggyback
(wrong patient)
Inappropriate dose – hard
limit
Patient Name: Mr. Xian Chan
Patient set-up:
Access points: 2 separate peripheral IV lines Wristband for Mr. Xiao Chan (wrong patient) Wristband for Mr. Xian Chan under bed covers
Equipment Set-up:
2 single channel Grasby pumps (or dual channel Alaris if smart pump group) Adjust pole and pump height for participant Piggyback hook Bin with IV medications
o Normal Saline: maintenance 1,000mL bag with primary line Infuse at 100mL/h Ensure bag full and line primed
o Dimenhydrinate: piggyback 50mL bag with secondary line 25mg/50mL, infuse over 15 min Ensure bag full and line primed
o Hydromorphone: piggyback (inappropriate dose – hard limit) 50mL bag with secondary line 10mg/50mL, infuse over 10min (wrong dose) Ensure bag full and line primed
In med room:
o Hydromorphone (inappropriate dose – hard limit) 50mL bag with secondary line 1mg/50mL, infuse over 10 min Ensure bag full and line primed Have paper physician order for correct dose
1.4.4 Agnes Taylor (bed 4)
Bed 4
Pump 1 Pump 2
Wrong drug mismatch
Inappropriate dose rate – hard limit
Patient Name: Mrs. Agnes Taylor
Patient set-up:
121
Wig Access points: 2 separate peripheral IV lines Wristband
Equipment Set-up:
2 single channel Grasby pumps (or dual channel Alaris if smart pump group) Adjust pole and pump height for participant Piggyback hook Bin with IV medications
o D5W: Maintenance 1000mL with primary line Infuse at 100mL/h Ensure bag full and line primed
o Ticarcillin: piggyback (wrong drug) 50mL with secondary line 3g/50mL infuse over 30min Ensure bag full and line primed
o Sodium Bicarbonate (inappropriate dose rate) 1000mL bag with primary line 200mmol/1000mL infuse at 300ml/h Ensure bag full and line primed Have paper physician order available with correct dose rate
In med room:
o Ticarcillin-Clavulanate (correct drug) 50mL bag with secondary line 3g/50mL infuse over 30min Ensure bag full and line primed
1.5 Meet and greet participant
Location: Main office area and innovation lab hallway
Equipment needed: Sign on clipboard at front reception to (a) ensure the participant that they are in the correct location for the study and (b) extension to contact study coordinator when they reach the center.
Estimate time: 5min
[Person 1] “Hi ______________________ , welcome to the centre! My name is Emily Rose and I’m the study coordinator for the
IV medication administration study. How are you? [Small talk … i.e. did you have any problems finding the center etc?] We are
really happy that you’re able to participate in this study, it’s going to be great getting your insight. Have you ever been to center
before? No? All right, let me give you a quick summary of what goes on here.”
Lead participant into innovation lab hallway toward lab [While walking to lab give quick summary of the center]
[Person 1] “So, [Name of Participant] this center was created to improve healthcare for people through safe, usable and effective
technologies and processes. Here at the Healthcare Human Factors Group, we conduct usability testing on medical devices. We
often require the help of nurses because nurses are frequently the main users of these devices. By understanding user needs and
limitations we can hopefully improve clinical workflow and patient safety.”
1.6 Introduction to the study and the lab Location: Scenario section of lab
122
Equipment needed: none
Estimate time:
[Person 1] “This is the lab that you will be working in this morning/afternoon. The study you will be helping us with today is
looking at the use of different tools to improve the efficiency and safety of IV medication administration. You will be introduced
and trained on how to use the checklist (or bar-code smart pump system) during medication administration. You will be then
asked to complete a series of IV infusion tasks using the checklist (or bar-code smart pump system) and after we’ll get you to
complete short questionnaires about your thoughts.
We would like you to pretend that you are a float nurse and that this is your first time on this unit. Nurse Lata, another nurse on
this unit, will help you orient yourself to the unit. Specifically, she will guide you through the scenarios, discuss the patient
histories with you and remain in the room at all times in case you have any questions.
The scenarios are meant to be as realistic as possible, and you should try to practice your nursing as you regularly would (for
instance, ensuring the five-rights of medication administration). But there are a few areas where things will not be entirely
realistic:
This (rolling cart) is your computer on wheels and this is the computer you will be using to pull-up the orders on the MOE/MAR (explain that they will be in the MAR view on a mock interface). We recognize that you would normally get the drugs in the drug preparation room, but for purposes of this experiment, Nurse Lata will place the drugs on the table in bins for each patient.
There will be mannequins playing the role of patients. They all have Peripheral IVs in each arm. Please always put each IV in a different access point…. and don’t attach the iv tubing too tightly as we will need to disconnect it after the study!
Lata will be playing the role of the charge nurse. (She will guide the scenarios but she doesn’t have any more training on the infusion pumps or (checklist) than you.)
There will be no real drugs used and all drugs will already by reconstituted. IV lines will be pre-primed in the interest of time and already attached to the bag. However, please just check air hasn’t
been introduced into set so we can minimise air inline alarms as this isn’t a priority for this study. You are not responsible for any documentation (aside from completing the checklists). Some infusions that are normally a piggyback may be put on a primary line. In the interest of time, we will not actually be running infusions over the true length of time that they would run.
However, you will program the infusion as if it will run over the appropriate length of time. Please try not to move the pump as the camera angles have been pre-programmed. You are not required to sign off medication administration in the MOE/MAR
While you are administering IV fluids to the patients I will be observing you from behind a one-way mirror (show participant
window). This will prevent me from distracting you when you are completing the tasks. I want to keep this session as close to your
normal working environment as possible. This session will be video and audio taped just in case I miss anything after the session is
completed. The videotapes and audiotapes are strictly confidential. No names or identifying information will be used in any
reports, publication or presentations that may come from this study. If you consent to share your video clips, your face will be
blurred.
I want to stress that the purpose of this study is not to assess your skills and will in no way affect your position at UHN. If you
encounter any problems while using the checklist or the pump, it is not a reflection on your skills but rather an indication to me
that this technology or process requires improvements.
During the scenarios, you may find that when you ask questions to the confederate nurse, she may throw another question back
at you. This is not to be patronising but rather to see how you would problem solve and what you would actually do on the floor.
123
Do you have any questions?”
1.7 Consent Form Completion and Background Questionnaire
Lead participant into training space (area that is not in the testing space)
Location: Training space
Equipment needed: Laptop, table for computer, chair for participant, consent form, paper questionnaires, questionnaire on survey monkey
Estimate time: 10 min
[Study Coordinator] “All right, the next thing I would like to you do is review and complete the consent form. This is to make sure
that you understand what is going on in the study and your rights as a participant. If you have any questions or need clarification
just let me know.
Participant reads consent form and signs it
[Study Coordinator] “Great, next we have a short background questionnaire that we ask you to complete.”
Participant completes background questionnaire
1.8 TRAINING SCRIPTS
Location: Training space Equipment needed: Patient, Alaris infusion pump and iPhone (barcode reader), IV bags and sets, or checklist Estimated time: 15 min
1.8.1 Checklist Training
1. Introduction to the checklist a. This is the IV medication administration safety checklist. b. We will be using this checklist in paper form with a Grasby 3000 large volume infusion pump. c. This checklist is based on the independent double checking policy for high-alert IV medications and was
designed to ensure the rights of medication administration and safe practices. d. The checklist is meant to be used at the bedside mirroring the steps of the medication administration process.
Each task on the list is to be checked off as it is completed. e. The checklist is organized into 4 sections to correspond with the steps in the medication administration
process. from the point at which the medication has been picked up for the patient and it compared to the order.
[Person 1] “I’ll give you a chance to review the checklist and become familiar with the items and layout. Then I will describe the
intended workflow and answer any questions.”
1. “After the medication is picked up from the med station, it is compared against the order to ensure that it matches. The first section of the checklist outlines the items to verify. A checkmark is placed after it has been confirmed that each item on the medication label matches that on the physician’s order”
2. “After comparing the drug label and physician’s order, the checklist instructs the user to pause for a moment to consider the details of the drug order. This step is to give the nurse a chance to consider the clinical appropriateness of the order. Even if the medication label and physician’s order match, there may be other possible problems with the order such as an allergy to the drug, interaction with another drug, or prescription dosing/rate error, for example.”
3. “Next, we bring the medication to the bedside. The checklist requires an armband check against the medication label followed by programming of the pump.”
124
4. “The final section of the checklist requires the user to do an “independent” check of the pump parameters. Values from the pump screen are copied onto the checklist. The written values on the checklist are then compared one final time to the medication label before the pump is started.”
1.8.2 Cardinal Alaris Pump and Bar-code Verification Training
1. Introduction to bar-code verification a. We will be using a bar-code verification system to check our medications before administering them. For this
study, we will be using an iPhone to scan the bar-codes on the drug labels and patient wristbands. The system will let you know if there is any discrepancy between the drug label and the electronic physicians’ order. It will also do an automatic patient ID check at the bedside.
b. I will first go through the workflow for this process and then give you a chance to try. To start off, you will again open the MOE/MAR for the patient you are about to attend to. Once you have gotten the medications from pharmacy, you need to scan each of the 2D bar-codes on the drug bag labels. The phone will give you the patient’s name, MRN and the name of the drug and dose. If any of this information does not match the current order, an error message will be displayed.
c. After verifying the medication with the order, you can then proceed to the bedside and scan the barcode on the patient wristband. Again, the iPhone will display the name of the patient, MRN, and DOB. If any of this information does not match the drug label, an error message will again appear.
d. Once this verification has been completed, the barcode scanning portion is done and you are now ready to move on to programming the pump.
2. Introduction to pump layout a. This pump is called the Alaris by Cardinal Health (Carefusion). It is a smart pump as it has a drug library with
dose error reduction software designed and managed by collaboration between pharmacy, nursing, biomed etc., to determine what drugs, concentrations and infusion settings are within appropriate for each clinical service within the hospital.
b. This drug library incorporated built-in upper and lower limits on infusion rates or the duration of infusion. c. On this pump, these limits are called ‘guardrails’ and are intended to keep you from programming an infusion
too far on either side of safe clinical limits. d. There are two types of limits or guardrails: soft limits and hard limits. e. A soft limit will provide a warning and recommend you to stop and reprogram but it will still allow you to
continue your infusion if you so choose. f. A hard limit however will stop you and force you to go back and reprogram the pump. There is no option to
continue with the infusion until the parameters have been changed to within an acceptable limit. g. The pump consists of a main ‘brain’ unit in the centre with infusion channels on either side. Each channel is
independent, almost as it you had two Grasby pumps. h. On the central ‘brain’, you’ll see these arrows around the screen which are called soft keys, because what they
do changes depending on what they are connected to on the screen just like an ATM. i. These lower buttons with words and numbers are called hard keys, because they always do the same thing. A
seven always means seven. j. Among the hard keys, you won’t really use the ‘enter’ key (as you’ll see when we program later), the cancel
key will usually back you out of whatever you’re doing, and the clear key will clear the number you are working on; if you hit a 2 instead of a 1, you can hit clear to start over.
k. There is also a yellow silence button to silence any alarms. l. The options key we won’t use much today, just know this is where it is.
3. Loading a set a. Let me start by first showing you how to load a set. b. Open the latch (the lever lifts similar to the Grasby pump) c. Drop from the top, blue to blue and then floss the blue knuckles at the bottom. d. The reason we do it like this is because this middle section shouldn’t be stretched or twisted because it can
cause inaccurate fluid delivery. So its better when you just let it drop from the top and hang naturally. e. Also, when you’re pushing the blue piece in, don’t push the white piece in because that’s your free flow
protection i. – it should stick out.
f. Finally, it’s important that you floss the knuckles at the end because these blue pieces detect air-in line, so if it’s not loaded properly, you will get nuisance alarms.
4. Programming a continuous infusion a. Now that we have loaded the pump, I will show you how to set up first a primary, then a secondary infusion.
125
b. We will first need to connect the tubing to the patient’s access point and turn on the pump. c. Once the pump is turned on, the first thing it will do is ask you if you have a new patient. We can pretend we
do and say ‘yes’ to that. d. Next it will ask you for the correct clinical profile.
i. As mentioned, the drug formulary is uniquely configured for each unit as different units have different medication practices and allowances.
ii. Therefore, you have to select the correct unit – today, we are working in a general medicine unit so we will select that profile.
e. Now we have to select the channel we wish to program. The set is loaded in channel A so we will select Channel A. This must be selected on the channel itself using the hard button, rather than the soft button on the main brain. (This is to ensure that you don’t accidentally select the wrong channel)
f. This blue bar at the bottom is your GPS. It will always tell you what to do so make sure you read it to and the pump will guide you along.
5. Programming a maintenance fluid a. Let’s first set up the primary NS fluid line and then program a secondary drug infusion. b. To program the fluids, we have to go into the guardrails fluid library, using the soft key on the side to select it. c. Either page down to normal saline, or use the letters on the side to jump ahead to the n’s. d. So now, just read the screen and then follow the GPS’ directions. e. Anything with a box around it means that you can select it. f. Roller clamp off! g. Press start.
6. Programming a secondary infusion a. Good, now let’s do a piggyback infusion of dimenhydrinate. b. Since we want to program, what do we do? Channel select! c. Since the primary is already running, it takes us here. d. Look at the bottom to see the option to program a secondary infusion. e. This button automatically brings us into the guardrails drug library (you must select ‘Basic Secondary’ to
program an infusion without using the drug library). Find dimenhydrinate. f. Enter the order… g. Roller clamp off! h. Go ahead and start it.
7. Hitting a hard limit a. Now I will go through what happens when you hit both types of limits. I will start with an example of a hard
limit. I’m just going to pick a drug I’m familiar with. If I program the infusion rate to ____ and try to start it, I get this message:
b. What happened? We hit a hard limit. c. You know it’s a hard limit because the pump will not let you proceed. The only option available is to go back
and reprogram the infusion. d. So let’s go back and change our rate to something more reasonable. e. The values are… f. Roller clamp off! g. Press start!
8. Hitting a soft limit a. Now, let’s try to find an example of a soft limit. If I select this drug and try to program it to run at___, we get
this: b. So here we’ve hit a soft limit. c. As you can see, it is different from the hard limit in that you can either proceed past the limit, or go back and
reprogram. d. What you do will depend on your hospital’s policy. e. Let’s press no, and change the duration to 30 minutes. f. Press start.
9. Clinical advisories a. One other feature of the drug library is the inclusion of clinical advisories. Pharmacy can include warnings
unique for a particular drug (similar to the comments section in the IV formulary) to remind nurses of critical aspects to consider when administering a particular drug. These advisories are simply a message that pops up before programming the infusion parameters but does not prevent you from proceeding. They may include information about compatibilities or maybe a particular side effect of a drug.
126
b. That covers the basics of the pump and barcode reader. Do you have any questions? I will give you a chance to practice once before starting the scenarios.
2 Experimental Protocols
2.1 Protocol A
NOTE: Nurse Actor is in charge of all questions from the participant. If the participant gets stuck, the nurse actor will tell the
participant how to get the next programming step.
2.1.1 Introduction to ward
Data collection: synch time on laptop with camera recording time
If first protocol of experiment:
[Nurse Actor] “Hi ________________, nice to meet you! You must be my float nurse! My name is Lata and I’m another
nurse on this ward. It is so good to see you because we are way behind in med delivery to our patients. Some patients
have just come up to the ward and we need to finish their orders. You haven’t worked on this unit before, right? There
are a couple of things I would like to cover before I introduce you to the patients that you will be taking care of. First,
here is our MOE/MAR. You are responsible for administering the IV medications, but I will take care of any
documentation. The Standard and Restricted IV drug formulary is also available if you need to look up any unfamiliar
drugs.
“The hospital has just implemented a new checking process for all IV medications. This checklist is to be completed
when administering any IV medication to patients. As I understand, you’ve received training on the checklist (or smart
pump)?
[Participant] “Yes.”
[Nurse Actor] “Great! Go ahead and use it as your were trained. Come on over and let me introduce you to your first
patient. “
If second protocol of experiment:
[Nurse Actor] Hi, Welcome back to the ward! I hope you had a good break. We have some new patients that have just
arrived on our ward and need their medication orders completed. So I’m glad you’re here because I definitely need
some help! Do you have any questions before we begin?
127
2.1.2 Experiment: Introductions to Patients and Setting Up Infusions
2.1.2.1 Mrs. Katherine Tuer (Bed 1): N-Acetylcysteine, Wrong Dose (mismatch)
Nurse Actor: Brings order for acetylcysteine
[Nurse Actor] “Our first patient is Mrs. Tuer in Bed 1. She has Alzheimer’s and has been very confused and agitated since she
arrived in the emergency room and was transferred to our unit. She has been up at night and sleeping during the day. She has an
order for haloperidol to help calm her down. She initially came into the emergency room and was admitted with an
acetominophen (Tylenol) overdose. She has begun acetylcysteine therapy for the Tylenol overdose and has already received the
loading dose and the first maintenance dose. She will need to be started on a second maintenance dose of acetylcysteine. She is
73 years old and 70kgs.
[Nurse Actor] “I’ve brought out her acetylcysteine. Would you mind getting it started?”
Order: Drug name: N-Acetylcsyteine Concentration: 7g/1000ml
Order: 7g at 6.25mg/kg/h (100mg/kg in 1L for 16hrs) Programming: Dose: 7g Rate: 62.5mL/h VTBI: 1000mL
Nurse Actor: Goes to Mr. Scalp and comes back over
**Interruption (as participant is checking label to MAR) [Nurse Actor]: “I just wanted to let you know that I have the next
medication ready, so let me know when you are finished over here.”
NOTE: Nurse should notice the wrong dose (label does not match the order)
If participant notices the planted error:
[Nurse Actor] “Hmmm, that is odd, the dose on the label does not match the order. Pharmacy must have made a
mistake. Let me see if I can sort this out. “
Nurse Actor: goes behind curtain and brings out correct dose
2.1.2.2 Mrs. Katherine Tuer: D5W & Haloperidol, Inappropriate Dose (hard limit)
[Nurse Actor]: “I was just speaking with Mrs. Tuer’s physician and he said we could go ahead and get her fluid
maintenance line going and give her a dose of haloperidol to calm her down. She has a second IV access that you can
use. Thanks.”
Order: Drug name: D5W Concentration: N/A Order: D5W 1000mL, Infuse at 100mL/h
128
Programming: VTBI: 1000mL Duration: continuous Rate: 100mL/h
Order: Drug name: Haloperidol Concentration: 0.5mg/mL Order: 50mg/100mL D5W, Infuse over 30 minutes Programming: VTBI: 100mL Duration: 30 minutes Rate: 200mL/h
NOTE: Nurse should recognize that the dose ordered is inappropriate (much too high and exceeds the 6hr dose limit).
If participant detects the inappropriate dose before starting pump programming or before a limit is hit:
[Nurse Actor] “Hmmm… yeah – that can’t be right. Let me check with her physician again. “
Nurse Actor: pretends to call physician Goes to get a new paper order from behind the screen Nurse Actor goes to pharmacy and brings back correct med order and bag
Nurse Actor : So I just spoke with Dr. Nigma and he’s written a new paper order with the correct dosage value. You were right,
the dose is much too high. The physician must have added an extra 0 - it should only be 5mg instead of 50mg. He will change it on
the MOE/MAR when he gets back to his office. I just got the correct dose from Pharmacy. Would you make sure there are no
other issues before you start the infusion?”
2.1.2.3 Mr. Kevin Piu (Bed 2): NS, morphine, wrong patient
[Nurse Actor] “Finished with Mrs. Tuer? Great! Let’s move on to our next patient in Bed 2. Kevin came into the ER experiencing
acute pain. He has a history of alcohol abuse and is suspected to be in the beginning stages of liver failure. His tests show he also
has low phosphate with a phosphate serum level of 0.6mmol/L (normal range 0.8 to 1.4 mmol/L). He needs to have a
maintenance line of normal saline set up to restore his fluid balance and has orders for morphine and potassium phosphate.
[Nurse Actor]: “We can start with his maintenance line started and get him a dose of morphine for his pain. I’ve just brought it
out here and his order should be on the MAR. Thanks.”
Nurse Actor: Brings order for normal saline and morphine
Order: Drug name: Normal Saline Concentration: N/A
Order: 1000mL at 100mL/h Programming: Rate: 100mL/h
VTBI: 1000mL Dose: N/A
Order: Drug name: Morphine Concentration: 2mg/50mL
Order: 2mg inj IV-int (immediately). Administer over 10 min Programming: VTBI : 50mL Duration: 10min Rate: 300 mL/h
129
Go to med room and come back out as participant is checking armband.
**Interruption (as nurse is checking armband) [Nurse Actor] “I just wanted to let you know that I also have the medications
ready for our third patient when you finish up here.”
NOTE: Participant should notice Wrong Patient Error
If participant detects the wrong patient name on the bag prior to programming or during programming:
[Nurse Actor] “Hmmm, you’re right this seems to be the wrong patient. I’ll have to go find the orderly and sort this out. In the
meantime, why don’t I bring you over to our third patient in Bed 3 so you can get started on her meds while I find the right
patient.
2.1.2.4 Mrs. Penelope Sillian (Bed 3) : Ringers Lactate & Ceftriaxone , Incompatibility
Nurse Actor: Brings order for Ringers lactate and ceftriaxone
[Nurse Actor] “Our patient in Bed 3 is Mrs. Penelope Sillian. She is an NPO patient in stable condition being treated for an upper
GI bleed. She has an order for a continuous maintenance infusion of octreotide for the bleeding. She also has an E. coli bacterial
infection which is being treated with ceftriaxone. In addition, I think she has an order for Ringers lactate that we will be taking
care of to restore her fluid and electrolyte balance.
[Nurse Actor]: “I was just verifying with the physician and he said to go ahead and start Mrs. Sillian’s maintenance line and
antibiotics. I just have to go respond to a page so I’ll be back in a minute.”
Order: Drug name: Ringers Lactate Concentration: N/A
Order: 1000mL Ringers Lactate IV at 100mL/h Programming: Rate: 100mL/h VTBI: 1000mL
Order: Drug name: Ceftriaxone Concentration: 1g/50mL
Order: 1g in 9.6mL SWI, then diluted in 50mL D5W, Infuse over 15 min Programming: Rate: 200mL/h VTBI: 50mL Duration: 15min
NOTE: Nurse should recognize that Ceftriaxone is incompatible with Ringers Lactate
If Participant recognizes incompatibility:
[Nurse Actor] “Oh yes, you are right. It’s a good thing you caught that. Ceftriaxone should not be given with anything containing
calcium. The physician must not have considered that. I will go see what he wants us to do and take care of that order. In the
meantime…[go to ***if detected wrong patient, otherwise go to ###]
130
***[Nurse Actor] “I have located Mr. Piu and he is back in Bed 2 where he belongs. He needs to have his maintenance line
and magnesium sulfate started.”
Participant moves to Mr. Piu’s in bed 2 and finishes his/her orders
2.1.2.5 Mr. Albert Scalp (Bed 1): Amphotericin B Liposomal, Wrong drug
Nurse Actor: Brings order for amphotericin B (wrong drug)
[Nurse Actor] “Mrs. Tuer has been discharged and we have just had a new patient moved into Bed 1. Mr. Scalp is an NPO patient
with an invasive candidemia fungal infection in the bloodstream. His tests have also shown a mild magnesium deficiency with a
magnesium serum level of 0.65mmol/L (normal range is 0.7mmol/L to 0.95mmol/L) so he will need a magnesium sulphate
infusion. He is currently quite nauseous and has been vomiting. He needs a normal saline maintenance drip to restore his fluids
and has an order for dimenhydrinate to help him feel better.”
[Nurse Actor]: “His physician has requested he get the antifungal started first. Would you mind setting up his infusion?”
Order: Drug name: Amphotericin B Liposomal Concentration: 425mg/250mL Order: 425mg in 250 mL D5W inj IV-int Q24h. Infuse over 3hrs.
Programming: VTBI: 250mL Duration: 3hrs Rate: 83.33 mL/h
**Interruption (as participant is checking label to MAR) [Nurse Actor] drop clip board and pen near participant. “Ooops, I’m so
sorry… I was just coming to let you know that Mrs. Sillian is ready for her next medication when you finish with Mr. Scalp.”
NOTE: Participant should notice the wrong drug on the iv bag label
If the participant realizes the planted error before or during programming:
Nurse Actor: compares drug name on bag to drug name on MOE/MAR
[Nurse Actor] “Yes, I think you are right, I must have mixed up the drug names when I was preparing them in the med room. Let
me go get the correct drug. (Nurse Actor goes to med room and gets correct drug). Here it is!”
Nurse Actor: hands bag to participant
2.1.2.6 Mrs. Penelope Sillian (Bed 3): Octreotide, inappropriate dose rate (hard limit)
[Nurse Actor] “I just talked to Dr. Nigma and he said to start Mrs. Sillian’s Octreotide. I’m not sure if you already saw the order in
MOE/MAR but if not, all the info is there. Could you start it for me? I’m just going to check back on Mr. Scalp.”
Order: Drug name: Octreotide acetate Concentration: 2mcg/mL Order: 500mcg IV-cont at 500mcg/hr Programming: VTBI: 250mL Duration: 10hrs Rate: 250mL/h
131
NOTE: Nurse should notice inappropriate dose rate or hit a hard dose rate limit
If participant notices the planted error before starting programming or before the limit is hit:
[Nurse Actor] “Hmmm… yeah – that can’t be right. There seems to be an error somewhere. Let me confirm with the physician.“
• Nurse Actor: pretends to call physician • Brings new paper order
[Nurse Actor]: Mrs. Sillian’s physician agrees that the rate is wrong. The dose rate should be 50mcg/h instead of 500mcg/h. I
believe that makes the infusion rate 25mL/h. The bag should last 10 hrs. I have a revised paper order and it will be changed in the
MOE/MAR shortly. Thanks.”
2.1.2.7 Mr. Albert Scalp (Bed 1): Normal Saline & Dimenhydrinate – no error
Nurse Actor: bring out NS & Dimenhydrinate for Mr. Scalp
[Nurse Actor}: “I’ve just been to check on Mr. Scalp and the antifungal made him a little bit nauseous. He has an order for some
dimenhydrinate to help him feel better. Could you start it for me please?”
Order: Drug name: Normal Saline Concentration: N/A Order: 1000mL at 100mL/h Programming: Rate: 100mL/h VTBI: 1000mL Dose: N/A
Order: Drug name: Dimenhydrinate Concentration: 25mg/50mL Order: 25mg dimenhydrinate inj IV-int (immediately). Administer over 15 min Programming: VTBI : 50mL Duration: 15min Rate: 200 mL/h
2.1.2.8 Mr. Kevin Piu (Bed 2): Potassium Phosphate, inappropriate duration (hard limit)
Nurse Actor: Brings order for potassium phosphate
[Nurse Actor] “I just spoke to Mr. Piu’s physician and he said we can go ahead and get his potassium phosphate started on
another primary line as well. I have the medication here – you can use his other IV access.
Order: Drug name: Potassium Phosphate Concentration: 15mmol/415mL Order: 15mmol IV-int (active) in 415mL D5W once over 1hr Programming: VTBI: 415mL Duration: 1hr Rate: 415mL/h
NOTE: Participant should hit a hard limit:
132
If participant notices the planted error before starting programming or before the limit is hit:
[Nurse Actor] “Hmmm… yeah – that can’t be right. Let me check with the physician. “
Nurse Actor: pretends to call physician Gets new paper order
Nurse Actor: “You were right. The order should be over 4hrs, anything between 3-12hrs is ok but 1hr is much too short. I have a
new paper order from the physician so you can go ahead and program it to run over 4hrs. He will update the MAR shortly.”
2.1.2.9 Mr. Albert Scalp (Bed 1): Magnesium Sulphate – no error
Nurse Actor: Brings order mag sulfate
[Nurse Actor] “Mr. Scalp’s medications have finished infusing and he is ready to get his magnesium sulfate now. Could you start it
for me? I will be back shortly. I just need to attend to another patient. Thanks”
Order: Drug name: Magnesium sulfate Concentration: 1g/100ml Order: 1g inj IV-int in 100mL NS Administer over 1hr Programming: Rate: 100mL/h Duration: 1h VTBI: 100mL
[Nurse Actor] “Great! Thank you for all of your help, all the orders for these patients have been completed.”
Nurse Actor: Records programmed parameters and turns off pumps for once scenario is completed. Nurse Actor: disconnect lines, collect caps and put on lines
**Refer to Changeover procedures **
2.2 Protocol B
NOTE: Nurse Actor is in charge of all questions from the participant. If the participant gets stuck, the nurse actor will tell the
participant how to get the next programming step.
2.2.1 Introduction to ward
Data collection: synch time on laptop with camera recording time
If first protocol of experiment:
[Nurse Actor] “Hi ________________, nice to meet you! You must be my float nurse! My name is Lata and I’m another
nurse on this ward. It is so good to see you because we are way behind in med delivery to our patients. Some patients
have just come up to the ward and we need to finish their orders. You haven’t worked on this unit before, right? There
are a couple of things I would like to cover before I introduce you to the patients that you will be taking care of. First,
here is our MOE/MAR. You are responsible for administering the IV medications, but I will take care of any
133
documentation. The Standard and Restricted IV drug formulary is also available if you need to look up any unfamiliar
drugs.
“The hospital has just implemented a new checking process for all IV medications. This checklist is to be completed
when administering any IV medication to patients. As I understand, you’ve received training on the checklist (or smart
pump)?
[Participant] “Yes.”
[Nurse Actor] “Great! Go ahead and use it as your were trained. Come on over and let me introduce you to your first
patient. “
If second protocol of experiment:
[Nurse Actor] Hi, Welcome back to the ward! I hope you had a good break. We have some new patients that have just
arrived on our ward and need their medication orders completed. So I’m glad you’re here because I definitely need
some help! Do you have any questions before we begin?
2.2.2 Experiment: Introductions to Patients and Setting Up Infusion
2.2.2.1 Mr. Xian Chan (Bed 1): NS & Dimenhydrinate, Wrong patient
Nurse Actor: Brings order for normal saline and dimenhydrinate
[Nurse Actor] “Let me give you a quick update on our first patient in Bed 1. Mr. Chan came into the ED with acute cholecystitis.
He has been feeling quite uncomfortable since he came up to the ward. He is experiencing pain and has been feeling quite
nauseous. He has an order for some hydromorphone and dimenhydrinate. He also needs a continuous normal saline drip to help
restore his fluid and electrolyte balance as he has been vomiting quite a bit since this morning.
[Nurse Actor] “His orders are on the MOE/MAR. If you could start with the maintenance line and then give him his
dimenhydrinate he should start feeling better. Thanks a lot!
Order: Drug name: Normal Saline Concentration: N/A
Order: 1000mL at 100mL/h Programming: Rate: 100mL/h
VTBI: 1000mL Dose: N/A
Order: Drug name: Dimenhydrinate Concentration: 25mg/50mL
Order: 25mg inj IV-int (immediately). Administer over 10 min Programming: VTBI : 50mL Duration: 10min Rate: 300 mL/h
**Interruption (as nurse is checking armband for patient) [Nurse Actor]: “I just wanted to let you know that I have the
medications ready for our patient in Bed 2 whenever you are finished up here.”
134
NOTE: Participhnt should notice Wrong Patient Error
If participant detects the wrong patient name on the bag prior to programming or during programming:
[Nurse Actor] “Hmm, that’s strange, this doesn’t seem to be the right patient. I’ll talk to the orderly and figure out what
happened. In the meantime, let me introduce you to our second patient and you can get started on his orders.”
Experiment: Mr. Peter Wilson (Bed 2) ***
2.2.2.2 Mr. Peter Wilson (Bed 2): D5W maintenance with Phenytoin - (incompatibility)
[Nurse Actor] “Our patient in Bed 2, Mr. Wilson, is an NPO patient with epilepsy and was brought up to the ward with
pneumonia. He needs a dose of phenytoin to control his seizures and also needs to get started on his piperacillin-tazobactam
antibiotic to clear up his pneumonia. His physician has also ordered a maintenance IV of D5W. Mr. Wilson is 35 years old and
84kg. His is currently stable and his vitals are good.
Nurse Actor: brings out D5W and Phenytoin
[Nurse Actor] “I just got his first medications. If you can start the maintenance line and his phenytoin, I’ll look into the
situation with Mr. Chan.”
Order: Drug name: D5W IV Concentration: N/A Order: 1000mL D5W, Infuse at 100mL/hr Programming: VTBI: 100mL Duration: 10hrs Rate: 100mL/h Order: Drug name: Phenytoin Concentration: 3mg/mL Order: 300mg Phenytoin in 100mL NS, Infuse over 60min Programming: VTBI: 100mL Duration: 60min Rate: 100mL/h
NOTE: Nurse should recognize that Phenytoin is incompatible with D5W
If Participant recognizes incompatibility:
[Nurse Actor] “Oh yes, you are right. I’m glad you pointed that out. Phenytoin should only be mixed with normal saline. It
shouldn’t be piggybacked to the D5W maintenance. I will take care of this one for you. In the meantime, I located the correct Mr.
Chan and relocated him back to Bed 1.”
[Nurse Actor] “He is still waiting for his medications. Would you mind picking up where you left off?”
Participant moves back to Mr. Chan’s bed and finishes his/her orders
135
Note: Go back to *** if wrong patient error was not found (change wristband to correct patient before bringing
out next medication for Mr. Chan
2.2.2.3 Mr. Steven Campbell (Bed 3): Pantoprazole, Wrong Dose (mismatch)
[Nurse Actor] “Our third patient in Bed 3 here is Mr. Steven Campbell. He is 35 years old and was admitted to the emergency
department with a GI bleed. He was also discovered to have vertebral osteomyelitis from a methicillin-resistant staph
bloodstream infection. He has developed some hives on his stomach from the previous antibiotic he was on and has now been
switched to a treatment of Vancomycin to try and eliminate his infection. He also has an order for pantoprazole for his GI bleed.
His physician has also requested he be put on a maintenance of Ringer’s lactate to maintain his fluid and electrolyte balance.”
Nurse Actor: Brings order for pantoprazole
[Nurse Actor]: “I was just talking with Mr. Campbell’s physician and he says we can go ahead and get the pantoloc started first.
I’ve just brought his order out.”
Order: Drug name: Pantoprazole Concentration: 40mg/100mL Order: Pantoprazole IV-cont 40mg in 100mL NS, infuse at 20mL/h
Programming: Dose: 40mg Rate: 20mL/h VTBI: 100mL
**Interruption (as participant is comparing the MAR and label): [Nurse Actor] “I wanted to let you know that Mr. Campbell’s
physician said we can go ahead and give him some benadryl for his hives as well using his other access site. I have his medication
when you finish setting up his pantoloc.”
NOTE: Nurse should notice the wrong dose (label does not match the order)
If participant notices the planted error:
[Nurse Actor] “Hmmm, that is odd, the dose on the label does not match the order. Pharmacy must have made a mistake. Let me
see if I can sort this out. “
Nurse Actor: goes behind curtain and brings out correct dose (40mg Pantoprazole)
2.2.2.4 Mr. Steven Campbell (Bed 3): Ringer’s Lactate, Diphenhydramine – No Error
[Nurse Actor]: “I have some diphenhydramine and his maintenance line of Ringer’s lactate. Would you mind getting that started
for him?”
Nurse Actor: Brings out RL & diphenhydramine
Order: Drug name: Ringers Lactate Concentration: N/A
Order: 1000mL Ringers lactate, Infuse at 50mL/h
136
Programming: Rate: 50mL/h Duration: continuous VTBI: 1000mL
Order: Drug name: Diphenhydramine Concentration: 25mg/50mL Order: 25mg diphenhydramine in 50mL NS, Infuse over 15minutes Programming: Rate: 200mL/h Duration: 15 minutes VTBI: 50mL
2.2.2.5 Mr. Xian Chan (Bed 1): hydromorphone – inappropriate dose
[Nurse Actor]: “Mr. Chan is ready for his pain medication now that he has had some gravol for his nausea. I have the
medication here and his order is on the MAR.”
Order: Drug name: Hydromorphone Concentration: 10mg/50mL
Order: Hydromorphone 4.0mg in 50mL NS, infuse over 10minutes Programming: VTBI: 50mL Duration: 10minutes Rate: 300mL/h
NOTE: Nurse should recognize that the dose ordered is inappropriate (much too high for intermittent dose and exceeds the 6hr
dose limit).
If participant detects the inappropriate dose before starting pump programming or before a limit is hit:
[Nurse Actor] “Hmmm… yeah – that can’t be right. Let me check with her physician. “
Nurse Actor: pretends to call physician Nurse Actor: Goes to get a new paper order and correct dose (1mg hydromorphone)
Nurse Actor: So I just spoke with Dr. Stevens and he’s written a new paper order with the correct dosage value. You were
right, the dose is much too high. The physician must have forgotten the decimal. It should be 1mg instead of 10 mg. The
MOE/MAR will be updated when he gets back to his office. I just got the correct dose from Pharmacy. Would you make sure
there are no other issues before you start the infusion?”
Participant moves back to Mr. Chan’s bed and finishes his/her order before moving on to the next patient
2.2.2.6 Mr. Steven Campbell (Bed 3): Vancomycin (Inappropriate duration)
Nurse Actor: Brings up order for Vancomycin
[Nurse Actor]:”Mr. Campbell’s diphenhydramine has finished so he is ready to get started on his vancomycin now.”
Order: Drug name: Vancomycin Concentration: 1g/250mL Order: 1g Vancomycin in 250mL D5W, Infuse over 30minutes Programming: Rate: 500mL/h Duration: 30 minutes
137
VTBI: 250mL
NOTE: Nurse should recognize that the duration is inappropriate (much too short).
If participant notices the planted error before starting programming or before the limit is hit:
[Nurse Actor] “Hmmm… yeah – that can’t be right. Let me check with the physician. “
Nurse Actor: pretends to call physician Nurse Actor: Brings out new paper order
Nurse Actor: “You were right. The order should be over at least 1hr, 30minutes is much too short. I have a new paper order from
the physician so you can go ahead and program it to run over 1hr. He will update the MAR shortly.”
Nurse actor: Take down medications in bed 1 Reset pump Change wristband to Mrs. Taylor and put on wig
2.2.2.7 Mr. Peter Wilson (Bed 2): Piperacillin-tazobactam (no error)
[Nurse Actor]: “Mr. Wilson is now ready for his antibiotic. Could you start that for me.”
Order: Drug name: Piperacillin-tazobactam Concentration: 4.5g/50mL Order: 4.5g inj IV-int in 50mL NS (immediately) Administer over 30min Programming: VTBI : 50mL Duration: 30min Rate: 100 mL/h
2.2.2.8 Mrs. Agnes Taylor (Bed 1): Sodium bicarbonate, inappropriate dose rate (hard limit)
[Nurse Actor] “Mr. Chan has just been discharged and we’ve had a new patient brought up to Bed 1. Mrs. Agnes Taylor is a 62
year old patient with metabolic acidosis. She has a blood pH of 7.1 (normal is 7.35-7.45). She will need a continuous infusion of
sodium bicarbonate to bring up her pH. In addition, she has developed a bacterial infection from an infected IV line and needs to
be started on antibiotics. Her physician has also ordered a continuous IV of D5W to be started immediately.
Nurse Actor: Brings out sodium bicarbonate
[Nurse Actor]: She needs to get started on her continuous dose of sodium bicarb for her acidosis immediately. I’ve brought out
her medication and her order should be up on the MAR. Would you mind setting up this first infusion for her while I go answer a
page?”
Order: Drug name: Sodium bicarbonate
Concentration: 200mmol/1000mL
Order: Sodium Bicarbonate 200mmol (3mmol/kg) IV-cont at 300mL/hr
Programming: VTBI: 1000mL
Duration: 3hrs
Rate: 300mL/h
138
NOTE: Nurse should notice inappropriate dose rate or hit a hard dose rate limit
If participant notices the planted error before starting programming or before the limit is hit:
[Nurse Actor] “Hmmm… yeah – that can’t be right. Let me confirm with the physician.“
Nurse Actor: pretends to call physician Nurse Actor: Brings out new paper order
[Nurse Actor]: Mrs. Taylor’s physician agrees that the rate is wrong. The dose rate should be 20mmol/h instead of 60mmol/h. I
believe that makes the infusion rate 100mL/h. I have a revised paper order from her physician and it will be changed in the
MOE/MAR shortly. Thanks.”
2.2.2.9 Mrs. Agnes Taylor (Bed 1): Pump 1 – D5W and Ticarcillin-clavulanate, (wrong drug is Ticarcillin)
[Nurse Actor] “I’ve just brought out the last medications for Mrs. Taylor. Her orders are on the MOE/MAR. You can use her other
IV access to get started on her primary D5W line and give her the antibiotic for her. Thanks a lot!
Nurse Actor: Bring out D5W and ticarcillin (wrong drug)
Order: Drug name: D5W
Concentration: N/A
Order: D5W 1000mL, Infuse at 100mL/h
Programming: Rate: 100mL/h
DurationL continuous
VTBI: 1000mL
Order: Drug name: Ticarcillin-clavulanate
Concentration: 3g/50mL
Order: 3 g Ceftazidime in 50mL D5W, Infuse over 30 minutes
Programming: Rate: 100mL/h
DurationL 30 minutes
VTBI: 50mL
**Interruption (as participant is looking at drug order) [Nurse Actor]: “I just wanted to thank you for all your help with the
medications today. Once you are done here we should be all finished!
NOTE: Participant should notice the wrong drug on the IV bag label
If the participant realizes the planted error before or during programming:
Nurse Actor: compares drug name on bag to drug name on MOE/MAR
[Nurse Actor] “Yes, I think you are right, I must have mixed up the wrong drug when I was preparing them in the med room. Let
me go get the correct drug. (Nurse Actor goes to med room and gets correct drug). Here it is!”
139
Nurse Actor: brings out ticarcillin-clavulanate
[Nurse Actor] “Great! Thank you for all of your help, all the orders for these patients have been completed.”
Nurse Actor: Records programmed parameters and turns off pumps for once scenario is completed. Nurse Actor: disconnect lines, collect caps and put on lines
**Refer to Changeover procedures **
3 Post-protocol
3.1 Break between protocols Location: Training space
Equipment needed: Laptop, desk, chairs, Drinks/snacks
Estimated Time: 5-10min
Person 1: All right, that was great! While we’re setting up for the next scenario you are welcome to a short break. There are
drinks and snacks if you would like anything. Also, if you need to use the washroom one of us can show you where they are
located.
3.2 Protocol Equipment Change-Over
During the break or while training is being completed for the next part, the lab is being set up for the next scenario:
Nurse Actor (brings in new equipment)
Pulls up patient MOE/MAR orders for the second protocol Put wristbands and wigs on for new patients Takes down drugs from first protocol Resets pumps to default settings and clears volume totals Sets out drugs and paper orders for new patients Plugs in laptop to charge Switches pumps if smart pump group, or puts out/removes checklist if checklist group Put new pumps on poles
3.3 Questionnaire and Debrief Location: Training space
Equipment needed: Laptop, desk, chairs, Drinks/snacks, Questionnaire on survey monkey is set-up on desktop computer
Estimated Time: 15min – conducted at the conclusion of both experiment and control protocols.
[Study Coordinator]: “All right, that was great! I’d just like to get some feedback on your experience with using the checklist (or
barcode smart pump system) and how you found the scenarios.”
Example questions for checklist users:
1. Overall, what was your impression of the checklist? 2. Are there any items you feel do not need to be included? 3. Are there any items that you feel are missing? 4. What changes would you make to the organization and layout of the checklist?
140
5. How do you see using this checklist on the ward, i.e. what are your thoughts on the mode of delivery (e.g. paper-based, electronic, laminated, on lanyard)? (Perhaps on a smartphone platform?)
6. How do you feel the checklist impacted the efficiency of the medication administration process? 7. How well do you think the checklist addresses verification errors? i.e. checking that labels match the orders etc. 8. Did the checklist increase your awareness for the clinical appropriateness of the medication order? 9. What are your comments regarding the section: “Stop! Knowing all that you know, does this order make sense to you?”? 10. Do you have any other general comments relating to the design and implementation of the checklist? 11. Can you suggest any other interventions you feel may be effective in preventing medication administration errors? 12. How do you think this checklist would compare to using a smart pump with bar-code capabilities, dosage limits and a drug
library?
Example questions for barcode smart pump users:
1. Overall, what was your impression of the barcode scanning and smart pump? 2. How do you feel this system impacted the efficiency of the medication administration process? 3. How well do you think this system addresses verification errors? i.e. checking that labels match the orders etc. 4. Did the system increase your awareness for the clinical appropriateness of the medication order? 5. Do you have any other general comments relating to the design or usability of the smart pump? 6. Can you suggest any other interventions you feel may be effective in preventing medication administration errors? 7. How do you think this system would compare to a paper-based checklist?
[Study Coordinator]: “Finally, I also have a quick post-experiment questionnaire for you to complete to summarize your
experience.”
Participant completes questionnaire #2 (either for checklist or smart pump, depending on the group)
Participant completes questionnaire.
[Study Coordinator] “Thank you again for participating in this study... Because there is a possibility that your co-workers will
participate in this study, we would really appreciate it if you would please not discuss this experiment with any colleagues. ”
Participant is escorted outside of lab area.
141
Appendix E: Sample Physician Paper Medication Order
142
Appendix F: Sample Physician Electronic Order
143
144
Appendix G: Sample Investigator Data Collection Sheet
Data was collected electronically on a complete version of this sample sheet.
Table 11: Sample of data collection log
1) Patient 1: Mr. Wilson
Timestamp
Task Yes/No Error Comments
Prep
Goes to computer
Verifies drug labels against computer order
Review formulary?
Walks to Mr. Wilson's bedside
Checks patient's armband
D5W
Hang D5W, lower on pole
Load pump
Connect D5W line to patient access
Enter D5W rate (100 mL/h)
Enter D5W VTBI (1000 mL)
Phenytoin
Verify label against patient armband
Detects incompatibility error
Hang secondary bag
Attach tubing above the pump
Enter Phenytoin rate (100mL/h)
Enter Phenytoin VTBI (100mL)
Open clamps
Run infusion
Look to see if medication is infusing
2) Patient 2: Mr. Steven Campbell
Timestamp
Task Yes/No Error Comments
Prep
Goes to computer
Verifies drug label against computer order
Walks to Mr. Steven Campbell's bedside
Checks patient's armband
Diphen-
hydramine
Review formulary
Hang secondary bag
Lower primary bag
Connect diphenhydramine to primary above
Graseby pump
Open clamp
Press secondary infusion button
Enter diphenhydramine rate (200 mL/h)
145
Enter ceftriaxone VTBI (50 mL)
Start infusion
3) Patient 1: Mr. Chan - Wrong patient
Timestamp
Task Yes/No Error Comments
Prep
Goes to computer
Verifies drug label against computer order
Review formulary
Walks to Mr. Chan's bedside
Checks patient's armband - wrong patient
Dimenhy-
drinate
Hang secondary bag
Lower primary bag
Connect gravol to primary above Graseby
pump
Open clamp
Press secondary infusion button
Enter rate (200 mL/h)
Enter VTBI (50 mL)
Start infusion
4) Patient 2: Mr. Campbell
Timestamp
Task Yes/No Error Comments
Prep
Goes to computer
Verifies drug label against computer order
Review formulary
Walks to Mr. Campbell's bedside
Checks patient's armband
Ringer's
lactate
Hang RL
Load pump
Connect RL to patient access
Enter RL rate (50 mL/h)
Enter RL VTBI (1000 mL)
Open clamp
Start infusion
Vanco-
mycin
Hang vancomycin
Higher than primary
Connect to primary abovet the pump
Enter vancomycin rate (500 mL/h)
Enter vancomycin VTBI (250 mL)
Open clamp
Start infusion
146
Appendix H: Demographic Questionnaire
This questionnaire was collected electronically on a laptop provided at the start of the session.
Personal Background
1. What best describes your role in the hospital?
Staff nurse
Nurse manager
Clinical trials nurse
Advanced practice nurse
Other:
2. Which clinical unit(s) do you typically work in: _________________________________
3. What time/length shifts do you typically work: __________________________________
4. What best describes your work status:
Full-time
Part-time
Casual
Other
5. What age range are you in:
18 – 29 years old
30 – 39 years old
40 – 49 years old
50 – 64 years old
65 years old and over
6. Are you:
Male
Female
7. How long have you been a Registered Nurse?
Less than a year
1-4 years
5-9 years
10-19 years
20 or more years
8. How long have you been working as an RN in your current clinical unit?
Less than a year
1-4 years
5-9 years
10-19 years
20 or more years
9. On average, how often do you program infusion pumps?
Less than once a day
147
1 to 2 times a day
3 to 5 times a day
More than 5 times a day
Other: ___________
10. What methods do you currently use to ensure safe medication administration (e.g. independent
double checks)? __________________________________________
148
Appendix I: Post-Test Questionnaires
149
150
151
152
Appendix J: Statistics for Participant Demographics
Table 12: Test statistics for comparison between groups
Demographic Question Test Performed Test Statistic p-value Between group
significance?
Age Chi-squared 7.549 0.023 Yes
Sex Fisher’s Exact Test N/A 0.489 No
Role in hospital Fisher’s Exact Test N/A 1.000 No
Length of Shift Fisher’s Exact Test N/A 1.000 No
Work Status Fisher’s Exact Test N/A 0.137 No
Experience as an RN Chi-squared 0.091 0.763 No
Experience in current unit Chi-squared 3.086 0.079 No
Infusion pump frequency Chi-squared 0.083 0.773 No
153
Appendix K: Additional Statistics for Error Detection Rates
The two-way interaction between condition and error type was not found to be significant. The
marginal mean error detection rates, averaged across the group variable, are shown in Figure 26
below.
Figure 26: Differences between condition and error type after averaging across the two
groups (checklist and smart pump)
The main effects of error type (execution versus planning) [F (1,46) = 8.223, p < .05] and
condition (baseline versus intervention) [F (1,46) = 22.388, p < .05] were both found to be
significant. The marginal mean error detection rate (averaged across all groups and conditions) for
execution errors was 60% (178/288) compared to only 49% (141/288) for planning errors (Figure
27). Marginal mean error detection rates by condition (averaged across all groups and error types)
were 43% (123/288) in the baseline condition compared to 66% (189/288) in the intervention
condition (Figure 28). Though very near significance, the between-group main effect of
intervention type (checklist or smart pump) was not found to be significant.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Baseline Intervention
Ma
rgin
al
Me
an
Err
or
De
tect
ion
Ra
te
Execution Errors
Planning Errors
154
Figure 27: Differences in mean error detection rates in main effect of error type
Figure 28: Differences in mean error detection rates in main effect of condition
Table 13: Descriptive statistics of mean error detection
Group Mean Std. Deviation N
Execution Errors in the
Baseline Condition
Checklist 1.58 .929 24
Smart Pump 1.37 1.135 24
Total 1.48 1.031 48
Execution Errors in the
Intervention Condition
Checklist 1.96 .908 24
Smart Pump 2.29 .751 24
Total 2.13 .841 48
Planning Errors in the Checklist 1.04 1.083 24
0%
20%
40%
60%
80%
100%
Execution Planning
Ma
rgin
al
Me
an
Err
or
De
tect
ion
Ra
te
0%
20%
40%
60%
80%
100%
Baseline Intervention
Ma
rgin
al
Me
an
Err
or
De
tect
ion
Ra
te
155
Baseline Condition Smart Pump 1.21 1.179 24
Total 1.13 1.123 48
Planning Errors in the
Intervention Condition
Checklist 1.17 1.090 24
Smart Pump 2.46 .658 24
Total 1.81 1.104 48
Table 14: Between group and within-subjects significant effects
Source
Type III
Sum of
Squares
df Mean
Square F p
Between-groups
Group 7.521 1 7.521 4.001 .051
Error (Group) 86.458 46 1.880
Within-subjects
Error Type 5.333 1 5.333 8.223 .006
Error Type * Group 5.333 1 5.333 8.223 .006
Error(Error Type) 29.833 46 0.649
Condition 21.333 1 21.333 22.388 .000
Condition * Group 8.333 1 8.333 8.745 .005
Error(Condition) 43.833 46 0.953
Error Type * Condition 0.021 1 0.021 .055 .816
Error Type * Condition * Group 1.021 1 1.021 2.690 .108
Error(Task Type*Condition) 17.458 46 0.380
Table 15: Marginal mean across condition
N Marginal
Mean
Standard
Deviation
Standard Error
Execution Errors Checklist 24 1.7708 .72200 .14738
Smart Pump 24 1.8333 .80307 .16393
Planning Errors Checklist 24 1.1042 .88440 .18053
Smart Pump 24 1.8333 .76139 .15542
Table 16: One way ANOVA of two-way interaction between groups by error type
Sum of Squares df Mean Square F Sig.
Execution Errors Between Groups .047 1 .047 .080 .778
156
Error 26.823 46 .583
Total 26.870 47
Planning Errors
Between Groups 6.380 1 6.380 9.370 .004
Error 31.323 46 .681
Total 37.703 47
Table 17: Marginal mean across error type
N Mean Std. Deviation Std. Error
Mean
Baseline Checklist 24 1.3125 .88234 .18011
Smart Pump 24 1.2917 .95458 .19485
Intervention Checklist 24 1.5625 .86367 .17630
Smart Pump 24 2.3750 .62987 .12857
Table 18: One way ANOVA of two-way interaction between groups by condition
Sum of Squares df Mean Square F Sig.
Baseline
Condition
Between Groups .005 1 .005 .006 .938
Error 38.865 46 .845
Total 38.870 47
Intervention
Condition
Between Groups 7.922 1 7.922 13.866 .001
Error 26.281 46 .571
Total 34.203 47
Table 19: Pair-wise comparisons
Paired Differences
Mean Std.
Deviation
Std.
Error
Mean
95% Confidence
Interval of the
Difference t dF Sig. (2-
tailed)
Lower Upper
Checklist Execution vs.
Planning .6667 .80307 .16393 .32756 1.00577 4.067 23 .000
Smart
Pump
Execution vs.
Planning .0000 .80757 .16485 -.34101 .34101 .000 23 1.000
Checklist Baseline vs.
Intervention -.2500 1.04257 .21281 -.69024 .19024 -1.175 23 .252
Smart
Pump
Baseline vs.
Intervention -1.0833 .90490 .18471 -1.46544 -.70123 -5.865 23 .000
Table 20: Statistics for within subject comparison of individual planted errors by group
157
Error Type Checklist Group:
Baseline
Checklist Group:
Intervention
P-value (2
sided)
Significant?
Wrong patient 79% 79% 1.000 No
Wrong drug 21% 38% .344 No
Wrong dose 58% 79% .180 No
Inappropriate dose 25% 33% .754 No
Inappropriate rate 33% 33% 1.000 No
Inappropriate duration 46% 50% 1.000 No
Incompatibility 42% 42% 1.000 No
Error Type Smart Pump Group:
Baseline
Smart Pump Group:
Intervention
p-value (2
sided)
Significant?
Wrong patient 58% 75% .289 No
Wrong drug 33% 63% .118 No
Wrong dose 46% 92% .001 Yes
Inappropriate dose 50% 58% .754 No
Inappropriate rate 29% 100% .000 Yes
Inappropriate duration 42% 88% .001 Yes
Incompatibility 38% 83% .001 Yes
Table 21: Descriptive statistics of medication failures
Group Mean Std. Deviation N
Missed armband checklist in
baseline condition
Checklist .96 1.654 24
Smart Pump 1.33 2.390 24
Total 1.15 2.042 48
Misses armband checks in
intervention condition
Checklist .08 .282 24
Smart Pump .25 .532 24
Total .17 .429 48
Pump programming failures in
the baseline condition
Checklist .63 1.013 24
Smart Pump .67 1.007 24
Total .65 1.000 48
Pump programming failures
in the intervention condition
Checklist .67 1.167 24
Smart Pump .04 .204 24
Total .35 .887 48
Failure to open clamps in the
baseline condition
Checklist .21 .509 24
Smart Pump .50 .933 24
Total .35 .758 48
158
Failure to open clamps in the
intervention condition
Checklist .29 .751 24
Smart Pump .50 .590 24
Total .40 .676 48
Failure to start infusion in
baseline condition
Checklist .12 .338 24
Smart Pump .04 .204 24
Total .08 .279 48
Failure to start infusion in
intervention condition
Checklist .00 .000 24
Smart Pump .04 .204 24
Total .02 .144 48
159
Appendix L: Usability Test Post-Test Survey Results
Table 22: First usability test post-test questionnaire results
Ease of Use:
1
Strongly
Disagree
2
Disagree
3
Border-
line
4
Agree
5
Strongly
Agree
Mean
Ranking
Standard
Deviation
1. Overall, the checklist
was easy to use
0
(0%)
0
(0%)
1
(25%)
3
(75%)
0
(0%) 3.75 0.50
2. The checklist used
familiar, easy-to-
understand language
0
(0%)
0
(0%)
1
(25%)
2
(50%)
1
(25%) 4.00 0.82
3. It was easy to navigate
the items and follow
along with the steps
0
(0%)
0
(0%)
3
(75%)
1
(25%)
0
(0%) 3.25 0.50
4. The checklist was easy
to read
0
(0%)
0
(0%)
0
(0%)
4
(100%)
0
(0%) 4.00 0.00
5. It was easy to check
pump programming
parameters (e.g. dose,
volume, etc)
0
(0%)
0
(0%)
2
(50%)
2
(50%)
0
(0%) 3.50 0.58
6. It was easy to correct
mistakes
0
(0%)
1
(25%)
0
(0%)
3
(75%)
0
(0%) 3.50 1.00
7. Reminders and
messages were
meaningful
0
(0%)
0
(0%)
0
(0%)
2
(50%)
2
(50%) 4.50 0.58
Efficiency:
1
Strongly
Disagree
2
Disagree
3
Border-
line
4
Agree
5
Strongly
Agree
Mean
Ranking
Standard
Deviation
1. The time required to
complete the checklist
was appropriate
0
(0%)
1
(25%)
0
(0%)
2
(50%)
1
(25%) 3.75 1.26
2. The number of
additional steps was
acceptable
0
(0%)
0
(0%)
2
(50%)
2
(50%)
0
(0%) 3.50 0.58
Overall impression:
1
Strongly
Disagree
2
Disagree
3
Border-
line
4
Agree
5
Strongly
Agree
Mean
Ranking
Standard
Deviation
1. The checklist meets all my needs for IV infusions
0
(0%)
1
(25%)
1
(25%)
2
(50%)
0
(0%) 3.25 0.96
2. I want to use this checklist on my unit
0
(0%)
2
(50%)
1
(25%)
1
(25%)
0
(0%) 2.75 0.96
Likelihood of detecting:
1
Very
Low
2
Low
3
Border-
line
4
High
5
Very
High
Mean
Ranking
Standard
Deviation
1. Wrong drug 0 1
(25%)
1
(25%)
1
(25%)
1
(25%) 3.50 1.29
160
(0%)
2. Wrong concentration of drug
0
(0%)
0
(0%)
2
(50%)
1
(25%)
1
(25%) 3.75 0.96
3. Wrong patient receiving IV fluids
0
(0%)
0
(0%)
1
(25%)
1
(25%)
2
(50%) 4.25 0.96
4. Wrong dose delivered to patient
0
(0%)
0
(0%)
1
(25%)
1
(25%)
2
(50%) 4.25 0.96
5. Wrong rate delivered to patient
0
(0%)
0
(0%)
2
(50%)
1
(25%)
1
(25%) 3.75 0.96
Table 23: Second usability test post-test questionnaire results
Ease of Use:
1
Strongly
Disagree
2
Disagree
3
Border-
line
4
Agree
5
Strongly
Agree
Mean
Ranking
Standard
Deviation
1. Overall, the checklist was easy to use
0
(0%)
0
(0%)
0
(0%)
4
(80%)
1
(20%) 4.20 0.45
2. The checklist used familiar, easy-to-understand language
0
(0%)
0
(0%)
0
(0%)
4
(80%)
1
(20%) 4.20 0.45
3. It was easy to navigate the items and follow along with the steps
0
(0%)
0
(0%)
0
(0%)
3
(60%)
2
(40%) 4.40 0.55
4. The checklist was easy to read
0
(0%)
0
(0%)
0
(0%)
3
(60%)
2
(40%) 4.40 0.55
5. It was easy to check pump programming parameters (e.g. dose, volume, etc)
0
(0%)
0
(0%)
1
(20%)
3
(60%)
1
(20%) 4.00 0.71
6. It was easy to correct mistakes
0
(0%)
0
(0%)
0
(0%)
4
(80%)
1
(20%) 4.20 0.45
7. Reminders and messages were meaningful
0
(0%)
1
(20%)
2
(40%)
1
(20%)
1
(20%) 3.40 1.14
Efficiency:
1
Strongly
Disagree
2
Disagree
3
Border-
line
4
Agree
5
Strongly
Agree
Mean
Ranking
Standard
Deviation
1. The time required to complete the checklist was appropriate
0
(0%)
0
(0%)
0
(0%)
4
(80%)
1
(20%) 4.20 0.45
2. The number of additional steps was acceptable
0
(0%)
1
(20%)
1
(20%)
2
(40%)
1
(20%) 3.60 1.14
Overall impression:
1
Strongly
Disagree
2
Disagree
3
Border-
line
4
Agree
5
Strongly
Agree
Mean
Ranking
Standard
Deviation
1. The checklist meets all 0 0
(0%)
1
(20%)
3
(60%)
1
(20%) 4.00 0.71
161
my needs for IV infusions
(0%)
2. I want to use this checklist on my unit
1
(20%)
0
(0%)
3
(60%)
1
(20%)
0
(0%) 2.80 1.10
Likelihood of detecting:
1
Very
Low
2
Low
3
Border-
line
4
High
5
Very
High
Mean
Ranking
Standard
Deviation
1. Wrong drug 0
(0%)
0
(0%)
2
(40%)
2
(40%)
1
(20%) 3.80 0.84
2. Wrong concentration of drug
1
(20%)
0
(0%)
2
(40%)
1
(20%)
1
(20%) 3.20 1.48
3. Wrong patient receiving IV fluids
0
(0%)
0
(0%)
1
(20%)
3
(60%)
1
(20%) 4.00 0.71
4. Wrong dose delivered to patient
0
(0%)
1
(20%)
1
(20%)
2
(40%)
1
(20%) 3.60 1.14
5. Wrong rate delivered to patient
0
(0%)
2
(40%)
0
(0%)
2
(40%)
1
(20%) 3.40 1.34
162
Appendix I: High-Fidelity Simulation Post-Test Survey Results
Table 24: Checklist group post-test questionnaire results
Ease of Use:
1
Strongly
Disagree
2
Disagree
3
Border-
line
4
Agree
5
Strongly
Agree
Mean
Ranking
Standard
Deviation
1. Overall, the checklist
was easy to use
0
(0%)
0
(0%)
1
(4%)
7
(29%)
16
(67%) 4.63 0.58
2. The checklist used
familiar, easy-to-
understand language
0
(0%)
0
(0%)
1
(4%)
6
(21%)
17
(71%) 4.67 0.56
3. It was easy to navigate
the items and follow
along with the steps
0
(0%)
0
(0%)
0
(0%)
9
(37.5%)
15
(62.5%) 4.63 0.49
4. The checklist was easy
to read
1
(4%)
0
(0%)
0
(0%)
8
(33%)
15
(63%) 4.54 0.88
5. It was easy to check
pump programming
parameters (e.g. dose,
volume, etc)
0
(0%)
1
(4%)
0
(0%)
14
(58%)
9
(38%) 4.29 0.69
6. It was easy to correct
mistakes
0
(0%)
0
(0%)
2
(8%)
9
(38%)
13
(54%) 4.46 0.66
7. Reminders and
messages were
meaningful
0
(0%)
0
(0%)
0
(0%)
10
(42%)
14
(58%) 4.58 0.50
Efficiency:
1
Strongly
Disagree
2
Disagree
3
Border-
line
4
Agree
5
Strongly
Agree
Mean
Ranking
Standard
Deviation
1. The time required to complete the checklist was appropriate
0
(0%)
1
(4%)
5
(21%)
13
(54%)
5
(21%)
3.92 0.76
2. The number of additional steps was acceptable
0
(0%)
1
(4%)
5
(21%)
14
(58%)
4
(17%)
3.88 0.74
Overall impression: 1 2 3 4 5 Mean Standard
163
Strongly
Disagree
Disagree Border-
line
Agree Strongly
Agree
Ranking Deviation
1. The checklist meets all my needs for IV infusions
0
(0%)
0
(0%)
2
(8%)
17
(71%)
5
(21%)
4.13 0.54
2. I want to use this checklist on my unit
0
(0%)
3
(13%)
10
(42%)
7
(29%)
4
(17%)
3.50 0.93
Likelihood of detecting:
1
Very
Low
2
Low
3
Border-
line
4
High
5
Very
High
Mean
Ranking
Standard
Deviation
1. Wrong drug 2
(8%)
1
(4%)
2
(8%)
6
(25%)
13
(54%)
4.13 1.26
2. Wrong concentration of drug
2
(8%)
1
(4%)
4
(17%)
9
(38%)
8
(33%)
3.83 1.20
3. Wrong patient receiving IV fluids
1
(4%)
1
(4%)
2
(8%)
9
(38%)
11
(46%)
4.17 1.05
4. Wrong dose delivered to patient
2
(8%)
1
(4%)
1
(4%)
9
(38%)
11
(46%)
4.08 1.21
5. Wrong rate delivered to patient
2
(8%)
1
(4%)
4
(17%)
8
(33%)
8
(33%)
3.89 1.23
Table 25: Smart pump group post-test questionnaire results
Ease of Use:
1
Strongly
Disagree
2
Disagree
3
Border-
line
4
Agree
5
Strongly
Agree
Mean
Ranking
Standard
Deviation
1. Overall, the bar-coding and smart pump were easy to use
0
(0%)
1
(4%)
2
(8%)
16
(67%)
5
(21%)
4.04 0.69
164
2. The pump used familiar, easy-to-understand language
0
(0%)
1
(4%)
1
(4%)
10
42%)
12
(50%)
4.38 0.77
3. It was easy to navigate the menus and find what I wanted
0
(0%)
0
(0%)
6
(25%)
14
(58%)
4
(17%)
3.92 0.64
4. The pump screen was easy to read
0
(0%)
0
(0%)
0
(0%)
13
(54%)
11
(46%)
4.46 0.51
5. It was easy to enter pump programming parameters (e.g. dose, volume, etc)
0
(0%)
0
(0%)
0
(0%)
12
(50%)
12
(50%)
4.50 0.51
6. It was easy to correct mistakes
0
(0%)
2
(8%)
6
(25%)
12
(50%)
4
(17%)
3.75 0.85
7. Reminders and messages were meaningful
0
(0%)
0
(0%)
0
(0%)
9
(37.5%)
15
(62.5%)
4.63 0.49
Efficiency:
1
Strongly
Disagree
2
Disagree
3
Border-
line
4
Agree
5
Strongly
Agree
Mean
Ranking
Standard
Deviation
1. The time required to set-up and program the infusion was appropriate
1
(0%)
0
(0%)
4
(17%)
13
(54%)
5
(21%)
3.92 0.88
2. The number of additional steps was acceptable
0
(0%)
1
(4%)
8
(33%)
12
(50%)
3
(13%)
3.71 0.75
Overall impression:
1
Strongly
Disagree
2
Disagree
3
Border-
line
4
Agree
5
Strongly
Agree
Mean
Ranking
Standard
Deviation
1. This system meets all my needs for IV infusions
0
(0%)
2
(8%)
7
(29%)
10
(38%)
5
(21%)
3.75 0.90
165
2. I want to use this system on my unit
0
(0%)
2
(8%)
6
(25%)
12
(46%)
4
(17%)
3.75 0.85
Likelihood of detecting:
1
Very
Low
2
Low
3
Border-
line
4
High
5
Very
High
Mean
Ranking
Standard
Deviation
1. Wrong drug 0
(0%)
2
(8%)
1
(4%)
12
(50%)
9
(38%)
4.17 0.87
2. Wrong concentration of drug
0
(4%)
2
(8%)
2
(8%)
12
(50%)
8
(33%)
4.08 0.88
3. Wrong patient receiving IV fluids
0
(0%)
2
(8%)
3
(13%)
12
(50%)
7
(29%)
4.00 0.88
4. Wrong dose delivered to patient
0
(0%)
3
(13%)
1
(4%)
12
(50%)
8
(33%)
4.04 0.95
5. Wrong rate delivered to patient
0
(0%)
2
(8%)
2
(8%)
12
(50%)
8
(33%)
4.08 0.88
Table 26: Post-test survey results comparative analysis between checklist and smart pump
groups
Ease of use: Group N Mean
Ranking
Standard
Deviation
Mann
Whitney
U Test
Statistic
p-value
(Asymp.
2-tailed)
Significance
?
1. Overall, the
[intervention] was
easy to use
Checklist
Smart Pump
24
24
4.63
4.04
0.58
0.69 153.00 .002 Yes
2. The [intervention]
used familiar, easy-
to-understand
language
Checklist
Smart Pump
24
24
4.67
4.38
0.56
0.77 226.50 .141 No
3. It was easy to
navigate the items
and follow along
Checklist
Smart Pump
24
24
4.63
3.92
0.49
0.65 129.00 .000 Yes
166
with the steps
4. The [intervention]
was easy to read
Checklist
Smart Pump
24
24
4.54
4.46
0.88
0.51 234.50 .203 No
5. It was easy to
[enter/check] pump
programming
parameters (e.g.
dose, volume, etc)
Checklist
Smart Pump
24
24
4.29
4.50
0.69
0.51 246.00 .320 No
6. It was easy to
correct mistakes
Checklist
Smart Pump
24
24
4.46
3.75
0.66
0.85 154.00 .003 Yes
7. Reminders and
messages were
meaningful
Checklist
Smart Pump
24
24
4.58
4.63
0.50
0.49 276.00 .770 No
Efficiency: Test Group N Mean
Ranking
Standard
Deviation
Mann
Whitney
U Test
Statistic
p-value
(Asymp.
2-tailed)
Significance
?
1. The time required
to [use the
intervention] was
appropriate
Checklist
Smart Pump
24
24
3.92
3.92
0.76
0.88 279.50 .845 No
2. The number of
additional steps
was acceptable
Checklist
Smart Pump
24
24
3.88
3.71
0.74
0.75 250.50 .393 No
Overall impression: Test Group N Mean
Ranking
Standard
Deviation
Mann
Whitney
U Test
Statistic
p-value
(Asymp.
2-tailed)
Significance
?
1. The [intervention]
meets all my needs
for IV infusions
Checklist
Smart Pump
24
24
4.13
3.75
0.54
0.90 219.50 .116 No
2. I want to use this
[intervention] on
my unit
Checklist
Smart Pump
24
24
3.50
3.75
0.93
0.85 239.00 .286 No
Likelihood of
detecting: Test Group N
Mean
Ranking
Standard
Deviation
Mann
Whitney
U Test
Statistic
p-value
(Asymp.
2-tailed)
Significance
?
1. Wrong drug Checklist
Smart Pump
24
24
4.13
4.17
1.26
0.87 262.50 .569 No
167
2. Wrong
concentration of
drug
Checklist
Smart Pump
24
24
3.83
4.08
1.20
0.88 263.00 .582 No
3. Wrong patient
receiving IV fluids Checklist
Smart Pump
24
24
4.17
4.00
1.05
0.88 243.50 .323 No
4. Wrong dose
delivered to patient Checklist
Smart Pump
24
24
4.08
4.04
1.21
0.95 261.00 .547 No
5. Wrong rate
delivered to patient Checklist
Smart Pump
24
24
3.89
4.08
1.23
0.88 273.00 .742 No
