A NOVEL USE OF THE ANESTHETISTS’ NON-TECHNICAL SKILLS ...
130
A NOVEL USE OF THE ANESTHETISTS’ NON-TECHNICAL SKILLS (ANTS) INSTRUMENT TO MEASURE CONGRUENCE OF GRADUATE NURSE ANESTHESIA STUDENT SELF-EVALUATION AND FACULTY EVALUATION A DISSERTATION SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY IN THE GRADUATE SCHOOL OF THE TEXAS WOMAN’S UNIVERSITY COLLEGE OF NURSING BY MARGARET ROSEANN DIEHL, BSN, MHS, DNP DENTON, TEXAS MAY 2016
Transcript of A NOVEL USE OF THE ANESTHETISTS’ NON-TECHNICAL SKILLS ...
A NOVEL USE OF THE ANESTHETISTS’ NON-TECHNICAL SKILLS (ANTS)
INSTRUMENT TO MEASURE CONGRUENCE OF GRADUATE
NURSE ANESTHESIA STUDENT SELF-EVALUATION
AND FACULTY EVALUATION
A DISSERTATION
SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS
FOR THE DEGREE OF DOCTOR OF PHILOSOPHY
IN THE GRADUATE SCHOOL OF THE
TEXAS WOMAN’S UNIVERSITY
COLLEGE OF NURSING
BY
MARGARET ROSEANN DIEHL, BSN, MHS, DNP
DENTON, TEXAS
MAY 2016
iii
DEDICATION
For my husband, Jon Diehl, your love and support were unending.
iv
ACKNOWLEDGMENTS
I would like to gratefully acknowledge the many individuals who have contributed to this
dissertation. I would like to thank my committee chair Dr. Constance Ayers for assisting me
every step of the way. I would like to thank my dissertation committee, Dr. Brenda Binder, Dr.
Sandra Cesario, and Dr. Teresa Maharaj. I would not have been able to successfully complete my
dissertation without your guidance. Your words of support, constructive comments, and
suggestions added valuable direction to my research and writing. I am also grateful to the faculty
at Texas Woman’s University, who encouraged me to think critically and challenged me to think
beyond my comfort zone. In addition I would like to thank the wonderful Graduate School staff;
they helped me to navigate through the forms and paperwork that accompanied the graduation
process.
v
ABSTRACT
MARGARET ROSEANN DIEHL
A NOVEL USE OF THE ANESTHETISTS’ NON-TECHNICAL SKILLS (ANTS) INSTRUMENT TO MEASURE CONGRUENCE OF
GRADUATE NURSE ANESTHESIA STUDENT SELF-EVALUATION AND FACULTY
EVALUATION
MAY 2016
The purpose of this study was to determine if a correlation existed between faculty
evaluation of students NTS and student self-evaluation of NTS after a simulated anesthesia
scenario using the ANTS tool. This protocol also explored written perceptions of students and
faculty using the ANTS tool. Thirty senior graduate nurse anesthesia students individually
participated in a high fidelity simulated anesthesia scenario that also included 3-4 live actors.
Post-simulation each student self-evaluated NTS performance using the ANTS tool and this
evaluation was compared to faculty evaluation of student performance using the ANTS tool.
Written perceptions regarding using the ANTS tool were collected from students and faculty and
transcribed verbatim. Data sources included demographic data, ANTS tool numeric score data
from both faculty and participants, and written comments from both faculty and participants using
the ANTS tool. Significant correlation between participant and faculty ratings was found in one
ANTS tool category, “Situation Awareness.” Two elemental variables also showed significant
correlations, anticipating in the “Situation Awareness” category and re-evaluating in the
“Decision-Making” category. Three major themes emerged from these comments by students and
faculty: adapting to changes, challenges in communicating, and hypervigilance. Participant
subthemes emerged within the themes adapting to changes (feeling rushed) and challenges in
vi
communicating (feeling alone). Other themes were identified unique to participant comments
(uncertainty) and faculty comments (seeing the big picture). Comparing student ANTS tool self-
evaluation to faculty evaluation of students’ simulated performance re-affirmed that students do
not have accurate insight into their own clinical performance and need faculty input. This study
re-confirmed that hypervigilance exists and is a natural part of learning in simulation. This study
also revealed that students need more practice in communication using dynamic simulated
scenarios. These scenarios brought out that students feel alone and use the surgical drape as a
physical and emotional barrier. The ANTS tool should be investigated further using high fidelity
simulation and could be useful in evaluating clinical NTS performance in the clinical milieu.
vii
TABLE OF CONTENTS Page
DEDICATION ....................................................................................................... iii
ACKNOWLEDGMENTS ....................................................................................... iv
ABSTRACT ............................................................................................................. v
LIST OF TABLES ................................................................................................... x
LIST OF FIGURES ................................................................................................. xi
Chapter
I. PROBLEM OF STUDY ....................................................................................... 1
Rationale for Study ........................................................................................... 4 Conceptual Framework ..................................................................................... 5 Kolb’s Experiential Learning Theory ........................................................... 5 Research Question(s) ........................................................................................ 8 Definition of Terms .......................................................................................... 9 Limitations...................................................................................................... 10 Summary ........................................................................................................ 12
II. REVIEW OF LITERATURE ............................................................................. 13
NTS Conceptual Framework Evolution: Crew Resource Management .......... 13 Anesthesia Crisis Resource Management ....................................................... 16 Anaesthetists’ Non-technical Skills Tool Development .................................. 19 Non-technical Skills Beyond Anesthesia ........................................................ 24
Self-assessment of Competence / Performance ........................................... 28 Non-technical Skills & Self-Evaluation ...................................................... 31
Summary ....................................................................................................... 31
III. PROCEDURE FOR COLLECTION AND TREATMENT OF DATA .............. 39
Setting .......................................................................................................... 41 Population and Sample ................................................................................. 41 Protection of Human Subjects ....................................................................... 42
viii
Instrument .................................................................................................... 42 Validity ........................................................................................................ 43 Reliability ..................................................................................................... 44 Data Collection ............................................................................................. 45 Pilot Study .................................................................................................... 48 Treatment of Data ......................................................................................... 48
IV. ANALYSIS OF DATA .................................................................................... 50
Description of Sample .................................................................................. 51 Findings ....................................................................................................... 53 Research Question One ............................................................................. 53 Research Question Two ............................................................................ 54 Research Question Three .......................................................................... 59 Summary of the Findings .............................................................................. 59
V. SUMMARY OF THE STUDY .......................................................................... 71
Summary ................................................................................................... 71 Discussion of the Findings ......................................................................... 72 Research Questions One and Two .......................................................... 73 Research Question Three ........................................................................ 76 Theoretical Framework Connection ........................................................ 86 Study Limitations ................................................................................... 89 Conclusions and Implications ...................................................................... 90 Recommendations for Further Study ........................................................... 92
REFERENCES ....................................................................................................... 94
APPENDICES ...................................................................................................... 112
A. ANTS Tool Front Page ........................................................................... 112
B. ANTS Tool Back Page ............................................................................ 114
C. Evolution of Crew Resource Management ............................................... 116
D. CRM Threat and Error Model .................................................................. 118
E. Anesthesia Crisis Resource Management Key Points ............................... 120
F. Annotated Bibliography of NTS Studies .................................................. 122
ix
G. Simulated Scenario Objectives and Key Events ....................................... 129
H. Demographic Questionnaire .................................................................... 134
I. TWU IRB Approval Letter ...................................................................... 136
x
LIST OF TABLES Table Page
1. Summary of Demographic Data for Gender and Age ......................................... 52 2. Summary of Demographic Data for Nursing Experience .................................... 52 3. Summary of Spearman’s Rank-Order Correlation for Participant and Faculty Evaluations at the Categorical Level .................................................................. 54 4. Summary of Spearman’s Rank-Order Correlation or Participant and Faculty Evaluations for Task Management Category at the Elemental Level .................. 55 5. Summary of Spearman’s Rank-Order Correlation for Participant and Faculty Evaluations for Team Working Category at the Elemental Level ........... 56 6. Summary of Spearman’s Rank-Order Correlation for Participant and Faculty Evaluations for Situation Awareness Category at the Elemental Level ............... 57 7. Summary of Spearman’s Rank-Order Correlation for Participant and Faculty Evaluations for Decision-Making Category at the Elemental Level .................... 58 8. Qualitative Analysis: Summary of Participant and Faculty Theme Emergence .. 67
xi
LIST OF FIGURES
Figure Page 1. Kolb’s Learning Theory Applied to Study Protocol ......................................................... 6 2. Kolb’s Learning Theory Applied to Study Results ......................................................... 88
1
CHAPTER I
PROBLEM OF STUDY
Eighty percent of anesthesia adverse incidents have been traced back to human error
(Koetsier, Boer, & Loer, 2011). This statement has been supported by the historic, “To Err is
Human” Institute of Medicine (IOM) report documenting the scope of preventable errors in
healthcare. The IOM report reported that over 90,000 individuals experienced adverse outcomes
that could have been prevented. The IOM also asserted “anesthesia is an area in which very
impressive improvements in safety have been made” (Kohn, Corrigan, & Donaldson, 2000, p.
32). Safety in healthcare has remained an endless process where improvements in error
prevention, education, and training have yet to be definitively developed (Rall, Gaba, Howard, &
Dieckmann, 2010).
Simulation in healthcare has emerged as one of the educational methodologies supported
by the IOM as well as the Agency for Health Care Research and Quality (AHRQ) (Aebersold &
Tschannen, 2010; Nishisaki, Keren, & Nadkarni, 2007). Integration of simulation into all levels
of nursing education has been recognized to provide no harm to patients and to immerse the
learner in a virtual, experiential learning environment (Aebersold & Tschannen, 2013; Aggarwal
et al., 2010). The report card for simulation as applied to patient safety supports training and
education using simulation as an essential strategy to improve patient safety (Nishisaki et al.,
2007). Simulation can promote competencies of clinical expertise, communication, and
collaboration, which include many aspects of non-technical skills (Aebersold & Tschannen, 2010;
Aggarwal et al., 2010; Dedy, Bonrath, Zevin, & Grantscharov, 2013; Gaba, 2000, 2004; Joint
Commission International Center for Patient Safety, 2009; Nishisaki et al., 2007; Wunder, 2016).
2
The role of human factors in anesthesia adverse incidents is increasingly acknowledged.
However, little attention has been paid to the behavioral components of safety in healthcare.
Safety research in high-reliability industries such as aviation has clearly demonstrated that the
causes of many accidents can be attributed to deficiencies in non-technical skills rather than a
lack of individual technical expertise (Patey, Flin, Fletcher, Maran, & Glavin, 2004). Non-
technical skills (NTS) including communication, teamwork, decision-making, and situational
awareness have been identified as important in high-risk industries such as industrial psychology,
aviation, nuclear power, and air traffic control, and have emerged as important safety
competencies in anesthesia (Flin, Patey, Glavin & Maran, 2010). NTS education and training is
virtually non-existent in current nurse anesthesia graduate curricula. Studies exploring anesthesia
education have demonstrated that NTS can be developed through simulated anesthesia
experiences (Gale et al., 2010; Yee et al., 2005; Zausig et al., 2009). Using andragogical
approaches appropriate for anesthesia learners and simulated scenarios emphasizing NTS could
be useful to the development and management of a safer anesthesia environment. NTS training
emphasizes self-awareness and self-evaluation of one’s own behavior in stressful clinical
situations offering insight for self-improvement and professional growth.
The Anaesthetists’ Non-Technical Skills (ANTS) tool offers a method to evaluate NTS,
improving self-awareness and quality of feedback to students during simulation and debriefing
(Patey et al., 2005). Observed performance of anesthesia practice using high fidelity simulation
and the ANTS tool could provide valuable information to nurse anesthesia students regarding
their development of NTS, a key aspect of patient safety.
This study focused on NTS evaluation for nurse anesthesia students. The ANTS tool was
used for student self-evaluation of NTS performance and was compared to faculty evaluation of
3
student NTS performance as a benchmark for validity of student self-evaluation (Appendix A and
B). Written student comments regarding the use of the ANTS tool were also compiled and
examined qualitatively to evaluate the value of this novel curricular approach.
Safety in anesthesia practice is comprised not only of tangible competencies (e.g.,
cognitive ability, critical reasoning, technical skills), but also tacit components such as NTS.
These tacit components are difficult to define, communicate, and evaluate. The ANTS system is
a tangible, structured tool and could be useful to identify strengths and weaknesses in NTS when
used by both students and faculty. Comparing faculty observations of student NTS and students’
self-evaluation of NTS after simulated experiences adds knowledge regarding the value of
different teaching/learning evaluation methods and impact on student learning (Arora et al.,
2011). Self-evaluation as a learning tool allows the learner to examine one’s own clinical and
professional performance with faculty guidance in order to gain a more accurate self-perspective
for future professional development (Pandy et al., 2008; Peyre, MacDonald, Al-Marayate,
Templeman, & Muderspach, 2010; Ward et al., 2003). Reflective capacity and accurate self-
evaluation are increasingly described as essential attributes of competent healthcare professionals
(Epstein, 1999; Epstein & Hundert, 2002; Mann et al., 2009; Schön, 1983; Ward et al., 2003). As
professional identity is developed during education and training, certain aspects of learning
require an understanding of one’s own knowledge, attitudes, values, and clinical expertise.
Reflection and self-assessment using simulation could offer an explicit approach to integrating
and evolving these professional attributes (Epstein, 1999; Mann et al., 2009). Exploring and
evaluating nurse anesthesia students’ NTS in a high fidelity simulated environment, using both
faculty observation and self-evaluation could positively impact anesthesia education, by
4
ultimately helping to develop a more mindful practitioner with more accurate self-understanding
and self-evaluation (Kanthan & Senger, 2011).
Rationale for the Study
Interest in the value of NTS in anesthesiology began in the 1990s with an impetus to
make anesthesia safer. Gaba and colleagues began to develop the foundation for crisis resource
management, using the aviation industry triumphs in safety as a foundation (Gaba, Fish &
Howard, 1994; Helmreich, 2000). Non-technical skills framework emerged from crisis resource
management in the 2000s and is continuing to evolve in several other healthcare specialties (e.g.,
surgeons and scrub technicians) (Fletcher, McGeorge, Flin, Glavin, & Maran, 2002; Hull et al.,
2012; Reader, Flin, Lauche, & Cuthbertson, 2006).
This growing awareness and interest of the value of NTS is now permeating the nurse
anesthesia profession (Larsson & Holmstrom, 2013; Lyk-Jensen, Jenseon, Spanager, Dieckmann,
& Ostergaard, 2014; Wunder, 2016). Therefore an impetus exists to become familiar with and
study NTS. High fidelity simulation (HFS) is the perfect environment to observe and evaluate
NTS, because complex clinical situations can be readily created, repeated, and debriefed.
Therefore graduate nurse anesthesia students enrolled in HFS curricula and exposed to NTS are a
logical cohort to study. Exposure to NTS and the ANTS tool in the simulated environment may
help graduate nurse anesthesia students understand how their own human factors affect
performance. This study of NTS and HFS is valuable to nursing and generates new science that
can be eventually explored in other nursing specialties. An improved understanding of NTS
observation and evaluation, as well as practice in self-evaluation in the simulated environment
may lead to higher quality advanced simulation curricula that could translate into better prepared
and safer nurse anesthetists.
5
Self-reflection can increase depth of understanding (Mann et al., 2009; Ward et al.,
2003). Self-reflection using the ANTS tool, combined with high fidelity simulation and
debriefing, has the potential to enhance the awareness of the essence of NTS (Najjar, Lyman, &
Miehl, 2015). Self-reflection is often nebulous and unstructured, but the ANTS tool provides a
structured rubric that could be an important foundation for professional growth and development
(Kanthan & Senger, 2011). Learning effectively from one’s experience is critical in developing
and maintaining competence throughout a practice lifetime. As profession-specific knowledge,
attitudes, values, and clinical expertise develop self- reflection offers an explicit approach to
weave these attributes into global professional competence (Mann et al., 2009). Therefore, an
active approach to learning linking new knowledge to existing knowledge is vital. High fidelity
simulation along with self-evaluation and debriefing offers such an approach and needs to be
explored.
Conceptual Framework
Experiential learning has been an integral part of high fidelity simulation curricula.
Therefore, Kolb’s experiential learning theory served as the conceptual framework for this
research protocol. Kolb emphasized the importance of adult learning through immersion and
experience. Post-scenario self-reflection utilizing the ANTS tool and debriefing is also an
important part of high fidelity simulation curricula and fits well with Kolb’s experiential learning
theory.
Kolb’s Experiential Learning Theory
Learning is a continuous process in Kolb’s experiential learning theory and is depicted
graphically as a dynamic cyclic structure where knowledge is created through experience. Kolb’s
learning theory is comprised of two dimensions that exist as a continuum: 1) grasping and
6
perceiving (vertical axis), and 2) transformation and processing (horizontal axis) (Kolb, 2015).
Four quadrants are formed representing four different types of experiential learning processes: 1)
concrete experience, 2) reflective observation, 3) abstract conceptualization, and 4) active
experimentation (Kolb, 2015). Moving clockwise through the quadrants starting at the upper
right it can be observed that concrete experiences are the basis for observations and reflections.
Reflections are distilled into abstract concepts from which new implications for action can be
drawn (Kolb, 2015).
High fidelity simulation curricula provide learning experiences reflecting core aspects of
Kolb’s experiential learning theory. The simulation environment evolves in real-time, immersing
the learner in a realistic environment, requiring the learner to reflect, analyze, and act as the
scenario unfolds. The learner moves through all four learning processes with each simulated
experience so that new knowledge is implemented and the cycle is restarted, but at a different
level. Therefore, Kolb’s cycle is better represented as a spiral-shape as depicted in Figure 1
(Armstrong & Parsa-Parsi, 2005; Kolb, 2015).
Figure 1. Kolb’s Theory Applied to Current Research Protocol (Kolb, 2015)
7
The cycle begins as the learner derives meaning and motivation regarding the simulation
environment and curricula through orientation and previous clinical experience. Nurse
anesthesia students are oriented to the simulated environment within the first few weeks of the
program, and reoriented to the high fidelity simulation environment for the advanced simulated
scenarios curricula that are a part of this research protocol. Orientation provides personal
meaning and motivation, which is the essence of this first learning process. Nurse anesthesia
students are adult learners and enter the nurse anesthesia program with previous educational and
clinical experiences. Reflecting, articulating, and using knowledge they have already learned and
experienced enables the graduate nurse anesthesia student to progress to the next learning process
in Kolb’s experiential learning theory (Armstrong & Parsa-Parsi, 2005; Kolb, 2015).
Acquisition of new knowledge and concepts and analyzing the learning experience is
important for progression through Kolb’s experiential learning theory. Nurse anesthesia students
will have acquired new knowledge prior to experiencing the advanced simulated environment
having completed didactic and clinical curricula. Acquisition and assimilation of new knowledge
enables nurse anesthesia students integrate and apply this learned knowledge with the simulated
experience (Armstrong & Parsa-Parsi, 2005; Kolb, 2015).
Practical application and a shift from thinking to doing is the next learning process
related to Kolb’s experiential learning theory. High fidelity simulation embodies this experiential
learning process as students individually apply knowledge and concepts to advanced simulated
scenarios. This quadrant of Kolb’s experiential learning theory is the most relevant to this
research protocol--theory moving to practice (Armstrong & Parsa-Parsi, 2005; Kolb, 2015).
Kolb’s experiential learning process cycle culminates by using synthesis and extending
the learned knowledge to future experiences. Self-reflection of the simulated experience through
8
ANTS tool utilization and structured debriefing allow for synthesis of the entire learning
experience. The ANTS tool offers a structured method to self-evaluate non-technical skills and
behaviors. The chance to incorporate knowledge, skills, and non-technical behaviors into future
clinical experiences extends the process of experiential learning process to a different level than
before. The cycle restarts by returning to the first quadrant through reflection (Armstrong &
Parsa-Parsi, 2005; Kolb, 2015).
Synthesizing one’s own performance can help bring forward skills that have never been
explicitly identified, described, or taught using traditional nurse anesthesia curricula, but are
described within the ANTS tool (Fletcher, Flin, & McGeorge, 2003a). The ANTS tool used
within Kolb’s conceptual framework can enhance the tacit part of anesthesia practice returning
the learner to the first quadrant, but with new experiences, therefore placing the learner at a
different vertical point on the spiral trajectory.
The following assumptions form the basis for this study:
1. NTS represent the tacit part of anesthesia practice.
2. NTS are observable.
3. Knowledge is created through the transformation of experience.
4. Learning is facilitated through self-awareness.
Research Question(s)
The study examined 3 research questions:
1. Is there a relationship between graduate nurse anesthesia students’ self-evaluation of
non-technical skills at the categorical level after participating in a high fidelity
simulation anesthesia scenario and faculty evaluations of student NTS at the
categorical level using the ANTS tool?
9
2. Is there a relationship between graduate nurse anesthesia students’ self-evaluation of
NTS at the element level after participating in a high fidelity simulation anesthesia
scenario and faculty evaluations of student NTS at the element level using the ANTS
tool?
3. What are student and faculty written perceptions of NTS using ANTS during high
fidelity simulation?
Definition of Terms
Anaesthetists’ non-technical skills are conceptually defined as day-to-day observable
behaviors of anesthesia providers in the operating room environment not directly related to the
use of medical expertise, drugs, or equipment, but encompassing interpersonal skills (Fletcher,
Flin, & McGeorge, 2003a) including task management, team working, situation awareness, and
decision-making. Anaesthetists’ non-technical skills are measured by the ANTS behavioral
marker tool (Gaba et al., 1998; Fletcher et al., 2003a; Flin, Glavin, Maran, & Patey, 2012, p. 3).
“Task Management” is conceptually defined as observable “skills for organizing
resources and necessary activities to achieve those goals including four specific elements: 1)
planning and preparing, 2) prioritizing, 3) providing and maintaining standards, and 4) identifying
and utilizing resources” (Flin et al., 2012, p. 8). “Task Management” is operationally defined as
1) score on the ANTS tool task management subscale and 2) the narrative comment data on the
ANTS tool task management subscale.
“Team Working” is conceptually defined as observable individual’s “skills for working
in a group to ensure effective joint task completion and team-member satisfaction including five
specific elements: 1) coordinating activities with team members, 2) exchanging information, 3)
using authority and assertiveness, 4) assessing capabilities, and 5) supporting others” (Flin et al.,
10
2012, p. 10). “Team Working” is operationally defined as 1) the score on the ANTS tool team
working subscale and 2) the narrative comment data on the ANTS tool team working subscale.
“Situation Awareness” is conceptually defined as observable “skills for developing and
maintaining overall awareness of the environment based on observing all relevant aspects of the
operating room environment, understanding what they mean and thinking ahead about what could
happen next” (Flin et al., 2012, p. 12). Three specific elements include 1) gathering information,
2) recognizing and understanding, and 3) anticipating (Flin et al., 2012). “Situation Awareness”
is operationally defined as 1) the score on the ANTS tool situation awareness subscale and 2) the
narrative comment data on the ANTS tool situation awareness subscale.
“Decision-Making” is conceptually defined as observable “skills for reaching a judgment
to select a course of action or make a diagnosis about a situation in both normal situations and
time-pressured crisis situations” (Flin et al., 2012, p. 13). Three specific elements include 1)
identifying options, 2) balancing risks and selecting options, and 3) re-evaluating (Flin et al.,
2012). “Decision-Making” is operationally defined as 1) the score on the ANTS tool Decision-
Making subscale, and 2) the narrative comment data on the ANTS tool Decision-Making
subscale.
Limitations
Limitations exist in this research protocol related to study design, and the choice of using
a non-experimental mixed-methods design in the form of a feasibility study. First, the study
cohort available was limited to 30 senior nurse anesthesia students in the nurse anesthesia
program. This smaller sample size limits generalizability of study findings. Other published
studies using the ANTS tool and similar in design to this study also use smaller sample sizes of
20-30 participants (Arora et al., 2011, Yee et al., 2005, Wunder, 2016; Zausig et al., 2009).
11
Second, no other studies can be found using self-evaluation and the ANTS tool in nurse
anesthesia students. Third, the novel use of the ANTS tool is supported by the authors of the tool
and therefore supports the rationale of the study design.
Other limitations exist related to using a high fidelity simulated environment, and the
ANTS tool itself. Evaluation of ANTS using a high fidelity simulated environment is an
emerging science. Several known simulation environmental factors currently exist that diminish
generalizability of data collected involving high fidelity simulation. (Fletcher et al., 2003b). First,
it is difficult to translate poor simulated performance to poor clinical performance, however there
is evidence to support using simulation to identify weaknesses in clinical performance (Fletcher
et al., 2002; Reader et al., 2006; Yee et al., 2005; Zausig et al., 2009). Second, a limited number
of valid and reliable tools exist to evaluate simulated performance of any kind, so there is limited
published experience with the ANTS tool used for self-evaluation of NTS performance in this
research protocol (Flin et al., 2012).
Several limitations have emerged regarding the difficulty in using the ANTS tool itself.
Individuals using the tool must have previous experience and familiarity with the tool in order to
accurately evaluate observable NTS (Flin et al., 2012). Some described behaviors within the tool
are not always present in every clinical setting, limiting feedback to the simulation participant.
Another limitation of the ANTS tool is that observed communication is not represented
as a category within the ANTS taxonomy. Communication in the ANTS tool pervades all of the
categorical behavioral markers as a means of indicating that the skill is being demonstrated. The
authors express that it is the content of the communication that is more important (Fletcher et al.,
2003a). It is also more useful having communication a part of all categories when providing
12
feedback to students, because communication feedback can be given for each separate category
(Fletcher et al., 2003b).
It is also difficult at times to clearly separate each ANTS tool skill when observing
performance, causing some ratings to overlap (Zausig et al., 2009). Additionally the skills
included are only those that can be observed, through behavior, actions, or interactions with
others. Items such as self-presentation, stress management, and perspective are not represented
within the ANTS conceptual framework. The authors state that accuracy and usability will
improve when assessing performance if observation is the primary method of assessment (Flin et
al., 2012).
A fourth limitation specific to the ANTS tool is that the tool evaluates individual
behavior and does not distinguish skills for working with different scenarios (e.g., the surgical
team and discussing the anesthetic plan with the patient or dealing with a crisis situation). The
authors believe the ANTS tool represents NTS that are necessary for all aspects of performance,
and should not be separated or different for specific scenarios (Fletcher et al., 2003b).
A fifth limitation focuses on the ANTS taxonomy. The ANTS taxonomy is designed
around the notion of flexible leadership. Leadership is considered both a task and situation that is
team dependent. When evaluating performance using the ANTS tool leadership should be
considered contextually depending on the scenario. The overall concept of leadership using the
ANTS tool should be examined in all other skill elements.
Summary
Non-technical skills are increasingly recognized as vital to professional anesthesia
practice. Studies show that teaching and learning NTS could improve safety in the operating
room environment (Fletcher et al., 2003b; Yee et al., 2005; Zausig et al., 2009). The ANTS tool
13
was developed to identify and evaluate anesthetists’ NTS in the simulated environment. The
effectiveness of using the ANTS tool among nurse anesthesia graduate students has not yet been
published regarding how NTS are perceived by both faculty observing NTS and student self-
perceptions of their own NTS. Reflective practice has been described as an essential attribute of
competent healthcare professionals (Arora et al., 2011; Epstein, 1999; Epstein & Hundert, 2002;
Mann, Jordon, & McLeod, 2009; Peyre et al., 2010). Using the ANTS tool to evaluate nurse
anesthesia students’ NTS may provide insight into the development of professional skills that are
needed on a daily basis in the operating room environment.
14
CHAPTER II
REVIEW OF LITERATURE
This literature review presents the evidence surrounding the development of the
Anaesthetists’ Non-technical Skills (ANTS) conceptual framework including the ANTS tool.
Anaesthetists’ non-technical skills contain scientific underpinnings from aviation’s crew resource
management (CRM) as well as anesthesia crisis resource management (ACRM) (Fletcher, Flin &
McGeorge, 2003a). The ANTS tool emerged from the ANTS conceptual framework and has been
used to evaluate observed non-technical skills of anesthesia providers.
A structured search was conducted using PubMed, Cumulative Index to Nursing and
Allied Health Literature (CINAHL), and Cochrane databases. Search terms included “ non-
technical skills,” “non-technical skills AND anesthesia,” “anaesthetists’ non-technical skills,”
“non-technical skills AND self-evaluation,” and “non-technical skills AND self-reflection.” No
restrictions were placed on language or publication date. A hand search reviewing references
from included studies specific to NTS was also performed. The authors of the ANTS tool have
extensive information on their website sharing exactly how the tool was developed. These data
were used to extract key pieces of the ANTS tool developmental history. Scientific
underpinnings for the ANTS system included aviation’s crew resource management and
anesthesia crisis resource management as scientific underpinnings uncovering pertinent
information and history from well over 20 years ago.
NTS Conceptual Framework Evolution: Crew Resource Management
Experiences from aviation crew resource management have provided a solid foundation
applicable to anesthesia to prevent, detect, and learn from accidents (Flin, O’Connor, & Mearns,
15
2002; Salas, Edens, & Bowers, 2001). In the 1970s the aviation industry became motivated to
develop human factors training after a series of airplane accidents were attributed to low
assertiveness, leadership, fatigue, decision-making, and communication (Weick, 1990). Aviation
accident analyses, simulator research, and cockpit voice recordings revealed that unsafe flight
conditions were frequently related to failures in pilots and flight attendants non-technical
(cognitive and social) skills, rather than a lack of technical knowledge, flying ability, or aircraft
malfunction (Flin & Maran, 2004). This inability of the crew to work together triggered a
philosophical change in the aviation industry to focus on human factors training with specific
concentration on leadership and decision-making (Wagner & Ison, 2014).
The roots of CRM in the United States can be traced back to a 1979 National Aeronautics
and Space Administration (NASA) workshop called Resource Management on the Flightdeck
(Cooper, White, & Lauber, 1980) that examined psychological research into aviation accidents.
The research presented at this meeting identified human error as the root cause of the majority of
air crashes as failures of interpersonal communications, decision-making, and leadership. This
meeting coined the term for this training as Cockpit Resource Management.
The first comprehensive cockpit resource management program was initiated by United
Airlines in 1981 (Helmreich, Merritt, & Wilhelm, 2001). These early cockpit resource
management programs emphasized the need to change individual pilot leadership styles,
correcting deficiencies in individual behavior such as lack of assertiveness by junior pilots and
authoritarian behavior by captains. Subsequent studies from the mid-1980s endorsed the need to
address aspects of entire crew behavior rather than just pilot behavior, to enhance pilot and crew
coordination and communication. As a result a new generation of training emerged, known as
16
crew resource management (CRM), focusing on group dynamics, situation awareness, and stress
management (Kanki, Helmreich, & Anca, 2010).
Crew resource management has been described and defined several ways as the science
evolved. One CRM definition has been cited as “using all available resources including
information, equipment, and people to achieve safe and efficient flight operations” (Lauber, 1984,
p. 20). A more recent definition has described CRM as a “structured process to provide members
with the knowledge, skills, and attitudes needed to respond to highly demanding situations in a
competent manner that proactively seeks to minimize the risk of errors” (Salas et al., 1999,
p.164). A goal of CRM training is to enable the crew team to operate effectively without being
overwhelmed through the optimal use of all available resources (Kanki et al., 2010). These
resources can include other team members, equipment, time, effort, or attention.
Crew resource management training has demonstrated that it is possible to reduce human
factor errors and improve teamwork behaviors and attitudes within a unified setting through
simulation (Morey et al., 2002; Wiener, Kanki, & Helmreich, 1993). CRM validity related to
producing a culture of aviation safety has been shown by an increase in acceptance of CRM
concepts and behavioral change. However, it is difficult to directly link CRM to a decrease in air
accidents/deaths, because the overall accident rate is already so low and training programs so
variable that it is impossible to draw strong conclusions about the impact of training during a
finite period of time (Helmreich, Chidester, Foushee, Gregorich, & Wilhelm, 1990).
Crew resource management science has evolved through six generations of training over
the last forty years and is depicted in Appendix C (Federal Aviation Administration, 2013). The
sixth generation focuses on threats and errors that must be managed by crews to ensure flight
safety. A threat and error model used in this sixth generation CRM depicts the relationship of
17
latent and individual threats on error management and development of further error (Appendix
D). Current CRM training embraces a multitude of human factors including human-technology
interface, timely acquisition of information interpersonal activities, leadership, team formation,
problem solving, decision-making, and situation awareness (Kanki et al., 2010; Wagner & Ison,
2014). Currently, no standardized methodology exists for the development of CRM training, and
airlines have been able to develop tailored fleet-specific courses (Salas et al., 1999). However,
CRM training is mandated by the Federal Aviation Agency (FAA) and is now used by all major
international airlines (Flin et al., 2002).
Specific behaviors commonly assessed in CRM include 28 behavioral markers within six
broad categories. Six broad categories include 1) team management crew communications; 2)
situational awareness, decision-making; 3) automation management; 4) special situations; 5)
technical proficiency; and 6) overall observations. Twenty-eight behavioral markers that are
subcategories of the six broader categories include briefing, leadership/ followership,
communication, decisions, interpersonal relationships, group climate, crew self-feedback,
preparation, planning, vigilance, inquiry, advocacy, assertion, workload and distractions
(Helmreich & Merritt, 1998; Wagner & Ison, 2014).
Anesthesia Crisis Resource Management
Anesthesia crisis resource management (ACRM) emerged in the late 1980s extracting
core principles from aviation’s crew resource management to better understand decision-making
by anesthesiologists during crises (Gaba et al., 1994). Anesthesia practice has often had
analogous comparisons to commercial aviation, comparing three phases of flight (takeoff,
cruising, and landing) to anesthesia (induction, maintenance, and emergence) (Gaba, 2000; Toff,
2010). One quote has been used with both specialties, as having “hours of boredom punctuated
18
by moments of sheer terror” (Shaw & Bennett, 1915, p. 979). Anesthesia and aviation have many
similarities and are considered to take place in dynamic environments, containing complex
problems that are poorly structured, and often develop with little time to act (Gaba, Howard, Fish,
Smith, & Sowb, 2001).
ACRM is one of the first models in healthcare emphasizing the importance of human
factors related to individual and team aspects of patient care in crisis situations and is still used as
a model today for behavioral assessment in anesthesia crisis situations (Gaba, 2004). The heart of
ACRM curriculum and training is in the use of high fidelity simulation to recreate anesthesia
crisis scenarios along with post-scenario debriefing. Key ACRM principles are emphasized
within this curriculum and are divided into two broad categories: 1) decision-making and
cognition, and 2) teamwork and resource management. Specific behavioral markers within each
of these two broad categories are located in Appendix E.
ACRM as a framework to improve patient safety has gained face and content validity by
participating anesthesia providers using questionnaires and surveys (Holzman et al., 1995;
Howard, Gaba, Fish, Yang, & Sarnquist, 1992; Kurrek et al., 1996). Anesthesia providers
repeatedly perceive that ACRM has value and that it has the potential to improve performance in
crisis situations, as well as patient safety. ACRM training has been shown to improve trainee
confidence and self-efficacy as well, however no consistent correlation to clinical competence has
been shown (Baker, Gustafson, Beaubain, Salas, & Barach, 2005; Blum et al., 2004; Gaba, 2010;
Isaak & Stiegler, 2015; Issenberg, McGaghie, Petrusa, Lee, & Scalese, 2005; Navarro-Martinez et
al., 2015; Reznek et al., 2003).
Just as it has been difficult to demonstrate a direct link between crew resource
management training and fewer flight accidents, it has been also difficult to demonstrate that
19
ACRM training leads to improved clinical performance. Various scoring systems have been
developed and tested within a high fidelity simulation environment in order to assess ACRM
performance (Baker, Gustafson, Beaubain, Salas, & Barach, 2005; Chopra et al., 1994; DeAnda
& Gaba, 1991; Devitt, Kurrek, & Cohen, 1998; Gaba et al., 1998; Henrichs, Rule, Grady, & Ellis,
2007; Kurrek, Devitt, Cohen & Szalai, 1999; Murray, Boulet, Kras, McAllister, Cox, 2005). The
plethora of scoring systems lies, in part, in the complex nature of clinical behavior and
competence, especially in crisis situations. Clinical experience, type of clinical crisis, and
specific curricular goals of the ACRM training create different scoring systems. Currently, no
standard measurement of ACRM clinical performance that translates into actual clinical practice
exists, so it is not known how well performance in a simulated environment predicts performance
during real crises.
High fidelity simulation (HFS) has been commonly used to teach ACRM. High fidelity
simulation has been described as a teaching/learning strategy or technique that mirrors or
amplifies real clinical situations using pre-structured scenarios that unfold in real-time (Altman,
Clancy, & Blendon, 2004; Baker et al., 2015; Gaba et al., 1994; Gaba, 2000; Murray et al., 2005;
Murray et al., 2007; Wunder, 2016). Scoring systems used in the HFS environment to teach
ACRM vary among each other depending on the skill level of the individual being evaluated and
the focus of the ACRM curricula (Baker et al., 2015; Gaba et al., 1998; Wunder, 2016). Scoring
has been conducted using discrete (performed or not performed), ranges of values. Overall
scoring has been documented as a numeric sum of several performance items or a more nominal
approach (pass or fail) (Murray et al., 2005; Murray et al., 2007).
Congruent, supportive evidence related to development of quality ACRM scoring
systems has been published and has helped establish a common denominator to improve ACRM
20
scoring systems validity and reliability. First, consensus expert opinion was gained using the
Delphi technique to establish content, face, and construct validity (Forrest, Taylor, Postlethwaite,
& Aspinall, 2002). Second, retrospective video review of simulated ACRM performance by
evaluators has improved evaluator accuracy (Devitt, Kurrek, Cohen, & Cleave-Hogg, 2001; Gaba
et al., 1998; Morgan & Cleave-Hogg, 2000; Murray et al., 2005; Murray et al., 2007; Schwid et
al., 2002; Wunder, 2016). Third, inter-rater reliability among evaluators has tended to be higher
for technical ratings than for behavioral ratings (Forrest et al., 2002; Gaba et al., 1998; Moorthy,
Munz, Adams, Pandy & Darzi, 2006; Wunder, 2016). This has been attributed to the more
tangible aspect of technical skill evaluation (Baker et al., 2015; Devitt et al., 2001; Gaba et al.,
1998). Fourth, establishing evaluator consensus on scoring items where evaluators disagreed has
helped improve the overall effectiveness of evaluation tools. Fifth, evaluator practice using the
tool has improved accuracy of the evaluation tool and inter-rater reliability (Baker et al., 2015;
Forrest et al., 2002; Morgan & Cleave-Hogg, 2000; Schwid et al., 2002; Wunder, 2016).
Observing and evaluating ACRM performance has allowed different behaviors, both technical
and nontechnical to evolve, and be assessed (Gaba et al., 1998; Morgan & Cleave-Hogg, 2000;
Wunder, 2016).
Anaesthetists’ Non-technical Skills Tool Development
Anesthetists’ non-technical skills evolved from a workgroup of industrial psychologists
and anesthetists from Aberdeen, Scotland who aimed to develop a taxonomy of ANTS that was
characteristic of experienced anesthetists’ performance (Fletcher et al., 2003b). This group
acknowledged that anesthesia was a high-reliability environment with dynamic situations that
unfold quickly involving multiple individuals, where goals continually shift and problems are
often ill structured (Fletcher et al., 2002). This group also demonstrated through an extensive
21
literature review that non-technical skills had an important influence on anesthesia practitioner
performance, potentially affecting patient safety (Fletcher et al., 2002; Fletcher, Flin, &
McGeorge, 2003b). The goal of developing a tool targeting the assessment of ANTS was to
maximize the understanding of ANTS performance measures so these could be infused into the
education and training of anesthesia providers. Therefore the ANTS tool was represented by a
scoring system that included both observable technical and non-technical skills to explain the
day-to-day behaviors and attributes of good anesthetists as well as those behaviors and attributes
of poorly practicing anesthetists.
Anaesthetists’ Non-technical Skills has been developed based on three main activities: 1)
observing anesthesia providers in the operating room (OR), 2) reviewing existing
anesthesia/aviation behavioral marker systems (i.e., CRM, ACRM), and 3) interviewing
practicing anesthesia providers. The role of the anesthesia provider in the operating room is
three-fold: 1) to ensure amnesia, 2) to provide optimal operating conditions for the surgeon, and
3) to ensure patient safety (Fletcher et al., 2003c). This requires constant vigilance and continual
manipulation and reassessment of the anesthesia plan. The anesthesia plan is manipulated in
conjunction with information obtained from sophisticated equipment, information technology,
and physiological responses to drugs, surgical procedure and other issues that may arise during
the procedure. Acute crises may occur requiring interaction and assistance from other members
of the OR team. The description above, as part of the first activity in the development of ANTS,
demonstrates that anesthesia providers clearly do not work in isolation in the OR, and there is a
high level of interdependency within the OR team. Therefore, good non-technical skills may be
just as important to anesthesia providers as good clinical skills (Fletcher, Flin, & McGeorge,
2003b).
22
The second activity undertaken to develop the ANTS tool was to review existing NTS
behavioral marker systems to ensure comprehensive identification of these behaviors and
attributes (Fletcher et al., 2003b). One unifying factor observed during the search for other
behavioral systems markers was that other existing lists/frameworks related to anesthesia (i.e.,
Anesthesia Crisis Resource Management) had been derived from the aviation industry (i.e., Crew
Resource Management, NOTECHS, LOFT, LOE). However, none of the existing systems had
been tested to ensure validity. Two main reasons were identified for the lack of validity testing
for the other behavioral marker systems: 1) used more as a framework for training rather than a
true assessment system, 2) focused on crisis management rather than routine behaviors (Fletcher,
Flin, & McGeorge, 2003a). Therefore, developing a tool that could actually be tested for validity
and reliability was a priority (Helmreich, Wilhelm, Kello, Taggart, & Butler, 1991; Fletcher et al.,
2003b). Specific aviation CRM training courses were used to develop the ANTS taxonomy (i.e.,
Line-Oriented flight training (LOFT) and Line Operational Evaluation (LOE)) as well as the
European CRM aviation training known as NOTECHS. NOTECHS offered a slightly different
approach than CRM because evaluation was placed on individual pilot NTS performance rather
than entire crew performance using a tangible behavioral marker system in the form of an actual
checklist (Helmreich et al., 1991).
NOTECHS framework divided skills into four main groups: 1) leadership and
management, 2) cooperation, 3) decision-making, and 4) situational awareness. The framework
of NOTECHS along with CRM concepts was integral in forming the hierarchal structure of the
ANTS tool (Fletcher et al., 2003b; Fletcher et al., 2003c). The four main groups identified were
considered to apply equally to anesthesia as to aviation. Each main group was divided into
further elements and observable behavioral markers. It is important to note that communication
23
was not included as a separate category because communication skills were important to all of the
four main groups (Fletcher et al., 2003c).
The third information gathering process for developing the ANTS tool included
interviewing 29 practicing anesthesia providers using a 3-part, semi-structured interview using
Cognitive Task Analysis (CTA) and the Critical Decision Method (CDM) (Klein, Calderwood &
MacGregor, 1989). First, participants were asked to describe a clinical situation that was
particularly challenging. Then participants were asked what skills or behaviors were considered
important for good anesthetists that might distinguish a novice from an expert anesthesia
provider. The last part of the interview was to have participants sort and rank the non-technical
skills identified from the existing behavioral rating systems. A list of 116 non-technical items
(skills, knowledge, and descriptors) in 8 broad groups emerged to help formulate the ANTS
taxonomy conceptual framework such as leadership, team working, communication, task
management and planning, situation awareness and recognition, decision-making, and personal
factors (Fletcher et al., 2003b).
The list from the existing anesthesia behavioral marker systems was cross-referenced
with the list from the interviews (Fletcher et al., 2003c). Criteria were established to ensure
suitability for use. The first criterion was that the framework needed to focus on skills and
behaviors that were observable. The second criterion was that the framework contains a
hierarchical structure using three description levels: category, element, and behavior. The third
criterion was that the framework was simple enough to use in the operating room environment
with minimal training. The fourth criterion was that the framework could easily be adopted as an
exemplar model to be used by not only anesthesia providers, but also other members of a team
24
that interact with anesthesia providers (Fletcher et al., 2003c). The final prototype ANTS
taxonomy has been included in Appendices A and B.
The ANTS framework and behavioral marker system has been published and
systematically evaluated and tested in five areas: 1) unit of assessment, 2) identification of CRM
skills and associated behavioral markers, 3) assessment method, 4) rater reliability, and 5) validity
(Fletcher et al., 2003b). These five areas represent the most prominent foci of published ANTS
evidence.
The unit of assessment for the ANTS tool was considered based on individual
performance rather than team performance because even though aviation CRM does emphasize
crew (team) performance, final assessment of pilots’ competence for licensure is at the individual
level and not the team level. Additionally, training individuals to develop effective NTS could
assist with individual performance and adaptation within different teams and different situations
(Fletcher et al., 2003b).
Identification of CRM skills for development of the ANTS framework was more
complex. Main CRM core skills seemed to be similar, but had been described differently
between aviation and anesthesia. Thirteen anesthesia programs and departments within eight
different countries participated in a survey to identify anesthesia behavioral markers systems
related to anesthesia performance. Six marker systems were identified to evaluate, compare, and
contrast to develop the ANTS tool: 1) CRM, 2) Kommunkations Status Team Performance
Indicators, 3) Operating Room Checklist, 4) ACRM Principle Markers, 5) Categories of behavior
defining aptitude for IOWA anesthesia residents, 6) Emergency Team Co-ordination Course
Teamwork Behavior Matrix (Fletcher et al., 2003a). From these six marker systems, cognitive
task analysis interviews were conducted with 29 expert anesthesia providers who were asked to
25
describe management of a particularly challenging case or crisis. The interview data were
analyzed using a grounded theory approach, to identify the NTS. A prototype of the ANTS tool
was refined by recoding interviews, reviewing anesthesia incident reports, and physical
observation in the operating room (Fletcher et al., 2003a).
Assessment methods used in the development and evaluation of the tool were limited to
the use of videotaped recordings of individual performance within a team using various simulated
anesthesia scenarios. Individual performance was assessed using the ANTS tool through direct-
videotaped observation of both individual observable actions and communication (Fletcher et al.,
2003a; Fletcher et al., 2003b; Morgan, Kurrek, Bertram, LeBlanc, & Przybyszewski, 2011;
Salvodelli et al., 2006; Yee et al., 2005; Zausig et al., 2009).
Inter-rater reliability (IRR) and validity is an especially important aspect of the ANTS
tool in order to be usable by different individuals during training of students with varying levels
of education and experience. Inter-rater reliability in most ANTS tool studies uses Intra-class
Correlation (ICC), comparing the variability of different ratings of the same individual to the total
variation across all ratings and participants (Fletcher et al., 2003b). Initial evaluative studies
using the ANTS system have suggested that it fulfills several aspects of validity (Fletcher et al,
2003b). Validity and reliability of the ANTS tool are addressed in more detail in Chapter III.
Non-technical Skills Beyond Anesthesia
Since initial development and testing, the ANTS tool and nature of NTS has continued to
be tested and explored beyond anesthesia practice. A combination of both technical and non-
technical skills has been established to be crucial for safe patient care, but the relationship
between these two aspects of anesthesia care has yet to be explained. An explorative, mixed
methods approach was conducted to explore correlation of NTS and technical skills in 27
26
anesthesia residents (Gjerra, Jepsen, Rewers, Ostergaard, & Dieckmann, 2015). Participants
attended a half-day course that consisted of a mix of didactic curricula, workshops, and
simulations related to TS and NTS and airway management. A modified ANTS tool was used for
the NTS portion and the Objective Structured Clinical Examination (OSCE) instrument was used
for the technical skills portion. No relationship was found between the NTS and TS. Good TS
were not always associated with good NTS performance. This finding was consistent with other
studies (Moorthy et al., 2006; Riem, Boet, Bould, Tavares, & Naik, 2012). The authors explained
that the lack of correlation might be due to the fact that some NTS are not present in all
anesthesia situations and that ANTS tool should be modified to fit each scenario. Technical skills
have been cited as more tangible to evaluate. NTS have been cited as easier to miss by
evaluators. An interesting finding in all of these studies was that technical skill was not
independent of NTS skill performance even though no definitive correlation existed (Gjerra et al.,
2015; Moorthy et al., 2006; Riem et al., 2012). The exact nature of this relationship has yet to be
addressed.
A recent systematic review examined the impact of NTS on technical performance in
surgeons rather than anesthesia providers with a hypothesis that NTS either enhances or
contributes to deterioration of surgeon technical performance. Several findings emerged: 1) No
evidence was found that poor communication negatively affects technical performance, 2) NTS
failures especially situational awareness are associated with a higher rate of technical errors, 3)
stress negatively impacts technical performance, 4) fatigue negatively affects performance, and 5)
technical performance feedback has a beneficial effect on task management (Hull et al., 2012).
The authors concluded that certain NTS (e.g., situational awareness) can have an effect on
technical performance and more robust tools need to be developed and tested.
27
Published evidence using the ANTS tool or NTS involving nurse anesthetists has been
scant. Only two qualitative studies and one quasi-experimental study are available. One was
conducted in Denmark and the other in Sweden, where the authors attempted to adapt the ANTS
tool to a specific national culture of anesthesia practice. The first study utilized semi-structured,
multi-disciplinary focus group interviews to identify NTS of Danish nurse anesthetists in the
operating room. The authors did not believe the ANTS tool applicable to Danish nurse anesthesia
practice “as is,” because of different scopes of practice between physician and nurse anesthetists
in Denmark (Lyk-Jensen et al., 2014). Overall, the original ANTS tool four main categories were
maintained (situation awareness, decision-making, task management, team working), but the
description of good and poor behavioral markers were morphed related to the differences in
professional and national culture. The authors concluded that other national cultures might need
to adapt the ANTS tool to make NTS evaluation more meaningful. The second study actually
used five focus groups of experienced anesthesia nurses (nurse anesthetists) to investigate how
excellent anesthetists (physician anesthetists) act and behave in the operating room (Larsson &
Holmstrom, 2013). Anaesthesia nurses were asked to participate because physician anesthetists
seldom have the opportunity to see other physicians at work, while the anaesthesia nurses often
work in parallel with the physician anesthetists. Six themes were found to describe good and poor
behavioral markers: 1) structured, responsible, and focused approaching tasks; 2) clear and
informative, briefing team about induction action plan; 3) humble to complexity of anesthesia,
admitting own fallibility; 4) patient-centered; 5) fluent in practical work without losing overview;
and 6) calm and clear in critical situations being able to change to a strong leading style.
A recent quasi-experimental study sought to determine the effect of an educational
intervention using ANTS concepts in 32 nurse anesthesia students and a high fidelity simulation
28
environment (Wunder, 2016). Each student experienced a high-risk low exposure anesthesia
scenario, watched a presentation related to ANTS, and then underwent a second anesthesia
scenario. Videotaped performance was evaluated by faculty using the ANTS tool. Non-technical
skills performance reflected by mean scores on the ANTS tool significantly improved. The
authors concluded that NTS are not acquired by experience, but are acquired through education
(Wunder, 2016).
Adapting the ANTS tool to specific national healthcare cultures and specialties has been
increasingly being studied (Spanager et al., 2013; Yule et al., 2008; Yule et al., 2006). Most
recently non-technical skills have become of interest to surgeons with an impetus to teach NTS to
surgical residents to potentially enhance patient safety in surgery (Hull et al., 2012). Therefore,
the same authors of the ANTS tool developed a taxonomy and behavioral rating system specific
to surgeons called the Non-technical Skills for Surgeons (NOTSS) (Yule et al., 2008; Yule et al.,
2006).
The emergence of the NOTSS tool has prompted several studies examining surgical
resident performance using high fidelity simulation and subsequent improvement in NTS
(Crossley, Marriott, Purdie, & Beard, 2011; Dedy et al, 2013; Heskin et al., 2015; Nguyen,
Elliott, Watson, & Dominguez, 2015; Yule et al., 2015). The impact of these studies has
highlighted several issues related to surgeons’ NTS. First, reinforcement of the concept that NTS
have been truly important to develop, and second, that NTS have been assessable (Crossley et al.,
2011). Third, simulation-based training appeared to be effective for improving NTS, and has
been often combined with procedural task training (Dedy et al., 2013; Nguyen et al., 2015).
Fourth training should begin early, often, and content adapted to level of training (Dedy et al.,
2013; Heskin et al., 2015; Yule et al., 2015).
29
Several limitations in these studies exist both inherent to the tool itself, as well as study
methodology. First, the ANTS tool requires evaluator training and/or experience using the tool.
No optimal time has been specifically studied or recommended. ANTS tool orientation has
ranged from 1 hour to 4 hours to an indefinite amount of time in order to achieve an IRR >0.9
(Fletcher et al., 2002; Fletcher et al., 2003a; Fletcher et al., 2003b; Morgan et al., 2011; Salvodelli
et al., 2006; Welke et al., 2009; Yee et al., 2005; Zausig et al., 2009). Second, the ANTS tool,
being a taxonomy consisting of a hierarchy of non-technical skills, has been considered complex
(Fletcher et al., 2002; Morgan et al., 2011). Some elements of the tool are more difficult to
observe, such as “Situation Awareness,” because this main category has been considered more of
a cognitive skill (Fletcher et al., 2003c). Future studies may reveal the need to cleave or change
some of the non-technical skills to create a more concise evaluation tool. Third, certain simulated
anesthesia scenarios may not include all categories and elements of the ANTS tool, either due to
the specific scenario or the participant behavior is too subtle and goes unnoticed by the evaluator.
This could affect the accuracy of the overall ANTS score (Fletcher et al., 2002). Fourth, the
accuracy in scoring the ANTS tool could be limited by not knowing where to set the boundaries
for each scaled category or element, especially with varied education and experience levels for
the anesthesia provider being evaluated (Fletcher et al., 2002). Lastly, the ANTS tool does not
directly address communication specifically in any main category or element, as the authors
believe it to permeate all categories and elements of the tool (Fletcher et al., 2002; Fletcher et al.
2003b). Therefore, communication specifically may be under-evaluated or unevaluated.
Overall, users of the ANTS taxonomy support the tool and adoption of this tool in
anesthesia curricula is increasing. The ANTS tool accurately represents the essence of
anesthesia non-technical skills. ANTS are important to teach and assess in both students and
30
anesthesia providers (Fletcher et al., 2002; Fletcher et al., 2003a; Fletcher et al., 2003b; Morgan
et al., 2011; Salvodelli et al., 2006; Welke et al., 2009; Wunder, 2016; Yee et al., 2005; Zausig et
al., 2009).
Anesthesia student NTS performance in the simulated environment has been shown to
significantly improve when using the ANTS tool to evaluate performance, supporting context
validity (Yee et al., 2005; Wunder, 2016). It is not known if this improved performance can be
translated into clinical practice. The ANTS tool has an acceptable inter-rater reliability from
studies assessing this aspect of the tool, with a greater correlation among the 4 major categories
than the sub-categorical elements (Fletcher et al., 2003a; Fletcher et al., 2003b; Morgan et al.,
2011; Wunder, 2016; Yee et al., 2005; Zausig et al., 2009). The ANTS tool requires further
research to establish more uses for the tool, among students and anesthesia practitioners.
Self-assessment of Competence / Performance
Accurate self-assessment of performance is fundamental to continued professional
competence and development of expertise (Gordon, 1991). Adult learners should have the ability
to accurately self-assess performance to facilitate additional learning and growth, especially
related to the philosophy related to life-long learning (Epstein & Hundert, 2002). Self-assessment
involves making judgments about one’s own learning and performance. The term self-assessment
encompasses two key elements: 1) identification and application of criteria or standards to be
applied to one’s work and 2) judging whether the extent of that work meets a certain criteria or
standard (Falchikov & Boud, 1989).
Self-assessment can be a formative or summative form of evaluation and has been
embraced by some healthcare educational programs to facilitate self-reflection and improve
performance (Davis et al., 2006). Formative evaluation can occur because self-assessment
31
contributes to the learning process and assists learners to recognize areas for improvement.
Summative evaluation can occur when the student determines or evaluates own performance in a
given area or more formally, may contribute to one’s own specific grade awarded (Boud &
Falchikov, 1989; Falchikov & Boud, 1989).
Self-assessment of performance and correlation to faculty observed measures of
competence have been studied with inconsistent results (Arora et al., 2011; Brewster et al., 2008;
Davis et al., 2006; Moorthy et al., 2006; Pandey et al., 2008; Peyre et al., 2010; Ward et al.,
2003). Self-assessment of technical skills tended to correlate (e.g., laparoscopic skills, suturing)
between trainee and faculty or expert evaluator and correlation became stronger with level of
clinical experience (Pandey et al., 2008; Peyre et al., 2010; Ward et al., 2003). Non-technical
skills evaluation lacked significant correlation in all studies (Arora et al., 2011; Brewster et al.,
2008; Davis et al., 2006; Moorthy et al., 2006).
The disparity in self-evaluation correlation among technical and non-technical skills
remains to be thoroughly explained. Increased trainee experience could lead to increased self-
assessment accuracy regarding technical skills. Likewise novices tend to have a more inaccurate
perception of technical skills, perhaps due to lack of performance expectations, as well as real
clinical experience (Pandey et al., 2008; Peyre et al., 2010; Ward et al., 2003). Technical skills
are also more tangible and easier to observe and perceive by both trainee and faculty expert.
Non-technical skills assessments are less tangible. Faculty experts and trainee
perceptions may vary depending on differing expectations of each party. Non-technical skills
tend to demonstrate higher correlation in novice trainees than more experienced trainees
(Moorthy et al., 2006). This may be due to more congruent faculty and trainee expectations early
in education and training. Students gain competence and confidence at varying rates, therefore
32
faculty may have increased disparities in student expectations as experience is gained (Brewster
et al., 2008; Davis et al., 2006; Moorthy et al., 2006).
Limitations also exist among self-assessment studies. No consistent valid and reliable
tool was used among all of the studies evaluating accuracy of self-assessment (Arora et al., 2011;
Brewster et al., 2008; Davis et al., 2006; Moorthy et al., 2006; Pandey et al., 2008; Peyre et al.,
2010; Ward et al., 2003). This was predominantly due to the heterogeneous nature of
populations sampled. Each study examined a different trainee population with varying degrees of
experience. Training among faculty experts and trainees also varied among self-assessment
studies.
A variety of preparatory curricula were used to prepare participants for self-evaluation.
Didactic education was included in only three of the studies (Brewster et al., 2008; Peyre et al.,
2010; Wunder, 2016). All other studies used varying time frames of experiential practice in a
simulation lab as preparatory curricula prior to evaluation or used pre-recorded videos of
benchmark performance (Arora et al., 2011; Davis et al., 2006; Moorthy et al., 2006; Pandey et
al., 2008; Ward et al., 2003). Regardless of teaching methodology, self-assessment of technical
skills possesses increased correlation between participant and faculty expert over self-assessment
of non-technical skills.
Non-Technical Skills & Self-Evaluation
Non-technical skills have evolved to be recognized as necessary for professional growth
and development in healthcare (Fletcher et al., 2002; Fletcher et al., 2003a; Fletcher et al., 2003b;
Morgan et al., 2011; Salvodelli et al., 2006; Welke et al., 2009; Yee et al., 2005; Zausig et al.,
2009). Self-evaluation has long-been studied with similar importance placed on professional
growth (Davis et al., 2006; Epstein & Hundert, 2002; Falchikov & Boud, 1989; Gordon, 1991).
33
Combining these two foci as a focus of primary research has not been thoroughly studied until
recently with mixed results (Arora et al., 2011; Brewster et al., 2008; Davis et al., 2006; Moorthy
et al., 2006; Pandey et al., 2008; Peyre et al., 2010; Ward et al., 2003).
What is known regarding NTS is that self-assessment of technical skills correlates
between trainee and expert evaluator. The correlation strengthens with level of clinical experience
(Pandey et al., 2008; Peyre et al., 2010; Ward et al., 2003). It is also known that self-assessment
of performance does not correlate with expert observer assessment of performance in all studies
(Arora et al., 2011; Brewster et al., 2008; Davis et al., 2006; Moorthy et al., 2006). An annotated
bibliography specific to published studies examining NTS is located in Appendix F (Fletcher et
al., 2003b; Graham, Hocking, & Giles, 2010; Larsson & Holmstrom, 2015; Yee et al., 2005;
Zausig et al., 2009).
It is not known what exactly causes the disparities between participant and observer,
especially when NTS are involved. More research needs to be performed comparing NTS
evaluation between trainee and observer to further explore this gap. Therefore, this research
protocol intended to explore the relationship further between observed and self-evaluated NTS.
Summary
Patient care in the operative environment is complex requiring a certain skills beyond the
technical realm. This skill set has been compared to those needed within the aviation industry
(Gaba et al., 1994; Helmreich et al., 2001). Daily both anesthesia providers and pilots are
involved in complex situations with multiple individuals and teams where a need exists to act
rapidly and effectively in time-pressured situations, characterized by vaguely-structured problems
with incomplete feedback (Fletcher et al., 2002).
34
The ANTS framework developed and evolved from aviation crew resource management
and ACRM (Fletcher et al., 2003a). The ANTS tool has been tested for validity and reliability
and is increasingly used in the education and training of anesthesia providers to more accurately
assess, explore, and explain these individuals’ non-technical skill behaviors (Fletcher et al., 2002;
Fletcher et al., 2003a; Fletcher et al., 2003b; Morgan et al., 2011; Salvodelli et al., 2006; Yee et
al., 2005; Zausig et al., 2009). High fidelity simulation is the ideal environment in which to
evaluate ANTS because it provides a patient-safe learning environment, repetitive experiences,
and ability to self-reflect upon individual performance (Salvodelli et al., 2006; Yee et al., 2005;
Zausig et al., 2009).
35
CHAPTER III
PROCEDURE FOR COLLECTION AND TREATMENT OF DATA
This non-experimental mixed-methods study sought to determine if student self-
evaluation of ANTS was a valid form of anesthesia student evaluation of NTS. In order to
determine the validity of student evaluations, the study sought to identify the relationship between
graduate nurse anesthesia students’ self-evaluation of non-technical skills (NTS) after
participating in a high fidelity simulation anesthesia scenario and faculty evaluations of student
NTS using the ANTS tool. Faculty evaluations were used as the benchmark to determine
agreement among student and faculty ratings and thus student ability to self-evaluate. Cross
validation of quantitative data was achieved by qualitatively exploring student and faculty written
perceptions of performance using the ANTS tool during high fidelity simulation (Polit & Beck,
2012).
Four different anesthesia scenarios were used based on existing simulation curricula in
place at a large university school of nurse anesthesia. Specific scenarios included the following:
1) hypertension during emergence from general anesthesia, 2) laryngospasm during emergence
from general anesthesia, 3) refractory hypotension during general anesthesia, and 4)
intraoperative oxygen tank failure during general anesthesia. Specific scenario objectives and
events are located in Appendix H. These scenarios have been considered complex situations
where rapid and effective actions should be taken that are time-sensitive (Gaba et al., 1998).
Each simulated scenario incorporated actors to enhance curricular fidelity. Each student
experienced only one of these scenarios, as four students attended the simulation curricula at one
time.
36
ANTS behaviors were observed and evaluated using the ANTS tool Likert-type scale
located on the back side of the tool: 0 (behavior not observed or present), 1 (poor performance),
2 (marginal), 3 (acceptable), or 4 (good) (Appendix A and B). The specific behaviors evaluated
include categories and elements within the ANTS tool such as 1) “Task Management” (planning
& preparing, prioritizing, providing and maintaining standards, identifying and utilizing resources
2) “Team Working” (coordinating activities with the team, exchanging information, using
authority and assertiveness, assessing capabilities, and supporting others, 3) “Situation
Awareness” (gathering information, recognizing and understanding, and anticipating, and 4)
“Decision-Making” (identifying options, balancing risks and selecting options, and re-
evaluating), (Fletcher et al., 2003b). The participant wrote a numeric value reflective of either
observed or self-assessed performance next to each category and element on the ANTS tool using
the Likert-type scale.
Qualitative information was also collected from student and faculty written perceptions
and comments regarding using the ANTS tool. The ANTS tool provided a space to engage in
written comments adjacent to each category and subcategory on the tool. Graduate student
participants and faculty were asked to comment on behaviors and perceptions specific to each
category and element on the ANTS tool (Polit & Beck, 2012). Written qualitative data were
transcribed, sorted, and coded using descriptive and in-vivo coding techniques (Saldana, 2012).
Validity was addressed using member checking. Final themes and subthemes were shared with
all participants to validate interpretation of comments. Faculty checking was utilized to ensure
reliability of theme development. Validity and reliability techniques used added to qualitative
data quality and provided new understanding regarding ANTS tool usefulness.
37
Setting
The physical conditions consistently present during the course of this study included 1) a
classroom setting and 2) a high fidelity simulated environment located within the college of
nursing. The classroom setting consisted of one large oval table seating 16 individuals and was
equipped with video and wireless technology to view didactic curricula and simulation lab
activity. The simulation lab was an exact replica of a real operating room containing surgical
instruments, sterile drapes, gowns, hats, gloves, masks, and other adjuvant supplies to ensure
scenario fidelity. The simulation environment also included 3-4 live actors (Kassab et al., 2010).
A one-way mirror separated the simulated operating room (OR) from a control room where
faculty controlled the scenario and viewed student activity during scenarios.
Population and Sample
Eligible participants included graduate nurse anesthesia students who had completed 12
months of didactic curricula, and a minimum of 3 months of clinical practicum enrolled in a
clinical management course that consisted of high fidelity simulated experiences. Thirty
participants met criteria for inclusion in the study. As a result a target population was proposed
changing this study design into a non-experimental, mixed-methods study.
Protection of Human Subjects
Compliance with Texas Woman’s University (TWU) Institutional Review Board (IRB)
as well as Texas Christian University (TCU) IRB and current rules and regulations were
followed. TWU and TCU IRB approval was obtained prior to initiation of this research protocol.
Student participation was purely voluntary. Students were scheduled for this simulated clinical
management course as part of normal curricula. The faculty of record was also the researcher as
38
well. Student progression in the program was not dependent on this simulated performance. A
copy of the TWU IRB approval letter has been included in Appendix H.
Instrument
The instrument utilized in this research protocol was the Anaesthetists’ Non-technical
Skills (ANTS) tool, a quantitative method for non-technical skills evaluation in the clinical
environment. The instrument contained a behavioral marker system with a hierarchical structure
outlining four main categories: task management, team working, situation awareness, and
decision-making. These categories were subdivided into a total of 15 skill elements (Patey et al.,
2005).
The ANTS tool represented key non-technical skills needed by anesthesia providers on a
daily basis to provide safe, effective anesthesia care. The ANTS tool represented the final product
of development of a behavioral marker system focusing on NTS. The final ANTS tool has four
categories: 1) situation awareness, 2) decision-making, 3) task management, and 4) team working
with component elements and examples of good and poor behavior for each element. A behavior
rating scale was incorporated into the framework, to help quantify observable behaviors. A
description of what constitutes poor and good performance was described for each element and
category for the participant, and a paper handbook was available during all aspects of training and
self-evaluation (Flin et al., 2003b).
Validity
The Anaesthetists’ Non-Technical Skills tool validity has been demonstrated using high
fidelity simulation training environments (Fletcher et al., 2002; Fletcher et al., 2003b; Gale et al.,
2010; Salvodelli et al., 2006; Yee et al., 2005; Zausig et al., 2009). The ANTS tool was initially
tested for face, content, and construct validity among 50 Scottish anesthesia providers. Eighty-
39
four percent of participants perceived the key ANTS behaviors were displayed in the tool and
81% perceived the ANTS listed behaviors were observable, establishing the foundation for face
and content validity. The participants in this initial study of the ANTS prototype perceived that
the tool could be used during anesthesia training, and also believed the most important NTS were
captured within the tool. Participants also perceived that the ANTS tool addressed an important
area of anesthesia practice (Fletcher et al., 2003b).
Subsequent studies have been performed examining validity of the ANTS tool. Yee et
al.’s (2005) prospectively studied the effects of repeated exposure to ANTS using the ANTS
scoring system with 20 anesthesia residents managing three different simulated anesthesia
scenarios. A significant improvement in ANTS was demonstrated between the first and second
sessions, and the first and third sessions (all p < 0.005), but no significant improvement between
the second and third sessions demonstrating support for construct validity (Welke et al., 2009;
Yee et al., 2005). These results could have potentially given support to faculty with limited
curricular resources offering that even 1 or 2 high fidelity experiences would have the greatest
impact on learning NTS.
More recently, 67 practicing anesthesiologists were prospectively studied to determine
whether high fidelity simulation educational debriefing sessions would improve subsequent NTS
during ACRM scenarios using the ANTS tool. Previous studies using the ANTS tool had only
been performed using anesthesia residents. No significant improvement in non-technical skills
using the ANTS tool and debriefing sessions in between scenarios was detected after two sessions
among the cohort. Even though results were different in this recent study, support for the ANTS
tool face and content validity was deemed present (Morgan et al., 2011).
40
Reliability
ANTS tool reliability has been tested among evaluators observing individuals’ NTS and
between evaluators observing individual NTS and individual self-evaluation. Initial studies using
the ANTS tool exposed evaluators to 4 hours of orientation to the tool and yielded an ICC of 0.56
– 0.65 for the 4 main category levels, and an ICC of 0.55 – 0.67 at the element level. Within the
four main categories “Situation Awareness” had the lowest IRR scores, while “Task
Management” and “Team Working” had the highest level of agreement. The element level,
recognizing and understanding, had the lowest IRR scores, while utilizing resources had the
highest agreement (Fletcher et al., 2003b; Wunder, 2016). Subsequent studies have reported
similar IRR scores (Salvodelli et al., 2006). Yee and colleagues reported an overall single-rater
IRR averaged score for the ANTS tool of 0.64 and an IRR of 0.58 for the main category level.
Evaluators using the tool only received 1 hour of ANTS tool orientation. Yee et al. (2005),
reported a single rater ICC of 0.53 across the main categories level, and a single rater ICC of 0.50
across the subcategory elemental level for evaluators who had only received one hour of ANTS
tool orientation and training. Zausig et al. (2009), increased the training time to 3.5 hours and
yielded a single rater ICC of 0.96 for the overall ANTS tool averaged score. Morgan et al. (2011)
trained evaluators using the ANTS tool until the IRR correlations reached > 0.9, however, the
actual ICC in this study was low (0.44) as compared to previous studies, despite the initially high
correlation after training. This lower correlation was possibly due to deterioration of ANTS tool
evaluation skills over time or possible evaluator inattentiveness, as a result of having to review 59
videos over a course of one year (Morgan et al., 2011). Overall the inter-rater reliabilities of
evaluators using the ANTS tool were originally deemed acceptable, and further subsequent
studies further support the reliability of the ANTS tool, considering the different times allotted for
41
ANTS tool training (Fletcher et al., 2002; Fletcher et al., 2003a; Fletcher et al., 2003b; Morgan et
al., 2011; Salvodelli et al., 2006; Yee et al., 2005; Zausig et al., 2009).
Data Collection
This research study sought to determine graduate nursing students’ self-evaluation of
performance using an existing tool called the ANTS tool evaluating non-technical skills (NTS)
after a simulated anesthesia scenario. Validity of student self-evaluations was determined by
examining the degree to which the self-evaluations agree with the primary investigator’s
evaluation of non-technical skills. The simulated experience was already a part of the school of
nurse anesthesia graduate curricula.
All participants participated in a 1-hour interactive presentation related to the concepts
and components of the ANTS framework and tool prior to using the tool. All students practiced
using the ANTS tool as part of the interactive presentation by watching previously recorded
simulated vignettes. At the conclusion of the interactive presentation each student participated as
the primary anesthesia provider in a simulated clinical scenario. One of four scenarios was used:
1) hypertension during emergence from general anesthesia, 2) laryngospasm during emergence
from general anesthesia, 3) refractory hypotension during general anesthesia, and 4)
intraoperative oxygen tank failure during general anesthesia. Scenarios lasted approximately 10
minutes each. Specific scenario curricular objectives and events are available in Appendix G.
One faculty was the primary investigator (PI) observed participant performance through a
one-way glass mirror in the control room. At the conclusion of the simulated clinical scenario the
participant was escorted to a quiet office where self-evaluation using the ANTS tool was
performed. No time limit was placed on the self-evaluation period. The participant also had
access to a paper ANTS tool reference guide during the self-evaluation period. The ANTS tool
42
reference guide was an 8-page description of each of the 4 major categories and elements within
the ANTS tool.
A comment section existed for each elemental variable on the ANTS tool. Participants
were asked to comment on their own non-technical skills in this area. A debriefing session was
held after the ANTS tool self-evaluation was complete where technical and non-technical aspects
of performance were discussed in a non-threatening environment.
Detail related to the research procedure is as follows:
1. PowerPoint presentation focusing on the nontechnical skills (60 minutes)
a. Definition of affective domain of anesthesia (e.g., nontechnical skills, crisis resource
management)
b. Introduction of the complex clinical milieu of the perioperative environment
c. Importance of professional development of nontechnical skills
d. Introduction of the ANTS tool
e. Video vignettes and discussion of each of the four domains of the ANTS tool
f. Practice using the ANTS tool
g. Discussion, questions, and answers period
2. Written consent to participate (30 minutes)
3. Simulated experience (10 minutes)
a. One student at a time participated in a video-taped anesthesia-related simulated
scenario. The student could have called for help (second student) if needed. The primary
investigator ended the scenario after approximately 10 minutes had unfolded in the
scenario.
4. Post-simulation self-evaluation using the ANTS tool (30 minutes)
43
a. The primary student participating in the scenario completed a post-simulation
self-evaluation using the ANTS tool in a quiet empty classroom.
b. The primary investigator also used the ANTS tool to complete a post-simulation
evaluation of each student participating in the scenario at the same time as the
student participants were completing their self-evaluation.
5. A debriefing session was held with student participants, actors (local nurse anesthetists
trained for each scenario), and primary investigator using video feedback (30 minutes).
Pilot Study
The ANTS tool was pilot tested on 8 graduate nurse anesthesia students. Students
participated in a one-hour didactic presentation regarding non-technical skills provided by the
researcher and within 1-1.5 months attended a clinical simulation session lasting for
approximately 10 minutes. Immediately after the conclusion of a simulated session each student
completed a self-evaluation using the ANTS tool. The faculty also evaluated each student using
the ANTS tool. No significant relationship was found between faculty and student evaluation
using the ANTS tool (r = -.38; p=0.1). Qualitatively three themes were noted from written
comments: unexpected situation, lack of confidence, wanting more simulated experiences.
Limitations identified from the pilot study were that it had been more than one month
since exposure to the tool and students may have been hesitant to self-evaluate themselves using
the ANTS tool. For this study, orientation to the tool was changed to the day of the simulated
experience.
Treatment of Data
This protocol was conducted as a mixed methods design. The rationale for using mixed-
method research stemmed from the novel use of the ANTS tool. This non-experimental approach
44
had not been studied among nurse anesthesia students and data obtained from this research design
was thought to be complementary. Using a mixed method design provided a new opportunity for
ANTS tool utilization within HFS (Polit & Beck, 2012).
Quantitative and qualitative data related to participants’ self-evaluation and faculty
assessment of participants’ performance were concurrently collected. Demographic data were
compiled, and descriptive statistics for all measures were calculated using SPSS version 17.0
(SPSS, Inc., Chicago, IL). Correlation between student self-assessment and faculty assessment of
each student was determined using Spearman’s rho correlation coefficient. Correlations between
student and faculty ratings were determined for each self-assessed elemental categorical and
elemental variable. For all analyses significance was set for p values of < 0.05.
Qualitative data were collected and transcribed directly from written participant comments on the
ANTS tool. Data were individually coded and categories identified. A combination of descriptive
and in-Vivo coding was used to help summarize words and phrases and identify distinctive
perspectives. Themes were identified and validated using member checking and faculty analysis
of qualitative data.
45
CHAPTER IV
ANALYSIS OF DATA
This non-experimental mixed-methods research protocol sought to determine if a
relationship existed between faculty evaluation of students NTS and student self-evaluation of
NTS after a simulated anesthesia scenario using the ANTS tool. This protocol also explored
perceptions of students and faculty using the ANTS tool.
Study participants were educated regarding the ANTS taxonomy and framework and
subsequently participated in one of four anesthesia scenarios: 1) hypertension during emergence
from general anesthesia, 2) laryngospasm during emergence from general anesthesia, 3)
refractory hypotension during general anesthesia and 4) intraoperative oxygen tank failure during
general anesthesia. After the scenario concluded participants self-evaluated their own
performance using the ANTS tool. Faculty also evaluated the student’s performance using the
ANTS tool. Study results found significant relationships between faculty and student scores in
the “Situation Awareness” category and the re-evaluating element of “Decision-Making” on the
ANTS tool. Three themes emerged consistent in both student and faculty comments regarding
assessment using the ANTS tool: 1) adapting to change, 2) communication, and 3)
hypervigilance. One unique participant theme, uncertainty, and one unique theme from faculty
(seeing the big picture), emerged from the data. Subthemes within adapting to change and
communication were also identified.
Description of the Sample
Study participants included 30 graduate nurse anesthesia students who had completed 12
months of didactic curricula, a minimum of 3 months of clinical practicum, and enrolled in a
46
clinical management course with high fidelity simulated learning experiences. This study
represented a novel use of the ANTS tool.
All participants were enrolled in a Bachelor of Science to Doctor of Nursing Practice
program at a southwestern university school of nurse anesthesia. The participants were already
enrolled in this clinical management course, which consisted of high fidelity simulated
experiences. The study protocol was already a part of the normal curriculum.
The sample included 20 females with a mean age of 28.7 (range 25 – 31 years) and 10
males with a mean age of 33.3 (range 28 – 46 years). Table 1 depicts the demographic summary
for participant gender and age. Participants had an average of 3.2 years of experience as a nurse
(range 1-7 years) and an average of 2.4 years of experience as an ICU nurse (range 1 – 5 years).
Most participants (26 of 30 participants) had one to three years of ICU nursing experience. Table
2 depicts the demographic summary for nursing experience.
Table 1
Summary of Demographic Data for Gender and Age
Variable N Percentage*
Age (years)
25 - 30
31 – 35
36 – 46
20
7
3
67.7
23.2
10.0
Gender
Male
Female
20
10
67.7
33.3
47
Table 2 Summary of Demographic Data for Nursing Experience
Variable N Percentage
Years RN Experience
1 – 2
3 – 5
> 5
11
16
3
36.7
53.3
10
Years ICU Experience
1 – 2
3 – 5
> 5
17
12
1
56.7
40.0
33.3
Findings
Research Question One
The first research question asked, is there a relationship between graduate nurse
anesthesia students’ self-evaluation of non-technical skills at the categorical level after
participating in a high fidelity simulation anesthesia scenario and faculty evaluations of student
NTS at the categorical level using the ANTS tool? The ANTS tool data was comprised of coded
variables on a Likert-type scale of 0 to 4. Spearman’s rank-order correlation coefficient was used
to assess the relationship between graduate nurse anesthesia students’ self-evaluation and faculty
evaluations of students.
All correlations in category ratings between student and faculty ratings were positive,
however, only “Situation Awareness” ratings reached significance. A significant correlation
between the students’ and faculty “Situation Awareness” ratings, rs(28) = .425, p < .05 was
48
revealed. Table 3 shows the results of the Spearman rank-order correlation for ANTS at the
categorical level.
Table 3
Summary of Spearman's Rank-Order Correlation for Participant and Faculty Evaluations at the Categorical Level (N = 30)
Faculty Task
Management
Faculty Team
Working
Faculty Situation
Awareness
Faculty Decision- Making
Participant Task Management
rs .165
P .384
Participant Team Working
rs .006
P .975 Participant Situation Awareness
rs .425*
P .019 Participant Decision-Making
rs .352
P .057 *. p < 0.05 level (2-tailed). **.p <0.01 level (2-tailed).
Research Question Two
The second research question asked, is there a relationship between graduate nurse
anesthesia students’ self-evaluation of NTS at the ANTS tool element level after participating in a
high fidelity simulation anesthesia scenario and faculty evaluations of student NTS at the element
level using the ANTS tool? A Spearman rank-order correlation was conducted on study variables
at the elemental level to assess the relationship between the participants’ and faculty’s evaluations
All correlations between student and faculty ratings were positive, except the elemental
variable, exchanging information, which was negative (rs(28) = -0.074, p > .05). Statistical
significance was reached for the elemental variable, anticipating in the “Situation Awareness”
category,( rs(28) = .0451, p = 0.012), and for the elemental variable re-evaluating, in the
49
“Decision-Making” category, (rs(28) = .0408, p = 0.025) (See Tables 3-6). All other correlations
for faculty and participant ratings of elemental variables in any ANTS tool category were not
significant.
Table 4
Summary of Spearman's Rank-Order Correlation for Participant and Faculty Evaluations for Task Management Category at the Elemental Level (N = 30)
Faculty Planning & Preparing
Faculty Prioritizing
Faculty Providing & Maintaining Standards
Faculty Utilizing
Resources Participant Planning & Preparing
rs 0.301
P 0.106
Participant Prioritizing
rs 0.129
P 0.495 Participant Providing & Maintaining Standards
rs 0.038
P 0.840
Participant Utilizing Resources
rs 0.164
P 0.388 *. Correlation is significant at the 0.05 level (2-tailed). **. Correlation is significant at the 0.01 level (2-tailed).
50
Table 5 Summary of Spearman's Rank-Order Correlation for Participant and Faculty Evaluations for Team Working Category at the Elemental Level (N = 30)
Faculty Coordinating Activities w/
Team
Faculty Exchanging Information
Faculty Using Authority &
Assertiveness
Faculty Assessing
Capabilities
Faculty Supporting
Others Participant Coordinating Activities w/ Team
rs 0.109
p 0.568
Participant Exchanging Information
rs -0.074
p 0.699
Participant Using Authority & Assertiveness
rs 0.135
p 0.328
Participant Assessing Capabilities
rs 0.001
p 0.998
Participant Supporting Others
rs 0.083
p 0.662
*Correlation is significant at the 0.05 level (2-tailed). **Correlation is significant at the 0.01 level (2-tailed).
Table 6 Summary of Spearman's Rank-Order Correlation for Participant and Faculty Evaluations for Situation Awareness Category at the Elemental Level (N = 30)
Faculty Gathering
Information
Faculty Recognizing & Understanding
Faculty Anticipating
Participant Gathering Information
rs 0.253
P 0.177
Participant Recognizing & Understanding
rs 0.175
P 0.396
Participant Anticipating
rs 0.451*
P 0.012
51
Table 7
Summary of Spearman's Rank-Order Correlation for Participant and Faculty Evaluations for Decision-Making Category at the Elemental Level (N = 30)
Faculty Identifying
Options
Faculty Balancing Risks
Selecting Options
Faculty Re-evaluating
Participant Identifying Options
rs 0.316
P 0.089
Participant Balancing Risks Selecting Options
rs 0.265
P 0.157
Participant Re-evaluating
rs 0.408*
P 0.025 *. Correlation is significant at the 0.05 level (2-tailed). **. Correlation is significant at the 0.01 level (2-tailed).
Research Question Three
Qualitative descriptive methods were used to study student and faculty perceptions of
NTS using the ANTS tool during high fidelity simulation. Each participant reflected on his or her
performance in each of the elemental areas on the ANTS tool after participation in the simulation
scenario. One faculty person also commented on each student’s performance in each of the
elemental areas on the ANTS tool after completion of a simulated scenario. Participants and
faculty were asked to discuss in writing any thoughts, feelings, or comments related to their
performance for each ANTS variable once they completed the scenario. Participants and faculty
were not given any verbal hints or examples.
52
ANTS tool written data from all 30 student participants were transcribed verbatim into an
electronic word document. The data were organized so that each element from the ANTS tool
was transcribed and coded on a separate page. Comments from each participant were sorted
according to the specific element and pooled together. Three major themes emerged from the data
congruent with both participant and faculty comments: adapting to changes, challenges in
communicating, and hypervigilance. One unique participant theme ,uncertainty, and one unique
faculty theme (seeing the big picture) were also identified. One subtheme, feeling rushed, was
identified under the theme, adapting to changes, and another subtheme, feeling alone, was
identified under the theme, communication.
Overall, 44 descriptive/ in-vivo codes were identified related to student ANTS tool
comments and 38 codes were identified related to faculty ANTS tool comments. The number of
codes directly reflected the number and volume of comments for each element. Students and
faculty did not always comment in each element.
Theme I: adapting to changes. Adapting to changes was a major theme for both
participants and faculty in two of the four ANTS categories, “Task Management,” and “Team
Working.” Participants were caught off guard with several different events unfolding
simultaneously. Statements such as, “…how do you plan in 30 seconds” and “didn’t have
succinylcholine ready,” showed how difficult it was for students to adapt to these acute changes.
Many comments related to this theme focused on challenges in adapting and controlling the
simulated OR environment. Comments such as, “I was not familiar with resources available,”
“did not know where syringes were,” and “could not find oxygen tank opener,” focused on
unfamiliarity and lack of environmental control. Many participant reflective statements, such as,
“could have asked the surgeon status of the case, “could have used the circulator more,” “I could
53
have turned the lights on,” “could have delayed emergence and increased the gas,” and “could
have allocated tasks to team members quicker” centered on what they could have done differently
to adapt. Other participant comments focused on what they did to try to manipulate their
environment such as “attempted to get information from others,” “planned to give long-acting
opioid and told team,” and “ was not fully prepared for case.” It was also difficult for participants
to manipulate the OR team member environment with comments like, “it was loud in the room,”
“could have asked the surgeon status of the case,” “difficult for me being newer and not knowing
the team,” and “used crisis book to ensure steps were followed.”
Faculty comments also indicated the difficulty participants had adapting and controlling
their environment as evidenced by, “unsure about environment because the lights remain off,”
“radio playing entire scenario,” and “inquired how case was going.” Other comments such as
“hesitant to defibrillate,” “did not see another oxygen tank directly behind them,” and “could not
find a way to open the tank,” indicated challenges the participants faced in controlling the
environment. At times participants attempted to control the environment, but with uncertainty,
with statements such as, “immediately assumed bronchospasm and followed treatment for that,”
“did not know OR staff or environment,” “looking for phenylephrine on cart- can’t find it,”
“stopped surgery to take care of patient,” and “appears not sure how to handle OR team.”
Part of adapting to changes included acting and reacting to the OR team. Some
participants relied on “Team Working” with the OR team or a secondary helper to adapt to
changes. “Told OR staff what he needed for help,” “secondary anesthesia provider took over for
primary to help stabilize patient,” and “the circulating nurse read out loud what needed to happen
next.” Other participants did not adapt well using their resources in the “Team Working”
category: “he tried to coordinate activities, but team not helpful,” and “did not engage anyone
54
across the drape.” Adapting to changes brought out participants’ emotions, especially in the
“Situation Awareness” category. Participants commented about “Too much drama to focus on
changes,” “Looking around the room for help, just called for help without engaging any particular
person,” “Stated quietly just wanted it to be over.” Part of adapting to changes in the “Decision-
Making” category included taking action and re-evaluating. Faculty comments included,
“reassessed, but did not treat it,” and “did not follow through with helper to ensure that
ventilation was adequate.”
Participant subtheme: Feeling Rushed. Feeling rushed as a participant subtheme also
emerged under the adapting to changes theme in three out of the four ANTS categories, “Task
Management,” “Situation Awareness,” and “Decision-Making.” Participants were not always
ready for patient changes to unfold. Simulated scenarios generally concluded after approximately
10 minutes or when the learning objectives were accomplished. Therefore, feeling rushed was
not an uncommon participant experience. Participant comments in the “Task Management”
category included, “rushed nature of emergence,” “needed to slow down,” “everything felt
rushed,” and “happening so fast,” and “did not have time to understand and anticipate situation.”
Participants also felt rushed when trying to share the current situation with others in the
“Situation Awareness” category with statements such as, “report was rushed,” and “tried to get
2nd helper ready to help, but did not have time.” “Decision-Making” was difficult when feeling
rushed, with comments like, “hard to focus when moving so fast,” “could not keep up with vital
sign changes,” and “the team was pushing me to extubate too soon…not ready.” Another
participant recalled, “ I could not process, I was going through the motions with everything
unfolding so rapidly and not processing the true essence of what was going on.”
55
Theme II: Challenges in communicating. Challenges in communicating was a major
theme for both participants and faculty in the “Team Working” ANTS category. Participants
were thrust into a situation that began with a structured hand-off in a surgical case where they did
not know anyone in the room. Written comments regarding hand-off communication included
“needed better exchange of information at report / change of shift,” and “I let the CRNA give a
bad report while changes were occurring.” Some participant comments reflected a sense of their
own lack of communication such as, “should have asked for help earlier,” “lack of
communication to staff when scenario began unfolding,” “needed more assertive communication
skills,” and “worked well to coordinate team but needed improvement on communication.” Other
statements indicated participants were reflecting on OR team communication, “asked scrub to
wait a minute,” “felt weak in this area,” “no preop sheet available, scrub interrupts report,” “no
one listened when I delegated,” and “pushy surgeon made me have to validate problem and
explain.” Statements also highlighted secondary helper communication such as, “gave report to
second person succinctly and delegated tasks immediately,” and “when secondary helper came to
room I gave him an idea what was going on. However when giving tasks I jumped around and
should have stayed on one task at a time.” Participants’ statements also revealed in-the-moment
communication with comments like, “communication with team about patient status was
prompt,” “explained the situation to the team faster,” “felt the team heard me and could have
increased assertiveness,” and “under stress pretty much stress affected my communication.”
Faculty comments paralleled student comments related to a lack of communication.
Comments such as, “no communication with secondary help,” “decreased communication with
team,” and “hesitant to advise team what to do” centered on observed communication deficits.
Faculty comments also revealed a more positive aspect to student communication and included,
56
“student asked anesthesia provider to stay until all information received,” “good thorough report
with important issues stressed,” and “quietly communicated with team in a positive way.”
Faculty also commented regarding a total communication absence, not observed in student
comments, such as “not aware enough to share thoughts about what was going on,” “ states
everything is okay, even though it is not,” “just functioned without verbal communication” and
“quiet.” Faculty comments also revealed factors related to engaging the OR team, such as,
“asked for second helper and did not even look up at surgeon or OR staff,” and “did not engage
anyone across the drape.”
Participant subtheme: Feeling alone. Feeling alone emerged from the theme,
challenges in communicating, because the sense of feeling alone often stemmed from a deficit of
effective verbal and / or nonverbal communication in participant comments. Feeling alone was
identified as a subtheme within the “Team Working” and “Situation Awareness” categories.
During the timeframe the study was conducted participants had approximately 3-6 months of
clinical anesthesia experience and did not know the confederates in each scenario. Comments
from the “Team Working” category such as “felt like I was by myself,” “had to think alone,” and
“ had no one to share ideas” demonstrated the participants’ concerns about feeling isolated. Team
members (confederates / actors) were scripted to be unhelpful in certain scenarios, perpetuating
the feeling of being alone. In the “Situation Awareness” category participants stated, “looked
around the room and everyone was staring at me and saying nothing,” and “as I scanned the
monitors I knew that I was the only one responsible in this situation.” Statements like, “did what
I could, but no one listened,” “team was not helpful,” “ was new and did not know anyone in the
room,” and “ignored others in the room,” helped demonstrate that feeling alone was a real
perception for participants. Two students helping each other experienced feeling alone together,
57
and conveyed that the surgical drape was obscuring their interactions with the team, “and that was
okay,” “the team was intimidating.” The surgical drape became a physical barrier, enhancing
participants’ feeling of aloneness.
Theme III: Hypervigilance. Hypervigilance emerged from the “Situation Awareness”
category and was congruent between participants’ and faculty comments. Participants had to
quickly scan the environment and dive into the scenario as it unfolded. Participants’ statements
regarding the imperativeness to gather information included, “I needed to gather information
more quickly,” “took little time recognizing scenario,” “recognized oxygen desaturation quickly,”
“immediately checking vital signs,” and “needed to respond faster to patient change.” Trying to
gather information rapidly resulted in comments such as, “should have rechecked the pipeline
connection,” “in that moment during report it was hard for me to think,” “I needed blood, but the
situation at hand needed immediate resolution,” and “panicking can cause mistakes.” Participants
knew something was going to happen. Statements that reflect this natural anticipation included,
“saw it coming,” “tried to anticipate, but was wrong,” “think to anticipate calmly,” and
“anticipated extubation.”
Anticipation and trying to gather information quickly were reflected in faculty comments
as well as participant comments. Faculty comments also reflected either observed hypervigilance
or failure to act appropriately with the stress of the scenario. Statements related to gathering
information were “gathering correct information, but expecting more to happen,” “was quick to
treat BP, but did not see ECG changes,” and “gathering correct information, but expecting
something more to happen.” Comments that reflected anticipation included, “waiting for
something to happen,” and “anticipated entire scenario,” and “moving at hyper-speed when
nothing is unfolding.” Faculty statements also revealed participant behaviors that were not
58
reflective of hypervigilance such as, “not buying into the vitals changing,” “went down wrong
path for scenario and stayed there,” “disorientation,” “situation unfolded quickly—over-
stimulated,” and “froze and just recycled the BP cuff.”
Participant theme: Uncertainty. Uncertainty emerged as a participant theme in three of
the four ANTS categories, “Task Management,” “Situation Awareness,” and “Decision-
Making.” Students did not know hemodynamic changes would occur so rapidly as exemplified by
statements in the Situation Awareness category such as “did not know what to expect,” and “not
sure what was causing the change.” Participants’ uncertainty was reflected in comments about
self-confidence in “Decision-Making” such as, “not sure how to handle circulating nurse,” and “I
have always been good in acute situations until now.” In the “Task Management” category
uncertainty was reflected by participants after they realized they had made erroneous decisions, “I
was fixated on bronchospasm when the patient really needed to just be ventilated better,” “several
interventions did nothing, I did not know what to do,” and “unable to think after I knew I did the
wrong thing.” One student reflected that her uncertainty as the simulated scenario unfolded made
her question every decision she made, and even after the scenario was over, she could not
remember exactly what happened.
59
Table 8
Qualitative Data Analysis: Summary Participant and Faculty Theme Emergence
Category Element Participant Faculty Task Management
Planning & preparing Theme: Adapting to changes Subtheme: Feeling rushed Participant theme: Uncertainty
Theme: Adapting to change Faculty theme: Seeing the big picture
Prioritizing
Providing & maintaining standards Identifying & utilizing resources
Team Working
Coordinating activities with team
Theme: Challenges in communicating, Subtheme: Feeling alone Theme: Adapting to changes
Theme: Challenges in communicating, Theme: Adapting to changes
Exchanging information
Using authority & assertiveness
Assessing capabilities
Supporting others
Situation Awareness
Gathering information Theme: Hypervigilance Theme: Adapting to changes Subtheme: Feeling rushed Participant theme: Uncertainty
Theme: Hypervigilance Theme: Adapting to changes Faculty theme: Seeing the big picture
Recognizing & understanding
Anticipating
Decision-Making
Identifying options Theme: Adapting to changes Subtheme: Feeling rushed Participant theme: Uncertainty
Theme: Adapting to changes Faculty theme: Seeing the big picture
Balancing risks & selecting options Re-evaluating
Faculty theme: Seeing the big picture. Paying attention to the big picture emerged in
three out of the four ANTS tool categories, “Task Management,” “Situation Awareness,” and
“Decision-Making.” It was often difficult for participants to grasp the larger pathophysiological
issue that was unfolding. In the “Task Management” category faculty observed participants
focusing on singular tasks and repeating them with a lack of prioritization, “recycled the BP cuff
60
4 times before looking under the drape to notice that it was disconnected,” and “focused on SpO2
probe,” “saw PVC’s, but only increased oxygen flow.” Participants called for help after one or
two single interventions did not bring about patient stability, “she called second provider when
BP not increasing after one reading,” “he could not find the oxygen tank in the dark, so called for
help immediately.” The “Situation Awareness” category highlighted faculty comments regarding
participants’ difficulties in gathering information to see the bigger picture. For example, “ dark in
room, so makes gathering information more difficult,” “focused on ventilator too much to see
other issues going on,” and “gathering good information, but not what is needed to solve the
problem.” Once participants gathered information, it was necessary for them to progress into the
“Decision-Making” category, acting and re-evaluating to see and solve the big picture issue.
Many participants “stayed with the course of action, did not re-evaluate, and did not consider
other options.”
One common thread observed related to participants seeing the big picture took place
repeatedly during this study and generally occurred in three segments. An example occurred
during an emergence hypoxemia scenario. First, the participant identified the big picture, but it
was the wrong big picture. Faculty commented, “she thinks its bronchospasm, but no intervention
is working.” Second, when help arrived, the secondary provider was brought into the wrong
pathophysiological picture and bought into it, “she came to help and thinks it’s bronchospasm
too. Can’t she see that the patient is hypoventilating and the tidal volumes are just too low?”
Third, the participant and secondary helper finally notice their error, and corrected the problem,
ending the scenario, “He adjusted the tidal volume and frequency to help the oxygen saturation
improve.”
61
Summary of the Findings
This study examined participant self-evaluation of non-technical skills in graduate nurse
anesthesia students by presenting a discrete method to measure tacit skills through the ANTS
tool. The study used a novel method of comparing student and faculty quantitative ratings and
written comments to find meaning and new potential uses for the ANTS instrument. The
relationship between the study variables indicated that at the ANTS tool categorical level there
was only one ANTS tool category, “Situation Awareness” that displayed significant correlation
between participant and faculty ratings. At the elemental level anticipating in the “Situation
Awareness” category and re-evaluating in the “Decision-Making” category significant
correlations between student and faculty ratings were found.
Qualitative analysis of participant and faculty comments using the ANTS tool identified
three common themes between participants and faculty: 1) adapting to changes, 2) challenges in
communicating, and 3) hypervigilance. Themes unique to participant or faculty were also
identified. A participant subtheme within the theme of adapting to changes was feeling rushed
and another participant subtheme within the theme challenges in communicating was feeling
alone. One theme unique to participant comments was uncertainty and one theme unique to
faculty comments was seeing the big picture.
62
CHAPTER V
SUMMARY OF THE STUDY
Non-technical skills have emerged as important safety competencies in anesthesia, but
education and training is virtually non-existent in current nurse anesthesia graduate curricula
(Fletcher et al., 2002; Flin et al., 2010). Studies exploring anesthesia education and NTS have
demonstrated that NTS can be developed through simulated anesthesia experiences (Gale et al.,
2010; Wunder, 2016; Yee et al., 2005; Zausig 2009). The Anaesthetists’ Non-technical skills
(ANTS) tool represents a tangible, structured instrument offering a method to evaluate NTS
(Patey et al., 2005).
Faculty observations of participant NTS and students’ self-evaluation of NTS after
simulated experiences added knowledge of the correlation and accuracy of ANTS evaluation
methods (Arora et al., 2011). Analyzing qualitative comments and perspectives of ANTS using
the ANTS tool could have offered new insights regarding the difference between perceptions of
clinical performance.
Summary
This study was conducted in a high fidelity simulated environment as part of graduate
nurse anesthesia students’ normal curricula to assess and compare student assessment of NTS as
compared to faculty assessment of student NTS. Students participated in a simulated anesthesia
scenario after a didactic introduction to the ANTS tool and concepts. Immediately after the
scenario concluded students used the ANTS tool to self-evaluate their own performance. Faculty
also used the ANTS tool to assess student performance. The ANTS tool was comprised of items
ranked on a Likert-type scale to rate four categories and fifteen elements of NTS behaviors.
63
Correlations among rankings were examined using Spearman’s rho statistic. The ANTS tool also
contained an area for written comments within each of the 15 elemental subcategories. Written
comments were collected from both students and faculty to glean perceptions of using ANTS.
Using this mixed-methods approach and the ANTS tool to its fullest extent provided rich
information for analysis.
Discussion of the Findings
Research Questions One and Two
This study demonstrated significant correlation between participant and faculty in one
ANTS tool category, “Situation Awareness.” Two elemental variables also showed significant
correlations: anticipating in the “Situation Awareness” category and re-evaluating in the
“Decision-Making” category. The ANTS tool category, “Situation Awareness,” contained the
elements, 1) gathering information, 2) recognizing and understanding, and 3) anticipating.
Categorical significance might be attributed to several factors. First, because each scenario was
pre-planned and structured, faculty already had an idea of what the situation was going to hold
and how it would unfold for the student. Actors were either facilitating “Situation Awareness” or
were inhibiting it. Participants had a certain amount of pre-existing “Situation Awareness” and
knew each scenario was going to start with a handoff. Participants were oriented to the simulated
environment prior to the scenarios and scenarios chosen were those that some participants could
have already experienced clinically. Therefore, participant ratings were more on target in the
“Situation Awareness” category than other ANTS tool categories. Faculty observations also
commented about hypervigilant behaviors in participants while participants also perceived their
own hypervigilance in this category. Hypervigilant behavior related to gaining situation
awareness has been well established as a common finding in simulated scenarios and could have
64
supported significant correlation in this category (Henrichs et al., 2007; Hotchkiss, Biddle, &
Fallacaro, 2002; Issenberg et al., 2005).
Finally, it is possible that participants could have shared scenarios with others.
Comparison of ANTS tool ratings was not performed between participants, but sharing of what
the simulated curriculum was about could have resulted in increased consistency between
participant and faculty ANTS tool ratings. Participants agreed to confidentiality of simulation
scenarios as a part of the curriculum, however, so sharing of information was not expected.
However, because there were no significant relationships in the other categories, this explanation
is unlikely.
A significant relationship was also found in the elemental variable, anticipating, within
the “Situation Awareness” category. The authors of the ANTS tool described anticipating as
asking ‘what-if” questions and thinking ahead about potential outcomes or running mental
projections of the current situation to predict what might happen (Flin et al., 2012). A high
degree of anticipation to the point of excessiveness has been reported from both students and
faculty observations (Hotchkiss et al., 2002). This could help explain the significant correlation
in this study and ultimately the significant relationship shown in the “Situation Awareness”
category.
A significant correlation between participant and faculty ratings in the elemental variable
re-evaluating in the “Decision-Making” category was found. The authors of the ANTS tool
described re-evaluating as continually reviewing the suitability of the options identified, and
selected, and reassessing the situation after an action (Flin et al., 2012). Re-evaluation was
relatively straightforward to assess, therefore this might explain the significant correlation. For
example, observing participants recycle a blood pressure after giving a medication, or turn to the
65
monitor to re-evaluate the heart rhythm or recheck breath sounds. Self-assessments related to re-
evaluation could have also improved over time as participants came with more clinical experience
and were more accurate about their own abilities.
Lack of correlation related to the other ANTS tool categories and elements in this study
may have been due to several factors. First, this study was conducted using a convenience
sample of only 30 students. However, qualitatively observing the similarity between participant
and faculty comments across the cohort, tends to support the sample sized used in this study.
Participants were novices regarding ANTS concepts and the tool. Curricular time only allowed
the students 60 minutes of interactive didactic education with ANTS including practice with the
ANTS tool. A recent study using a similar-sized cohort of graduate nurse anesthesia students
oriented them to the ANTS tool using 60-minute didactic interactive curricula. Student NTS
performance did improve, between the first and second simulated experiences, but there was no
comparison between student and faculty evaluations (Wunder, 2016). Prior studies examining
ANTS tool training and orientation revealed no optimal timeframe to become proficient in ANTS.
Curricular time in these studies ranged from 1 hour to 4 hours to an indefinite period of time
before an IRR > 0.9 could be achieved (Fletcher et al., 2002; Fletcher et al., 2003a; Fletcher et al.,
2003b; Morgan et al., 2011; Salvodelli et al., 2006; Welke et al., 2009; Yee et al., 2005; Zausig et
al., 2009). It has been also been reported that the ANTS tool was inherently difficult to use, even
with ample education and training (Byrne et al., 2015).
Further, only one individual faculty rated participant performance using the ANTS tool.
A majority of studies examining ANTS tool reliability have utilized 2-3 raters to establish rater
score consensus (Morgan et al., 2011; Salvodelli et al., 2006; Wunder, 2016; Yee et al., 2005;
Zausig et al., 2009). This was a limitation of this study and due to limited faculty resources for
66
simulation curricula. However, using one faculty rater in this study with over 8 years experience
using the ANTS tool may have led to improved rating consistency.
Additionally, learning in a high fidelity simulated environment has been reported to
invoke heightened emotional responses in participants (Issenberg et al., 2005). It might have
been possible that student participants were emotionally charged immediately post-scenario
performance and both quantitative and qualitative responses became skewed, creating a disparity.
Students have tended to be critical about themselves post-scenario in oral debriefing sessions
according to faculty observations, but this was not a consistent finding.
Finally, there could have been disparity between faculty and participant performance
expectations. Assessment disparity has been speculated as an issue in previous studies regarding
faculty assessment of student performance, especially as experience has been gained without
specific knowledge of individual skill, experience, and maturity level (Brewster et al., 2008;
Davis et al., 2006; Moorthy et al., 2006). Students acquire competence and confidence at varying
rates depending on type of clinical site, clinical experiences and individual learning styles.
Research Question Three
The third research question in this study qualitatively explored faculty and participant
perspectives related to their performance as indicated by the ANTS tool. The qualitative aspect of
this study protocol and comparison of participant self-evaluation to rater observed evaluation had
not been published, or explored at the time of the study proposal. Authors of the tool also
confirmed this. The ANTS tool allowed for written comments in each elemental item. Three
major themes emerged from these comments by students and faculty: adapting to changes,
communication, and hypervigilance. Participant subthemes emerged within the themes adapting
67
to changes (feeling rushed) and communication (feeling alone). Other themes were identified
unique to participant comments (uncertainty) and faculty comments (seeing the big picture).
Adapting to changes emerged as a theme in the categories “Task Management” and
“Team Working.” The ANTS framework had identified behaviors for good practice related to
adapting to changes including 1) preparing equipment and drugs prior to the start of the case, 2)
identifying resources available and requesting additional resources as needed, 3) allocating tasks
to appropriate members of the team, and 4) using team working as a resource to help control the
environment (Flin et al., 2012). Students came into this scenario with a handoff, so immediately
they did not have basal control of their own practice environment. Members of the OR team had
been instructed to act out a certain personality and play a specific role. Therefore, it was not
unexpected that this theme emerged from the participant and faculty comments.
Student comments included “it was loud in the room,” “could have asked the surgeon
status of the case,” “could have delayed emergence and increased the gas,” “could have used the
circulator more.” The majority of students’ comments related to environmental control related a
lack of control, not knowing how to control it or simply avoiding any manipulation of the
environment, reflecting a difficulty in adapting to changes.
Adapting to changes from a faculty perspective represented a double-edged sword.
Purposeful environmental manipulation to improve authenticity and enhance learning (e.g., room
lighting, certain equipment and monitors, actors, hemodynamic changes) was specifically
addressed in the development of the scenarios for the research study. An objective of using high
fidelity simulation in this study was to challenge the student to manipulate the environment (OR
environment and OR team) and to create a scenario where the student had no choice but to adapt
to changes and suspend disbelief.
68
An example in this study involved a scenario where a patient became progressively more
hypoxemic. The pulse oximeter tone slowly decreased and should have instilled an emotional
response and sense of urgency to investigate the issue—increase FIO2, assess breath sounds,
check to see that the pulse oximeter did not fall off of the finger. Those would have been some of
the first initial assessment interventions expected. The scenario continued to unfold, but the
participant did not touch or manipulate the surgical drape to look underneath for the pulse
oximeter. Another situation involved some minor hemodynamic perturbations. The participant
had just received the handoff and was scanning around the room. Nothing was changing at the
moment, but the participant was already gathering the defibrillator and preparing for a cardiac
arrest.
Another aspect of adapting to changes that emerged from faculty comments was
controlling the human environment (i.e., circulating nurse, scrub technician, and surgeon). Part
of the study protocol was to develop the participants’ NTS and manipulate the human
environment in the operating room. This may have been actually more difficult than controlling
the physical environment because participants had limited experience in this area of professional
growth. Most participants had not dealt with enough situations where they have controlled or
utilized the OR team for information or assistance. Faculty comments indicated that the “surgical
drape was an interactive barrier” and acted as both a physical and emotional barrier for accessing
information and assistance during the simulated scenarios. The participant tended to “act alone
and often ignored the OR team.”
Adapting to changes brought out participants’ emotions, creating a ripple effect and
changing how they continued to adapt within the scenario. Faculty comments related to
participants with limited clinical experience or those struggling in their clinical practicum
69
reflected a more limited adaptation to change and more negative comments documented within
each category. Negative comments such as “did not treat or did not follow through” were
observed more frequently in those participants who had minimal adaptation to changes and were
an unexpected finding. Negative comments or observing what was not done may have been
easier to take note of rather than documenting what was done correctly.
Feeling rushed was a participant subtheme that captured the essence of adapting to
changes related to scenario acuity and time compression. Some scenarios in the study contained
an abbreviated hand-off; other scenarios were designed for hemodynamics or oxygenation to
change rapidly. The very nature of the scenario was rushed. Participant comments about feeling
rushed contained emotion. It was as if the rushed nature of the scenario diffused into their body.
Participants commented in the “Task Management” category that they could not keep up, they did
not have time. Even when patient changes did not occur rapidly there was a perception that they
needed to slow down, but could not.
When the participant was thrown into a 10-minute scenario as opposed to an 8-hour day,
it created a feeling of being rushed. This feeling had been described in earlier high fidelity
simulation studies (Henrichs et al., 2007; Hotchkiss et al., 2002). Participants were aware of the
study timeline prior to participating. It was clear that acute anesthesia events were going to
unfold. Feeling rushed naturally emerged as an authentic participant emotion.
Challenges in communicating as a theme emerged in the “Team Working” category. The
ANTS framework identified behaviors for good practice related to communication including 1)
discussing case with colleagues, 2) giving situation updates/reports for key events, 3)
communicating plans to appropriate individuals, 3) giving clear orders to team members 4)
calling for assistance as needed, and 5) providing reassurance/encouragement (Flin et al., 2012).
70
Participants came to the simulation curricula after 3-6 months of clinical anesthesia practicum.
Participants also at this point in their education, training, and experience had likely not had to
assertively and so dramatically confront and coordinate activities with an OR team. This was a
new experience for them, and programmed into the scenarios. Therefore, it was not unexpected
that communication emerged as a major theme. Assessing communication directly using the
ANTS tool has been cited as a weakness by other investigators because it does not exist as a
distinct category. The creators of the ANTS tool did not include communication separately
because communication was thought to permeate all categories of the ANTS framework and tool.
The results of this study support the findings from other studies that communication does cross all
categories of the ANTS tool (Fletcher et al., 2003a; Fletcher et al., 2003b; Flin et al., 2010).
The creators of the ANTS tool believed that the content of the communication was more
important than just skill demonstration (Fletcher et al., 2003a; Fletcher et al., 2003b; Zausig et al.,
2009). It became evident that content of communication was important during several key aspects
of the scenarios: 1) hand-off; 2) acute unfolding of events; 3) secondary anesthesia provider; and
4) OR team. Thoroughness of the hand-off tended to set the emotional tone for the scenario and
participants often reflected on a sense of their own lack of communication. It was a relief to
participants when the secondary anesthesia provider entered into the scenario. Once the scenario
was acutely unfolding, communication was not always filled with quality information, especially
when directed towards the OR team.
For example, in this study faculty commented that the surgical drape was used by
participants as an interactive communication barrier, and acted as both a physical and emotional
barrier for accessing information and assistance during the simulated scenarios. The surgical
drape evolved to represent the communication quality deficit. The drape’s mere presence
71
encouraged participants to “act alone and ignore the OR team” and “not engage anyone across the
drape.”
Feeling alone emerged as a participant subtheme within the challenges in communicating
theme. It was interesting to note that feeling alone was the only subtheme distinct to the “Team
Working” category. Participants had not been left alone in the operating room yet at the time the
study was conducted. They had always been partnered with a nurse anesthetist or
anesthesiologist, so participation in this study also gave them a chance to develop experience as
an autonomous practitioner.
The distinctiveness of the “Team Working” category described skills for working in a
group, exchanging information, and using authority and assertiveness, many skills that had not
been developed or even tapped into at this point during the study participants’ anesthesia
experiences. Participants written expressions reflected a deficit of ANTS in all elements of the
“Team Working” category. Feeling alone was characterized by thinking and making decisions
alone, not feeling like a part of the team, not feeling heard, and not engaging the team. One
participant commented that she looked around the room and realized that she was the only one
responsible for this situation.
Feeling alone prompted help to be called early. Participants naturally did not want to be
the only one to make a mistake, especially in the company of an OR team that they did not know.
Two participants helping each other did not always solve the problem of aloneness. A participant
commented that she thought he would have been helpful, but she did not fully convey what she
needed to her secondary helper. Comments related to being a better communicator and taking the
time to explain to the secondary helper what was going on would have improved feeling alone.
72
Feeling alone prompted ignoring others in the room such as OR team and even the
second helper. Four situations evolved: 1) the participant would act purely alone and not engage
any individual throughout the scenario; 2) the participant would only engage the secondary
helper, excluding the OR team; 3) the participant would attempt to engage the OR team
unsuccessfully; and 4) the participant and the second helper both avoided engaging the OR team.
Faculty observed participants entering the scenario without introducing themselves to the OR
team. Sometimes the participant would not even make eye contact with others in the room.
A natural environmental barrier that fostered feeling alone was revealed focusing on the
presence of the surgical drape. The surgical drape became a physical and emotional barrier to
engagement and communication that enabled the participant to remain alone. Feeling alone was
associated with each participant’s personality and the absolute necessity to cross over the drape
barrier and obtain information and assistance from the OR team. Two students were experiencing
feeling alone together and avoided looking up and over the drape even though the patient was
bleeding. They commented that it was intimidating and that they were “doing okay” functioning
together.
Acting autonomously was purposefully planned into the protocol for this study. It was
imperative that participants gain experience in managing more complex anesthetic cases alone.
Therefore, it was not a surprise that this subtheme emerged. It was an unexpected finding that the
surgical drape became a communication barrier both physically and emotionally.
Hypervigilance emerged as a theme, contained in the “Situation Awareness” category.
Hypervigilance has been described as a consistently observed and accepted learner behavior in
the simulated environment. (Gaba et al., 2001; Hotchkiss et al., 2002; Issenberg et al., 2005;
Lupien, 2007; Zausig et al., 2009). It has also been described as a limitation to learning in a
73
simulated environment versus learning in a true clinical environment (Hotchkiss et al., 2002).
Participants in this study tended to display anxious behaviors in the form of often very rapid and
sometimes mechanical movements or verbal sentences. After hand-off and a quick, jerky scan of
the environment, one participant immediately grabbed the crash cart, put the defibrillator pads on
the patient, and began drawing up emergency drugs. The patient’s vital signs were stable.
Nothing acute was unfolding. This was a classic hypervigilant response to a simulated scenario
and very characteristic of the “Situation Awareness” category. Another participant kept adjusting
ventilator settings and hand-ventilating the patient every couple of minutes. Again the patient
was stable and no acute changes had occurred.
Participants’ hypervigilance may be fostered in the simulated environment because of the
very nature that they are being observed and evaluated. Even though hypervigilant behaviors are
not truly representative of typical behaviors in the clinical milieu, experiencing high-risk, low-
exposure anesthesia scenarios in the safety of a simulated environment is far more beneficial to
learning than trying to eliminate hypervigilance (Hotchkiss et al., 2002; Issenberg et al., 2005;
Lupien, 2007). Hypervigilance has become accepted and acknowledged as a known behavior in
any simulated teaching and learning environment where individuals are being observed and /or
evaluated.
The participant theme uncertainty, in this study, attempted to mirror the unpredictability
of the real clinical environment, but when the participant was thrown into a 10-minute scenario as
opposed to an 8-hour day, it changed the intensity considerably. Uncertainty proved to be a
prominent participant theme in “Task Management,” “Situation Awareness,” and “Decision-
Making” categories. The lack of being able to plan and prepare for unknown changes and
making the right decisions to take care of the patient permeated participant comments from the
74
ANTS tool. Self-confidence was questioned by some participants in whether or not they were
making the right decision in the moment or on the fly. Uncertainty triggered a call for help early
on in the scenarios. Faculty were surprised how quickly participants wanted someone else in the
room to help clarify and support assessment, interventions, and re-evaluation of patient
management. One participant in the study called for the secondary anesthesia provider
immediately after hand-off and just looked around the room. She commented in the “Situation
Awareness” category that she had always been good in assessing acute situations until now.
Part of uncertainty in this study revealed the potential to make a wrong decision. A
participant noted that she was fixated on bronchospasm, when the patient really just needed more
adequate ventilation. After recognizing what was wrong, the participant commented on the
ANTS tool in the “Decision-Making” category that she could not remember exactly what
happened. Uncertainty may have fostered emotions that clouded thinking during the scenario.
Seeing the big picture was a unique faculty theme in this study and was identified in three
out of the four ANTS tool categories, “Task Management,” “Situation Awareness,” and
“Decision-Making.” One of the purposes of using high fidelity simulation in teaching and
learning is to foster development of seeing the big picture. Participants in this study became
focused and overwhelmed, had difficulty prioritizing, and went down erroneous decision
pathways. All of these issues observed by faculty revealed participant deficits in seeing the big
picture.
A participant was observed assessing, treating and managing what was thought to be
bronchospasm. The secondary helper arrived, a brief report was shared, and both participants
continued to treat the patient for bronchospasm, which was not responding to any therapies. The
big picture and correct identification of the problem was finally attained, both participants were
75
visibly enlightened and the scenario was terminated by faculty at that point. This scenario
example in this study was observed and commented on by the faculty.
Participants often bought into a certain set of physiological variables that were changing
and then focused on one that seemed most familiar. One participant explained that she had just
had a patient who had a severe bronchospasm. She just knew that was what was happening in
this scenario. Faculty observed that she had an emotional imprint from that experience that
carried her through this scenario and convinced the secondary helper too. She commented that
she would never forget this experience and the importance of seeing the larger picture.
This study incorporated more than one type of data collection to answer three research
questions. Triangulation of study data revealed rich information. Significant congruence
between participant and faculty ANTS tool ratings in the “Situation Awareness” category was
identified, especially within the element, anticipating. Qualitative descriptive data analysis
confirmed that hypervigilance, supporting these quantitative findings.
Non-significant correlations between participant and faculty ANTS tool ratings were
found in the other three categories, “Task Management,” “Team Working,” and “Decision-
Making.” This mirrored the unique perspectives revealed within the subthemes between
participant and faculty comments. The disparity and lack of correlation reinforced the difference
in perspective between students and faculty related to clinical performance. Participants’
subthemes revealed more emotion and vulnerability related to the study protocol. Faculty
subthemes reflected a more global perspective and outcome of what participants should achieve
by taking part in the scenario.
Non-significant correlations between participant and faculty ANTS tool ratings as well as
unique subthemes between participants and faculty could have reflected lack of insight into self-
76
evaluation of performance. This study was conducted after 3-6 months of clinical anesthesia
experience. Student self-evaluation might have improved with more clinical experience.
Theoretical Framework Connection
Kolb’s Experiential Learning Theory served as the conceptual framework for this
research protocol and truly came to life as part of the participants’ simulated experiences.
Learning is a continuous process in Kolb’s learning theory and is depicted as a dynamic, cyclic
structure where knowledge is created through experience. This part of Kolb’s theory was
congruent with the overall simulated experience and ANTS performance.
Participants individually applied their knowledge and concepts to advanced simulated
scenarios, moving Kolb’s experiential learning theory into practice (Armstrong & Parsa-Parsi,
2005; Kolb, 2015). Then participants evaluated their own performance using the ANTS tool that
helped bring forward skills they had never explicitly identified or described (Fletcher et al.,
2003c). Self-reflection of the simulated experience using the ANTS tool allowed for
multifactorial synthesis of the entire learning experience. Participant self-evaluation data and
faculty evaluation data supported the return back to Kolb’s first quadrant, with new experiences,
therefore placing the participant at a different vertical point on the spiral learning trajectory
Armstrong & Parsa-Parsi, 2005; Kolb, 2015).
For example, a participant experienced an acute hypoxemia scenario where the oxygen
saturation decreases precipitously. The OR team was not helpful, stressed the participant, and
stretched the participant’s NTS abilities, and the second anesthesia helper was called and arrived.
The primary participant engaged the helper about what had transpired and then both participant
and second helper were treating the patient as if bronchospasm was present. They both
eventually realized that the patient was simply hypoventilating and correctly treated to resolve the
77
situation. The participant returned to Kolb’s first quadrant with new knowledge and experience,
and was therefore not on the same experiential plane from where he / she started.
This example represents the spiral-learning trajectory. Participants take with them not
only new experiences, but new tools and resources related to NTS. In turn these experience help
them continue the Kolb’s experiential learning theory growth spiral.
Figure 2. Kolb’s Learning Theory Applied to Study Results
While participants gained knowledge through this simulated experiential learning,
disparities in participant and faculty evaluations reflected that participants had difficulty moving
from Kolb’s third to fourth quadrant, to better engage in the spiral-learning trajectory. In other
words participants had difficulty accurately reflecting and extracting cognitive information and
integrating it into the simulated experience, even if it was a safe learning environment, therefore
application in the clinical environment and escalation to a higher plane of professional growth
may have been limited. Participants’ level of self-evaluation of NTS behavior was not congruent
78
with faculty evaluations of that behavior. This inaccuracy in student-self evaluation supported the
need for faculty input in student self-evaluation to progress effectively from reflecting, to
thinking, to doing to evolve to a higher level within Kolb’s learning theory growth spiral.
Study Limitations
Even though this study revealed new information regarding a novel use of the ANTS
tool, there were four main limitations observed. First, participants had limited exposure to the
ANTS framework and to the ANTS tool. Participants received 60 minutes of didactic and
interactive education to ANTS. Other studies have cited 1 hour to 4 hours to an indefinite amount
of time (Fletcher et al., 2002; Fletcher et al., 2003a; Fletcher et al., 2003b; Morgan et al., 2011;
Salvodelli et al., 2006; Welke et al., 2009; Yee et al., 2005; Zausig et al., 2009).
Also, it was difficult for one faculty to comprehensively observe participant NTS in a
real-time environment. No video-replay was used to complete the ANTS tool ratings just as
participants did not have video-replay to view their performance. This could have contributed to
poor correlation of quantitative ANTS tool ratings. A majority of published studies using the
ANTS tool used video replay, but only for faculty inter-rater reliability of student performance,
and not directly to faculty ANTS tool ratings to student self-evaluation of ANTS tool ratings
(Fletcher et al., 2002; Fletcher et al., 2003a; Fletcher et al., 2003b; Morgan et al., 2011; Salvodelli
et al., 2006; Welke et al., 2009; Yee et al., 2005; Zausig et al., 2009).
Additionally, certain aspects of communication were difficult to attribute to a specific
ANTS tool category. For example, a participant asked for the circulating nurse for help, but the
circulating nurse was scripted not to help and said nothing. This lack of communication and
resultant participant behavior translated across “Situation Awareness,” “Decision-Making,” and
“Team Working” categories. As a result participants and faculty likely commented and rated in
79
different categories about this same communication interaction. As a result, it would have been
difficult to apply any correlation or congruency across groups. ANTS tool authors documented
this aspect when the tool was released for public use and it has remained a current limitation of
the ANTS tool (Fletcher et al., 2003a; Fletcher et al., 2003b).
Finally, a limitation specific to the ANTS tool was that the tool evaluated individual
behavior and did not distinguish skills for working with different scenarios (e.g., the surgical
team and discussing the anesthetic plan with the patient or dealing with a crisis situation).
Therefore, because four different scenarios were used for this study, it was likely that different
behaviors were highlighted in the various ANTS categories and elements. This could have
skewed the ANTS tool ratings and comments because different scenarios evoked different
participant behaviors. The authors believed the ANTS tool represented NTS that are necessary
for all aspects of performance, and should not be separated or different for specific scenarios
(Fletcher et al., 2003a).
Conclusions and Implications
Conclusions
Quantitative and qualitative data in “Situation Awareness” category confirm current
understanding that anticipation is part of the construct of the simulated environment.
Hypervigilance is an overarching theme that not only emerges from the “Situation Awareness”
category, but also crosses multiple ANTS tool categories, consistent with current literature. The
Decision-Making category additionally revealed a significant relationship between faculty and
participant ratings of the element re-evaluating, indicating that participants are on target in this
area of self-evaluation. Disparities between participant self-evaluation and faculty evaluation of
80
NTS exist consistent with current literature related to student self-evaluation emphasizing the
importance of faculty benchmarking and input in student self-evaluations.
The majority of faculty written perspectives related to using the ANTS tool highlighted
observations about what was not performed as opposed to actual performance. This is an
important factor for faculty to consider in providing feedback to students. Communication as an
overarching theme crosses all ANTS tool categories consistent with the ANTS developers’
claims. Feeling alone is an important factor that relates back to communication for students
participating in intense, high fidelity simulated experiences. The surgical drape represents an
interactive communication barrier that is an important factor and can limit ability to adapt from a
faculty perspective in high fidelity simulated experiences. The overall lack of correlation and
congruence between participant and faculty ANTS tool evaluations in other categories does not
decrease the value of self-evaluation in simulated experiences as indicated in qualitative
comments on the ANTS instrument.
Implications
The research study has the following implications for teaching and learning NTS in a
simulated environment:
1. Because of the lack of correlation and congruency in participant and faculty evaluation,
more time needs to be allotted to educating students about NTS in anesthesia education.
2. Scenarios focusing on communication should be incorporated in simulation curriculum.
3. Student self-evaluation requires faculty feedback to improve insight and accuracy into
professional growth.
4. Because student self-evaluations were inaccurate more self-evaluation experiences are
needed along with faculty feedback.
81
Recommendations for Further Study
Retrospection upon this research protocol generated several potential future studies that could be
useful to this body of information including:
1. Replication of this study using a larger sample size to fulfill the power analysis to
determine if truly there is any correlation between student and faculty ANTS tool scores.
2. Utilization of video-taped student performance to facilitate self-assessment skills and
determine if correlations exist between student and faculty ANTS tool scores.
3. Examination of whether more than one exposure for students will lead to increased
ANTS tool self-evaluation accuracy.
4. Exploration of one-to-one interaction with faculty and students post-scenario and whether
this would lead to improved correlation of ANTS tool ratings after a second simulated
exposure.
5. Utilization of the ANTS tool in the clinical setting at the end of the day between student
and clinical preceptor / faculty to rate student performance of NTS together and student
perceived development of ANTS.
6. Exploration of scenarios where no surgical drape is present and students’ perceived
ANTS self-evaluations compared to faculty evaluations.
This study demonstrated the ANTS tool could provide useful insight into student self-
evaluation of simulated performance. Comparing student ANTS tool self-evaluation to faculty
evaluation of students’ simulated performance re-affirmed that students do not have accurate
insight into their own clinical performance and need faculty input. This study re-confirmed that
hypervigilance exists and is a natural part of learning in simulation. This study also revealed that
students need more practice in communication using dynamic simulated scenarios. These
82
scenarios brought out that students feel alone and use the surgical drape as a physical and
emotional barrier. The value of collecting and analyzing qualitative data to formulate a deeper
understanding cannot be underestimated. This rich data provides new information and knowledge
regarding exposure and development of ANTS in graduate nurse anesthesia students and supports
further development of simulation curricula to enhance clinical and professional development.
83
REFERENCES
Aebersold, M. & Tschannen, D. (2013). Simulation in nursing practice: The impact on patient
care. The Online Journal of Issues in Nursing,18(2), Manuscript 6. doi:
10.3912/OJIN.Vol18No02Man06.
Aggarwal, R., Mytton, O. T., Derbrew, M., Hananel, D., Heydenburg, M., Issenberg, B., et al.
(2010). Training and simulation for patient safety. Quality and Safety in Healthcare,
19(Suppl 2), i34-i43. doi:10.1136/qshc.2009.038562.
Altman, D. E., Clancy, C., & Blendon, R. J. (2004). Improving patient safety: Five years after the
IOM report. New England Journal of Medicine, 351(20), 2041-3. doi: 10.4103/0972-
5229.42559.
Armstrong, E. & Parsa-Parsi, R. (2005). How can physician’s learning styles drive educational
planning. Academic Medicine, 80(7), 680-684. doi: 10.1097/00001888-200507000-00013.
Arora, S., Miskovic, D., Hull, L., Moorthy, K., Aggarwal, R, Johannsson, H., et al. (2011). Self
vs expert assessment of technical and non-technical skills in high fidelity simulation.
American Journal of Surgery, 202(4), 500-506. doi: 10.1016/j.amsurg.2011.01.024.
Baker, D, Gustafson, S., Beaubain, M. Salas, E., Barach, P. (2005). Medical team training
programs in healthcare. In: Henriksen, K. Battles, J. B., Marks, E. S., Lewin D. I. (Eds.),
Advances in Patient Safety: From Research to Implementation. Rockville, MD: Agency
for Healthcare Research and Quality.
84
Baker, V. O., Cuzzola, D., Knox, C., Liotta, C., Cornfield, C. S., Tarkow, R. D., & Carlson, J. N.
(2015). Teamwork education improves trauma team performance in undergraduate health
professional students. Journal of Educational Evaluation for Health Professions, 12, 1-6.
doi: 10.3352/jeehp.2015.12.36.
Blum, R. H., Raemer, D. B., Carroll, J. S., Sunder, N., Felstein, D. M., & Cooper, J. B. (2004).
Crisis resource management training for anaesthesia faculty: A new approach to
continuing education. Medical Education, 38(1), 45-55.
Boud, D. Falchikov, N. (1989). Quantitative studies of student self-assessment in higher
education: a critical analysis of findings. Higher Education, 18, 529-549.
Brewster, L. P., Risucci, D. A., Joehl, R. J., Littooy, F. N., Temeck, B. K., Blair, P G., et al.
(2008). Comparison of resident self-assessments with trained faculty and standardized
patient assessments of clinical and technical skills in a structured education module.
American Journal of Surgery, 195, 1-4.
Byrne, A., J., Turner, M., Lewis, E., Murphy, A., Chawathe, M. P., & Thomas, P. (2015).
Training for non-technical skills measurement. Anaesthesia, 70(3), 368-369. doi:
10.111/anae.13021.
Chopra, V., Gesink, B., De Jong, J., Bovill, J., Spierdijk, J., & Brand, R. (1994). Does training on
an anesthesia simulator lead to improvement in performance? British Journal of
Anaesthesia, 73(3), 293-297. doi: 10.1093/bja/73.3.293.
Cooper, G. E., White, M. D., & Lauber, J. K. (1980). Resource Management on the Flightdeck:
Proceedings of a NASA/Industry Workshop. (NASACP-2120). Moffett Field, CA: NASA-
Ames Research Center.
85
Crossley, J., Marriott, J., Purdie, H., Beard, J. D. (2011). Prospective observational study to
evaluate NOTSS (Non-technical skills for surgeons) for assessing trainees’ non-technical
performance in the operating theatre. British Journal of Surgery, 98, 1010-1020. doi:
10.1002/bjs.7478
Davis, D. A., Mazmanian, P. E., Fordis, M., Harrison, R. V., Thorpe, K. E., & Perrier, L. (2006).
Accuracy of physician self-assessment compared with observed measures of competence:
A systematic review. JAMA, 296(9), 1094-1102. doi: 10.1001/jama.296.9.1094.
DeAnda, A., & Gaba, D. (1991). The role of experience in the response to simulated critical
incidents. Anesthesia Analgesia, 72(3), 308-315.
Dedy, N. J., Bonrath, M., Zevin, B., & Grantscharov, T. P. (2013). Teaching nontechnical skills
in surgical residency: A systematic review of current approaches and outcomes. Surgery,
154(5), 1000-1008. doi: 10.1016/j.surg.2013.04.034
Devitt, J., Kurrek, M., & Cohen, M. (1998). Can a simulator based evaluation be used to assess
anesthesiologists (abstract)? Anesthesiology, 89(3), A1172.
Devitt, J. H., Kurrek, M. M., Cohen, M. M. Cleave-Hogg, D. (2001). The validity of performance
assessments using simulation. Anesthesiology, 95(1), 36-42.
Epstein, R. M. (1999). Mindful practice. JAMA, 282(9), 833-839.
Epstein, R. M., & Hundert, E. M. (2002). Defining and assessing professional competence.
JAMA, 287(2), 226-235. doi: 10.1001/jama.287.2.226.
Falchikov, N. & Boud, D. (1989). Student self-assessment in higher education: A meta-analysis.
Review of Educational Research, 59, 395-430.
86
Federal Aviation Administration. (2013). Crew resource management training in Part 135 training
programs and OpSpec A037 and A038 for Basic Part 135 operators. Retrieved from
www.faa.gov/documentLibrary/media/Notice/N_8900.209.pdfhttp://www.faa.gov/docu
mentLibrary/media/Notice/N_8900.209.pdf
Fletcher, G. C. L., Flin, R. H., & McGeorge, P. (2003a). The Identification and measurement of
Anaesthetists’ non-technical skills: Review of human factors research in anesthesia,
workpackage 1 report. Retrieved from http://www.abdn.ac.uk/iprc/ants/
Fletcher, G. C. L., Flin, R. H., & McGeorge, P. (2003b). The Identification and measurement of
Anaesthetists’ non-technical skills: Review of behavioral marker systems in anaesthesia,
workpackage 2 report. Retrieved from http://www.abdn.ac.uk/iprc/ants/
Fletcher, G. C. L., Flin, R. H., & McGeorge, P. (2003c). The Identification and measurement of
Anaesthetists’ non-technical skills: Interview study to identify anaesthetists’ non-technical
skills, workpackage 3 report. Retrieved from http://www.abdn.ac.uk/iprc/ants/
Fletcher, G. C. L., Flin, R. H., & McGeorge, P., Glavin, R., Maran, N., & Patey, R. (2003a). The
Identification and measurement of Anaesthetists’ non-technical skills: Development of a
prototype behavioral marker systems for anaesthetists’ non-technical skills (ANTS),
workpackage 5. Retrieved from http://www.abdn.ac.uk/iprc/ants/
Fletcher, G., Flin, R., McGeorge, P, Glavin, R., Maran, N., & Patey, R. (2003b). Anaesthetists’
non-technical skills (ANTS): Evaluation of a behavioural marker system. British Journal
of Anaesthesia, 90(5), 580-588. doi: 10.1093/bja/aeg112.
Fletcher, G. C. L., McGeorge, P., Flin, R. H., Glavin, R. J., & Maran, N. J. (2002). The role of
non-technical skills in anaesthesia: review of current literature. British Journal of
Anaesthesia, 88(3), 418-429. doi: 10.1093/bja/88.3.418.
87
Flin, R. & Maran, N. (2004). Identifying and training non-technical skills for teams in acute
medicine. Quality and Safety in Healthcare, 13(Suppl 1), i80-i84.
Flin, R., O’Connor, P., Mearns, K. (2002). Crew resource management: improving safety in high
reliability industries. Team Performance Management, 8, 68-78.
Flin, R., Patey, R., Glavin, R., & Maran, N. (2010). Anaesthetists’ non-technical skills. British
Journal of Anaesthesia, 105(1), 38-44. doi: 10.1093/bja/aeq134.
Flin, R., Glavin, R., Maran, N., & Patey, R. (2012). ANTS Handbook. Retrieved from
http://www.abdn.ac.uk/iprc/uploads/files/ANTS%20Handbook%202012.pdfhttp://www.a
bdn.ac.uk/iprc/uploads/files/ANTS Handbook 2012.pdf
Forrest, F. C., Taylor, M. A., Postlethwaite, K., Aspinall, R. (2002). Testing validity of a high
fidelity simulator for assessment of performance: Development of a technical performance
scoring system and its application in the assessment of novice anaesthetists. British
Journal of Anaesthesia, 88, 228-44. doi: 10.1093/bja/88.3.338.
Gaba, D., Howard, S., Flanagan, B., Smith, B., Fish, K., & Botney, R. (1998). Assessment of
clinical performance during simulated crises using both technical and behavioral ratings.
Anesthesiology, 89(3), 8-18.
Gaba, D. M. (2000). Anesthesiology as a model for patient safety in healthcare. British Medical
Journal, 320, 785-788.
Gaba, D. M. (2004). The future of simulation in healthcare. Quality and Safety in Health Care,
13(Suppl 1), i2-i10. doi: 10.1136/qshc.2004.009878.
Gaba, D. M., Fish, K. J., & Howard, S. K. (1994). Crisis Resource Management in
Anesthesiology. Churchill Livingstone: Philadelphia.
88
Gaba, D. M., Howard, S. K., Fish, K. J., Smith, B. E., & Sowb, Y. A. (2001). Simulation-based
training in anesthesia crisis resource management (ACRM): a decade of experience.
Simulation & Gaming, 32(2), 175-193.
Gaba, D. M. (2010). Crisis resource management and teamwork training in anaesthesia. British
Journal of Anaesthesia, 105(1), 3-6. doi: 10.1093/bja/aeq124.
Gale, T. C., Roberts, M. J., Sice, P. J., Langton, J. A., Patterson, F. C., et al. (2010). Predictive
validity of a selection centre testing non-technical skills for recruitment to training in
anaesthesia. British Journal of Anaesthesia, 105(5), 603-309. doi: 10.1093/bja/aeq228.
Gjerra, K., Jepsen, R. M., Rewers, M., Ostergaard, D., & Dieckmann, P. (2015). Exploring the
relationship between anesthesiologists’ technical and non-technical skills. Acta
Anaesthesiologica Scandinavica, Advance online publication.doi:10.1111/aas.12598.
Gordon, M. J. (1991). A review of the validity and accuracy of self-assessments in health
professions training. Academic Medicine, 66(12), 762-769.
Graham, J., Hocking, G., & Giles, E. (2010). Anaesthesia non-technical skills: Can anaesthetists
be trained reliably to use this behavioural marker system in one day? British Journal of
Anaesthesia, 104(4), 440-445. doi: 10.1093/bja/aeq032.
Helmreich, R. L., Chidester, T. R., Foushee, H. C., Gregorich, S. E., & Wilhelm, J. A. (1990).
How effective is Cockpit Resource Management training? Issues in evaluating the impact
of programs to enhance crew coordination. Flight Safety Digest, 9(5), 1-17. Arlington, VA:
Flight Safety Foundation.
Helmreich, R. L., Wilhelm, J. A., Kello, J., Taggart, W. R., & Butler, R. E. (1991). Reinforcing
and evaluating crew resource management: Evaluator/LOS instructor reference manual.
Austin: NASA/UT Technical Manual 90-2. Austin, TX: University of Texas Publishing.
89
Helmreich, R. L. (2000). On error management: lessons from aviation. British Medical Journal,
320(7237), 781-785.
Helmreich, R. L., & Merritt, A.C. (1998). Culture at Work in aviation and Medicine. Aldershot,
Hants: Ashgate.
Helmreich, R. L., Merritt, A. C., Wilhelm, J. A. (2001). The evolution of crew resource
management training in commercial aviation. International Journal of Aviation
Psychology, Retrieved from
http://www.psy.utexas.edu/psy/helmreich/Evolution_IJAP_for_Dist.htm
Henrichs, B., Rule, A., Grady, M., & Ellis, W. (2007). Nurse anesthesia students’ perceptions of
the anesthesia patient simulator: a qualitative study. AANA Journal, 70(3), 219-225.
Heskin, L., Mansour, E, Lane, B., Kavanagh, D., Dicker, P., Ryan, D., et al. (2015). The impact
of a surgical boot camp on early acquisition of technical and nontechnical skills by novice
surgical trainees. American Journal of Surgery, 210(3), 570-577. doi:
10.1016/j.amjsurg.2014.12.046
Holzman, R. S., Cooper, J. B., Gaba, D. M., Philip, J. H., Small, S. D., & Feinstein, D. (1995).
Anesthesia crisis resource management: Real-life simulation training in operating room
crises. Journal of Clinical Anesthesia, 7(8), 675-687.
Hotchkiss, M. A., Biddle, C., Fallacaro, M. (2002). Assessing the authenticity of human
experience in anesthesiology. AANA Journal, 70(6), 470-473.
Howard, S., Gaba, D., Fish, K., Yang, G., & Sarnquist, F. (1992). Anesthesia crisis resource
management training: Teaching anesthesiologists to handle critical incidents. Aviation
Space and Environmental Medicine, 63(9), 763-770.
90
Hull, L., Arora, S., Aggarwal, R., Darzi, A., Vincent, C., Sevdalis, N. (2012). The impact of
nontechnical skills on technical performance in surgery: A systematic review. Journal of
the American College of Surgeons, 214(2), 214-230. doi:
10.1016/j.jamcollsurg.2011.10.016
Isaak, R. S. & Stiegler, R. P. (2015). Review of crisis resource management (CRM) principles in
the setting of intraoperative malignant hyperthermia. Journal of Anesthesia. doi:
10.1007/s00540-015-2115-8.
Issenberg, S. B., McGaghie, W. C., Petrusa, E. R., Lee, G. D., & Scalese, R. J. (2005). Features
and uses of high fidelity simulations that lead to effective learning: A BEME systematic
review. Medical Teacher, 27(1), 10-28. doi: 10.1080/01421590500046924.
Joint Commission International Center for Patient Safety. (2009). Communication: a critical
component in delivery quality care. Commission on Accreditation of Healthcare
Organizations. Oakbrook Terrace, IL.
Kanki, B. G., Helmreich, R. L., & Anca, J. (eds.). (2010). Crew Resource management (2nd ed.).
San Diego, CA: Press/Elsevier.
Kanthan R. & Senger, J. B. (2011). An appraisal of students’ awareness of self-reflection in a
first-year pathology course of undergraduate medical/dental education. BMC Medical
Education, 11(67). Retrieved from bmcmededuc.biomedcentral.com/articles/10.1186/1472-
6920-11-67.
Kassab, E. S., King, D., Hull, L. M., Arora, S., Sevdalis, N., Kneebone, R. L., et al. (2010). Actor
training for surgical team simulations. Medical Teacher, 32, 256 – 258.
Klein, G. A., Calderwood, R., & MacGregor, D. (1989). Critical Decision Method for Eliciting
Knowledge. IEEE Transaction on Systems, Man and Cybernetics, 19(3), 462-472.
91
Koetsier, E., Boer, C., & Loer, S. A. (2011). Complaints and incident reports related to anesthesia
service are foremost attributed to nontechnical skills. European Journal of Anesthesiology,
28, 29-33. doi: 10.1097/EJA.0b013e32833e364c.
Kohn, L. T., Corrigan, J., & Donaldson, M. S. (2000). To err is human: Building a safer
healthcare system. Washington, DC: National Academy Press.
Kolb, D. A. Experiential Learning Experience as the Source of Learning and Development (2nd
ed.). (2015). Upper Saddle River, NJ: Pearson Education, Inc.
Kurrek, M.,Devitt, J., Cohen, M., & Szalai, J. (1999). Inter-rater reliability between live-
scenarios and video recordings in a realistic scenario. Paper presented at the Society for
Technology in Anesthesia Annual Meeting, San Diego, CA.
Kurrek, M., Small, S., Feinstein, D., Russell, R., Fish, K., & Gaba, D. (1996). Anaesthesia crisis
resource management (ACRM) instructor course: Teaching the teachers (abstract).
Anesthesia Analgesia, 82(3), S256.
Larsson, J., & Holmstrom, I. K. (2013). How excellent anaesthetists perform in the operating
room theatre: a qualitative study on non-technical skills. British Journal of Anaesthesia,
110(1), 115-121. doi: 10.1093/bja/aes359.
Lauber, J. (1984). Resource management in the cockpit. Air Line Pilot, 53, 20-23.
Lupien, A. E. (2007). High fidelity patient simulation. In M.J. Bradshaw & A. J. Lowenstein
(Eds.), Innovative Teaching Strategies in Nursing and Related Health Professions (4th ed.).
(197-215). Sudbury, MA: Jones & Bartlett Publishers.
Lyk-Jensen, H. T., Jepsen, H. G., Spanager, L., Dieckmann, P., & Ostergaard, D. (2014).
Assessing nurse anaesthetists’ non-technical skills in the operating room. Acta Anaesthesiologica
Scandinavica, 58, 794-801. doi: 10.1111/aas.12315.
92
Mann, K. Gordon, J., MacLeod, A. (2009). Reflection and reflective practice in health
professions education: a systematic review. Advances in Health Science Education: Theory
and Practice, 14(4), 595-621.
Mann, K. V., Mcfetridge-Durdle, R., Martin-Misener, R., Clovis, J., Rowe, R., Beanlands, H., et
al. (2009). Interprofessional education for students of the health professions: The seamless
care model. Journal of Interprofessional Care, 23(3), 224-233. doi:
10.1080/13561820802697735.
Moorthy, K., Munz, Y, Adams, S., Pandy, V. Darzi, A. (2006). Self-assessment of performance
among surgical trainees during simulated procedures in a simulated operating theater.
American Journal of Surgery, 192(1), 114-118. doi: 10.1016/j.amjsurg.2005.09.017.
Morey, J., Simon, R., Jay, G., Wears, R. L., Salisbury, M., et al. (2002). Error reduction and
performance improvement in the emergency department through formal teamwork training:
Evaluation results of MedTeams project. Health Services Research, 37(6), 1553-1581.
Morgan, P. J., Cleave-Hogg, D. (2000). Evaluation of medical students’ performance using the
anesthesia simulator. Medical Education, 34, 42-45.
Morgan, P. J., Kurrek, M. M., Bertram, S., LeBlanc, V., & Przybyszewski, T. (2011).
Nontechnical skills assessment after simulation-based continuing medical education.
Simulation in Healthcare, 6(5), 255-259. doi: 10.1097/SIH.ob013e31821dfd05.
Murray, D. M., Boulet, J. R., Kras, J. F., McAllister, J. D., Cox, T. E. (2005). A simulation-based
acute skills performance assessment for anesthesia training. Anesthesia Analgesia, 101(4),
1127-1134.
93
Murray, D. M., Boulet, J. R., Avidan, M., Kras, J. F., Henrichs, B., Woodhouse, J., Evers, A. S.
(2007). Performance of residents and anesthesiologists in a simulation-based skills
assessment. Anesthesiology, 107(5), 705-713. doi: 10.1097/01.anes.0000286926.01083.9d.
Najjar, R. H., Lyman, B., & Miehl, N. (2015). Nursing students’ experiences with high fidelity
simulation. International Journal of Nursing Education Scholarship, 12(1), 27-35.
Navarro-Martinez, J., Galvez, C., Rivera-Cogollos, M. J., Galiana-Ivars, M., Bolufer, S., &
Martinez-Adsuar, F. (2015). Intraoperative crisis resource management during a non-
intubated video-assisted thoracoscopic surgery. Annals of Translational Medicine, 3(8), 1-
9. doi: 10.3978/j.issn.2305-5839.2015.04.14.
Nguyen, N., Elliott, J. O., Watson, W. D., & Dominguez, E. (2015). Simulation improves
nontechnical skills performance of residents during the perioperative and intraoperative
phases of surgery. Journal of Surgical Education, 72(5), 957-963. doi:
10.1016/j.jsurg.2015.03.005
Nishisaki, A., Keren, R., Nadkarni, V. (2007). Does simulation improve patient safety: Self-
efficacy, competence, operational performance, and patient safety. Anesthesiology Clinics,
25, 225-236. doi: 10.1016/j.anclin.2007.03.009.
Pandy, V. A., Wolfe, J. N., Black, S. A., Cairols, M., Liapis, C. D., & Bergqvist, D. (2008). Self-
assessment of technical skill in surgery: the need for expert feedback. Annals of the Royal
College of Surgeons of England, 90, 286 – 290. doi: 10.1308/003588408X286008
94
Patey, R., Flin, R., Fletcher, G., Maran, N., Glavin, R. (2005). Developing a Taxonomy of
Anesthetists’ Nontechnical Skills (ANTS), in K. Henriksen, J. Battles, E. Marks, et al
(Eds.), Advances in Patient Safety: From Research to Implementation (Volume 4:
Programs, Tools, and Products) (pp. 326-336). Rockville: Agency for Healthcare Research
and Quality.
Peyre, S. E., MacDonald, H., Al-Marayati, L., Templeman, C. & Muderspach, L. I. (2010).
Resident self-assessment versus faculty assessment of laparoscopic technical skills using a
global rating scale. International Journal of Medical Education, 1, 37-41. doi:
10.5116/ijme.4bf1.c3c1
Polit, D. E., & Beck, C. T. (2012). Nursing Research: Generating and Assessing Evidence for
Nursing Practice (9th ed.). Philadelphia: Wolters Kluwer Health/Lippincott Williams &
Wilkins.
Rall, M., Gaba, D. M., Howard, S. K., & Dieckmann, P. (2010). Human performance and patient
safety. In Miller, R. D. (Ed.), Miller’s Anesthesia (pp 93-149). Philadelphia, PA: Churchill
Livingstone.
Reader, T., Flin, R., Lauche, K., Cuthbertson, B. H. (2006). Non-technical skill in the intensive
care unit. British Journal of Anaesthesia, 96(5), 551-559. doi: 10.1093/bja/ael067.
Riem, N., Boet, M. D., Bould, D., Tavares, W., & Naik, V. N. (2012). Do technical skills
correlate with non-technical skills in crisis resource management: a simulation study.
British Journal of Anaesthesia, 109(5), 723-728. doi: 10.1093/bja/aes256
95
Reznek, M., Smith-Coggins, R., Howard, S, Kiran, K., Harter, P., Sowb, Y., Gaba, D., &
Krummel, T. (2003). Emergency medicine crisis resource management (EMCRM): Pilot
study of a simulation-based crisis management course for emergency medicine. Academic
Emergency Medicine, 10(4), 386-389. doi: 10.1111/j.1553-2712.2003.tb01354.x.
Salas, E., Prince, C., Bowers, C., Clint, A., Stout, R., Oser, R., Cannon-Bowers, J. (1999). A
methodology for enhancing crew resource management training. Human Factors, 41(1),
161-172.
Salas, E., Edens, E., Bowers. (2001). Applying Resource Management in Organizations: A guide
for training professionals. Mahway, NJ: Lawrence Erlbaum Associates.
Saldana, J. (2013). The Coding Manual for Qualitative Researchers (2nd ed.). Los Angeles, CA:
Sage Publications, Ltd.
Savoldelli, G. L., Naik, V. N., Park, J., Joo, H. S., Chow, R., et al. (2006). Value of debriefing
during simulated crisis management: oral versus video-assisted oral feedback.
Anesthesiology, 105, 279-85. doi: 10.1097/00000542-200608000-00010.
Schön, D. A. (1983). The Reflective Practitioner: How Professionals Think in Action. New York:
Basic Books.
Schwid, H. A., Rooke, G. A., Carline, J., Steadman, R. H., Murray, W. B., Olympio, M., et al.
(2002). Evaluation of anesthesia residents using mannequin-based simulation.
Anesthesiology, 97(6), 1434-1444. doi: 10.1097/00000542-200212000-00015.
Shaw, G. B., Bennett, A. (1915). Current History: The European War. New York Times, Vol. 1,
p 979. Retrieved from
https://play.google.com/books/reader?id=50FIAQAAIAAJ&printsec=frontcover&output=r
eader&authuser=0&hl=en&pg=GBS.PP5 .
96
Spanager, L., Beier-Holgersen, R., Dieckmann, P., Konge, L., Rosenberg, J., & Ostergaard, D.
(2013). Reliable assessment of general surgeons’ non-technical skills based on video-
recordings of patient simulated scenarios. The American Journal of Surgery, 206(4), 810-
817. doi: 10.1016/j.amsurg.2013.04.002.
Toff, N. J. (2010). Human factors in anaesthesia: Lessons learned from aviation. British Journal
of Anaesthesia, 105(1): 21-25. doi: 10.1093/bja/aeq127.
Wagner, F. and Ison, D. C. (2014). Crew resource management application in commercial
aviation. Journal of Aviation Technology and Engineering, 3(2), 2-14. doi:
10.7771/2159-6670.1077.
Ward, M., MacRae, H., Schlachta, C., Mamazza, J., Poulin, E. Reznick, R. et al. (2003). Resident
self-assessment of operative performance. American Journal of Surgery, 185, 521 – 524.
Weick, K. (1990). The vulnerable system: an analysis of the Tenerife air disaster. Journal of
Management, 16, 571-593.
Welke, T. M., LeBlanc, V. R., Savoldelli, G. L., Joo, H. S., Chandra, D. B., Crabtree, N. A.,
Naik, V. N. (2009). Personalized oral debriefing versus standardized multimedia
instruction after patient crisis simulation. Anesthesia Analgesia, 109(7), 183-189. doi:
10.1213/ane.0b013e3181a324ab.
Wiener, E., Kanki, B., Helmreich, R. (1993). Cockpit Resource Management. Gulf Professional
Publishing: Oxford, United Kingdom.
Wunder, L. L. (2016). Effect of a non-technical skills intervention on first-year student registered
nurse anesthetists’ skills during crisis simulation. AANA Journal, 84(1), 46-51.
97
Yee, B., Naik, V. N., Joo, H. S., Savoldelli, G. L., Chung, D. Y., Houston, P. L. (2005).
Nontechnical skills in anesthesia crisis management with repeated exposure to
simulation-based education. Anesthesiology, 103(2), 241-248. doi: 10.1097/00000542-
200508000-00006.
Yule, S., Flin, R, Maran, N., Rowley, D., Youngson, G., & Paterson-Brown, S. (2008). Surgeons’
Non-technical skills in the operating room: Reliability testing of the NOTSS behavior
rating system. World Journal of Surgery, 32(4), 548-556. doi: 10.1007/s00268-007-9320-
z.
Yule, S., Flin, R., Paterson-Brown, S., Maran, N., & Rowley, D. (2006). Development of a rating
system for surgeons’ non-technical skills. Medical Education, 40, 1098-1104. doi:
10.1111/j.1365-2929.2006.02610.x
Yule, S., Parker, S. H., Wilkinson, J., McKinley, A., MacDonald, J. Neill, A., et al. (2015).
Coaching non-technical skills improves surgical residents’ performance in a simulated
operating room. Journal of Surgical Education, 72(6), 1124-1130. doi:
10.1016/j.jsurg.2015.06.012
Zausig, Y. A., Grube, C., Boeker-Blum, T., Busch, C. J., Bayer, Y., Sinner, B. (2009).
Inefficiency of simulator-based training on anesthesiologists’ non-technical skills. ACTA
Anaesthesiologica Scandinavica, 53, 611-61. doi: 10.1111/j.1399-6576.2009.01946.x.
98
APPENDIX A
ANTS Tool Front Page
99
ANTS Tool Front Page (Patey et al., 2005)
100
APPENDIX B
ANTS Tool Back Page
101
ANTS Tool Back Page (Patey et al., 2005)
102
APPENDIX C
Evolution of Crew Resource Management
103
Evolution of Crew Resource Management (Helmreich, Merritt, Wilhelm, 2001; Wagner & Ison, 2014) First Generation: Curriculum was psychological in nature and referred to as cockpit resource management, emphasizing changing individual styles of behavior and interpersonal skills, encouraging assertiveness by junior pilots. Employed games and exercises unrelated to aviation to illustrate concepts. Second Generation: Curriculum focused on concepts such as team building, specific aviation concepts, situation awareness, briefing strategies and stress management and was referred to as crew resource management. Employed training as an intensive seminar with specific modules and exercises unrelated to aviation to demonstrate concepts. Third Generation: Curriculum began to proceed down multiple paths to reflect characteristics of the aviation system, organizational safety culture. Training extended to check airmen, dispatchers, and maintenance personnel. Fourth Generation: The Federal Aviation Administration initiated the voluntary Advanced Qualification Program (AQP) allowing air carriers to develop innovative training fitting the needs of the specific organization. Curricula requirements included CRM training, Line-Oriented Flight Training (LOFT), and integration of CRM concepts into technical training. Fifth Generation: Curricula aimed at the normalization of error and the developing strategies for managing error. Training acknowledged that effective error management was the hallmark of effective crew performance. Sixth Generation: Curricula focuses on threats and errors that crews manage to ensure flight safety. Human-technology interface, timely acquisition of information interpersonal activities, leadership, team formation, problem solving, decision-making, and situation awareness are included as part of the training.
104
APPENDIX D
CRM Threat and Error Model
105
Figure 5: CRM Threat and Error Model (Helmreich, 2000)
106
APPENDIX E
Anesthesia Crisis Resource Management Key Points
107
Anesthesia Crisis Resource Management Key Points (Gaba et al., 2001)
Decision-making & Cognition Teamwork & Resource management
Know the environment Anticipate and plan Use all available information & cross check Prevent/manage fixation errors Use cognitive aids
Exercise leadership & followership Call for help early Communicate effectively Distribute the workload Mobilize all available resources
108
APPENDIX F
Annotated Bibliography of NTS Studies
109
Annotated Bibliography of NTS Studies
Author, Date
Study Design/ Research
Question or hypothesis
Sample & Setting
Description, Size (n)
Data Collection Methods
Study Tools Results
Arora et al.,
(2011)
Prospective, observational,
non-experimental
design
Explore relationship
b/w self-assessment and expert
assessment of both technical
and NTS
26 surgical residents
in a simulated OR setting
with 4 professional actors
Direct observation and also via video tape of both
technical skills and NTS using 2 faculty assessors
Technical skills: Objective Structured
Assessment of Technical Skills (OSATS) tool NTS: Non-
Technical Skills for Surgery
(NOTSS) tool
Interrater reliability for
technical skills (ρ = .622, P
<.05), NTS, (ρ = .684, P <
.05) Overall strong
correlation among
technical skill assessments,
but no significant correlation with NTS
Fletcher, Fletcher
et al., (2003b)
Prospective, observational,
non-experimental
design
Investigate experimental
validity, reliability, and
usability of ANTS
tool/system
50 experience
d anesthetist
s with 4 hours of
ANTS tool training
observed 8 HFS
videos and rated
performance using
ANTS tool
Direct observation of
video-tape performance of participants in
HFS
Written completion of questionnaire
regarding ANTS system usage
ANTS tool
Questionnaire about ANTS tool
Interrater reliability (IRR) (r = 0.55-0.67)
average for all elements/categ
ories Accuracy >
88% to 1 scale pt
Internal consistency (Cronbach α 0.79-0.86)
Overall
usability and acceptability was > 90%
110
Gjerra et al.,
(2015)
Explorative, mixed-
methods design
Are the ratings
of trainee anesthesiologists’ NTS and
TS correlated?
What characterizes
the trainee anesthesiologi
sts’ use of good or poor
NTS?
What concrete TS
characterize the good
management of unexpected
difficult airway?
Convenience sample
of 25 trainee
anesthesiologists
attending a 3-day
mandatory course
related to high
fidelity simulation
Direct observation of
previously recorded video performance of
participants
Modified ANTS tool called the
ANTSdk assessment
11-item
technical skills checklist
Demographic
data
NTS ICC (0.66)
TS ICC 0.73
6 themes were
identified relating to
good and poor difficult airway
management
Overall no correlation
between NTS and TS.
Graham, Hocking, & Giles, (2010)
Prospective, observational,
non-experimental
design
Investigate whether an 8-hour training
period can yield IRR (r >
0.7) using ANTS tool
26 practicing anesthetist
s participated in an 8-
hour ANTS
workshop, after
which 5 test videos
using ANTS tool were used to evaluate
IRR and
Direct observation of
video-tape performance of participants in
HFS
Written completion of questionnaire
before and after workshop
ANTS tool
Questionnaire (pre and post workshop)
ICC for each element of ANTS tool
never reached accepted level of 0.7 (range 0.11-0.62)
Poor match
b/w rater and expert scores for individual
elements especially decision-
making and
111
intra-class correlation (ICC) and compared to experts in using
ANTS tool
team working elements.
Good match
for categorical scores (all r >
0.7)
Koetsier, Boer, &
Boer, (2011)
Retrospective analysis
Investigate
whether anesthesia-
related complaints and incident
reports can be related to non-technical skills as defined by CanMEDS
All complaints
and incident reports
from the VU
University Medical Centre
Amsterdam between 2001-2007
were evaluated
Report analysis related to 7 roles as defined by CanMEDS: 1. Medical-expert 2. Communicator 3. Collaborator 4. Manager 5. Health advocate 6. Scholar 7. Professional
CanMEDS Physician
Competency Framework
169 out of 84,252 (0.2%) complaints/inci
dents were located related to anesthesia
77% of all
reports could be attributed to
roles of medical expert and manager where NTS
Riem et al.,
(2012)
Retrospective analysis
Determine
relationship between
technical skills (TS) and
nontechnical skills (NTS)
during a simulated
intraoperative crisis scenario
50 anesthesia residents’ prerecorde
d simulation
videos
Direct observation of
video tape, ANTS tool, and an
independently constructed TS
tool by 2 assessors who rated TS, and another 2 who
rated NTS
ANTS tool
TS tool/checklist developed by researchers
TS checklist (ICC = 0.92) ANTS tool
(ICC = 0.76)
TS/NTS correlation
0.45 suggesting there were statistically significant
correlations in total scores
from each tool
Wunder, (2016)
Quasi-experimental
32 nurse anesthesia students
Videotaped scenarios viewed
by faculty and scored using ANTS tool
ANTS tool Significant improvement
in NTS second exposure to
scenario (P = 0.025)
112
NTS are learned from training, not
from professional experience
Yee et al.,
(2005)
Prospective, observational,
non-experimental
design
Investigate effects of repeated ACRM
simulation on NTS skills
using ANTS tool
20 anesthesia residents
experienced 3
different HFS
scenarios (each 1 month apart)
starting with a
handoff with a post-
scenario debriefing following
each scenario.
Direct observation and video tape using
two faculty assessors and ANTS tool
ANTS tool (four main categories
evaluated: 1) Task
management 2) Team working
3) Situation awareness
4) Decision- making
Significant improvement
in mean scores b/w first and
second sessions (all P < 0.005) and first and third sessions (all P < 0.005)
No significant differences in mean category
scores b/w second and
third sessions.
Zausig et al.,
(2009)
Experimental, pre-test, post-
test design
Two different types of training (medical
management (MM) vs
NTS) would lead to a
difference in NTS
performance in a post-training
45 participant
s were randomly
assigned to different training (MM vs
NTS) in a HFS
scenario environment with a pretest
scenario offered before
Direct observation and
videotape of participant
performance using ANTS tool
and Delphi method to attain group consensus.
ANTS tool
Pre-test: NTS did not differ
between groups
Post-test: No statistical difference
between groups
No definitive improvement demonstrated after a single exposure to
NTS
113
simulator scenario
training and a post-
test scenario offered
after training
Larsson, &
Homstrom,
(2013)
Qualitative descriptive
phenomenological design
Describe the way excellent anaesthetists
act/perform in the operating room as seen
by experienced anaesthesia
nurses.
18 anesthesia
nurses participate
d in 5 focus
groups (4-5
participants per focus group) in Sweden from 5
different hospitals
Structured interview that was audio-taped using 8 pre-determined
questions
Audio-taped, structured
interview data
Six themes were identified: 1)Focused way
to approach work tasks
2) Clear and informative
plan for team and induction 3)Humility to
anesthesia complexity 4)Patient-centered
5)Fluent in practical work 6) Calm and
clear in critical situations
114
APPENDIX G
Simulated Scenario Objectives and Key Events
115
Simulated Scenario Objectives and Key Events
Scenario Learning Objectives Key Events
Emergence Hypertension
1. Demonstrate appropriate actions to resolve emergence hypertension
2. Discuss patient condition/changes with operating room team and colleagues
3. Differentiate among other causes of emergence hypertension (e.g., pain, essential hypertension, full bladder)
● Handoff ● Laparoscopic
cholecystectomy case is near the end
● Surgeon ready to move on to the next case...in a hurry
● Mild hypertension ensues (140/95 mmHg)
● Moderate hypertension develops (170/100 mmHg)
● Severe hypertension develops (210/120 mmHg)
● Resolution to mild hypertension as participant treats blood pressure with multiple modalities
Emergence Laryngospasm
1. Demonstrate appropriate actions to resolve laryngospasm
2. Discuss patient condition/changes with operating room team and colleagues
3. Differentiate among causes of emergence laryngospasm
● Handoff ● Tonsillectomy case is near
the end ● Surgeon is pushing to
extubate patient ● After extubation patient
cannot be ventilated and desaturation begins
● Severe hypoxemia develops rapidly
● Resolution after positive pressure ventilation given and consideration of succinylcholine
Refractory hypotension
1. Demonstrate appropriate actions to resolve hypotension
2. Discuss patient condition/changes with operating room team and colleagues
3. Differentiate among causes of hypotension related to this specific
● Handoff ● Orthopedic lower
extremity case is just beginning. Surgeon is trying out a new technique
● Blood loss begins as red lap pads are thrown about and suctioning
● Blood pressure exhibits mild hypotension (90/55
116
case mmHg) and tachycardia (105 beats/minute)
● Blood loss continues and moderate hypotension (78/40 mmHg) and worsening tachycardia continues (120 beats/ minute)
● Severe blood loss continues yielding severe hypotension (58/20 mmHg) and worsening tachycardia (140 beats/ minute)
● Resolution with crystalloid, colloid, assessing if type and cross has been performed, and sending for blood products
Intraoperative oxygen tank failure on anesthesia machine
1. Demonstrate appropriate actions to provide oxygen to the patient
2. Discuss patient condition/changes with operating room team and colleagues
3. Differentiate among causes of oxygen tank failure
● Handoff ● Laparoscopic
appendectomy is in the early phases and room is dark
● Whistle indicating no oxygen left in tank sounds on machine
● Resolution with locating extra tank in room, opening tank, attaching circuit to provide oxygen to patient and realizing that alternative methods of maintaining anesthetic depth must be also sought
117
APPENDIX H
Demographic Questionnaire
118
Demographic Questionnaire
1. What is your age? __________ 2. Are you male or female? (Circle one) 3. How many years have you been a nurse? _________ 4. How many years have you worked in a critical care setting? _________
Name: ____________________________
119
Appendix I
TWU IRB Approval Letter
120
