Post on 14-Dec-2015
Steven Yule, Rhona Flin, George Youngson University of Aberdeen Simon Paterson-Brown, Nikki Maran Royal
Infirmary of EdinburghDavid Rowley University of Dundee
Surgeons non-technical skills: Surgeons non-technical skills: Reliability testing of the NOTSS Reliability testing of the NOTSS
systemsystem
Project sponsors:Royal College of Surgeons of Edinburgh (RCSEd)
NHS Education for Scotland (NES)
Association for Surgical Education, Washington DC, April 2007
Right hepatic arteryRight hepatic artery
Common bile ductCommon bile duct
right hepatic ductright hepatic duct
Portal veinPortal vein
“High technical proficiency cannot guarantee safety”
Surgeons’ non-technical skills defined:“Behavioural aspects of performance in the operating theatre which underpin medical expertise, use of equipment and drugs. They are cognitive (i.e. situation awareness) and social (i.e. communication & teamwork) skills”
Research on non-technical skills
• Adverse events in surgery are primarily caused by failures in perception, judgement, communication and teamwork» Wilson et al (1999) - communication breakdown in 43% of
errors » de Leval et al (2000) – cardiac switch operations» Way et al (2003) – 97% of bile duct injuries had perception
failures» Gawande et al (2003; 2004) – insurance claims in the US
• Good non-technical skills can lead to positive outcomes for the team and patient» Edmondson (2003) – effective leadership» Carthey et al (2003) – behavioural markers in cardiac surgery» Healey et al (2004) – observing teamwork in surgery» Moorthy et al (2005) – team skills in operating theatre simulator
NOTSS project background
• Rationale» Support safe practice in the operating theatre and
surgeons’ professional training• Aim
» Identify the essential non-technical skills for surgeons
» Develop system for surgeons to rate skills in operating theatre and give structured feedback for training
» Establish reliability & usability of system• Scope
» Develop an education tool» Intraoperative environment only» For surgeons to observe, rate, and provide feedback
Developing the NOTSS system(based on Gordon, 1993, p.12)
Phase 1: Task analysis 1. Literature review (Surgery, 2006) 2. Adverse event analyses 3. Cognitive Interviews (n=27 surgeons, n=12 nurses) 4. Observation in theatre 5. Attitude survey (The Surgeon, 2006)
Phase 2: Design and development (Medical Education, 2006)
• Develop skills taxonomy• Refine categories and elements (4 panels of surgeons)• Write behavioural markers (n=16 surgeons)• Prototyping and usability testing
Phase 3: System evaluation• Finalize evaluation criteria – reliability and usability• Design evaluation strategy: rwg, in-theatre trials• Collect and analyse data (44 surgeons; c. 100 cases)• Modify system if necessary
Categories Elements
Situation Awareness
Gathering InformationUnderstanding InformationProjecting and anticipating future state
Decision Making Considering optionsSelecting and communicating optionImplementing and reviewing decisions
Communication and Teamwork
Exchanging informationEstablishing a shared understandingCo-ordinating team
Leadership Setting and maintaining standardsCoping with pressureSupporting others
NOTSS skills taxonomy v1.2(Yule et al, Medical Education, 2006)
NOTSS System Structure
Situation Awareness
Decision Making
Communication and Teamwork
Leadership
Gathering Information
Understanding Information
Projecting and anticipating future state
Surgeons’ Non-Technical Skills
Skill Categories
Skill Elements
Behavioural Markers
Good: Keeps anaesthetist informed about procedure (e.g. to expect bleeding) Poor: Waits for a predicted problem to arise before responding
NOTSS rating scale
1 Poor Performance endangered or potentially endangered patient safety, serious remediation is required
2 Marginal Performance indicated cause for concern, considerable improvement is needed
3 Acceptable Performance was of a satisfactory standard but could be improved
4 Good Performance was of a consistently high standard, enhancing patient safety; it could be used as a positive example for others
N/A Not Applicable. Skill was not required or not relevant in this case or scenario
NOTSS reliability study
Purpose» Test how reliably consultant surgeons use NOTSS to rate
behaviours in standardized scenariosScenario development
» Good & poor behavioural examples in 9 scenarios» ‘case mix’ of orthopaedic & general surgical scenarios » Simulator used as patient, surgeons and nurses as actors
Participants» n=44 consultant surgeons (mean experience 8.9 yrs (sd
7.5))» 2-3 hour training session
Analysis» Within group agreement (rwg)» Intraclass correlations (ICC)» Reference/ expert ratings
NOTSS video scenarios
Leadership
Situation AwarenessCommunication & Teamwork
Decision Making
Reliability study - results
Categories *rwg ICC(2)S ICC(2)A Elements rwg
Situation Awareness
.59 .95 .29 Gathering InformationUnderstanding InformationProjecting and anticipating future state
.65
.59
.58
Decision Making
.67 .99 .60 Considering optionsSelecting and communicating optionImplementing and reviewing decisions
.65
.63
.67
Communication and Teamwork
.71 .99 .39 Exchanging informationEstablishing a shared understandingCo-ordinating team
.59
.69
.63
Leadership .70 .99 .66 Setting and maintaining standardsCoping with pressureSupporting others
.59
.69
.72
*rated separately, not an average of element ratings
NOTSS system sensitivity
0
10
20
30
40
50
60
70
80
90
100
SituationAwareness
Decision Making Leadership Communication& Teamwork
All categories
NOTSS category
Mea
n %
of
par
tici
pan
ts a
gre
ein
g w
ith
th
e re
fere
nce
rat
ing
% of raters
Agreement by scenario
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Scenario 1General
Scenario 3General
Scenario 6General
Scenario 2Orthopaedic
Scenario 5Orthopaedic
Scenario 7Orthopaedic
Scenario number and surgery specialism depicted
rwg
(w
ith
in g
rou
p a
gre
emen
t: 0
-1 s
cale
)General (n=18)
Orthopaedic (n=11)
Critical level
Limitations
• Novel task for surgeons• Limited training for participants• Calibration would improve reliability• Inter-rater agreement will improve with
experience of using the system and non-technical language
Current research
• Controlled usability trial in the OR, using NOTSS to observe, rate and debrief on performance
• Examine the relationship between non-technical and technical skills
Contact: s.j.yule@abdn.ac.ukProject website: www.abdn.ac.uk/iprc/notss