Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation...
-
Upload
baldwin-pearson -
Category
Documents
-
view
221 -
download
0
description
Transcript of Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation...
![Page 1: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/1.jpg)
EVALUATING SURGICAL SKILL AND OR PERFORMANCE
Neal E. Seymour, MDBaystate Medical Center
Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab
2010 APDS Annual Meeting, San Antonio, Texas
![Page 2: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/2.jpg)
DisclosuresNo disclosures
![Page 3: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/3.jpg)
Evaluation of Surgical Skill Skills that are uniquely surgical
Evidenced in ORSurgical patient management
Observed during training eventsClinical (e.g. OR)Laboratory
Occurs throughout residencyProgression of skills
![Page 4: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/4.jpg)
Improved Task Performance
0
50
100
150
200
1 3 5 7 9 11 13 15 17
Tota
l Sco
re
Trial Number
VR Laparoscopic Suturing and Knot-Tying
Residents
Experts
Effect of VR Training on Laparoscopic Suturing Performance: Video Analysis
0
10
20
30
40
50
60
70
Pre-VR Post-VR
Video Score
Res.1Res.2Res.3Res.4Res.5Res.6Res.7Res.8Res.9
Effect of VR Training on Laparoscopic Suturing Performance: Time
0
50
100
150
200
250
300
350
400
Pre-VR Post-VR
Time (sec)
Res.1Res.2Res.3Res.4Res.5Res.6Res.7Res.8Res.9
![Page 5: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/5.jpg)
Proficiency-Based Lab Training
![Page 6: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/6.jpg)
p > 0.006*
Skills Transfer: “VR to OR” Study
0
1
2
3
4
5
6
7
8
9
10M
ean
num
ber o
f erro
rs
VR Trained Standard Training
*
![Page 7: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/7.jpg)
Assessment of Performance in OR
Procedure specific instruments GOALS OSATS Analysis of video recorded tasks Differences between investigative
instruments and assessment in frequent, routine use.Limitations on obtrusive methodsResource intensive
![Page 8: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/8.jpg)
Design of New Assessment Tool
User-friendly in routine use, valid, reliable.Opportunity for delivery of feedbackLeverage technology to manage large
volumes of data generated.Generate useful reports on resident
performance.
![Page 9: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/9.jpg)
Resident OR Performance Assessment at Baystate Medical Center
Objective: Develop OR performance assessment tool for routine use
OpRate: Dept. of Surgery and IS project:Multi-item global assessment of skillAttending surgeon raterEase of use = compliance with use.Interfaced to other databases
![Page 10: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/10.jpg)
Evaluated Items
3 preop (preparedness) questionsPatient, technical and disease-specific knowledge
6 open technical skills questionsTissue, instrument handling, knot tying, efficient
planning and ability to function independently 6 laparoscopic technical skills questions 4 point frequency-based responses for
specific technical performance behaviors
![Page 11: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/11.jpg)
![Page 12: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/12.jpg)
![Page 13: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/13.jpg)
Assessment of Residents with OpRateJuly 2004 - June 2006
15 29 21 36 3863
146101
237
113
050
100150200250
Appen
decto
my
Ventra
l Herni
a
Colon R
esec
tion
Cholecy
stecto
my
Inguin
al Hern
ia
Case
Num
ber
2004-5 2005-6
799 OpRate evaluations logged
579 cases reviewed 29 residents
33 attending surgeon evaluators◦ 6 contributed 75% of
assessments
![Page 14: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/14.jpg)
Preop Preparedness
0
20
40
60
80
100
120
140
160
PGY1 PGY2 PGY3 PGY4 PGY5
Training Year
Num
ber o
f Cas
es
3.5
3.6
3.7
3.8
3.9
4
Perfo
rman
ce S
core
Cases
Performance
Cronbach’s α (3 test items) = 0.80
![Page 15: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/15.jpg)
Laparoscopic Appendectomy
Cronbach’s α (4 test items) = 0.80
0
5
10
15
20
25
30
35
PGY1 PGY2 PGY3 PGY4 PGY5Training Year
Num
ber o
f Cas
es
2
2.2
2.4
2.6
2.8
3
3.2
3.4
3.6
3.8
4
Perfo
rman
ce S
core
Cases
Performance
![Page 16: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/16.jpg)
Laparoscopic Cholecystectomy
Cronbach’s α (4 test items) = 0.80
05
101520253035404550
PGY1 PGY2 PGY3 PGY4 PGY5Training Year
Num
ber o
f Cas
es
22.22.42.62.833.23.43.63.84
Perfo
rman
ce S
core
Cases
Performance
![Page 17: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/17.jpg)
Colon Resection
Laparoscopic (n = 37): Cronbach’s α (4 test items) = 0.82 Open (n = 36): Cronbach’s α (6 test items) = 0.84
0
5
10
15
20
25
30
35
40
PGY1 PGY2 PGY3 PGY4 PGY5Training Year
Num
ber o
f cas
es
2.5
2.7
2.9
3.1
3.3
3.5
3.7
3.9
Perfo
rman
ce S
core
CasesPerformance
![Page 18: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/18.jpg)
VR laparoscopic gall bladder excision (time) vs laparoscopic OR performancep = 0.0066, r2 = 0.4450 (Spearman correlation)
J Surg Res 2009;154:177-178
![Page 19: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/19.jpg)
Conclusions Experience with OpRate confirms predictions of
experience-dependent resident performance increases, but does not validate attending rating behaviors.
Still Required:Systematic rater trainingInter-observer agreementConcurrent validity studies (lab and OR)Mandatory compliance
![Page 20: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/20.jpg)
OpRate Build 2 CQL Server database Edit redundant items Add important items
Bimanual skillsAbility to assistDepth perception in laparoscopic cases
![Page 21: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/21.jpg)
![Page 22: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/22.jpg)
![Page 23: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/23.jpg)
Status of Use Remediation decisions 2 SD below mean for peer group
High likelihood of performance problems Compliance to be augmented
Variable compensation linked to useCompliance monitored and posted monthlyExpectation that residents will advocate for
use
![Page 24: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/24.jpg)
Patient Management and Teamwork Skills
![Page 25: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/25.jpg)
Electrical Burns VTE Septic Shock Hemorrhagic Shock Spinal shock Post-Op MI Anaphylaxis
FPS Curriculum Thoracic trauma Hyperkalemia Intraabdominal
catastrophe Pediatric trauma Pheochromocytoma
![Page 26: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/26.jpg)
Recognition of importance of teamwork orientation to effective clinical careStrong foundation principlesAlignment with growing body of work on “non-
technical skills” Effort to take advantage of simulated patient
care, simulated hospital environments (e.g. OR)Limited precedent in surgery
ACS-APDS Phase III Curriculum
![Page 27: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/27.jpg)
Unique communications rating instrument
Checklisted trauma assessment SPIKES protocol (delivering bad news) NOTECHS Ottawa Global Scale Need to strengthen and consolidate
assessment
Assessment
![Page 28: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.](https://reader035.fdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/28.jpg)
Conclusions Assessment of surgical skills and OR
performance:Opportunities
Expanded uses:Identification of performance outliers.Definition of areas where focused training is required
for specific residents.Measurement of effects of educational interventions.
Measurement of curriculum effectiveness