ProMIS Validation

Validated! PPPrrrooovvveeennn wwwiiittthhh FFFLLLSSS

BBBeeetttttteeerrr ttthhhaaannn VVVRRR

PPPrrreeefffeeerrrrrreeeddd bbbyyy uuussseeerrrsss

Augmented Reality combines VR

with a physical model allowing

virtual bleeding and real haptics.

ProMIS has pure VR Modules, eg

for Instrument Handling. Users

still use real instruments.

Vision technology enables

tracking of errors and automatic

calculation of dissected tissue.

ProMIS assesses performance on

real models where real haptics

are important, eg in Suturing and

Knot-tying

ProMIS and FLS

• ProMIS metrics are “excellent predictors of scores

in the standard FLS simulator” and “predict

readiness for FLS Certification”1 2 3

• ProMIS metrics are valid on peg transfer, pre-tied

loop placement and knot-tying tasks4

• “Initial trials of the metrics on the FLS Precision-

Cutting Task show that ProMIS is as accurate as

the current method” 5

• FLS tasks are transferable to the ProMIS simulator

with traditional FLS scoring and intrinsic ProMIS

metrics being good measurement tools. A ProMIS

total path length <4000 mm or total smoothness

<6000 reliably predicts a passing FLS score. 25

ProMIS vs pure VR

• ProMIS out-performed the virtual reality

simulators24

• Scores for ProMIS were significantly higher than

for SurgicalSIM for overall realism, thread

behavior, reflection of clinical ability, and overall

educational value.6

• In comparison with LapSim, ProMIS was regarded

by all participants as a better simulator for

laparoscopic skills training on all tested features7

• “Only [ProMIS] was able to distinguish between

advanced trainees and beginners (and) was

graded more realistic (70% vs 33%) and more

useful (83% vs 62%)” than Xitact8

• The ability of performance metrics of [ProMIS] to

discern predicted performance differences

between experts and non-experts was better

than for SimSurgery’s robotic surgery simulator and

SurgicalSIM 9

ProMIS: the preferred Simulator

• ProMIS can be used effectively with the DaVinci

robot to obtain performance data with robotic

instrumentation10

• Residents believe that ProMIS is easy to use and

improved their operative skills11

1 SAGES 2006 S064 Ritter et al,

2 SAGES 2006 P237 McCluney MD, et al,

3 SAGES 2007 P279 McCluney et al,

4 SAGES 2006 S065 Vuong et al,

5 SAGES 2007 ETP057 Young et al,

6 SAGES 2006 P224 Fellinger, et al,

7 World J Surg. 2007 Apr;31(4):764-72. Botden et al,

8 SAGES 2007 P270 Hahnloser et al,

9 SAGES 2007 S077 Lin et al,

10 SAGES 2006 Narula et al,

11 SAGES 2006 P219 Chang et al,

24 Heinrichs, et al 2007

25 Hungness, et al 2008

ProMIS surgical simulator

ProMIS Modules range from Basic Laparoscopic Skills to

MIS procedures like LapColectomy. For more information

on ProMIS, please contact us at:

Email: [email protected]

U.S. tel: +1 617 342 7270

RoW tel: +353 (0)1 676 7310

Validation

www.haptica.com | email: [email protected] | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310

1. SAGES 2006 Scientific Session S064

CONCURRENT VALIDITY OF AUGMENTED REALITY METRICS

APPLIED TO THE FUNDAMENTALS OF LAPAROSCOPIC

SURGERY (FLS)

E. Matt Ritter MD, Tamara W Kindelan MD, Curtis Michael,

Elisabeth A Pimentel BA, Mark W Bowyer MD, 1NCA Medical

Simulation Center, Department of Surgery, Uniformed Services

University, 2Division of General Surgery, National Naval Medical

Center, Bethesda Maryland

Objective

Current skills assessment in the Fundamentals of Laparoscopic

Surgery (FLS) program is labor intensive requiring one proctor for

every 1-2 subjects. The ProMIS Augmented Reality (AR) simulator

(Haptica, Dublin IRE) allows for objective assessment of physical

tasks through instrument tracking technology. We hypothesized

that the ProMIS metrics could differentiate between ability

groups as well as standard FLS scoring with fewer personnel

requirements.

Methods

We recruited 60 volunteer subjects. Subjects were stratified

based on their laparoscopic surgical experience. Those who had

performed more than 100 laparoscopic procedures were

considered experienced (n=8). Those with less than 10

laparoscopic procedure were considered novices (n=44). The rest

were intermediates (n=8). All subjects performed up to 5 trials

of the peg transfer task from FLS in the ProMIS simulator. FLS

score, instrument path length, and instrument smoothness

assessment were generated for each trial.

Results

For each of the 5 trials, experienced surgeons outperformed

intermediates who in turn out performed novices. Statistically

significant differences were seen between the groups across all

trials for FLS score (p < 0.001), ProMIS path length (p <0.001) and

ProMIS smoothness (p < 0.001). When the FLS score was

compared to the path length and smoothness metrics, a strong

relationship between the scores was apparent for novices (r =

0.78, r = 0.94 , p < 0.001) respectively), intermediates (r = 0.5, p

= 0.2 , r = 0.98, p < 0.001), and experienced surgeons (r = 0.86, p

= 0.006, r = 0.99, p < 0.001)

Conclusions

The construct that the standard scoring of the FLS peg transfer

task can discriminate between experienced, intermediate, and

novice surgeons is validated. The same construct is valid when

the task is assessed using the metrics of the ProMIS. The high

correlation between these scores establishes the concurrent

validity of the ProMIS metrics. The use of AR for objective

assessment of FLS tasks could reduce the personnel requirements

of assessing these skills while maintaining the objectivity.

2. SAGES 2006 Education/Outcomes–P237

VALIDATION OF THE PROMIS HYBRID SIMULATOR USING

A STANDARD SET OF LAPAROSCOPIC TASKS

A L McCluney MD, L S Feldman MD, G M Fried, Steinberg-

Bernstein Centre for Minimally Invasive Surgery, McGill

University Health Centre, Montreal, QC, Canada

Introduction

SAGES Fundamentals of Laparoscopic Surgery (FLS) tasks are

validated measures of technical skills, however FLS scoring

requires a trained proctor. The ProMIS simulator (Haptica;

Dublin, IR) is a ‘hybrid’ system with physical and virtual

reality tasks. It has the flexibility to incorporate any

physical task and score it with ProMIS metrics. Metrics are

automated and report motion analysis data as instrument

path length (PL) and instrument smoothness (IS). The

purpose of this study was to test for construct and

concurrent validity using FLS tasks in the ProMIS simulator.

Methods

5 laparoscopic novices and 5 experts performed FLS tasks in

both the standard FLS simulator box and the ProMIS

simulator. Assessments were made based on FLS metrics, as

well as PL and IS. Student’s t-test was used to compare the

mean (SD) of total scores for novices and experts. Pearson’s

correlations were calculated for standard FLS scores in

relation to ProMIS FLS scores, total PL, and total IS.

Significance was defined as p < 0.01 (*).

Results

Standard FLS scores correlated strongly with ProMIS FLS

scores (r=0.90), total PL (r=-0.83), and total IS (r=-0.78)

(p< 0.01).

Conclusions

FLS tasks performed in ProMIS, when scored by either

traditional FLS metrics or by intrinsic ProMIS metrics,

discriminate effectively between novices and experts. Based

on the observed correlations, ProMIS FLS scores, total PL,

and total IS are excellent predictors of scores in the

standard FLS simulator.

Validation


3. SAGES 2007 Education/Outcomes – P279

AUTOMATED PROMIS SIMULATOR METRICS PREDICT

READINESS FOR FLS CERTIFICATION

Anthony L McCluney MD, J Cao, G N Polyhronopoulos MD, D D

Stanbridge, L S Feldman MD, G M Fried MD, Steinberg-Bernstein

Centre for Minimally Invasive Surgery, McGill University,

Montreal, QC, Canada

Introduction

SAGES Fundamentals of Laparoscopic Surgery (FLS) tasks are

validated measures of technical skills. Certification requires

travel to a testing site and a fee, thus a reliable method of

predicting readiness for the exam would be advantageous. The

ProMIS simulator (Haptica) provides automated scoring. FLS tasks

can be placed in the ProMIS simulator and scored using time

(TT), as well as motion analysis metrics: instrument path length

(PL) and instrument smoothness (IS). This study was designed to

evaluate these automated ProMIS metrics and their ability to

predict readiness for FLS certification.

Methods

33 subjects (12 students, 16 residents PGY 1-4, and 5 experts)

performed FLS tasks in the standard simulator and in ProMIS.

Tasks were scored by FLS and ProMIS metrics. For each ProMIS

metric, the total score was calculated by summing the scores for

the 5 FLS tasks. Pearson’s correlations were calculated for

ProMIS metrics versus standard FLS scores. Multivariate

regression analysis identified independent predictors of standard

FLS performance. These variables were then used for sensitivity

and specificity calculations in order to establish a ProMIS pass-

fail score for predicting readiness for FLS certification.

Significance was defined as p<0.05.

Results

TT (r= -0.82), PL (r= -0.56), and IS (r= -0.75) all correlated

significantly with standard FLS score. Multivariate regression

analysis identified TT as the strongest predictor of FLS score. A

TT score of 1000 maximizes sensitivity and specificity and was

identified as the pass-fail for reliably predicting FLS

performance.

Conclusions

Automated ProMIS metrics correlate well with standard FLS

performance. In this study sample, a TT score less than 1000

reliably predicted a passing FLS certification score.

4. SAGES 2006 Scientific Sessions S065

WHAT CAN MOTION DERIVATIVES TELL US ABOUT SKILL

PERFORMANCE?

Laurel N Vuong BS, Steven D Schwaitzberg MD, Caroline G

Cao PhD, Tufts University School of Medicine, Cambridge

Health Alliance, Tufts University School of Engineering

Surgical simulators are a popular topic of discussion on

training in laparoscopic surgery. They reduce the need to

use human cadavers or animal models for skills

development. A subset of the MISTELS methodology has

been employed in the manual skills assessment for the

Fundamentals Laparoscopic Skills (FLS) program because it

was shown to be a valid discriminator of surgical

experience. Pure performance outcome, such as time to

task completion and number of errors, is used for scoring,

which is dependent on the consistency of the scorer. A new

simulator environment has been created which uses motion

tracking for measurement of performance outcome

measures and motion derivatives such as smoothness and

efficiency.

The purpose of the study was to determine if the motion

derivatives can be used to automatically and objectively

discriminate experience levels. Twenty-one subjects (6

medical students, 14 surgical residents, and 1 expert

surgeon) were recruited to perform the following tasks: peg

transfer, pattern cutting, pre-tied loop placement,

extracorporeal and intracorporeal knot-tying in the new

simulator environment. Subjects were evaluated on time to

completion, errors, smoothness and total path length (used

to calculate efficiency).

Results show that experience level is still distinguishable

when using task-dependent parameters to evaluate

performances during peg transfer (p= 0.035), pre-tied loop

placement (p= 0.022), extracorporeal (p= 0.0006) and

intracorporeal (p= 0.025) knot tying in this new simulator

environment.

Evaluation of performance using task-independent

parameters significantly distinguished training level in three

tasks: (1) smoothness of the left instrument was significant

as a function of experience level in extracorporeal knot-

tying (p= 0.016), (2) efficiency (total path length divided by

time to completion) was also significant in the movement of

the right tool as a function of experience level in peg

transfer (p= 0.0011) and (3) pre-tied loop placement (p=

0.013979).

This preliminary analysis shows that automatic and

objectively measured motion derivatives can be associated

with the level of experience. These results indicate a

potential for the application of an automatic and objective

means of skills evaluation.

Validation


5. SAGES 2007 ETP057

OBJECTIVE MEASUREMENT OF FLS PRECISION CUTTING TASK

Derek Young, Fiona Slevin, Derek Cassidy, Donncha Ryan, Haptica

Inc

The Precision-Cutting Task in the SAGES/ACS FLS Program

requires the user to dissect a circle of specific size and shape

from a marked piece of mesh. Currently, measurement of the

accuracy and area dissected is done by observation and by

measuring the dissected mesh on a measurement grid. Using

advanced vision-tracking, the ProMIS surgical simulator takes an

image of the dissected mesh and automatically generates a

metric, indicating the accuracy of the shape and area dissected.

Method

1. Once the user has completed the Precision-Cutting Task,

ProMIS takes an image of the dissected mesh. The image is

converted to binary image and then scanned using a blob

detection algorithm which produces a list of blobs.

2. The blob with the largest area is taken as the cut out area and

the number of pixels are counted inside this area.

3. The actual measurement for area is given in cm2. This is

calculated by counting the number of pixels in a known area of

the image and then using ratios to determine the area of the cut

out. (Note: this calibration step is achieved by taking the tissue

off the tray and running the blob detection and pixel count on

the uncovered black foam of which the exact area is known)

Results

Initial trials of the metrics on the FLS Precision- Cutting Task

show that ProMIS is as accurate — and frequently more accurate

— than the current human observation method.


COMPLEX LAPAROSCOPIC TASK PERFORMANCE ON TWO

NEW COMPUTER-BASED SKILLS TRAINING DEVICES

Erika K Fellinger MD, Michael E Ganey MD, Anthony G

Gallagher PhD, Daniel J Scott MD, Ron W Bush BS, Neal E

Seymour MD, Department of Surgery, Baystate Medical

Center, Springfield, MA

Introduction

New computer-based skills training devices can simulate and

measure performance of complex surgical tasks. The aim of

this study is to determine basic face and construct validity

characteristics of two new devices configured for

laparoscopic suturing and knot-tying tasks.

Methods

At the 2005 SAGES meeting, Learning Center attendees

evaluated two computer-based skills training platforms:

SurgicalSIM (SS), a virtual reality (VR) device (METI,

Sarasota, FL; SimSurgery, Oslo, Norway) and ProMIS (PM), a

computer-enhanced video trainer (Haptica, Ltd., Dublin,

Ireland). Demographic and training data were collected

from 73 subjects. All were asked to perform 2 iterations of

laparoscopic suturing and intracorporeal knot-tying (10-

minute time limit) on each device. A 6-question survey was

used to define impressions of task realism, relevance, and

execution using a 5-point Likert scale. Performance data

(SS: time, path length, errors; PM: time, path length,

smoothness) were collected on both devices and

comparisons made between user-defined expert and

nonexpert (intermediate and novice) groups (ANOVA and

Mann Whitney U test).

Results

46 subjects used SS and 56 used PM. Task completion rate

was 80% for SS and 93% for PM. Experts performed better

than nonexperts for all performance measures on SS

(composite score 496±41 vs 699±60, p < 0.005) and PM

(974±111 vs 1466±89, p < 0.005). Post-task survey scores for

PM were significantly higher for perceived realism (overall

realism and thread behavior), reflection of clinical ability,

and overall educational value. Perception of educational

value was not significantly different between the devices

among subjects with prior VR experience.

Conclusions

Using subject-defined expert and nonexpert groups,

construct validity was demonstrated for all performance

measures on both training devices. Surveyed face validity

measures favored the non-VR device, but results also

suggest that subjects with prior VR training experience are

more apt to accept a new VR surgical training platform.

Validation


7. World J Surg. 2007 Apr;31(4):764-72.

AUGMENTED VERSUS VIRTUAL REALITY LAPAROSCOPIC

SIMULATION: WHAT IS THE DIFFERENCE? : A COMPARISON

OF THE PROMIS AUGMENTED REALITY LAPAROSCOPIC

SIMULATOR VERSUS LAPSIM VIRTUAL REALITY

LAPAROSCOPIC SIMULATOR.

Botden SM, Buzink SN, Schijven MP, Jakimowicz JJ. Catharina

Hospital, Eindhoven, The Netherlands.

Background

Virtual reality (VR) is an emerging new modality for laparoscopic

skills training; however, most simulators lack realistic haptic

feedback. Augmented reality (AR) is a new laparoscopic

simulation system offering a combination of physical objects and

VR simulation. Laparoscopic instruments are used within an

hybrid mannequin on tissue or objects while using video tracking.

This study was designed to assess the difference in realism,

haptic feedback, and didactic value between AR and VR

laparoscopic simulation.

Methods

The ProMIS AR and LapSim VR simulators were used in this study.

The participants performed a basic skills task and a suturing task

on both simulators, after which they filled out a questionnaire

about their demographics and their opinion of both simulators

scored on a 5-point Likert scale. The participants were allotted

to 3 groups depending on their experience: experts,

intermediates and novices. Significant differences were

calculated with the paired t-test.

Results

There was general consensus in all groups that the ProMIS AR

laparoscopic simulator is more realistic than the LapSim VR

laparoscopic simulator in both the basic skills task (mean 4.22

resp. 2.18, P < 0.000) as well as the suturing task (mean 4.15

resp. 1.85, P < 0.000). The ProMIS is regarded as having better

haptic feedback (mean 3.92 resp. 1.92, P < 0.000) and as being

more useful for training surgical residents (mean 4.51 resp. 2.94,

P < 0.000).

Conclusions

In comparison with the VR simulator, the AR laparoscopic

simulator was regarded by all participants as a better simulator

for laparoscopic skills training on all tested features.


COMPARISON AND VALIDATION OF TWO DIFFERENT

SURGICAL SKILLS SIMULATORS

Dieter Hahnloser MD,Rachel Rosenthal MD,Christian

Hammel,Daniel Oertli,Markus Müller,Pierre-Alain Clavien,

Department of Visceral and Transplantation Surgery,

University Hospital Zurich, Switzerland

Background

Simulators are increasingly incorporated in surgical training

and validation is important. The simulations need to

resemble the task they are based upon (face validity) and

the simulator should be able to differentiate between levels

of experience (construct validity).

Aim

To assess two different types of computer-based simulators:

the fully computerised virtual reality (VR) simulator Xitact

LS500 (VR-simulator) and the hybrid ProMisTM simulator.

Methods: 146 participants (61%) of the 22nd Davos

International Gastrointestinal Surgery Workshop performed

on a voluntary basis three similar exercises (camera

navigation, clip and cut, and dissection) on the two

different simulators. Objective performance parameters

recorded by either simulator and subjective evaluation by

questionnaire were compared between beginner (n=73) and

advanced participants (n=73).

Results

The camera navigation exercise was completed by 52% of

the participants on the VR- and by 47% on the hybrid

simulator with no difference in performance parameters

between beginners and advanced trainees. The hybrid

simulator was graded more realistic (70% vs. 20%, p=.001)

and more useful (65% vs. 36%, p=.043) than the VR-

simulator. Participation was higher at the clip and cut

exercise (75% VR- and. 52% hybrid simulator) and advanced

trainees performed significantly better (shorter tool-tip-

travel distance, smoother, quicker and with higher score) on

both simulators compared to beginners. The clip and cut

exercise was graded more realistic on the hybrid (81% vs.

44%, p=.007) and similar useful on both simulators (77% vs.

72%). The dissection exercise was completed more often on

the hybrid simulator (47% vs. 23%, p=0.002). Only the hybrid

simulator was able to distinguish between advanced

trainees and beginners, with significantly higher scores for

all performance parameters for the latter. The hybrid

simulator was graded more realistic (70% vs. 33%, p=.016)

and more useful (83% vs. 62%, p=.12). Overall, acceptance

of requirement to train on and to be evaluated by such

simulators is still low (53% and 50%, respectively).

Conclusion

Fully computerized VR- or hybrid simulator performance

parameters can distinguish between beginner and advanced

trainees for perceptual motor skills (proving construct

Validation


validity), but not for visuo-spatial exercises such as the camera

navigation.

__________________________________________

9. SAGES 2007: S077

COMPUTER-BASED LAPAROSCOPIC AND ROBOTIC SURGICAL

SIMULATORS: PERFORMANCE CHARACTERISTICS AND

PERCEPTIONS OF NEW USERS

David W Lin MD, John R Romanelli MD, Renee E Thompson

MD,Michael E Ganey MD, Ron W Bush BS, Neal E Seymour MD,

Baystate Medical Center, Department of Surgery

The expanding inventory of advanced surgical training devices

now includes simulators for laparoscopic and robotic surgery. In

order to define perceptions of the need and value of such

devices, we evaluated the initial experience of surgeons using

both in the course of performance of an advanced laparoscopic

skill.

Methods

At the 2006 SAGES meeting, 62 Learning Center attendees evaluated a

new virtual reality (VR) robotic surgery simulator (RS) [SimSurgery, Oslo,

Norway] as well as either a computer-enhanced laparoscopic [ProMIS

(PM), Haptica, Ltd, Dublin, Ireland] or a VR simulator [SurgicalSIM (SS),

SimSurgery and METI, Inc, Sarasota, FL]. Demographic and training data

were collected and all were assessed during one iteration of

laparoscopic suturing and knot-tying on RS and either PM or SS. An 8-

question survey was used to determine users? impressions of task

realism, interface quality, and educational value (5-point Likert scale).

Performance data [time, path length, smoothness (PM), errors (SS/RS)]

were collected and comparisons made between user-defined groups and

different simulation platforms (Mann-Whitney Test, ANOVA).

Results

Task completion rate was greater for experts than nonexperts on all

platforms (PM 100% vs 75%; SS 100% vs 36%; RS 93% vs 36%). Experts

performed better than nonexperts on all performance measures on PM

(p<0.05: time 154±16 vs 205±12; path length 820±97 vs 1287±97;

smoothness 952±111 vs 1582±127). There were no significant differences

between experts and nonexperts for SS and RS performance measures.

Perception of value of haptic features was less for subjects with prior robot

experience (n=10; p<0.05). Otherwise realism, interface quality, and

educational value scores did not differ on the basis of prior simulator or

robot use. Nonexperts found that robotic simulation better reflected

clinical skill than did experts. Overall, subjective quality was scored higher

for PM than for SS or RS.

Conclusions

The ability of performance metrics of the computer-enhanced simulator

to discern predicted performance differences between experts and

nonexperts was better than for VR devices with a single task iteration.

Initial use of VR devices was associated with a lower overall perception

of realism and educational value as compared to use of physical objects

in the non-VR simulator. This may reflect the need for familiarization

with the computer-generated environment before the educational

potential of VR can be realized.

__________________________________________

10. 11. SAGES 2006 Scientific Sessions S096 A COMPUTERIZED ANALYSIS OF ROBOTIC VERSUS

LAPAROSCOPIC TASK PERFORMANCE

V K Narula MD, W C Watson MD, S S Davis MD, K Hinshaw BS,

B J Needleman MD, D J Mikami MD, J W Hazey MD, J H

Winston MD, P Muscarella MD, M Rubin, V Patel MD, W S

Melvin MD, The Ohio State University. CMIS. Columbus, OH

Introduction

Robotic technology has been postulated to improve

performance in advanced surgical skills. We utilized a novel

computerized assessment system to objectively describe the

technical enhancement in task performance comparing

robotic and laparoscopic instrumentation.

Methods and Procedures

Advanced laparoscopic surgeons (2- 10 yrs experience)

performed 3 unique task modules using laparoscopic and

Telerobotic Surgical Instrumentation (Intuitive Surgical,

Sunnyvale, CA). Performance was evaluated using a

computerized assessment system (ProMIS, Dublin, Ireland)

and results were recorded as time (sec), total path (mm)

and precision. Each surgeon had an initial training session

followed by two testing sessions for each module. A Paired

Student’s T-Test was used to analyze the data.

Results

10 surgeons completed the study. Objective assessment of

the data is presented in the table below. 8/10 surgeons had

significant technical enhancement utilizing robotic

technology Laparoscopic vs Robotic Time (sec) Total

Path(mm) Precision Module 1210 vs 161 # 11649 vs 5571 *

1434 vs 933 * Module 2119 vs 68 * 5573 vs 1949 * 853 vs 406 *

Module 377 vs 55 * 4488 vs 2390 * 552 vs 358 * # = p < 0.009 *

= p<0.001 Conclusions: The ProMIS computerized assessment

system can be modified to objectively obtain task

performance data with robotic instrumentation. All the

tasks were performed faster and with more precision using

the robotic technology than standard laparoscopy.

Validation



INTEGRATING SIMULATION LAB TRAINING INTO A SURGICAL

RESIDENCY PROGRAM: IS VOLUNTARY PARTICIPATION

EFFECTIVE

Lily Chang MD, James Petros MD, Donald Hess MD, Caroline

Rotondi BA, Timothy Babineau MD, Boston Medical Center

Objective

Surgical training programs nationwide are struggling with the

integration of simulation training into their curriculum given the

constraints of the 80-hour work week. We examine the

effectiveness of voluntary training in a simulation lab as part of

the surgical curriculum. Methods: The ProMIS simulator was

introduced into the general surgery residency at Boston

University Medical Center. All categorical residents (28) were

required to attend a 2-hour training session and curriculum

review. Non-categorical residents (23) were given the option to

complete training. After the introductory session, time spent in

the lab was encouraged, but voluntary. Use of the simulator was

tracked for all residents. Participation in the simulation

curriculum was defined as 3 or more uses of the simulator. After

3 months, all residents completed a survey regarding the

simulation lab and their simulator usage. Results: 26 (93%)

categorical residents and 3 (6%) non-categorical residents

completed the introductory simulator training session. Over a 3

month period, use of the simulator at least once was 31% among

all eligible residents; 80% of PGY1, 40% of PGY2, 60% of PGY3,

and 0% of PGY4 and PGY5. Four residents (14%) participated in

the simulation curriculum. 71% of simulator usage was during

working hours while 29% was completed post-call or off duty.

Most residents agreed that the simulator was easy to use and

improved their operative skills, but did not think it was a good

substitute for actual operative experience. Reported reasons for

not using the simulator included off-site rotation (44%), no time

(30%), and no interest (11%).

Conclusions

Voluntary use of a surgical simulation lab leads to minimal

participation in a training curriculum. Participation should be

mandatory if it is to be an effective part of a residency

curriculum.

13. SAGES 2007: S080 THE IMPACT OF KNOWLEDGE OF RESULTS IN SURGICAL

SKILLS TRAINING

A. O’Connor MD, C. Cao PhD, S. Schwaitzberg MD,

Department of Mechanical Engineering, Tufts University.

Background

Concerns about the adequacy of advanced laparoscopic

training continue to be raised despite a proliferation of

training systems exist. The manner in which the training

modules are structured to maximize learning has not been

examined. There are many aspects to the accumulation of

laparoscopic skills during training, one of which is

Knowledge of Results (KR), i.e. the information provided to

individuals about the outcomes of their motor responses in

their environment. We studied the effects of KR on the

learning curve of laparoscopic suturing and knot tying.

Aims

We evaluated the learning curves of 9 medical students with

no previous laparoscopic surgical experience under three

different conditions, each with different levels of

knowledge of results.

Methods

Subjects were randomly assigned to one of three groups.

Each subject attended a training session for 1 hour each

day, 6 days a week for 4 weeks. Group 1 (No feedback)

received no knowledge of results (KR) and no performance

feedback. Group 2 (feedback only) received factual KR

following each training session, but no coaching. Group 3

(feedback and coaching) received KR and coaching. Learning

curves were plotted based on task time, smoothness of

instruments and instrument’s path length. The task used

was an intracorporeal suture/knot tying in the ProMIS

laparoscopic simulator. Perceived workload for each session

was recorded using a standardized NASA TLX workload

score.

Results

The variability across each session for each student was

calculated for each of the three parameters. There was

statistical significance between the groups for all

parameters (p-values 0.0002, 0.0002 and 0.009). Significant

differences were found between groups 2 and 3 and group 1

(p values 0.0314-0.0410) Groups 2 and 3 learned

significantly faster than those in Group 1, reaching

performance plateaus at earlier sessions. There were no

significant differences between groups 2 and 3 (p-values

0.1211, 0.1758 and 0.1375). Providing individuals with

knowledge of results lowered their perceived workload,

adding instructional feedback lowered this even further.

These results demonstrate that KR is essential for efficient

surgical skill acquisition. Individual coaching, a labor

intensive proposition, reduces workload but has NO added

beneficial effect on the speed of learning. These results

Validation


provide a useful basis for developing efficient and cost effective

surgical skills training curriculum.

____________________________________________________

14. SAGES 2007: Education/Outcomes – P267

THE CONSTRUCT VALIDITY OF COMPUTER-DERIVED

PERFORMANCE METRICS FOR SELECTED SIMULATED

LAPAROSCOPIC TASKS

J. A Oostema MD, Matthew Abdel BS, Jon C Gould MD, University

of Wisconsin School of Medicine and Public Health, Department

of Surgery

Introduction

A surgical skills assessment tool is said to demonstrate evidence

of construct validity if users with more experience, and by

inference more skill, perform better or more efficiently.

Computer derived motion metrics such as smoothness (the

number of times an instrument tip changes velocity during a

task) and path length may be more sensitive measures of skill for

a particular task than traditional metrics such as time.

Methods

Twenty-four medical students (third year), 19 surgical residents

(PG1-5), and 3 attending surgeons were asked to perform four

different tasks 3 times in a hybrid computer-based physical

laparoscopic trainer (ProMIS, Haptica Inc., Dublin). The 4 tasks in

order of complexity were laparoscopic orientation (Task 1),

object positioning (Task 2), sharp dissection (Task 3), and intra-

corporeal knot tying (Task 4). Metrics recorded were time, path

length, and smoothness. Laparoscopic operative experience for

each user was quantified using case logs. Correlations were

determined using regression analysis and ANOVA.

Results

A statistically significant correlation was observed between

experience and performance for all three metrics for tasks 2-4

(all p< 0.01). Smoothness was the only metric to correlate in the

laparoscopic orientation task. Within tasks, time and smoothness

correlate much more strongly with experience and to a similar

degree. The strongest correlation was observed for the knot

tying task (r2=0.60 for time and 0.59 smoothness). An r2=1.0

would represent a perfect correlation between experience and

the specified metric.

Conclusions

The computer-derived metrics measured by the hybrid trainer

correlate with laparoscopic experience. Further study is

necessary to determine if specific metrics are better indicators

of actual skill.


ABSTRACT VIRTUAL REALITY TRAINING DEVELOPS CORE

LAPAROSCOPIC SKILLS COMPARABLE TO EXPERIENCED

LAPAROSCOPIC SURGEONS: RESULTS OF A PROSPECTIVE

RANDOMIZED TRIAL COMPARING TWO VIRTUAL REALITY

TRAINERS

E. Matt Ritter MD, Elisabeth A Pimentel BA, Ryan E Earnest

BS, Randy S Haluck MD, Mark W Bowyer MD, National Capital

Area Medical Simulation Center, Uniformed Services

University, Bethesda, Maryland / Department of Surgery,

Pennsylvania State College of Medicine, Hershey,

Pennsylvania

Introduction

While simulation is becoming more widely accepted in

surgical training, comparative trials on the training

effectiveness of these simulators are lacking. We sought to

compare the effectiveness of two abstract virtual reality

trainers to train laparoscopic skills as assessed by the

Fundamentals of Laparoscopic Surgery (FLS). We then

compared the post training performance of the novice

subjects with a group of experienced surgeons.

Methods and Results

20 novice medical students were recruited. Each subject

performed a pre-test consisting of 3 FLS tasks - Peg Transfer

(PT), Pattern Cut (PC) and Intracorporeal Suture (IS) -

placed in the ProMIS augmented reality simulator (Haptica,

Ireland). They were then randomized to train to

predetermined levels of proficiency on 3 tasks of the

Minimally Invasive Surgical Trainer-Virtual Reality (MIST-VR)

(Mentice, Sweden) or the Rapid Fire/Smart Tutor (RFST)

(Verefi, Elizabethtown, PA). After reaching the proficiency

levels, both groups then took a post test consisting of 3

trials of the same tasks used for the pre-test. Post test

performance by both groups was then compared to a control

group, composed of 10 experienced surgeons who had

completed the same post test.

MIST-VR and RFST groups demonstrated statistically

significant improvement from the pre-test to the post test

on all 3 FLS tasks (p < 0.0001). There was no significant

difference in post test performance between the MIST-VR

and RFST groups. When the simulation trained groups were

compared to experienced controls there was no significant

difference in performance with respect to PT. The

experienced controls did significantly outperformed the

MIST-VR group in PC (p<0.01) and IS (p<0.05), but

differences between the experienced controls and the RFST

group did not reach statistical significance.

Conclusion

Simulation based training on either the MISTVR or the RFST

simulator improves the skill level of novices as assessed by

FLS. The post training skill level of these novices compares

favorably with a group of experienced surgeons. Virtual

Reality trainers, such as RFST and MISTVR, train

Validation


fundamental laparoscopic skills equally and to a level

comparable to a group of experienced practicing surgeons.

____________________________________________________

16. SAGES07 Ergonomics/Instrumentation P307

A COMPUTERIZED ANALYSIS OF STANDARD VERSUS HIGH

DEXTERITY LAPAROSCOPIC INSTRUMENTATION IN TASK

PERFORMANCE

V K Narula MD,K M Reavis MD,D R Renton MD,D J Mikami MD,B J

Needleman MD,J W Hazey MD,K E Hinshaw BS,W S Melvin MD,

THE OHIO STATE UNIVERSITY HOSPITAL, CENTER FOR MINIMALLY

INVASIVE SURGERY

Introduction

Minimally invasive surgery is becoming the standard of care for

the majority of abdominal procedures. Laparoscopic

instrumentation is constantly undergoing improvements to give

surgeons an advantage. Articulated instrumentation provides a

distinct advantage in the field of robotic surgery. Applying the

same principles to standard laparoscopic instrumentation could

offer increased degrees of freedom to make complex

laparoscopic tasks easier to perform. We utilized a novel

computerized assessment system to objectively evaluate task

performance comparing Standard and High Dexterity (HD)

laparoscopic instrumentation.

Methods

Advanced laparoscopic surgeons (2-12yrs experience) performed

3 unique task modules utilizing Standard and HD laparoscopic

instrumentation (Novare Surgical Systems, Cupertino, CA).

Performance was evaluated using a computerized assessment

system (ProMIS, Dublin, Ireland) and results were recorded as

time (sec), path (mm), and precision. Each surgeon had an initial

training session followed by two testing sessions for each

module. A Paired Student’s T-Test was used to analyze the data.

Results: Nine surgeons completed the study. Objective

assessment of the data is presented in the table below. Module 1

was statistically significant, whereas Module 2 and 3 showed no

difference in task performance with the HD instrumentation.

Conclusion

HD instrumentation is in its infancy. Results showed no

advantage using HD instrumentation. This could be due to the

learning curve associated with new instrumentation and

technology. With future developments in HD technology and

training, the user interface will improve and may offer an

advantage over standard laparoscopic instrumentation.

17. SAGES 2006 Emerging Technologies P036

AUGMENTED REALITY SIMULATOR FOR HAND-ASSISTED

LAPAROSCOPIC COLECTOMY

Derek Young, Derek Cassidy, Fiona Slevin, Donncha Ryan,

Haptica Ltd, Dublin, Ireland.

Training in Hand-Assisted Laparoscopic Colectomy (HALC)

has largely been done using cadavers and porcine models.

These have drawbacks in terms of realism, logistics and lack

of performance measurement. A Simulator would provide

consistent instruction and practice and provide feedback on

performance. However, given the range of instruments used

in the procedure, and especially the use of a hand, pure

virtual reality could not be considered as a solution.

The ProMIS Augmented Reality simulator platform – by

combining physical and virtual reality - enables interaction

and tracking of real instruments with a physical model. And

because of its technological approach (vision-tracking) also

enables the hand to be tracked. In the new ProMIS HALC

simulator, 3D models or graphical objects are overlaid on

the physical model to provide instruction and guidance. For

example, a 3D animation may be used to demonstrate how

to complete a step; a graphical guideline ‘A – B’ may be

used to indicate a target area for dissection. ProMIS HALC

measures surgical skill by gathering data on the movement

of commercial laparoscopic instruments while completing a

standardized task. The main performance metrics are time

taken, total path length and economy of movement.

Additionally metrics specific to a step are calculated to

measure performance associated with a specific instruction

in a specific region of the physical model. Following the

simulated procedure, the user completes a self-assessment

which contributes to the metrics for the full procedure. A

full analysis is of performance is presented to the user on

completion of the procedure and self-assessment.

Results

Initial trials of the ProMIS HALC simulator indicate that

practice on the simulator improves performance as

measured by the metrics gathered by the simulator. While

detailed validation studies remain to be done, initial

indicators are that the HALC simulator represents an

“unparalleled opportunity to practice, step by step, a Hand-

assisted laparoscopic sigmoid resection” and “a huge step

forward in surgical training”.

Validation


18. May 2006 North of England Surgical Society Annual Registrar’s Meeting Winner of the George Feggeter Gold Medal

LAPAROSCOPIC SKILLS ACQUISITION: IS PSYCHOMETRIC

MOTION ANALYSIS A VALID ASSESSMENT TOOL?

Pellen MGC1;2, Barton JR2, Horgan LF1, Attwood SE1 Northumbria

Upper Gastrointestinal Team of Surgeons1; University of

Newcastle upon Tyne2, Newcastle upon Tyne, United Kingdom

Aims

In an evolving climate of competency-based assessment, reliable

and validated methods of objective skills assessment are

required for trainee surgeons. We aimed to assess whether the

ProMIS Simulator (Haptica, Dublin, Ireland) offers a method of

assessing laparoscopic psychomotor performance.

Methods

Volunteers comprising 17 experienced laparoscopists ( >100

laparoscopic cholecystectomies) and 38 medical students novices

(no laparoscopic experience) performed 3 simulated tasks

comprising virtual reality camera navigation, object transfer and

sharp dissection task (glove over balloon). A further group of 28

basic surgical trainees (experience limited to 1st assistant)

attending BSS Courses were assessed on the same tasks before

and after training in laparoscopic skills. Data metrics of time,

smoothness and path length were measured via optical tracking

of instrument movement. Objective observations of specific

errors were also recorded.

Results

Non-parametric analysis demonstrated experienced

laparoscopists performed all 3 tasks significantly faster,

smoother and with more economy of movement (p<0.05),

excluding camera navigation path length. Experienced

participants performed sharp dissection more accurately (p<0.01)

although no difference in balloon puncture was seen. Repeat

assessment of BSS Course Trainees showed significant

improvements in simulator metrics (Paired T test, P<0.05),

although smaller yet significant improvements in “untrained”

student performance was also seen.

Conclusions

Gross analysis of these metrics can distinguish between

experience levels supporting construct validity of these simulator

tasks. These results suggest potential for objectively measuring

baseline skill level and response to training. Further work will

examine the effect of interface familiarisation and defining

target levels of performance in simulated tasks.

19. EAES 2006, Poster DEVELOPING PSYCHOMETRIC ASSESSMENT OF

LAPAROSCOPIC SKILLS USING THE PROMIS SIMULATOR

Pellen MGC1;2, Barton JR2, Horgan LF1, Attwood SE1

Northumbria Upper Gastrointestinal Team of Surgeons1;

University of Newcastle upon Tyne2, United Kingdom

Aims

Reliable and validated methods of objective skills training

and assessment are required for trainee surgeons. The

ProMIS Simulator (Haptica, Ireland) potentially offers a

method of assessing laparoscopic psychomotor performance.

We present initial data from our Centre and Royal College of

Surgeons Basic Surgical Skills (BSS) Courses.

Methods and results

Volunteers comprising 17 experienced laparoscopists (>100

laparoscopic cholecystectomies) and 38 medical students

novices (no laparoscopic experience) were assessed on a

complex sharp dissection task (glove over balloon). A

further group of 28 basic surgical trainees (experience

limited to 1st assistant) attending BSS Courses were assessed

on the same task before and after training in laparoscopic

skills. Data metrics of time, smoothness and path length

were measured via optical tracking of instrum-ent

movement as well as observations of specific errors.

Data analysis (ANOVA) demonstrated experienced

laparoscopists performed target dissection at least 50%

faster, smoother and with more economy of instrument

movement than students (p<0.05). Experienced participants

performed sharp dissection more accurately (p<0.01)

although no difference in balloon puncture frequency was

seen. Similarly significantly better performance over

trainees was demonstrated. Trainees showed only

significantly smoother instrument handling when compared

to students, possibly reflecting greater baseline dexterity in

this selective group. Repeat assessment following course

training showed significant improvements in all metrics by

32-40% (Paired T test, P<0.05). Whilst significant

improvements were also demonstrated in repeat assessment

of the untrained student group, these were less marked (15-

18%).

Conclusions

The gross analysis of these metrics can distinguish between

experience levels supporting the construct validity of this

simulator task. These results suggest a potential role for

objectively measuring baseline skill level and response to

training in distinct psychomotor challenges. Further work in

progress is examining the effect of interface familiarisation

and repeated task performance on novice learning curves

and defining target levels of performance in a range of

simulated tasks.

Validation


20. Surg Endosc (May 2006) 20: 900–904 CONSTRUCT VALIDATION OF A NOVEL HYBRID SURGICAL

SIMULATOR

D. Broe, P. F. Ridgway, S. Johnson, S. Tierney, K. C. Conlon

Department of Surgery, Professorial Surgical Unit, Level 4, The

Adelaide and Meath Hospital, incorporating the National

Children’s Hospital, Tallaght, Dublin 24, Ireland

Background

Simulated minimal access surgery has improved recently as both

a learning and assessment tool. The construct validation of a

novel simulator, ProMis, is described for use by residents in

training.

Methods

ProMis is a surgical simulator that can design tasks in both virtual

and actual reality. A pilot group of surgical residents ranging

from novice to expert completed three standardized tasks:

orientation, dissection, and basic suturing. The tasks were tested

for construct validity. Two experienced surgeons examined the

recorded tasks in a blinded fashion using an objective structured

assessment of technical skills format (OSATS: task-specific

checklist and global rating score) as well as metrics delivered by

the simulator.

Results

The findings showed excellent interrater reliability (Cronbach_s

a of 0.88 for the checklist and 0.93 for the global rating). The

median scores in the experience groups were statistically

different in both the global rating and the task-specific

checklists (p < 0.05). The scores for the orientation task alone

did not reach significance (p = 0.1), suggesting that modification

is required before ProMis could be used in isolation as an

assessment tool.

Conclusions

The three simulated tasks in combination are construct valid for

differentiating experience levels among surgeons in training.

This hybrid simulator has potential added benefits of marrying

the virtual with actual, and of combining simple box traits and

advanced virtual reality simulation.

21. Surgical endoscopy ISSN: 0930-2794 (Paper) 1432-2218 (Online)

CONSTRUCT VALIDATION OF THE PROMIS SIMULATOR

USING A NOVEL LAPAROSCOPIC SUTURING TASK

K. R. Van Sickle1, D. A. McClusky III1, A. G. Gallagher and

C. D. Smith1

Background

The use of simulation for minimally invasive surgery (MIS)

skills training has many advantages over current traditional

methods. One advantage of simulation is that it enables an

objective assessment of technical performance. The

purpose of this study was to determine whether the ProMIS

augmented reality simulator could objectively distinguish

between levels of performance skills on a complex

laparoscopic suturing task.

Methodology

Ten subjects — five laparoscopic experts and five

laparoscopic novices — were assessed for baseline

perceptual, visio-spatial, and psychomotor abilities using

validated tests. After three trials of a novel laparoscopic

suturing task were performed on the simulator, measures

for time, smoothness of movement, and path distance were

analyzed for each trial. Accuracy and errors were evaluated

separately by two blinded reviewers to an interrater

reliability of >0.8. Comparisons of mean performance

measures were made between the two groups using a Mann-

Whitney U test. Internal consistency of ProMIS measures was

assessed with coefficient α.

Results

The psychomotor performance of the experts was superior

at baseline assessment (p < 0.001). On the laparoscopic

suturing task, the experts performed significantly better

than the novices across all three trials (p < 0.001). They

performed the tasks between three and four times faster (p

< 0.0001), had three times shorter instrument path length

(p < 0.0001), and had four times greater smoothness of

instrument movement (p < 0.009). Experts also showed

greater consistency in their performance, as demonstrated

by SDs across all measures, which were four times smaller

than the novice group. Observed internal consistency of

ProMIS measures was high (α = 0.95, p < 0.00001).

Conclusions

Preliminary results of construct validation efforts of the

ProMIS simulator show that it can distinguish between

experts and novices and has promising psychometric

properties. The attractive feature of ProMIS is that a wide

variety of MIS tasks can be used to train and assess technical

skills.

Validation


22. EAES 2004 Abstract nr.: O207 RELATIONSHIP BETWEEN MOTION ANALYSIS, TIME,

ACCURACY, AND ERRORS DURING PERFORMANCE OF A

LAPAROSCOPIC SUTURING TASK ON AN AUGMENTED REALITY

SIMULATOR

Author: D.A.M. McClusky, Emory University School of Medicine,

Atlanta Georgia, United States of America. Co-author(s): K. Van

Sickle, Emory University School of Medicine, Atlanta Georgia,

United States of America

A.G. Gallagher, Emory University School of Medicine, Atlanta

Georgia, United States of America

Background

Time, efficient movement, accuracy, and safety are reliable and

discriminative metrics of proficiency during virtual reality and

box-trainer based minimally invasive surgical (MIS) training. The

role these metrics may serve during more advanced skills training

are not well understood. Using a novel augmented reality

simulator, we sought to gain an understanding of the relationship

between these metrics during an advanced MIS suturing task.

Methods: Eleven subjects completed 3 trials of a suturing task

designed for a box-trainer and adapted for the ProMIS (Haptica,

Dublin, Ireland) simulator. Time, tool path, and smoothness of

movement were assessed using computer algorithms. Measures of

accuracy during suture placement and errors in performance

were assessed by two blinded reviewers trained to assess

performance with inter-rater reliability > 0.8. A Pearson’s

correlation coefficient was used to assess the strength of the

relationship between ProMIS metrics and suturing task

performance. Results: Of the ProMIS metrics, time correlated

with tool path distance and smoothness of movement in three

trials (range 0.914 – 0.957, p < 0.0001). When the suturing task

was analyzed, accuracy and error score demonstrated an equally

strong relationship (range -0.726 - -0.84, all p < 0.0001).

Combining all metrics, path distance correlated strongest with

accuracy (2 trials significant, range -0.67 - -0.93), and error

score (3 trials significant, range 0.54 – 0.61). Smoothness of

movement significantly correlated with accuracy in 2 trials

(range -0.63 - -0.88), and time correlated with error score in 2

trials (range 0.56 – 0.60).

Conclusion

Metrics based on movement efficiency and time, and those based

on task accuracy and error scores strongly correlate when

grouped independently. At this time, a proficiency curriculum

should incorporate both forms of analysis, however further

validation work is needed to replicate these findings and give

further insight into how ProMIS metrics relate to real-world

performance.

23. EAES 2004 abstract nr.: O208 PSYCHOMOTOR SKILLS ASSESSMENT IN PRACTICING

SURGEONS PERFORMING ADVANCED LAPAROSCOPIC

PROCEDURES II: DEMOGRAPHICS AND PERFORMANCE

PROFILES

Author: A.G. Gallagher, Emory University, Atlanta, United

States of America. Co-author(s): C.D. Smith, Emory

University, Atlanta, United States of AmericaR.M. Satava,

University of Washington, Seattle, United States of America

Background

This study reports on the objectively assessed psychomotor

performance of minimally invasive surgeons on a box-trainer

and a virtual reality (VR) task as a function of handedness,

gender, sight corrected status, and age. Methods: Two

hundred and ten surgeons attending the 2001 annual

meeting of the American College of Surgeons (ACS) in New

Orleans who reported having completed more than 50

laparoscopic procedures participated. Subjects completed a

box-trainer laparoscopic cutting task and a similar virtual

reality task twice. Demographic and laparoscopic

experience data was also collected. Results: There were no

significant differences between subjects performance on

either tasks in terms of handedness, gender or whether they

were sight corrected or not. A clear and consistent linear

trend emerged in terms of age. Older subjects (ages 60 – 69)

performed significantly worse than younger subjects (ages

30 – 39, 40 – 49) on the box-trainer task for correct incisions

(13.1 Vs 19.3, p < 0.008) and incorrect incisions (12.3 Vs 2.5,

p > 0.05). They also performed worse on the VR task for

time (132 Vs 71, p < 0.05), error (99 Vs 41, p < 0.05) and

economy of movement (22.8 Vs 11.7, p < 0.05). Conclusions:

Increasing age was found to be associated with a decline in

objectively assessed psychomotor performance on two well

validated laparoscopic tasks.

Acknowledgements

This study was supported with grants from the ACS, SAGES,

SLS, TATRC, and Emory University Endosurgery Unit.

Validation


24. JSLS, Journal of the Society of Laparoendoscopic Surgeons, Vol. 11, No. 3. (September 2007), pp. 273-

302.

CRITERION-BASED TRAINING WITH SURGICAL SIMULATORS:

PROFICIENCY OF EXPERIENCED SURGEONS

Heinrichs, Wm, Lukoff, Brian, Youngblood, Patricia, Dev, Parvati,

Shavelson, Richard, Hasson, M Harrith, Satava, M Richard,

Mcdougall, M Elspeth, Wetter, Paul Alan

Objective

In our effort to establish criterion-based skills training for

surgeons, we assessed the performance of 17 experienced

laparoscopic surgeons on basic technical surgical skills recorded

electronically in 26 modules selected in five commercially

available, computer-based simulators.

Methods/Procedures

Performance data were derived from selected surgeons randomly

assigned to simulator stations, and practicing repetitively during

three one-half day sessions on five different simulators. We

measured surgeon proficiency defined as efficient, error-free

performance and developed proficiency score formulas for each

module. Demographic and opinion data were also collected.

Results

Surgeons’ performance demonstrated a sharp learning curve with

the most performance improvement seen in early practice

attempts. Median scores and performance levels at the 10th,

25th, 75th, and 90th percentiles are provided for each module.

Construct validity was examined for two modules by comparing

experienced surgeons’ performance with that of a convenience

sample of lessexperienced surgeons.

Conclusions

A simple mathematical method for scoring performance is

applicable to these simulators. Proficiency levels for training

courses can now be specified objectively by residency directors

and by professional organizations for different levels of training

or post-training assessment of technical performance. But data

users should be cautious due to the small sample size used in this

study and the need for further study into the reliability and

validity of the use of surgical simulators as assessment tools.

Summary comment

The simulators used included pure Virtual Reality Simulators and

two simulators with real haptics (including ProMIS): Lap Mentor

from Simbionix (pure Virtual Reality); LapSim from Surgical-

Science AB (pure Virtual Reality); SurgicalSIM from METI (pure

Virtual Reality); ProMIS from Haptica (real haptics with metrics);

LTS2000 ISM60 from RealSim (real haptics with metrics).

ProMIS out-performed the virtual reality simulators with a mean

effectiveness rating of 3.56 versus 3.22 and 3.11 for LapSim and

SurgicalSim (LapMentor also scored 3.56).

25. SAGES 2008 P174 FLS TEST IS TRANSFERABLE TO PROMIS SIMULATOR

Eric S Hungness MD, Albert Amini BA, Deb E Rooney MS, Eric

T Volckman MD, Nathaniel J Soper MD, Feinberg School of

Medicine, Northwestern University, Chicago, IL

Introduction

The McGill Inanimate System for Training and Evaluation

of Laparoscopic Skills (MISTELS) comprises five tasks with

an objective scoring system, and has been incorporated

by SAGES in their Fundamentals of Laparoscopic Surgery

(FLS) program. MISTELS has high inter-rater and test-

retest reliability and correlates with operative skill.

However, the FLS program is labor intensive, requiring a

trained proctor. The ProMIS simulator allows for

assessment of physical tasks (instrument path length and

instrument smoothness) through instrument tracking

technology. We hypothesized that the FLS scores

obtained in the ProMIS simulator as well as ProMIS

metrics would correlate with standard FLS scoring.

Methods

Twenty general surgery residents (13 junior and 7 senior)

had baseline laparoscopic skills assessed using MISTELS in

the standard FLS and ProMIS simulators (pre-test). Nine

junior and 4 senior residents had a post-test after four

weeks of training. Tasks were scored by FLS and ProMIS

metrics. Total path length (TP) and total smoothness (TS)

were calculated by adding the path lengths and

smoothness of each individual task. ANOVA was used to

compare the mean (SD) of total and individual task scores

for pre- and post-tests in the FLS and ProMIS simulators.

Student's t-test was used to compare ProMIS metrics.

Pearson's correlations were calculated for standard FLS

scores in relation to ProMIS FLS scores, TP and TS.

Significance was defined as p < 0.01.

Results

All residents showed statistically significant improvement in

post-test total and individual task FLS scores on either the

FLS or ProMIS simulator. 100% and 88% of residents achieved

passing post-test scores on the FLS and ProMIS simulator,

compared to 30% and 29.2% on the pre-test. There was no

difference in junior and senior resident posttest scores (87.6

vs 79.1). ProMIS path length and smoothness were

significantly reduced across all tasks (range 14- 68%). Total

ProMIS FLS scores (0.729), TP (-0.753) and TS (- 0.769)

significantly correlated with total standard FLS simulator

scores. All residents with TP < 4000mm or TS < 6000

achieved a passing total FLS score.

Conclusions

All surgical residents achieved a passing FLS score after a

4-week laparoscopic skills curriculum. FLS tasks are

transferable to the ProMIS simulator with traditional FLS

scoring and intrinsic ProMIS metrics being good

measurement tools. A ProMIS total path length <4000 mm

or total smoothness <6000 reliably predicts a passing FLS

score.

ProMIS Validation

Health & Medicine

Transcript of ProMIS Validation