8th Annual Meeting of the Consortium of ACS-Accredited Education Institutes

Special Panel: Advances in Summative

Assessment of Surgeons Using Simulation

Simulation-Based

Summative Assessments

in Orthopaedic Surgery

Robert A. Pedowitz, MD, PhD

Professor Emeritus of Orthopaedic Surgery

David Geffen School of Medicine at UCLA

Disclosure: Robert Pedowitz, MD, PhD

Consultant: Virtamed

Chair of the Fundamentals of

Arthroscopic Surgery Training

(FAST) Program • Arthroscopy Association of North America

• American Academy of Orthopaedic Surgeons

• American Board of Orthopaedic Surgery

Summative Assessments: Ortho

ASSET (Arthroscopic Surgical Skill

Evaluation Tool): Diagnostic Knee Scope

Virtual Reality Simulation Training:

Diagnostic Knee Arthroscopy

FAST (Fundamentals of Arthroscopic

Surgery Training Program): Knot Tying

Copernicus Initiative: Proficiency-based

Progression Training (Bankart Procedure)

Assessment Strategies

ASSET: Likert scales

Virtual Reality: VR to OR (Likert scales)

FAST: Biomechanical measures

Copernicus Initiative: Binary scores

ASSET (Arthroscopic Surgical

Skill Evaluation Tool) Am J Sports Med. 2013 June ; 41(6): 1229–1237.

Koehler, Amsdell, Arendt, Bisson, Braman,

Butler, Cosgarea, Harner, Garrett, Olson,

Warme, Nicandri

ASSET

Task deconstruction: Diagnostic knee

arthroscopy

Content validity: Delphi panel, 7 experts

Construct validity: Video recordings; 28

residents & 2 attending surgeons

Strong inter-rater reliability and test-

retest reliability

Max

Score

40

Koehler et al, AJSM 2013

Various Assessment Strategies

ASSET: Likert scales

Virtual Reality: VR to OR (Likert scales)

FAST: Biomechanical measures

Copernicus Initiative: Binary scores

Improving Residency Training in

Arthroscopic Knee Surgery with

Use of a Virtual-Reality Simulator A Randomized Blinded Study

J Bone Joint Surg Am. 2014;96:1798-806

Cannon, Garrett, Hunter, Sweeney,

Eckhoff, Nicandri, Hutchinson, Johnson,

Bisson, Bedi, Hill, Koh, Reinig

Cannon et al., 2014

48 PGY3 Ortho residents randomized:

VR training to proficiency (mean 11

hours on the VR trainer)

Standard educational routine

Video recorded diagnostic knee scope

on a live patient

Each recording reviewed by a pair of

blinded raters

Cannon, JBJS 2014

Cannon et al., JBJS 2014

Global Ratings of Arthroscopy Performance

Mean + SD (%)

Measure

Simulator

Trained (N=27)

Non-Simulator

Trained (N=21)

P

Value

Procedural

checklist

63 + 20 52 + 21 0.031

Visualization

scale

61 + 20 58 + 21 0.34

Probing

scale

64 + 24 48 + 24 0.016

Global

ratings

64 + 20 57 + 19 0.061

Cannon et al., JBJS 2014

Various Assessment Strategies

ASSET: Likert scales

Virtual Reality: VR to OR (Likert scales)

FAST: Biomechanical measures

Copernicus Initiative: Binary scores

Enhancement of arthroscopic

knot-tying performance with

the FAST workstation and

knot-tester

Pedowitz, Nicandri, Angelo, Ryu, Gallagher

Arthroscopic Knot Tying

A challenging surgical skill, best

learned with a deliberate sequence

Rope

Suture

Cannulas

Video control

Patient care (fluids) Assessment:

Observation

Arthroscopic knot

performance

(As opposed to appearance)

Good suture

repair constructs

demonstrate

knot security

and

loop security

Knot

Loop

Construct failure defined as

> 3 mm of loop expansion

FAST Knot Tester

Study 1

5 sequential knots with #2 Fiberwire

Knot of choice: All with 3 reversed hitches & posts

Faculty: 20 surgeons at OLC course (19.9 + 8.25

years in practice; 381 + 150 scopes per year).

Study 1

5 sequential knots with #2 Fiberwire

Knot of choice: All with 3 reversed hitches & posts

Faculty: 20 surgeons at OLC course (19.9 + 8.25

years in practice; 381 + 150 scopes per year).

Orthopaedic Residents (All PGY4 / PGY5)

Copernicus Study (Bankart Procedure)

Group A: Standard resident’s course

Group B: Structured curriculum

Group C: Proficiency based progression

Faculty

(n = 20)

Group A

Residents

(n = 14)

Group B

Residents

(n = 14)

Group C

Residents

(n = 16)

Knot Performance

0/5 Failed 12 3 3 11

1/5 Failed 0 3 5 2

2/5 Failed 4 7 2 2

3/5 Failed 2 0 1 1

4/5 Failed 0 1 1 0

5/5 Failed 2 0 2 0

Knots that failed

(> 3mm displacement)

24 / 100 =

24%

21 / 70 =

30%

26 / 70 =

35%

9 / 80 =

11%

Logistic Regression Analysis

Faculty

(n = 20; tied 100 knots)

Group A Residents

(n = 14; tied 70 knots)

Group B Residents

(n = 14; tied 70 knots)

Odds Ratio = 0.53

p = 0.07

Odds Ratio = 0.74

p = 0.38 Knots Passed

Binary

Outcome •Pass (< 3 mm)

•Fail (> 3 mm)

Reference group

30%

Failed

24%

Failed

35%

Failed

Group C Residents

(Proficiency-Based Progression)

(n = 16; tied 80 knots)

Odds Ratio = 2.84

p = 0.01

12%

Failed

Study 2

5 sequential knots with #2 Fiberwire

Arthroscopic knot of choice with 3

reversed half-hitches & posts

Faculty and surgeon-attendees at the

2013 AANA Fall Course (n = 30)

Faculty

(n = 10)

Attendee

(n = 12)

Not Defined

(n = 8)

Total

(n = 30)

Year in Practice:

Mean + SD (range)

20.5 + 7.6

(3 – 30)

14.6 + 12.4

(1 – 40)

19.0 + 9.3

(9 – 32)

17.1 + 19.3

(1 – 40)

Knot Performance

0/5 Failed 5 3 4 12

1/5 Failed 3 3 3 9

2/5 Failed 1 3 0 4

3/5 Failed 1 3 1 5

4/5 Failed 0 0 0 0

5/5 Failed 0 0 0 0

Knots that failed

(> 3 mm displacement)

8 / 50 =

16%

18 / 60 =

30%

6 / 40 =

15%

32 / 150 =

21.3%

Combined Data:

Experienced

Surgeons

Study 1

Faculty at AANA

Copernicus Course

(n = 20)

Study 2

Faculty and Attendees at

AANA Fall Course

(n = 30)

0/5 Failed 12 12

1/5 Failed 0 9

2/5 Failed 4 4

3/5 Failed 2 5

4/5 Failed 0 0

5/5 Failed 2 0

Knots that failed

(> 3mm displacement)

24 / 100 = 24.0% 32 / 150 = 21.3% Overall failure rate: 56/250 knots = 22.4%

Setting a

performance

benchmark

Knot Performance

Study 1

Faculty at AANA

Copernicus Course

(n = 20)

Study 2

Faculty and Attendees at

AANA Fall Course

(n = 30)

0/5 Failed 12 12

1/5 Failed 0 9

2/5 Failed 4 4

3/5 Failed 2 5

4/5 Failed 0 0

5/5 Failed 2 0

Knots that failed

(> 3mm displacement)

24 / 100 = 24.0% 32 / 150 = 21.3% If perfection: 52% would fail the test

Knot Performance

Study 1

Faculty at AANA

Copernicus Course

(n = 20)

Study 2

Faculty and Attendees at

AANA Fall Course

(n = 30)

0/5 Failed 12 12

1/5 Failed 0 9

2/5 Failed 4 4

3/5 Failed 2 5

4/5 Failed 0 0

5/5 Failed 2 0

Knots that failed

(> 3mm displacement)

24 / 100 = 24.0% 32 / 150 = 21.3% If threshold < 1: 34% would fail the test

Knot Performance

Study 1

Faculty at AANA

Copernicus Course

(n = 20)

Study 2

Faculty and Attendees at

AANA Fall Course

(n = 30)

0/5 Failed 12 12

1/5 Failed 0 9

2/5 Failed 4 4

3/5 Failed 2 5

4/5 Failed 0 0

5/5 Failed 2 0

Knots that failed

(> 3mm displacement)

24 / 100 = 24.0% 32 / 150 = 21.3% If threshold < 2: 18% would fail test

Based upon AANA expert faculty and

experienced arthroscopic surgeons:

“Acceptable” Performance:

< 2/5 failures using FAST protocol

Various Assessment Strategies

ASSET: Likert scales

Virtual Reality: VR to OR (Likert scales)

FAST: Biomechanical measures

Copernicus Initiative: Binary scores

A new educational approach:

From the apprenticeship model to

proficiency-based progression

Development Process

Task deconstruction: Bankart procedure

Step & error definition (face/content validity)

Metrics: Stress testing & metric refinement

Inter-rater reliability

Delphi panel

Construct validity (joint models and cadavers)

Proficiency benchmarks

Randomized controlled trial

Bankart Procedure (45 Surgical Steps) Arthroscopy Portals

Arthroscopic Instability Assessment

Capsulolabral Mobilization / Glenoid Preparation

1st Anchor Preparation / Insertion

1st Suture Delivery / Management

1st Knot Tying

2nd Anchor Preparation / Insertion

2nd Suture Delivery / Management

2nd Knot Tying

3rd Anchor Preparation / Insertion

3rd Suture Delivery / Management

3rd Knot Tying

Arthroscopic Instability Assessment

Metric Development / Validation - Bankart Procedure

Task Deconstruction

Operational Definitions: Steps & potential intra-operative errors

Construct validation of

metrics on cadaver

Stress test & refine steps & intra-operative errors

Delphi Panel: Consensus on steps (45) and intra-op errors (77)

Construct validation of

shoulder model

Didactic Bankart

Orientation Video

Simulation Model

Benchmark

Didactic Proficiency

Benchmark Bankart Outcome

Benchmark

Binary Assessments (Y/N)

Procedure-Specific Shoulder Model

Construct Validity: Shoulder Model

Group A:

Control

Group B:

Simulation

Group C:

PBP

Lectures / Knot

Tying Practice

Random allocation

Lectures / Knot

Tying Practice

Perform Assisted / Unaided Bankart Repair on Fresh Cadaver (video)

Baseline assessment: visio-spatial, perceptual, and psychomotor performance

Prospective, randomized, blinded comparisons for

intra-operative performance of a Bankart procedure

Lectures / Knot Tying: Test to Proficiency

Train on Simulator: Test to Proficiency

Train on Simulator

Practice Arthroscopic Bankart Skills on Cadaver Shoulders

Random allocation

Online Orientation Video & Didactic Material Test to Proficiency

Independent reviews of videos done in pairs blinded to group IRR = 0.93

Multicenter, prospective, randomized, blinded

trial of proficiency-based progression simulation

training for an arthroscopic Bankart procedure:

PGY 4 and PGY 5 residents

Residency training

program

No. of

residents

1. East Amherst 1 2. Loyola University Medical

Center 1

3. New York Medical College 1 4. Northwestern University 3 5. NS-LIJ 3 6. NS-LIJ - Plainview Hosp 4 7. Ohio State University 1 8. Philadelphia Coll of Osteopathic

Medicine 1

9. Rush University Medical Center 2 10. St Louis 1 11. Tufts 4 12. Tulane University 2 13. UCSF 3 14. University of Illinois at Chicago 1 15. University of Louisville 1 16. University of Miami School of

Medicine 1

17. University of Michigan 1 18. University of Mississippi 2 19. University of Tennessee 3 20. University of Utah Medical

Center 6

21. Yale University Medical Center 2 Total 44

Motor Skill Assessments

Baseline psychomotor parameters Knot tying

Shoulder model Cadaver training Full Procedure on a

Cadaver Shoulder:

Summative Assessment

N = 44

A N = 14

B N = 14

C N = 16

Random

assignment

On-line

material

On-line

material

On-line

material to

proficiency

Traditional

weekend

AANA

Resident

Course

Suturing

and knot

tying

course

Suturing and

knot tying

course to

proficiency

Bankart

Shoulder

Model

Bankart

Shoulder

Model to

proficiency

Assisted but Unaided Video-recorded Cadaver Bankart Repair

Independently Assessed by two Experienced Faculty

Failed to

demonstrate

proficiency

n = 0

n = 1

Ʃ = 2 N = 12 N = 14 N = 12 Ʃ = 4

Trial design W

eeke

nd c

ours

e

Did not

complete index

procedure

n = 4*

N =

6

N = 2

34

36

38

40

42

44

A B C

Bankart procedure steps completed

Mean

no. of

ste

ps

com

ple

ted

an

d

95%

Con

fid

ence

In

tervals

Groups

Steps completed p < 0.000

p < 0.000

A = Traditional training

B = New

curriculum but no

PBP requirement

C = New

curriculum with a

PBP requirement

ALL Group C trainees included

0

1

2

3

4

5

6

7

8

9

10

Mean

no. of

err

ors

an

d

95%

Con

fid

ence

In

tervals

A B C

Groups

Total errors

p < 0.01

p = 0.039

~56%

~41%

A = Traditional training

B = New

curriculum but no

PBP requirement

C = New

curriculum with a

PBP requirement

Objectively assessed intra-operative errors

ALL Group C trainees included

Groups

A B C

0.5

1.0

1.5

2.0

2.5

3.0

0

Mea

n n

o. o

f se

nti

nel

err

ors

an

d

95

% C

on

fid

ence

In

terv

als

Sentinel errors

p < 0.023

p < 0.351

~69%

~44%

A = Traditional training

B = New

curriculum but no

PBP requirement

C = New

curriculum with a

PBP requirement

(all trainees

included)

ALL trainees included in Group C

Proficiency Benchmark

• Benchmark based on mean

faculty performance with

outliers excluded (> 2 SD)

• Pass score defined as:

• Completion of a 3 anchor repair

• No more than 3 procedural errors

• No more than 1 sentinel error

Outcome: Achieved Benchmark?

No Yes Total Group A 10 4 14

71% 29% 100%

Group B 9 5 14

64% 36% 100%

Group C 5 11 16

31% 69% 100%

Outcome: Achieved Benchmark? No Yes Total Group A 10 4 14

71% 29% 100%

Group B 9 5 14

64% 36% 100%

Group C (Met all 3 9 12

proficiency steps) 25% 75% 100%

Group C- (Failed a 2 2 4

proficiency step) 50% 50% 100%

Group C (Overall) 5 11 16

31% 69% 100%

Group A

Group B

Group C

(All C subjects)

Logistic Regression Analysis:

Odds Ratio = 5.5

p = 0.033

Odds Ratio = 1.4

p = 0.121

Proficiency

Benchmark

Demonstrated

(Y/N)

Reference Group

(Traditional Course)

1.4 times more likely

>5 times

more likely

69%

36%

29%

Group C

(PBP only)

Odds Ratio = 7.5

p = 0.036

>7 times

more likely

75%

Summative Assessments: Challenges

ASSET: Video analysis, experts, Likert

scales (Is tool reliable if scored real time?)

VR Knee Scope: Video, experts, Likerts

(Will it transfer to VR computer scoring?)

FAST Knot Tying: Biomechanical test,

direct assessment (Benchmarks correct?)

Copernicus Study: Binary scores, steps &

errors (Is development process scalable?)