Creation of the Simulator Value Index ToolAdapted from workshop on 4.21.14 presented byAmerican College of Surgeons Accreditation Education Institutes, Technologies & Simulation Committee)

Deborah Rooney PhD

James Cooke MD

Yuri Millo MD

David Hananel

MEDICAL SCHOOLUNIVERSITY OF MICHIGAN

Disclosures

o David Hananel, No Disclosures

o Yuri Millo, No Disclosures

o James Cooke, No Disclosures

o Deborah Rooney, No Disclosures

Overview of Main Topics

o Introduction of project

o Overview of 2014 IMSH Survey results

o Summary of 2014 ACS Consortium results

o Working meeting to refine the AVI algorithm

o Apply AVI algorithm in group exercise

o Discuss next steps

Introduction: How it all started

o ACS AEI, Technologies and Simulation Committeeo Guidelines for Simulation Development

(Millo, George, Seymour and Smith)

o University of Michigano Need to support faculty in sim purchase/decision-making

process(Cooke)

o Discourse o Definition of “value”o Differences across stakeholder role

(institution, administration, clinician, educator, researcher...)

Introduction: How it all started

o Reached consensus on factors used when considering a simulator purchaseo Survey 1

o IMSH general membership, N=2800o January, 2014

o Workshop 1, n=16o IMSH, January, 2014

o Survey 2o ACS AEI Consortium membership, N = 455o March, 2014

o Workshop 2, n = ?o ACS AEI-March, 2014

Introduction: The Instrument

o Began with 31-item survey accessed via www (Qualtrics)

o 4-point rating scale o (1 = not considered/not important 4= critical to me when I

consider a simulator purchase)o 6 Domains

o Cost, Impact, Manufacturer, Utility, Assessment, Environment/Ergonomics)

o Demographicso Country/Institutiono Stakeholder role o Involvement o Follow-up

44

5

2

1= Grenada

1= Chile1= Peru

1=Czech Republic

2

2=Singapore

3 = NewZealand

11

95 total respondents, 72 individuals completed survey approximately 2+% of IMSH membership (2,800), 7 undesignated/16 incomplete

IMSH Survey Sample: 67 institutions x 12 Countries

6

1

1

3

4

4

1 1

1= Massachusetts3 = Rhode Island1 = New Jersey

1

3

12

1

3

1

1

1

1

1

50 participants from US

IMSH Survey Sample: 44 institutions x 22 States/US

Acad./U

niv. H

ospita

l

Medica

l Sch

ool

Teachin

g Hosp

ital

Healthca

re S

yste

m

Comm

ercia

l Skil

ls Cente

r

Gov.

/Milit

ary C

enter

Indust

ry05

101520253035404550

4658%

2835% 26

33% 2025%

68%

45%

11%

n = 791 undesignated

IMSH Survey Sample: Institution Affiliation

o Costo Commercial Skills Centers (CSC) rated C1 (Purchase cost) lower than

each of the other institutions, p = .001.

o Manufacturero CSCs rated M1 (Reputation of manufacturer) lower than each of the other

institutions, p = .001.

o Utilityo CSCs rated U3 (Ease of data management) and o U11 (portability) lower than each of the other institutions, p = .001.

o Ergonomicso Medical Schools rated item E2 (Ergonomic risk factor) much higher thank

other institutions), p = .05. CSCs rated E3 (Ease of ergonomic setup) lower than each of the other

institutions, p = .001.

IMSH Survey Results: Rating Differences by Institutional Affiliation

AdministrationClinicianInstitutionTechnicianEducator

3139%

1924%

79%

810%

1418%

n = 791=undesignated

IMSH Survey Sample: Stakeholder Role

o Costo Clinicians rated C2 (Cost of warranty) lower than the

other stakeholders, p = .048.

o Utilityo Clinicians rated U11 (portability of simulator) higher than

other stakeholders, p = .037.

IMSH Survey Results: Rating Differences by Stakeholder Role

Contribute to DecisionLead/ResponsibleApproval ProcessNot Involved

3746%37

46%

45%

23%

n = 80

IMSH Survey Sample: Involvement in Decision

o Although there are no differences across level of involvement,

o There are different considerations during simulator purchasing process across;o Country o Institutional affiliation (commercial skills center may

have unique needs)o Stakeholder role (Clinicians may have unique needs)

o Keeping this in mind, let’s review the top factors considered

IMSH Survey Results: Summary

Average Factor (survey item number, item description) Domain

3.8 21- Technical stability/reliability of simulator Utility3.7 10- Customer service Manufacturer

3.4 16- Ease use for instructor/administrator Utility

3.4 19- Ease of use for learner Utility

3.3 6- Relevance of metrics to real life/clinical setting  Impact

3.2 11- Ease of delivery and installation, orientation to sim Manufacturer

3.2 26- Reproducibility of task/scenario/curriculum Assmnt/Res

3.2 1- Purchase cost of simulator Cost

3.2 9- Reputation of manufacturer Manufacturer

3.1 8- Scalability Impact

3.1 20- Quality of tutoring/feedback from sim to learners Utility

3.1 7- Number of learners impacted Impact

3.0 2- Cost of warranty Cost

3.0 3- Cost of maintenance Cost

3.0 17- Ease of configuration/authoring sim's learning management system Utility

- Physical durability Utility

The SVI Factors: Top 15+1 Factors Ranked

ACS Consortium Survey: Introduction

o Identical Survey items, ratingso Added durability of simulator questiono 31 32-item survey accessed www (Qualtrics)o 4-point rating scale

o (1 = not considered/not important 4= critical to me when I consider a simulator purchase)

o 6 Domainso Cost, Impact, Manufacturer, Utility, Assessment,

Environment/Ergonomics)o Demographics

o Country/Institutiono Stakeholder role o Involvement o Follow-up

49

1

2

1=UK1=France1=Italy

1

65 total respondents, 54 individuals completed survey approximately 12% of ACS membership (455), 2 undesignated

ACS Survey Sample: 41 institutions x 7 Countries

1=Greece

1=Sweden

8

1

1

3

5

4

1 1

8 = Massachusetts1 = Rhode Island1=Delaware1 = Maryland

1

1

44

1

3

2

1

2

1

3

49 participants from US47 indicated institution

ACS Survey Sample: 36 institutions x 17 States/US

1

1

1

Acade

mic

(Univ

ersit

y) H

ospit

al

Teach

ing H

ospit

al

Med

ical S

choo

l

Health

Car

e Deli

very

Sys

tem

Gover

nmen

t or M

ilitar

y Cen

ter

Comm

ercia

l Skil

ls Cen

ter

Indu

stry

0

5

10

15

20

25

30

35

40

3767%

2851% 24

44%

1629%

24% 0

0%

n = 55

ACS Survey Sample: Institution Affiliation

00%

InstitutionAdministrationClinicianTechnicianEducatorResearcherCoordinator

916%

2748%

1<2%

3<6%

1323%

n = 56

ACS Survey Sample: Stakeholder Role

2<4%1

<2%

Contribute to Decision

Lead/Responsible

Approval Process

Not Involved

2545%29

52%

23%

n = 56

ACS Survey Sample: Involvement in Decision

o Although there are no differences across;o institutiono stakeholder role

o There are different considerations during simulator purchasing process across;o Level of involvemento (Self-reported “Responsible” folks are more concerned

about number of learners impacted and Scalability)

ACS Survey Results: Summary

But are there differences across

IMSH and ACS membership?

ACS Survey Results: Summary

1. C1

4. C4

7. I2 10. M2

13. U2

16. U5

19. U8

22. U11

25. A1

28. A4

31. E2

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

1. IMSH2. ACS

Aver

age

Obs

erva

tion

Survey Results: IMSH v. ACS

4 (C2)

7 (I2) 11 (M3) 15 (U4) 22 (U11)

o Costo ACS members rated C2 (Cost of warranty) higher

than the IMH members, bias = .40, p = .04.

o Impacto ACS members rated I2 (Number of learners) higher than other

stakeholders, bias = .53, p = .01.

o Utilityo ACS members rated U4 (Ease of report generation) higher than

the IMH members, bias = .43, p = .02.o ACS members rated U11 (Portability of simulator) higher than

other stakeholders, bias = .48, p = .01.

Survey Results: Rating Differences by Conference

The SVI Factors: Top 15+1 Factors Ranked

Applying the SVI Toolo General impressions? What stood out?

o What worked well?

o What could have gone better?

o Any surprises?

o Usefulness? How might you use the SVI Tool at your institution?

o Please complete the questions on “Feedback” Tab on the SVI Worksheet

Thank you: Our Contact Information

o Deb Rooney University of Michigandmrooney@med.umich.edu

o Jim Cooke University of Michigano cookej@med.umich.edu

o David Hananel SimPORTAL & CRESTUniversity of Minnesota Medical Schoolo dhananel@umn.edu

o Yuri Millo Millo Groupyuri.millo@millo-group.com

o Olivier Petinaux ACS American College of Surgeon, Division of Education opetinaux@facs.org