Post on 14-Dec-2015
Creation of the Simulator Value Index ToolAdapted from workshop on 4.21.14 presented byAmerican College of Surgeons Accreditation Education Institutes, Technologies & Simulation Committee)
Deborah Rooney PhD
James Cooke MD
Yuri Millo MD
David Hananel
MEDICAL SCHOOLUNIVERSITY OF MICHIGAN
Disclosures
o David Hananel, No Disclosures
o Yuri Millo, No Disclosures
o James Cooke, No Disclosures
o Deborah Rooney, No Disclosures
Overview of Main Topics
o Introduction of project
o Overview of 2014 IMSH Survey results
o Summary of 2014 ACS Consortium results
o Working meeting to refine the AVI algorithm
o Apply AVI algorithm in group exercise
o Discuss next steps
Introduction: How it all started
o ACS AEI, Technologies and Simulation Committeeo Guidelines for Simulation Development
(Millo, George, Seymour and Smith)
o University of Michigano Need to support faculty in sim purchase/decision-making
process(Cooke)
o Discourse o Definition of “value”o Differences across stakeholder role
(institution, administration, clinician, educator, researcher...)
Introduction: How it all started
o Reached consensus on factors used when considering a simulator purchaseo Survey 1
o IMSH general membership, N=2800o January, 2014
o Workshop 1, n=16o IMSH, January, 2014
o Survey 2o ACS AEI Consortium membership, N = 455o March, 2014
o Workshop 2, n = ?o ACS AEI-March, 2014
Introduction: The Instrument
o Began with 31-item survey accessed via www (Qualtrics)
o 4-point rating scale o (1 = not considered/not important 4= critical to me when I
consider a simulator purchase)o 6 Domains
o Cost, Impact, Manufacturer, Utility, Assessment, Environment/Ergonomics)
o Demographicso Country/Institutiono Stakeholder role o Involvement o Follow-up
44
5
2
1= Grenada
1= Chile1= Peru
1=Czech Republic
2
2=Singapore
3 = NewZealand
11
95 total respondents, 72 individuals completed survey approximately 2+% of IMSH membership (2,800), 7 undesignated/16 incomplete
IMSH Survey Sample: 67 institutions x 12 Countries
6
1
1
3
4
4
1 1
1= Massachusetts3 = Rhode Island1 = New Jersey
1
3
12
1
3
1
1
1
1
1
50 participants from US
IMSH Survey Sample: 44 institutions x 22 States/US
Acad./U
niv. H
ospita
l
Medica
l Sch
ool
Teachin
g Hosp
ital
Healthca
re S
yste
m
Comm
ercia
l Skil
ls Cente
r
Gov.
/Milit
ary C
enter
Indust
ry05
101520253035404550
4658%
2835% 26
33% 2025%
68%
45%
11%
n = 791 undesignated
IMSH Survey Sample: Institution Affiliation
o Costo Commercial Skills Centers (CSC) rated C1 (Purchase cost) lower than
each of the other institutions, p = .001.
o Manufacturero CSCs rated M1 (Reputation of manufacturer) lower than each of the other
institutions, p = .001.
o Utilityo CSCs rated U3 (Ease of data management) and o U11 (portability) lower than each of the other institutions, p = .001.
o Ergonomicso Medical Schools rated item E2 (Ergonomic risk factor) much higher thank
other institutions), p = .05. CSCs rated E3 (Ease of ergonomic setup) lower than each of the other
institutions, p = .001.
IMSH Survey Results: Rating Differences by Institutional Affiliation
AdministrationClinicianInstitutionTechnicianEducator
3139%
1924%
79%
810%
1418%
n = 791=undesignated
IMSH Survey Sample: Stakeholder Role
o Costo Clinicians rated C2 (Cost of warranty) lower than the
other stakeholders, p = .048.
o Utilityo Clinicians rated U11 (portability of simulator) higher than
other stakeholders, p = .037.
IMSH Survey Results: Rating Differences by Stakeholder Role
Contribute to DecisionLead/ResponsibleApproval ProcessNot Involved
3746%37
46%
45%
23%
n = 80
IMSH Survey Sample: Involvement in Decision
o Although there are no differences across level of involvement,
o There are different considerations during simulator purchasing process across;o Country o Institutional affiliation (commercial skills center may
have unique needs)o Stakeholder role (Clinicians may have unique needs)
o Keeping this in mind, let’s review the top factors considered
IMSH Survey Results: Summary
Average Factor (survey item number, item description) Domain
3.8 21- Technical stability/reliability of simulator Utility3.7 10- Customer service Manufacturer
3.4 16- Ease use for instructor/administrator Utility
3.4 19- Ease of use for learner Utility
3.3 6- Relevance of metrics to real life/clinical setting Impact
3.2 11- Ease of delivery and installation, orientation to sim Manufacturer
3.2 26- Reproducibility of task/scenario/curriculum Assmnt/Res
3.2 1- Purchase cost of simulator Cost
3.2 9- Reputation of manufacturer Manufacturer
3.1 8- Scalability Impact
3.1 20- Quality of tutoring/feedback from sim to learners Utility
3.1 7- Number of learners impacted Impact
3.0 2- Cost of warranty Cost
3.0 3- Cost of maintenance Cost
3.0 17- Ease of configuration/authoring sim's learning management system Utility
- Physical durability Utility
The SVI Factors: Top 15+1 Factors Ranked
ACS Consortium Survey: Introduction
o Identical Survey items, ratingso Added durability of simulator questiono 31 32-item survey accessed www (Qualtrics)o 4-point rating scale
o (1 = not considered/not important 4= critical to me when I consider a simulator purchase)
o 6 Domainso Cost, Impact, Manufacturer, Utility, Assessment,
Environment/Ergonomics)o Demographics
o Country/Institutiono Stakeholder role o Involvement o Follow-up
49
1
2
1=UK1=France1=Italy
1
65 total respondents, 54 individuals completed survey approximately 12% of ACS membership (455), 2 undesignated
ACS Survey Sample: 41 institutions x 7 Countries
1=Greece
1=Sweden
8
1
1
3
5
4
1 1
8 = Massachusetts1 = Rhode Island1=Delaware1 = Maryland
1
1
44
1
3
2
1
2
1
3
49 participants from US47 indicated institution
ACS Survey Sample: 36 institutions x 17 States/US
1
1
1
Acade
mic
(Univ
ersit
y) H
ospit
al
Teach
ing H
ospit
al
Med
ical S
choo
l
Health
Car
e Deli
very
Sys
tem
Gover
nmen
t or M
ilitar
y Cen
ter
Comm
ercia
l Skil
ls Cen
ter
Indu
stry
0
5
10
15
20
25
30
35
40
3767%
2851% 24
44%
1629%
24% 0
0%
n = 55
ACS Survey Sample: Institution Affiliation
00%
InstitutionAdministrationClinicianTechnicianEducatorResearcherCoordinator
916%
2748%
1<2%
3<6%
1323%
n = 56
ACS Survey Sample: Stakeholder Role
2<4%1
<2%
Contribute to Decision
Lead/Responsible
Approval Process
Not Involved
2545%29
52%
23%
n = 56
ACS Survey Sample: Involvement in Decision
o Although there are no differences across;o institutiono stakeholder role
o There are different considerations during simulator purchasing process across;o Level of involvemento (Self-reported “Responsible” folks are more concerned
about number of learners impacted and Scalability)
ACS Survey Results: Summary
But are there differences across
IMSH and ACS membership?
ACS Survey Results: Summary
1. C1
4. C4
7. I2 10. M2
13. U2
16. U5
19. U8
22. U11
25. A1
28. A4
31. E2
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
1. IMSH2. ACS
Aver
age
Obs
erva
tion
Survey Results: IMSH v. ACS
4 (C2)
7 (I2) 11 (M3) 15 (U4) 22 (U11)
o Costo ACS members rated C2 (Cost of warranty) higher
than the IMH members, bias = .40, p = .04.
o Impacto ACS members rated I2 (Number of learners) higher than other
stakeholders, bias = .53, p = .01.
o Utilityo ACS members rated U4 (Ease of report generation) higher than
the IMH members, bias = .43, p = .02.o ACS members rated U11 (Portability of simulator) higher than
other stakeholders, bias = .48, p = .01.
Survey Results: Rating Differences by Conference
The SVI Factors: Top 15+1 Factors Ranked
Applying the SVI Toolo General impressions? What stood out?
o What worked well?
o What could have gone better?
o Any surprises?
o Usefulness? How might you use the SVI Tool at your institution?
o Please complete the questions on “Feedback” Tab on the SVI Worksheet
Thank you: Our Contact Information
o Deb Rooney University of Michigandmrooney@med.umich.edu
o Jim Cooke University of Michigano cookej@med.umich.edu
o David Hananel SimPORTAL & CRESTUniversity of Minnesota Medical Schoolo dhananel@umn.edu
o Yuri Millo Millo Groupyuri.millo@millo-group.com
o Olivier Petinaux ACS American College of Surgeon, Division of Education opetinaux@facs.org