Virtual Experiences Research Group

• PhD Students– Kyle Johnsen, Aaron Kotranza, John Quarels, Andrew Raij, Xiyong Wang, Brent Rossen

• Undergraduates– Joshua Horton, Harold Rodriguez

• Funding– National Science Foundation (CAREER, REU), University of Florida Colleges of Engineering and

Medicine, Medical College of Georgia, Keele University, School of Pharmacy

Graduating Students

• Kyle Johnsen– http://www.cise.ufl.edu/

~kjohnsen

• Andrew Raij– http://www.cise.ufl.edu/~raij

• Harold Rodriguez– Undergraduate

Immersive Virtual Humans for Educating Medical and Pharmacy Communication Skills

K. Johnsen, A. Raij, B. Rossen, A. Kotranza, X. Wang, B. LokComputer and Information Science and EngineeringM. Cohen, A. StevensSurgeryJ. CendanCommunity Health and Family Medicine R. FerdigEducation

A. Deladisma, D. S. LindSurgical Oncology

S. Chapman, L. BracegirdlePharmacy

Can Virtual Humans Enable…

VOSCE Project Overview• Started Spring ’04• n > 250 students

– Medical– Nursing– Physician Assistant– Pharmacy

• One of the most popular VH/VR experiences

• Three institutes– University of Florida– Medical College of Georgia– Keele University (U.K.)

• Team– VR/HCI/CS – 1 PhD, 6 grad

students, 2 undergrads– Medicine – 6 MDs, 2 medical

students– Education – 2 PhDs

• Research focus on interfaces

• Play Video

Why Virtual Humans?• Students

– Repetition– Feedback– Longitudinal learning

• Educators– Standardization– Dynamic

• Abnormal findings• Cultural competency

– Aggregate performance• Researchers

– Study the extent of impact of VHs– Easy to run studies (Twiddle one thing)

What can a Virtual Human do?

• Respond to a sneeze• Show empathy

• “I’m scared, can you help me?”• “Could this be cancer?”• Sneezing

• Evoke emotion (e.g. anxiety)• Physiological effects• Social conventions• Similar to a human• So what happens with VH’s of

different backgrounds?

VHs and Bias• VHs elicit racial/ethnic bias

• Working to mitigate effects of bias

Mixed Reality Humans

• Virtual humans have limitations– Open research problems (e.g. AI, speech)– No tactile feedback

• Merge the real and virtual spaces– Real tools– Real simulators

• Complete patient interaction• Physical Exams (e.g. eye exam)

– Point at eye chart– Ophthalmoscope– Follow my finger– 1 or 2 fingers?

Breast Simulator Integration

• Breast simulator integrated– Dr. Carla Pugh, Northwestern– Student does a patient history– Asks to remove gown

(physiological measures)– Performs a breast exam– VP winces at too much pressure

• Future work– Pelvic simulator– Central line simulator

• [Kotranza (submitted) VR2008]

Potential Applications for VHs• Training Interpersonal Scenarios• Employee Training

– Social situations• Sexual Harassment• Cultural Sensitivity/Competency

– Business Dealings• Patron diversity (physical,

emotional, mental)• Park Visitor Experience

Enhancement– Kiosks for patrons to interact with

their own culturally knowledgeable guide

– Personalized “ride” that allows patrons to have immersive experiences

IPS System• Inputs

– Natural speech– Tracking data (head, hands, chair, tools)– Video– Physiological measures

• Outputs– Speech and animation– Life-size projection (or HMD)– Perspective correct rendering – Reactive virtual human

• COTS components– 2 PCs– 2-4 video cameras– Data projector or HMD– Wireless microphone– Bodymedia Sensewear– < $10,000(USD)

• Potential:– Every Hospital

Bias• Study (n=9), people reported

that the African-American VH had less education and money

• Identical: animations, words spoken

• Differences: skin-tone, voice• Other biases to study

– Race/Ethnicity– Gender– Age– Weight– Social ‘hot topics’

Student interaction with VH1

Student interaction with VHn

Behavior CueAnalysis

After-Action VisualizationΔ?

Studying diversity issues with VHs

Abnormal Findings• Conditions difficult to represent with existing

education methods• Conditions

– Psychomotor– Neurological– Age/race/ethnicity dependent conditions

• Blurry vision scenario– Cranial Nerve III (due to brain tumor)– Corneal ulcer– Retinal detachment

• Benefits– Curricular planning of medical student exposure– Supplement SP experiences– Leverage the dynamic nature of VPs

• [Wang ISMAR 2007, submitted VR 2008]• Play Video

Current Work• Classroom

incorporation– Communication course

• 2nd year MS• Surgical rotation 3rd year

MS• Pharmacology 1st year

– n>120 per year– Potentially 30,000

interactions a year @ existing institutions

• Virtual Instructor• Real-time response to

tracked cues– Physiological measures– Posture cues– Verbal cues

Levelof

Interactivity

Normal Abnormal

Standardized Patients

Virtual Patients

Physical Simulators(Harvey, HPS, etc.)

Patient Condition

Textbooks and Journal Articles

Scenarios and VPs

• Acute abdominal pain• Breast mass• Dyspepsia• Sexual history• Eye exam• Patients

– DIgital ANimated Avatar (DIANA)– Elderly Diana (Edna)– Manniquin Diana (Mandi)

• Building the following VPs– Male – Personality (e.g. irate)– Intelligence (e.g. mentally

retarded)– Appearance (e.g. disfigurement,

limbs, burns)

Join Us!

• Application domain outside medicine– Additional locations to install and test system

• Support– Research directions– Students

• Thank you!• http://www.cise.ufl.edu/research/vegroup

• Questions?

Virtual Experiences Research Group

• PhD Students– Kyle Johnsen, Aaron Kotranza, John Quarels, Andrew Raij, Xiyong Wang, Brent Rossen

• Undergraduates– Joshua Horton, Harold Rodriguez

• Funding– National Science Foundation (CAREER, REU), University of Florida Colleges of Engineering and

Medicine, Medical College of Georgia, Keele University, School of Pharmacy

After Action Review of VH Interactions

• Taking a cue from flight simulators

• End-Users– Self-reflection– Understanding– Feedback

• Educators– Quantitative analysis– Identify trends and outliers

• Researchers– Capturing H-VH interaction– Analysis

• Process, Filter, Visualize• [Raij (submission) VR 2008,

SIGCHI 2008]