imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of...
Transcript of imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of...
![Page 1: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/1.jpg)
IMGD 5100: Immersive HCI
Augmented Reality
Robert W. Lindeman Associate Professor
Interactive Media & Game Development Department of Computer Science Worcester Polytechnic Institute
![Page 2: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/2.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
2
Motivation ! Augmented Reality
" Mixing of real-world (RW) and computer-generated (CG) stimuli
" Graphical overlays on the real world " Adding information to real experiences
! Much work on visual sense ! Can be extended to auditory sense
" Other senses?
! For the user to merge RW and CG, attributes must be matched " Visual: Lighting & shadows, level of fidelity " Audio: CG and RW sound occlusion and reflection
![Page 3: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/3.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
3
! Direct
Real-World Stimulus Paths
1 2 3 4
Real-World Signal Environment Sensory
Subsystem Nerves Brain
Real-World Signal Environment Capture
Device Post-
Processing Captured
Signal
! Captured/Mediated
![Page 4: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/4.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
4
Visual Sense
PC
! Projection " Mixing in the environment (far)
![Page 5: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/5.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
5
Visual Sense (cont.)
PC
! Optical-see-through AR " Mixing in the environment (near)
![Page 6: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/6.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
6
Visual Sense (cont.)
PC
! Video-see-through AR " Mixing in the Computer
![Page 7: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/7.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
7
Video-See-Through HMD
(Image: Fuchs, et al., Medical Image Computing and Computer-Assisted Intervention (MICCAI) ’98, LNCS, 1998, Vol. 1496/1998, 934)
![Page 8: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/8.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
8
Video-See-Through HMD (cont.)
(Image: Fuchs, et al., Medical Image Computing and Computer-Assisted Intervention (MICCAI) ’98, LNCS, 1998, Vol. 1496/1998, 934)
![Page 9: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/9.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
9
Video-See-Through HMD (cont.) ! NVIS: nVisor MH60-V (2010)
http://www.nvisinc.com/product2009.php?id=57
![Page 10: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/10.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
10
Using Visual AR: SDKs ! ARToolKit
" http://www.hitl.washington.edu/artoolkit/ " Earliest usable kit " Now Open Source (free) " Commercial versions for iPhone & Android
! http://www.artoolworks.com/
! Studierstube ES & Tracker " http://studierstube.icg.tu-graz.ac.at/handheld_ar/ " ES sits on top of Tracker " Not free
![Page 11: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/11.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
11
Using Visual AR: SDKs Examples ! ARToolKit
" http://www.youtube.com/watch?v=5M-oAmBDcZk " (local clip)
! Studierstube ES " http://www.youtube.com/watch?v=JwluCuVKO9c " (local clips)
![Page 12: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/12.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
12
Using Visual AR: Tools ! Google SketchUp + ARMedia Plugin
" http://www.youtube.com/watch?v=wsQ-YGgVUT0 " (local clip) " http://sketchup.google.com/
! Layar for mobile devices " http://www.layar.com/ " Layering tool for layar browser
! "Like HTML for AR" " (local clip)
![Page 13: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/13.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
13
Using Visual AR: Tools (cont.) ! Cereal?
" http://www.youtube.com/watch?v=jGdSslAJRwM " (local clip)
! Slot Cars? " http://www.youtube.com/watch?v=WMWEYqYPDfc " (local clip)
! Magic Tricks? " http://www.youtube.com/watch?v=Mk1xjbA-ISE " (local clip)
! Heads-up Display in Cars (play GE clip) ! Mobile AR (play Nokia clip) ! Mobile 3rd Party
" http://news.bbc.co.uk/2/hi/technology/8193951.stm
![Page 14: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/14.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
14
Sound Paths & Mixing Points
Real-World Sound Environment Outer
Ear Nerves Brain
! Typical VR/AR systems use speakers (1) or headphones (2a)
! Our approach performs the mixing at the cochlea (2b)
Inner Ear
Middle Ear
Ear System
1 2 3 4 a b
![Page 15: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/15.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
15
Auditory Sense ! Acoustic-Hear-Through AR (Speakers)
" Mixing in the environment (far)
PC
![Page 16: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/16.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
16
Auditory Sense (cont.) ! Mic-Through AR
" Mixing in the computer
PC
![Page 17: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/17.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
17
Auditory Sense (cont.) ! Hear-Through AR
" Bone conduction " Mixing at the sensory subsystem
PC
![Page 18: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/18.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
18
Bone-Conduction Example ! The sound of your own voice is a
combination of: " Sound reaching your ears through the air " Vibrations reaching your cochlea though your
head
! Example Sound heard through the air Sound heard through the head Combined sound
Mauldin & Scordilis, 2004
![Page 19: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/19.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
19
Research Questions ! How well can people localize sound using
bone conduction? ! What types of sound works best?
" Ambient sound " Spoken voice " Sound FX " Music
! We looked at basic sounds (sine waves) of various frequencies " Stationary and moving sounds
![Page 20: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/20.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
20
Design of the User Study ! 24 Computer science students (22 male) ! 3 Main treatments (Audio Devices)
! Speakers, Headphones, Bone-Conduction Device
! Each subject performed 63 trials with each device " 3 Frequencies
! Low (200Hz), Medium (500Hz), High (1kHz) " 7 sound samples (5 sound locations + 2 directions)
! Left, Center-Left, Center, Center-Right, Right ! Moving, right-to-left moving
" 3 repetitions of each combination " 3 * 7 * 3 = 63
![Page 21: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/21.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
21
User Study ! Physical/Virtual sound locations
Ө = 45o
Ө Ө
Ө LEFT RIGHT
CENTER
CENTER- LEFT
CENTER- RIGHT
r = 1m
![Page 22: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/22.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
22
User Study (cont.) ! Each sample was played for 1 second ! Subjects wore a blindfold ! No HRTFs used ! Subjects had to identify location/direction
![Page 23: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/23.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
23
Results ! Accuracy for Stationary Sounds
" Speakers > headphones > bone conduction " High-Freq. == Low Freq., both > Medium Freq.
! Accuracy for Moving Sounds " Speakers == Bone conduction " Bone Conduction == Headphones " Speakers > headphones (α = .05)
ns ns Interaction ns HIGH LOW MED Frequency
S B H S H B Audio Device Moving Stationary
![Page 24: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/24.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
24
Results (cont.) ! Problems with the "in-between" locations
" Center-Left/Center-Right
![Page 25: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/25.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
25
Analysis ! Real-world sound
" High fidelity " Low control
! Computer-generated sound " Low(er) fidelity " Complete control
! Later mixing point = Closer to the brain " More personalized, but " More processing for transforming and mixing
![Page 26: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/26.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
26
Analysis (cont.) ! Bone-conduction/headphone approaches
" Require head tracking for CG sound " Require processing for spatialization (e.g.,
HRTF or BRTF)
! Speaker-based " Allows for shared experience (like projection
systems in visual field)
![Page 27: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/27.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
27
Haptic Sense
Mixing at Sensory Subsystem (Novint Falcon)
Mixing in Computer (teleoperation) or in Environment (Immersion CyberGrasp)
![Page 28: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/28.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
28
Haptic Sense (cont.) Mixing in the Environment
(Lindeman, VRST 2004)
![Page 29: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/29.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
29
Olfactory Sense Mixing in the Environment (far) [AirCanon (Yanagida et al., 2004)]
Mixing in the Computer (Hirose et al. 1997)
![Page 30: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/30.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
30
Olfactory Sense (cont.)
Mixing in the Environment (mid) [AirCanon (Yanagida et al., 2004)]
Mixing in the Environment (near) (Nakamoto & Min, 2007)
![Page 31: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/31.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
31
Gustatory Sense ! Bite interface
" Really haptics (near)
Iwata, 2004 (photos: Sid Fels)
![Page 32: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/32.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
32
Gustatory Sense (cont.) ! Edible bits ! Straw-like interface
" Mixing in the env.
(Maynes-Aminzade, 2005) (Nakamoto, 2007)
![Page 33: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/33.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
33
Gustatory Sense (cont.)
![Page 34: imgd5100 09 AugmentedReality - WPIgogo/courses/imgd5100/... · Augmented Reality " Mixing of real-world (RW) and computer-generated (CG) stimuli " Graphical overlays on the real world](https://reader036.fdocuments.us/reader036/viewer/2022081406/5f145d4a27c0d54cbe2dbf61/html5/thumbnails/34.jpg)
R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development
34
Final Thoughts ! What about a 3D printer+robot arm? ! RW stimuli
" High fidelity / low control
! CG stimuli " Low(er) fidelity / complete control
! Later mixing point = more "personal" stimuli " Closer to the brain
! Multi-sensory approaches are interesting " Compensate for weaknesses in one sense with
another sense " Use speakers for environmental, bone-conduction for
virtual characters