Post on 16-Apr-2017
Introduction to Optical See-Through Head-Mounted Displays in Augmented Reality
Yuta Itoh23th May, 2016
~2016.03: Ph.D. at TU Munich,@Germany
2016.04~: Project Assistant Prof.@Sugimoto lab
Yuta Itoh
Research topics:OST-HMDs, Eye tracking, AH
Research: http://campar.in.tum.de/Main/YutaItoh
Augmented Reality (AR)
©IKEA, 2013
Augmented Reality (AR)
Feiner, Steven, Blair Macintyre, and DoréeSeligmann. "Knowledge-based augmented reality." Communications of the ACM 36.7 (1993): 53-62.
Hirota, Gentaro, et al. "Superior augmented reality registration by integrating landmark tracking and magnetic tracking." ACM SIGGRAPH 1996.
Fuse virtual contents into the reality
Augmented Reality (AR)
[1] Klinker et al. ISMR’99, [2] Kato and Bilinghurst, [3] Kaufmann and Schmalstieg 2003 [3] IWAR’99, Sony Computer Entertainment, [4] Schwerdtfeger et al. VRST 2008, [5] Bichlmeier et al. ISMAR 20017
Augmented Reality (AR)
Research collections from FAR, TU Munich (http://campar.in.tum.de/Chair/ResearchAr)
Fuchs, Henry, et al. "Optimizing a head-tracked stereo display system to guide hepatic tumor ablation." Studies in health
technology and informatics 132 (2007): 126-131.
Augmented Reality (AR)
Fuchs, Henry, et al. Augmented reality visualization for laparoscopic surgery. Springer Berlin Heidelberg, 1998.
Non-Optical (Video) See-Through
Head-Mounted Displays (HMD)
Optical See-Through (OST-HMD)
VR HMDs
VideoSee-Thro.
OpticalSee-Thro.
Opaque display
Scene camera
Sight ©Robot Genius Film, 2012https://vimeo.com/46304267
Hugo Gernsback, TV Glasses
1963
Ivan Sutherland, The Sword of Damocles
1968
Ivan Sutherland, The Sword of Damocles
1968
2000- (VR HMD)
Toshiba, 2006 Oculus Rift DK2, 2014Photo from IFIXIT
2013-
2013-
Google Glass, 2013Photo by Mikepanhu
Don't Be A Glasshole | Mashable
2016 (VR HMDs)
Oculus Rift, 2016HTC Vive, 2016Photo from Tom's Hardware
2016
Meta 2, 2016 ODG R-7, 2016
EPSON BT300, 2016
Kotaku, Minecraft Hololens demo at E3 2015https://www.youtube.com/watch?v=xgakdcEzVwg
2016-
https://twitter.com/ksasao/status/715569511684911105
Microsoft, Hololens
2016-
Magic Leap, 2016
©Microsoft, 2015, 2016
How to make AR real?
Issues on OST-HMDsDesign Factors
Photo by Mikepanhu
Field of ViewIssues on OST-HMD
https://twitter.com/ksasao/status/715569511684911105
Field of ViewIssues on OST-HMD
“Pinlight Displays: Wide Field of View Augmented Reality Eyeglassesusing Defocused Point Light Sources” Maimone et al., TOG’14
Issues on OST-HMD
“Pinlight Displays: Wide Field of View Augmented Reality Eyeglassesusing Defocused Point Light Sources” Maimone et al., TOG’14
Field of View
Realism in AR (with OST HMDs)
Visual
[Kiyokawa ’01] [Liu ’08] [Lee ’09]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08]
Temporal
[Zheng ’14] [Lincoln ’16]
Realism in AR (with OST HMDs)
Visual
[Kiyokawa ’01] [Liu ’08] [Lee ’09]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08]
Temporal
[Zheng ’14] [Lincoln ’16]
Temporal Realism in OST-HMDs
“How fast is fast enough? : a study of the effects of latency in direct-touch pointing tasks” Jota et al. CH’13
https://www.youtube.com/watch?v=PCbSTj7LjJg
“latencies down to 2.38 ms are required to alleviate user perception when dragging”
Temporal Realism in OST-HMDsDigital Light Processing Projector
“Minimizing Latency for Augmented Reality Displays: Frames Considered Harmful” Zheng et al. ISMAR’14
Temporal Realism in OST-HMDs
Lincoln, Peter, et al. "From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality." (2016), IEEE VR 2016 (Best paper award)
Realism in AR (with OST HMDs)
Visual
[Kiyokawa ’01] [Liu ’08] [Lee ’09]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08]
Temporal
[Zheng ’14] [Lincoln ’16]
Visual Realism (including photorealism)
Occlusion, Depth, Color, etc..
Lee and woo, 2009
Kiyokawa et al. 2003Liu et al. 2008
Occlusion (physical “Z-buffer”)
Kiyokawa et al. 2003
Depth (eye accommodation)
Liu et al. 2008
Konrad et al. 2016
Depth (eye accommodation)
The Light Field Stereoscope,Huan et al., SIGGRPAH 2015
Realism in AR (with OST HMDs)
Visual
[Kiyokawa ’01] [Liu ’08] [Lee ’09]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08]
Temporal
[Zheng ’14] [Lincoln ’16]
Grubert, J., Tümler, J., & Mecke, R. (2008). Untersuchungen zur optimierung der see-through-kalibrierung fuer mobile augmented reality assistenzsysteme. Michael Schenk, Hrsg, 6(7).
Spatial Realism in OST-HMDs
Spatial Realism in OST-HMDs
Optics Designs of OST-HMDs
(Free-form)prism
Point source
Waveguide
Laser source
Half-mirror
Eye
Prism-based
Waveguide-based
Half-mirror
Hainich and Bimber, Displays Fundamentals and Applications, 2011
Retinal Scanning
Eye-HMD Calibration
3D2D
Eye-HMD Calibration
3D
Off-axis Pinhole camera
3D2D
Manual method (De facto standard)
[Tuceryan ’00] [Genc ’02]
SPAAMSingle Point Active Alignment Method
SPAAM (Manual Calibration) video
Track the position!
Itoh and Klinker, IEEE 3DUI 2014, IEEE ISMAR 2014
Automated method
OST-HMD with eye-tracking
Hua and Gao, SPIE, 2012
VR-HMD with eye-tracking
Stengel et al., ACM Multimedia, 2015
Better understandings ofHMD and Eye-ball optics
Locational Realism revisited
EyeEnvironment
Medium
View-Dependent Lens Distortion
Distorted ray
HMDoptics
Image source
Itoh and Klinker IEEE TVCG ‘15 (VR ’15) & IEEE ISMAR ‘15
Light Field Calibration
World camera
User-view camera
Itoh and Klinker IEEE TVCG ‘15 (VR ’15) & IEEE ISMAR ‘15
Learned distortion maps
IEEE TVCG ‘15 (VR ’15) & IEEE ISMAR ‘15
View-Dependent Point-Spread Func.
Corneal-Imaging Calibration
Plopski et al., IEEE TVCG ‘15 (VR ’15)
Schematic Eye for Virtual Environments
Jones et al., IEEE 3DUI 2016
What would happen in 20XX???
Sight ©Robot Genius Film, 2012
Vision Augmentation
Analysis of individual eyes
Computational photography
+
Low-vision devices
Peli, Eli. "Vision multiplexing: an engineering approach to vision rehabilitation device development." Optometry & Vision Science 78.5 (2001): 304-315.
Peli, Eli. "Treating with spectacle lenses: a novel idea!?." Optometry & Vision Science 79.9 (2002): 569-580.
Enhancing Color
Toshiyuki Amano, Oliver Bimber, and Anselm Grundhöfer. "Appearance Enhancement for Visually Impaired with Projector Camera Feedback.“, TECHNICAL REPORT, BAUHAUS-UNIVERSITY WEIMAR, JANUARY 2009
Broecker, Markus, Ross T. Smith, and Bruce H. Thomas. "Adaptive substrate for enhanced spatial augmented reality contrast and resolution." transport 3.6 (2011): 22.
Enhancing Contrast
Huang, Fu-Chung, et al. "Computational light field display for correcting visual aberrations." ACM SIGGRAPH 2013 Posters. ACM, 2013.
De-blurring displays
De-blurring OST-HMD screen
Oshima et al. ISMAR’15
De-blurring eye via OST-HMD
Itoh and Klinker, Augmented Human ‘15
Compensation layer
Blurred
Enhanced
Itoh and Klinker, Augmented Human ‘15
Sharp
In Science Fiction, a precog sees his short-term future…
Next, Nicholas Cage, TM & © Paramount (2007)
Vision Predication
Itoh et al., Augmented Human ‘16
A proof-of-concept system
Tracking system
User-view Camera
OST-HMD
Virtual Image plane
Tracking system
User-view Camera
OST-HMD
Virtual Image plane
A proof-of-concept system
Realism in OST-HMDs Better understandings of
Tracking, Graphics, HMD optics, Eye balls, etc.
Summary
Vision Augmentation Extend human capability
Questions & comments: yuta.itoh@keio.jpResearch: http://campar.in.tum.de/Main/YutaItoh
Resources related to OST-HMD&AR
“Optical See-Through AR Calibration: Methods for Current & Next Generation Head-Mounted Displays”, Kenneth Moserhttps://sites.google.com/site/krm104/project-updates/ieeevr2016tutorialslidesandmaterials
“How to calibrate your Optical See-Through Head Mounted Displays”, Jens Grubert and Yuta Itohenneth Moserhttp://www.slideshare.net/JensGrubert/google-glass-the-meta-and-co-how-to-calibrate-your-optical-seethrough-head-mounted-displays
“2016 AR Summer School ”, Mark Billinghurst,http://www.slideshare.net/marknb00/2016-ar-summer-school-lecture1