➢Introduction: History and Physiology Display Taxonomy Multi-view Rendering using OpenGL/GLSL Designing Content for Glasses-free 3D Displays Emerging Technology

Stereo and 3D Displays

Monocular Depth Cues Supported by Conventional Displays relative and familiar size perspective and occlusion texture gradient, shading and lighting, atmospheric effects

Limitations of Conventional Displays

Limitations of Conventional DisplaysLimitations of Conventional Displays

Monocular Depth Cues with Conventional Displays relative and familiar size perspective and occlusion texture gradient, shading and lighting, atmospheric effects

Additional Monocular Depth Cues motion parallax [Hermann von Helmholtz, 1866] accommodation

What is missing?

Binocular Depth Cues retinal disparity [Charles Wheatstone, 1838] convergence

“It being thus established that the mind perceives an object of three dimensions by means of the two dissimilar pictures projected by it on the two retinae, the following question occurs: What would be the visual effect of simultaneously presenting to each eye, instead of the object itself, its projection on a plane surface as it appears to that eye?”

Binocular Depth Cues

American Civil War-era stereoscopic photos

• Available from the US library of congress•http://www.loc.gov/pictures/search - Search for “stereographs civil war prints”

• Lincoln in 3D• Selection of stereographs converted to red-

cyan anaglyph images•John J. Richter: ISBN 978-0811872317

Interesting Historical Example

l The HVS can ignore conflicting or missing depth cuesl Understand depth in 2D (monocular) videol Perceive shape in “noise”

Ponzo Illusion: © Walt Anthony 2006 magiceye.com

Conflicting Cues

Stereo and 3D Displays

Introduction: History and Physiology

➢Display Taxonomy Multi-view Rendering using OpenGL/GLSL Designing Content for Glasses-free 3D Displays Emerging Technology

Taxonomy of 3D Displays:Glasses-bound vs. Unencumbered Designs

Glasses-boundStereoscopic

Immersive(blocks direct-viewing of real world)

See-through(superimposes synthetic images onto real world)

Head-mounted(eyepiece-objective and microdisplay)

Multiplexed (stereo pair with same display surface)

Spatially-multiplexed (field-concurrent)(color filters, polarizers, autostereograms, etc.)

Temporally-multiplexed (field-sequential)(LCD shutter glasses)

UnencumberedAutomultiscopic

Parallax-based(2D display with light-directing elements)

Volumetric(directly illuminate points within a volume)

Holographic(reconstructs wavefront using 2D element)

Parallax Barriers(uniform array of 1D slits or 2D pinhole arrays)

Integral Imaging(lenticular sheets or fly’s eye lenslet arrays)

Multi-planar(time-sequential projection onto swept surfaces)

Transparent Substrates(intersecting laser beams, fog layers, etc.)

Static(holographic films)

Dynamic(holovideo)

Taxonomy adapted from Hong Hua

Taxonomy of 3D Displays:Immersive Head-mounted Displays (HMDs)

Glasses-boundStereoscopic

Immersive(blocks direct-viewing of real world)

Head-mounted(eyepiece-objective and microdisplay)

Multiplexed (stereo pair with same display surface)

Taxonomy of 3D Displays:See-through Head-mounted Displays (HMDs)

Glasses-boundStereoscopic

Immersive(blocks direct-viewing of real world)

See-through(superimposes synthetic images onto real world)

Head-mounted(eyepiece-objective and microdisplay)

Multiplexed (stereo pair with same display surface)

Taxonomy of 3D Displays:Spatial Multiplexing (e.g., Anaglyphs)

Glasses-boundStereoscopic

Immersive(blocks direct-viewing of real world)

See-through(superimposes synthetic images onto real world)

Head-mounted(eyepiece-objective and microdisplay)

Multiplexed (stereo pair with same display surface)

Spatially-multiplexed (field-concurrent)(color filters, polarizers, etc.)

Taxonomy of 3D Displays:Temporal Multiplexing (e.g., Shutter Glasses)

Glasses-boundStereoscopic

Immersive(blocks direct-viewing of real world)

See-through(superimposes synthetic images onto real world)

Head-mounted(eyepiece-objective and microdisplay)

Multiplexed (stereo pair with same display surface)

Spatially-multiplexed (field-concurrent)(color filters, polarizers, autostereograms, etc.)

Temporally-multiplexed (field-sequential)(LCD shutter glasses)

Taxonomy of 3D Displays:Parallax Barriers

UnencumberedAutomultiscopic

Parallax-based(2D display with light-directing elements)

Volumetric(directly illuminate points within a volume)

Holographic(reconstructs wavefront using 2D element)

Parallax Barriers(uniform array of 1D slits or 2D pinhole arrays)

NewSight MV-42AD3 42''(1920x1080, 1x8 views)

Taxonomy of 3D Displays:Integral Imaging

UnencumberedAutomultiscopic

Parallax-based(2D display with light-directing elements)

Volumetric(directly illuminate points within a volume)

Holographic(reconstructs wavefront using 2D element)

Parallax Barriers(uniform array of 1D slits or 2D pinhole arrays)

Integral Imaging(lenticular sheets or fly’s eye lenslet arrays)

Alioscopy 3DHD 42''(1920x1200, 1x8 views)

Taxonomy of 3D Displays:Multi-planar Volumetric Displays

UnencumberedAutomultiscopic

Parallax-based(2D display with light-directing elements)

Volumetric(directly illuminate points within a volume)

Holographic(reconstructs wavefront using 2D element)

Parallax Barriers(uniform array of 1D slits or 2D pinhole arrays)

Integral Imaging(lenticular sheets or fly’s eye lenslet arrays)

Multi-planar(time-sequential projection onto swept surfaces)

Taxonomy of 3D Displays:Transparent-substrate Volumetric Displays

UnencumberedAutomultiscopic

Parallax-based(2D display with light-directing elements)

Volumetric(directly illuminate points within a volume)

Holographic(reconstructs wavefront using 2D element)

Parallax Barriers(uniform array of 1D slits or 2D pinhole arrays)

Integral Imaging(lenticular sheets or fly’s eye lenslet arrays)

Multi-planar(time-sequential projection onto swept surfaces)

Transparent Substrates(intersecting laser beams, fog layers, etc.)

Taxonomy of 3D Displays:Static Holograms

UnencumberedAutomultiscopic

Parallax-based(2D display with light-directing elements)

Volumetric(directly illuminate points within a volume)

Holographic(reconstructs wavefront using 2D element)

Parallax Barriers(uniform array of 1D slits or 2D pinhole arrays)

Integral Imaging(lenticular sheets or fly’s eye lenslet arrays)

Multi-planar(time-sequential projection onto swept surfaces)

Transparent Substrates(intersecting laser beams, fog layers, etc.)

Static(holographic films)

capture reconstruction

Taxonomy of 3D Displays:Dynamic Holograms (Holovideo)

UnencumberedAutomultiscopic

Parallax-based(2D display with light-directing elements)

Volumetric(directly illuminate points within a volume)

Holographic(reconstructs wavefront using 2D element)

Parallax Barriers(uniform array of 1D slits or 2D pinhole arrays)

Integral Imaging(lenticular sheets or fly’s eye lenslet arrays)

Multi-planar(time-sequential projection onto swept surfaces)

Transparent Substrates(intersecting laser beams, fog layers, etc.)

Static(holographic films)

Dynamic(holovideo)

Tay et al. [Nature, 2008]

MIT Media Lab Spatial Imaging Group[Holovideo, 1989 – present]

Stereo and 3D Displays

Introduction: History and Physiology Display Taxonomy

➢Multi-view Rendering using OpenGL/GLSL Designing Content for Glasses-free 3D Displays Emerging Technology

Overview:GLSL: Programmable Pipeline

Fixed Function Pipeline

Simple 1-Slide Explanati

on!

Drawing APIDrawing API

Process VerticesProcess Vertices

Process PixelsProcess Pixels

FramebufferFramebuffer

Programmable Pipeline

Vertex ProgramVertex Program

Fragment ProgramFragment Program

l Some graphics cards have support for stereo 3D (Not on mobile)l Double buffered stereo = Quad buffered

voiddisplay(void){ glDrawBuffer(GL_BACK_LEFT);

<Draw left eye here>

glDrawBuffer(GL_BACK_RIGHT); <Draw right eye here>

glutSwapBuffers();}

intmain(int argc, char **argv){ glutInit(&argc, argv); glutInitDisplayMode(

GLUT_DOUBLE | GLUT_RGB | GLUT_STEREO); glutCreateWindow("stereo example"); glutDisplayFunc(display); glutMainLoop(); return 0;}

Anaglyphic Model Viewer:Stereo 3D in OpenGL

Overview:Multi-View Rendering in OpenGL

OpenGL Draw Calls

Render

Standard Pipeline

Output

Multi-View Pipeline

Loop Over Views

BackbufferFramebuffer Object Array

Render View

Change Camera

Screen:Memory:

Overview:Multi-View Interlacing using GLSL Shaders

Framebuffer Object Array

Framebuffer Object Array

View 1View 2

View 3

GLSL ProgramTranslate views appropriately for

output device

Translate views appropriately for

output device

BackbufferBackbufferAnaglyph GlassesAnaglyph Glasses LenticularLenticular

Shown in this course…

The model can apply to many others

Multi-View Rendering in OpenGL:Off-Axis Perspective Projection with glFrustum()

Output

Anaglyphic Model Viewer:Anaglyph Compositing Algorithms

LL RR3x3 Color Transform Matrix Pair3x3 Color Transform Matrix Pair

Full Color

Half Color

Optimized

L= R=1 0 00 0 00 0 0

0 0 00 1 00 0 1

L= R=0.299 0 00.587 0 00.114 0 0

0 0 00 1 00 0 1

L= R=0 0 00.7 0 00.3 0 0

0 0 00 1 00 0 1

=

Source: http://3dtv.at/Knowhow/AnaglyphComparison_en.aspx

% read in imagesImL = imread('l.png');ImR = imread('r.png');

% define "half color" matrices (see slides)L = [.299 0 0 .587 0 0 .114 0 0];

R = [0 0 0 0 1 0 0 0 1];

% create a pixel x color arrayImL1d = double(reshape(ImL,prod(size(ImL(:,:,1))),3));ImR1d = double(reshape(ImR,prod(size(ImR(:,:,1))),3));

% perform per pixel color permutationImL1d = ImL1d*L;ImR1d = ImR1d*R;

Anaglyphic Model Viewer:Making an Anaglyph Image in MATLAB

% convert back to 2d x color imageImL = uint8(reshape(ImL1d,size(ImL)));ImR = uint8(reshape(ImR1d,size(ImR)));

% create outputIout = ImL + ImR;

anaglyph.m

Anaglyphic Model Viewer:Demonstration

Stereo and 3D Displays

Introduction: History and Physiology Display Taxonomy Multi-view Rendering using OpenGL/GLSL

➢Designing Content for Glasses-free 3D Displays Emerging Technology

• Stereo cameras (commercial and improvised) are common

Source DataStereo Cameras

• Many researchers/hobbyists have built their own solutions to capture light fields

• The PointGrey ProFusion is one of the few commercially available

PointGrey ProFusion

Stanford

Source DataLight Field Cameras

MIT

Focal Plane

Example in Anaglyph Viewer

Screen

Virtual Object

Placing objects farm from the plane of focus is uncomfortable

Displays with limited DOF: objects further from screen plane are blurred

Rendering TipsAccommodation & Object Placement

Focal Plane

Screen

Kirshnan, V. V., Stark, L. A heuristic model for the human vergence eye movement system, IEEE Trans. BioMed, 1977.

Limit distance of virtual object to viewerLimit rate of change in scene distance

<1 m/s for distant objects

Rendering TipsComfortable Vergence

Off-axis parallel projection Rotate and translate – Toe-in

• Puts ‘infinity’ at axis of rotation•Requires user to focus beyond infinity•Some find diverged eyes uncomfortable

Disadvantages of toe-in • Distortion between views•Camera distance to most objects change•Off axis objects will have different perspective projection

Rendering TipsCamera Model Choice

Avoid cases that cause a view to differ greatly from its neighbor Left Right

Pillar pointing at viewer

Left Right

Clipped by edge of screen

Also watch out for• Far objects clipped by near object• Edges of hallways, tunnels, tubes,

etc

Also watch out for• Don’t exit in front of screen plane• More comfortable behind screen• Avatar does a good job with this

Rendering TipsClipping and Degenerate Cases

Warzone 2100: GL Game Conversion

Stereo and 3D Displays

Introduction: History and Physiology Display Taxonomy Multi-view Rendering using OpenGL/GLSL Designing Content for Glasses-free 3D Displays

➢Emerging Technology

Tensor Display

Tensor Display

Stereo and 3D Displays Resources

SIGGRAPH 2010/2011 Course: BYO3Dhttp://web.media.mit.edu/~mhirsch/byo3d/index.html Long-form slides Code and examples

display blocks

Display Blocks bloghttp://displayblocks.org Tutorials Building blocks explained

Gordon Wetzsteingordonw@media.mit.edu

Matt Hirschmhirsch@media.mit.edu