3D Shape and Indirect Appearance by Structured Light Transport

Matthew O’Toole John Mather Kiriakos N. Kutulakos

Department of Computer ScienceUniversity of Toronto

http://www.dgp.toronto.edu/~motoole/slt

1. indirect-only imaging

1. indirect-only imaging

2. indirect-invariant imaging

1. indirect-only imaging

2. indirect-invariant imaging 3. one-shot, multi-pattern imaging

contributions

using epipolar geometry to analyze light transport

contributions

using epipolar geometry to analyze light transport

new live video channels for computer vision• indirect-only and direct-only imaging

• indirect-invariant imaging of structured patterns

• one-shot, multi-pattern imaging

contributions

using epipolar geometry to analyze light transport

new live video channels for computer vision• indirect-only and direct-only imaging

• indirect-invariant imaging of structured patterns

• one-shot, multi-pattern imaging

structured light transport for 3D shape acquisition

contributions

basic light paths

projector

basic light paths

projectorcamera

basic light paths

camera projector

basic light paths

direct light

camera projector

basic light paths

indirect light(scattered)

camera projector

mirror

basic light paths

indirect light(specular)

camera projector

mirror

basic light paths

1. indirect-only imaging

camera projector

mirror

basic light paths

camera projector

mirror

epipolar constraint & light transport

direct paths satisfy epipolar constraints

camera projector

mirror

epipolar constraint & light transport

indirect paths almost never satisfy constraints

camera projector

mirror

epipolar constraint & light transport

indirect paths almost never satisfy constraints

projectorcamera

blocking epipolar paths with patterns & masks

mirror

camera projector

mirror

projection pattern

mask pattern

blocking epipolar paths with patterns & masks

camera projector

mirror

blocking epipolar paths with patterns & masks

projection pattern

mask pattern

camera projector

mirror

blocking epipolar paths with patterns & masks

complementary random epipolar patterns

camera projector

mirror

blocking epipolar paths with patterns & masks

complementary random epipolar patterns

camera projector

mirror

blocking epipolar paths with patterns & masks

complementary random epipolar patterns

camera projector

mirror

blocking epipolar paths with patterns & masks

complementary random epipolar patterns

camera projector

mirror

1. open electronic shutter2. for i = 1 to N

use random epipolar mask &project complementary pattern

3. close electronic shutter

blocking epipolar paths with patterns & masks

complementary random epipolar patterns

live indirect-only and direct-only video stream

conventional imaging

live indirect-only and direct-only video stream

conventional imaging indirect-only imaging

live indirect-only and direct-only video stream

conventional imaging indirect-only imaging

direct-only imaging

live indirect-only and direct-only video stream

conventional imaging indirect-only imaging

direct-only imaging

live indirect-only and direct-only video stream

conventional imaging indirect-only imaging

direct-only imaging

live indirect-only and direct-only video stream

conventional imaging indirect-only imaging

direct-only imaging

live indirect-only and direct-only video stream

conventional imaging indirect-only imaging

direct-only imaging

separating direct & indirect components [e.g. Achar et al. 2013]• multiple shots, motion processing

• only low-frequency indirect light (i.e. diffuse)

transport-robust shape acquisition [e.g. Gupta et al. 2012]• assumption: only high-freq patterns can be robust to indirect light

• no specular transport

structured light transport• live video, fully-general transport, no motion processing

• any projection pattern can be made robust to indirect light

• new forms of one-shot 3D imaging

related work

structured light transport: theory

structured light transport: theory

dominance of non-epipolar transport

transport matrixmask

illumin.

deriving mask and illumination patterns

• exact derivation of patterns is NP-hard [Zhong 2012]

• instead, use random codes, and approximate imaging in expectation

structured light transport: theory

dominance of non-epipolar transport

2. indirect-invariant imaging

live indirect-invariant video stream

conventional structured light

indirect-invariant structured light

mirror

shape acquisition by structured light

mirror

cameraprojector

shape acquisition by structured light

mirror

cameraprojector

shape acquisition by structured light

conventional structured light

reconstructed 3D shape

indirect-invariant structured light

reconstructed 3D shape (same algorithm)

conventional structured light

reconstructed 3D shape

indirect-invariant structured light

reconstructed 3D shape (same algorithm)

3. one-shot, multi-pattern, indirect-invariant imaging

one-shot, multi-pattern, indirect-invariant imaging

one-shot, multi-pattern, indirect-invariant imaging

reconstructed 3D shape

• a new general imaging technique for computer vision

• combines epipolar geometry and light transport analysis

• theoretical analysis & a physical device for live video generation

• SIGGRAPH 2014: extension to time-of-flight imaging

structured light transport

3D Shape and Indirect Appearance by Structured Light Transport

Matthew O’Toole John Mather Kiriakos N. Kutulakos

Department of Computer ScienceUniversity of Toronto

http://www.dgp.toronto.edu/~motoole/slt

Poster ID: O-4A-4