Download - Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport

Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport

http://www.dgp.toronto.edu/~motoole/temporalprobing.html

Matt O’Toole1 Felix Heide2 Lei Xiao2 Matthias Hullin3 Wolfgang Heidrich2,4 Kyros Kutulakos1

1University of Toronto 2University of British Columbia 3University of Bonn 4KAUST

the time-of-flight (ToF) sensing revolution

estimating poses[Schwarz et al. 11]

Kinect for Xbox Onefemto-photography[Velten et al. 2013]

looking-around-corners[Velten et al. 2012]

gesture recognition[Droeschel et al. 11]

Google’s self-driving car

temporal probing for scene analysis

capture depthrobust to indirect light

reconstruct “light-in-flight” videowith high temporal resolution

key insight: complex-valued image formation model

contributions

the transient-frequency transport matrix

unified theory for light transport analysis analyze ToF images using classical techniques

ToF imaging using projectors & maskscapture depth robust to indirect light

reconstruct “light-in-flight” video with high temporal resolution

what is a ToF photo?

ToF camera


point lightToF camera


ToF camera ToF projector


ToF projectorToF camera



intensity


phase delay



transport coefficient


mirror


transport coefficient


mirror

camera projector

it is a vector of complex-valued pixels…

camera

photo

camera

it is a vector of complex-valued pixels…

camera

photo pattern

camera

… produced by a complex-valued matrix

camera

photo patterntransport matrix for

conventional image under ambient lighting

brightness

hue

visualizing a complex ToF image

ph

ase

camera

transient-frequency light transport equation

camera

photo patterntransport matrix for

transient-frequency light transport equation

conventional transport matrix [Ng et al. 03; …]

photo transport matrix for pattern

Techniques Reference(s)

transport equation [Ng et al. 03]

dual photography [Sen et al. 05; Sen and Darabi 09]

radiometric compensation [Wetzstein and Bimber 07]

radiosity equation [Goral et al. 84]

radiosity solution [Goral et al. 84]

inverse light transport [Seitz et al. 05; Bai et al. 10]

transport eigenvectors [O’Toole and Kutulakos 10]

structured light transport [O’Toole et al. CVPR 2014]

fast direct/global separation

[Nayar et al. SIGGRAPH 2006]

Techniques Reference(s) Conventional Analysis











Techniques Reference(s) Time-of-Flight Analysis












acquire direct & indirect ToF images using epipolar constraints

fast direct/global separation [Nayar et al. SIGGRAPH 2006]

acquire caustic indirect & diffuse indirect ToF images using radiometric constraints

mirror


structured light transport for freq.

mirror

direct paths obey epipolar geometry



mirror

indirect paths DONTobey epipolar geometry



mirror

projection pattern

mask pattern



mirror

mask pattern



projection pattern

mirror

projection pattern

mask pattern

1. open electronic shutter2. for i = 1 to N

use random epipolar mask &project complementary pattern

3. close electronic shutter



brightness

hue

conventional ToF imaging

brightness

hue

indirect ToF

structured light transport conventional ToF imaging

direct ToF

brightness

hue

indirect ToF


application: 3D from ToF

ToF depth maps & indirect illumination

direct-only

conventional vs direct-only ToF

conventional

conventional direct-only

3D from conventional vs direct-only ToF


conventional vs direct-only ToF


3D from conventional vs direct-only ToF


acquire direct & indirect ToF images using epipolar constraints

fast direct/global separation [Nayar et al. SIGGRAPH 2006]

acquire caustic indirect & diffuse indirect ToF images using radiometric constraints

mirror

effect of indirect transport on spatial frequencies


mirror


caustic high frequencynon-caustic low frequencyToF camera ToF projector

mirror



caustic high frequencynon-caustic low frequency

mirror


1. for i = 1 to Nproject high spatial-freq. patterncapture image

2. non-caustic = min of images


caustic high frequencynon-caustic low frequency

direct ToF

indirect ToF

brightness

hue


direct ToF non-caustic ToF

indirect ToF

structured light transport fast caustic/non-caustic separation

brightness

hue



caustic ToFindirect ToF

structured light transport fast caustic/non-caustic separation

brightness

hue



caustic ToF

application: light-in-flight imaging

mirror

“light-in-flight” imaging for time instant t

goal: reconstruct image for time t


“light-in-flight” imaging: prior work

femto-photography[Velten et al. 2013]

very high costwell-resolved wavefronts

PMD-based system[Heide et al. 2013]

many modulation frequenciesstrong scene priors

optical wavefronts not resolvable

our work

many modulation frequenciesweaker, transport-specific priors

resolvable optical wavefronts

conventional ToF for = 100 MHz conventional image

piecewise “light-in-flight” video construction

direct ToF for = 100 MHz light-in-flight video


direct (no regularization)

caustic ToF for = 100 MHz light-in-flight video



caustic indirect (no regularization)

light-in-flight videonon-caustic ToF for = 12 to 140 MHz



non-caustic indirect (regularizedusing [Heide et al. 2013])

caustic indirect (no regularization)

light-in-flight video (ours)light-in-flight video [Heide et al. 13]

comparison to [Heide et al. 2013]

conclusion

• a new paradigm for ToF sensing

• combines ToF sensors with projectors & masks

• can now leverage from decade of light transport research

• widespread implications for spatio-temporal transport analysis

come visit us at our E-Tech booth

visualizing light transport phenomena

Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport

http://www.dgp.toronto.edu/~motoole/temporalprobing.html

Matt O’Toole1 Felix Heide2 Lei Xiao2 Matthias Hullin3 Wolfgang Heidrich2,4 Kyros Kutulakos1

1University of Toronto 2University of British Columbia 3University of Bonn 4KAUST

temporal probing setup

max. exposure time of 8 ms, max. modulation frequency of 150 MHz