Post on 05-Jul-2015
description
Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport
http://www.dgp.toronto.edu/~motoole/temporalprobing.html
Matt O’Toole1 Felix Heide2 Lei Xiao2 Matthias Hullin3 Wolfgang Heidrich2,4 Kyros Kutulakos1
1University of Toronto 2University of British Columbia 3University of Bonn 4KAUST
the time-of-flight (ToF) sensing revolution
estimating poses[Schwarz et al. 11]
Kinect for Xbox Onefemto-photography[Velten et al. 2013]
looking-around-corners[Velten et al. 2012]
gesture recognition[Droeschel et al. 11]
Google’s self-driving car
temporal probing for scene analysis
capture depthrobust to indirect light
reconstruct “light-in-flight” videowith high temporal resolution
key insight: complex-valued image formation model
contributions
the transient-frequency transport matrix
unified theory for light transport analysis analyze ToF images using classical techniques
ToF imaging using projectors & maskscapture depth robust to indirect light
reconstruct “light-in-flight” video with high temporal resolution
what is a ToF photo?
ToF camera
what is a ToF photo?
point lightToF camera
what is a ToF photo?
ToF camera ToF projector
what is a ToF photo?
ToF camera ToF projector
what is a ToF photo?
ToF camera ToF projector
what is a ToF photo?
ToF camera ToF projector
what is a ToF photo?
ToF camera ToF projector
what is a ToF photo?
ToF camera ToF projector
what is a ToF photo?
ToF projectorToF camera
what is a ToF photo?
ToF projectorToF camera
what is a ToF photo?
ToF projectorToF camera
intensity
what is a ToF photo?
phase delay
ToF projectorToF camera
what is a ToF photo?
transport coefficient
ToF camera ToF projector
what is a ToF photo?
transport coefficient
ToF camera ToF projector
what is a ToF photo?
transport coefficient
ToF camera ToF projector
mirror
what is a ToF photo?
transport coefficient
ToF camera ToF projector
mirror
camera projector
it is a vector of complex-valued pixels…
camera
photo
camera
it is a vector of complex-valued pixels…
camera
photo pattern
camera
… produced by a complex-valued matrix
camera
photo patterntransport matrix for
conventional image under ambient lighting
brightness
hue
visualizing a complex ToF image
ph
ase
camera
transient-frequency light transport equation
camera
photo patterntransport matrix for
transient-frequency light transport equation
conventional transport matrix [Ng et al. 03; …]
photo transport matrix for pattern
Techniques Reference(s)
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global separation
[Nayar et al. SIGGRAPH 2006]
Techniques Reference(s) Conventional Analysis
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global separation
[Nayar et al. SIGGRAPH 2006]
Techniques Reference(s) Time-of-Flight Analysis
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global separation
[Nayar et al. SIGGRAPH 2006]
Techniques Reference(s) Time-of-Flight Analysis
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global separation
[Nayar et al. SIGGRAPH 2006]
Techniques Reference(s) Time-of-Flight Analysis
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global separation
[Nayar et al. SIGGRAPH 2006]
Techniques Reference(s) Time-of-Flight Analysis
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global separation
[Nayar et al. SIGGRAPH 2006]
structured light transport [O’Toole et al. CVPR 2014]
acquire direct & indirect ToF images using epipolar constraints
fast direct/global separation [Nayar et al. SIGGRAPH 2006]
acquire caustic indirect & diffuse indirect ToF images using radiometric constraints
structured light transport [O’Toole et al. CVPR 2014]
acquire direct & indirect ToF images using epipolar constraints
fast direct/global separation [Nayar et al. SIGGRAPH 2006]
acquire caustic indirect & diffuse indirect ToF images using radiometric constraints
mirror
ToF camera ToF projector
structured light transport for freq.
mirror
ToF camera ToF projector
structured light transport for freq.
mirror
direct paths obey epipolar geometry
ToF camera ToF projector
structured light transport for freq.
mirror
indirect paths DONTobey epipolar geometry
ToF camera ToF projector
structured light transport for freq.
mirror
projection pattern
mask pattern
ToF camera ToF projector
structured light transport for freq.
mirror
mask pattern
ToF camera ToF projector
structured light transport for freq.
projection pattern
mirror
projection pattern
mask pattern
1. open electronic shutter2. for i = 1 to N
use random epipolar mask &project complementary pattern
3. close electronic shutter
ToF camera ToF projector
structured light transport for freq.
brightness
hue
conventional ToF imaging
brightness
hue
indirect ToF
structured light transport conventional ToF imaging
direct ToF
brightness
hue
indirect ToF
structured light transport conventional ToF imaging
application: 3D from ToF
ToF depth maps & indirect illumination
direct-only
conventional vs direct-only ToF
conventional
conventional direct-only
3D from conventional vs direct-only ToF
conventional direct-only
conventional vs direct-only ToF
conventional direct-only
conventional vs direct-only ToF
conventional direct-only
3D from conventional vs direct-only ToF
structured light transport [O’Toole et al. CVPR 2014]
acquire direct & indirect ToF images using epipolar constraints
fast direct/global separation [Nayar et al. SIGGRAPH 2006]
acquire caustic indirect & diffuse indirect ToF images using radiometric constraints
mirror
effect of indirect transport on spatial frequencies
ToF camera ToF projector
mirror
effect of indirect transport on spatial frequencies
caustic high frequencynon-caustic low frequencyToF camera ToF projector
mirror
effect of indirect transport on spatial frequencies
ToF camera ToF projector
caustic high frequencynon-caustic low frequency
mirror
effect of indirect transport on spatial frequencies
1. for i = 1 to Nproject high spatial-freq. patterncapture image
2. non-caustic = min of images
ToF camera ToF projector
caustic high frequencynon-caustic low frequency
direct ToF
indirect ToF
brightness
hue
structured light transport conventional ToF imaging
direct ToF non-caustic ToF
indirect ToF
structured light transport fast caustic/non-caustic separation
brightness
hue
conventional ToF imaging
direct ToF non-caustic ToF
caustic ToFindirect ToF
structured light transport fast caustic/non-caustic separation
brightness
hue
conventional ToF imaging
direct ToF non-caustic ToF
caustic ToF
application: light-in-flight imaging
mirror
“light-in-flight” imaging for time instant t
goal: reconstruct image for time t
ToF camera ToF projector
“light-in-flight” imaging: prior work
femto-photography[Velten et al. 2013]
very high costwell-resolved wavefronts
PMD-based system[Heide et al. 2013]
many modulation frequenciesstrong scene priors
optical wavefronts not resolvable
our work
many modulation frequenciesweaker, transport-specific priors
resolvable optical wavefronts
conventional ToF for = 100 MHz conventional image
piecewise “light-in-flight” video construction
direct ToF for = 100 MHz light-in-flight video
piecewise “light-in-flight” video construction
direct (no regularization)
caustic ToF for = 100 MHz light-in-flight video
piecewise “light-in-flight” video construction
direct (no regularization)
caustic indirect (no regularization)
light-in-flight videonon-caustic ToF for = 12 to 140 MHz
piecewise “light-in-flight” video construction
direct (no regularization)
non-caustic indirect (regularizedusing [Heide et al. 2013])
caustic indirect (no regularization)
light-in-flight video (ours)light-in-flight video [Heide et al. 13]
comparison to [Heide et al. 2013]
light-in-flight video (ours)light-in-flight video [Heide et al. 13]
comparison to [Heide et al. 2013]
conclusion
• a new paradigm for ToF sensing
• combines ToF sensors with projectors & masks
• can now leverage from decade of light transport research
• widespread implications for spatio-temporal transport analysis
come visit us at our E-Tech booth
visualizing light transport phenomena
Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport
http://www.dgp.toronto.edu/~motoole/temporalprobing.html
Matt O’Toole1 Felix Heide2 Lei Xiao2 Matthias Hullin3 Wolfgang Heidrich2,4 Kyros Kutulakos1
1University of Toronto 2University of British Columbia 3University of Bonn 4KAUST
temporal probing setup
max. exposure time of 8 ms, max. modulation frequency of 150 MHz