CORNAR: Looking Around Corners using Trillion FPS Imaging
-
Upload
mit-media-lab-camera-culture-group -
Category
Technology
-
view
43.080 -
download
3
description
Transcript of CORNAR: Looking Around Corners using Trillion FPS Imaging
![Page 1: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/1.jpg)
Raskar, Camera Culture, MIT Media Lab
Camera Culture
Ramesh Raskar
MIT Media LabRamesh Raskar
Computational Light Transport:
CORNAR: Looking Around Corners using
Trillion FPS Imaging
http://raskar.info/cornar
![Page 2: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/2.jpg)
What isaround the corner ?
![Page 3: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/3.jpg)
Can you look around the corner ?
![Page 4: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/4.jpg)
Can you look around the corner ?
![Page 5: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/5.jpg)
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
![Page 6: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/6.jpg)
![Page 7: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/7.jpg)
![Page 8: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/8.jpg)
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
![Page 9: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/9.jpg)
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
![Page 10: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/10.jpg)
CORNAR: Femto-PhotographyFemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
•2012: 3D around a corner (NatureComm, Velten, et. al.)•2011: Material Sensing (Siggraph Asia, Naik, Zhao, Velten, Raskar, Bala)•2011: DARPA Young Faculty Award•2011: Motion Sensing (CVPR, Pandharkar, Velten, Bardagjy, Bawendi, Raskar)•2009: Hidden barcode (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009)•2008: Indirect depth (Hirsch, Raskar)•2008: Transient Light Transport (Raskar, Davis, March 2008)
With M Bawendi,
MIT Chemistry
![Page 11: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/11.jpg)
Inverting Light Transport
Multiple Scattering Direct/Global
Dual Photography
[Seitz , Kutulakos, Matsushita 2005] [Nayar, Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos, Steger 2005]
![Page 12: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/12.jpg)
Collision avoidance, robot navigation, …
![Page 13: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/13.jpg)
…, bronchoscopies, …
![Page 14: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/14.jpg)
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
![Page 15: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/15.jpg)
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
![Page 16: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/16.jpg)
Streak Photo
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
![Page 17: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/17.jpg)
Multi-Dimensional Light Transport
5-D Transport
![Page 18: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/18.jpg)
s1
Streak-camera
Occluder
C
3rd bounce
Why Pico-second Resolution?
s2ToF Diff = 0.15 mm
p1
p21st bounce
1cm
Curse of Pythagoras
![Page 19: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/19.jpg)
Streak Photo
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
![Page 20: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/20.jpg)
Trillion FPS
ToF Streak Tube = Inverse of CRT
1D camera: Single scan line stretched vertically in time~2 ps resolution, 480 lines ~= 1 ns
But for small samples in biochemistry
Very accurate sync
![Page 21: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/21.jpg)
Time-Image
Time Profile for a single pixel
R
![Page 22: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/22.jpg)
Third Bounce (First bounce not shown)
Space, 640 pixels
Tim
e, ~
2ns e
ach
row
Time Image of a single point
![Page 23: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/23.jpg)
![Page 24: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/24.jpg)
![Page 25: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/25.jpg)
Forward
Wavefront Propagation
Reconstruction
Non-linear Inversion
Invertibility
Scene Priors
Analysis
Resolution and dimensions
3D shape result from synthetic data
![Page 26: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/26.jpg)
Steady State 4D
Impulse Response, 5D
[Raskar and Davis, 2007]
[Kajiya, 1986] [Seitz.., 2005]
![Page 27: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/27.jpg)
Scene with hidden elements
Raw Time profiles
t
Signal Proc.
Novel light transport models and inference
algorithms
Photo, geometry, reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Time Resolved Multi-path Imaging
![Page 28: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/28.jpg)
Third Bounce (First bounce not shown)
![Page 29: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/29.jpg)
Third Bounce (First bounce not shown)
![Page 30: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/30.jpg)
Third Bounce (First bounce not shown)
![Page 31: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/31.jpg)
Third Bounce (First bounce not shown)
![Page 32: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/32.jpg)
Third Bounce (First bounce not shown)
![Page 33: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/33.jpg)
Capture Setup
Photos from Streak Camera
Hidden Scene
![Page 34: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/34.jpg)
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
![Page 35: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/35.jpg)
![Page 36: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/36.jpg)
Hidden 3D Shape Space-time Photos
![Page 37: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/37.jpg)
![Page 38: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/38.jpg)
![Page 39: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/39.jpg)
Motion beyond line of sight
Pandharkar, Velten, Bardagjy, Lawson, Bawendi, Raskar, CVPR 2011
![Page 40: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/40.jpg)
BRDF (reflectance) from Single Viewpoint and Time Images
Naik, Zhao, Velten, Raskar, Bala, (SIGGRAPH Asia 2011)
![Page 41: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/41.jpg)
![Page 42: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/42.jpg)
…, cardioscopies, …
Participating Media
![Page 43: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/43.jpg)
Space-time Photos
![Page 44: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/44.jpg)
![Page 45: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/45.jpg)
Trillion Frames Per Second Imaging
http://raskar.info/trillionfps
![Page 46: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/46.jpg)
Each frame = ~2ps = 0.6 mm of Light Travel
![Page 47: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/47.jpg)
Camera Time
![Page 48: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/48.jpg)
Camera Time
raskar.info/trillionfps
![Page 49: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/49.jpg)
World Time
![Page 50: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/50.jpg)
Lorentz transformation: distances, velocities, ordering
raskar.info/trillionfps
![Page 51: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/51.jpg)
Pixel Ray Wave Photons(ampli+phase)
Steady State TransientImpulse/Step•2012: 3D around corner (Nature Comm, Velten et al)•2011: Material Sensing (Siggraph Asia, Naik, Zhao, Velten, Raskar, Bala)•2011: DARPA Young Faculty Award•2011: Motion Sensing (CVPR, Pandharkar, Velten, Bardagjy, Bawendi, Raskar)•2009: Hidden barcode (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009)•2008: Indirect depth (Hirsch, Raskar)•2008: Transient Light Transport (Raskar, Davis, March 2008)
FemtoFlash
Trillion FPS camera
Computational OpticsSerious Sync
![Page 52: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/52.jpg)
• Collaborations Welcome• Dataset Available• Propose Configurations
raskar(at)media.mit.eduhttp://www.media.mit.edu/~raskar/cornar/
![Page 53: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.fdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/53.jpg)
Femto-PhotographyLooking Around the Corner
BRDF Detection
Trillion FPS Movies
Space-time Transforms
http://raskar.info