Mitsubishi Electric Research Laboratories Raskar 2007
Media Lab, MIT
Cambridge, MA
From 4D Capture to 6D Display: From 4D Capture to 6D Display: A mask-based approachA mask-based approach
Ramesh Raskar
P ro jec tor
T ags
P os =0
P os =255
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Discussion Topics
• What is the info content of a 3D scene?– Encoding appearance and geometric
complexity
• What are the dimensions beyond viewpt?– Lighting?
• What other optical sensors we can use?
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
R Raskar, H Nii, B de Decker, Y Hashimoto, J Summet, D Moore, Y Zhao, J Westhues, P Dietz, M Inami, S Nayar, J
Barnwell, M Noland, P Bekaert, V Branzoi, E Bruns
Siggraph 2007
Prakash: Lighting-Aware Motion Capture UsingPhotosensing Markers and Multiplexed Illuminators
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Vicon Motion Capture
High-speed IR Camera
Medical Rehabilitation Athlete Analysis
Performance Capture Biomechanical Analysis
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Imperceptible Tags under clothing, tracked under ambient light
Hidden Marker Tags
Outdoors
Unique Id
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006Labeling Space(Indoor GPS)
Each location receives a
unique temporal code
But 60Hz video projector
is too slow
Projector
Tags
Pos=0
Pos=255
Time
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Pattern
MSB
Pattern
MSB-1
Pattern
LSB
For each taga. From light sequence, decode x and y
coordinateb. Transmit back to RF reader (Id, x, y)
0 1 1 0 0 X=12
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Inside of Multi-LED Emitter
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Tag
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Analog Space Labeling
Multi-LED
Beacon1
Beacon2
Beacon3
Tag
N ?
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Imperceptible Tags Location
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Location Orientation
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
3D Overlay Orientation
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Imperceptible Tags Incident Illumination
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Inverse Optical Mo-Cap
High Speed Camera Detect blobs in each frame
Reflective/Emitting Marker Disambiguate in camera Only Location
High Speed Projector Label the 3D space
Photosensing Marker Find ego-position Location, Orientation, Illum
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
On-set MoCap: Location + Orientation + Incident Illumination
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Coded Illumination Sensor Skin• 500 Hz with Id for each Marker Tag• Capture in Natural Environment
– Visually imperceptible tags– Photosensing Tag can be hidden under clothes– Ambient lighting is ok
• Unlimited Number of Tags– Light sensitive fabric for dense sampling
• Non-imaging, complete privacy• Base station and tags only a few 10’s $
• Body scan + bio– Elderly, patients, athletes, performers
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Project Topics
• Structured Light Scanning– Fast Stripping
• Can you build a scanner using very low cost hardware?• Without full 2D cameras or video projectors?
– Global-direct Separation• Can you scan difficult (global effect) using direct/global
separation?
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Towards a 6D Display
Passive Reflectance Field Display
Martin Fuchs, Ramesh Raskar,Hans-Peter Seidel, Hendrik P. A. Lensch
Siggraph 2008
1 2
11
1 MPI Informatik, Germany 2 MIT
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Martin Fuchs <[email protected]>
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Martin Fuchs <[email protected]>
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Martin Fuchs <[email protected]>
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Improved Design
Martin Fuchs <[email protected]>
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Variance with Observer
Martin Fuchs <[email protected]>
recall:
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Martin Fuchs <[email protected]>
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Martin Fuchs <[email protected]>
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Martin Fuchs <[email protected]>
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Observer-Variance
Martin Fuchs <[email protected]>
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
6D Construction
Martin Fuchs <[email protected]>
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Illumination + Spatial Variation
Martin Fuchs <[email protected]>
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Variance with Observation Angle
Martin Fuchs <[email protected]>
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Towards 6D
Martin Fuchs <[email protected]>
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
6D Results
Martin Fuchs <[email protected]>
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Future Work
• Efficient manufacturing• scale• precision• How fine can we get our structures?
– is 6D really practical?• Extensions for local illumination ?
Martin Fuchs <[email protected]>
Coded Aperture CameraCoded Aperture Camera
The aperture of a 100 mm lens is modified
Rest of the camera is unmodifiedInsert a coded mask with chosen binary pattern
In Focus Photo
LED
Out of Focus Photo: Open Aperture
Out of Focus Photo: Coded Aperture
Captured Blurred Photo
Refocused on Person
Mask? SensorMask
SensorMask? Sensor
MaskSensor
Mask? Sensor
4D Light Field from 2D Photo:
Heterodyne Light Field Camera
Full Resolution Digital Refocusing:
Coded Aperture Camera
Light Field Inside a CameraLight Field Inside a Camera
Lenslet-based Light Field cameraLenslet-based Light Field camera
[Adelson and Wang, 1992, Ng et al. 2005 ]
Light Field Inside a CameraLight Field Inside a Camera
Stanford Plenoptic Camera Stanford Plenoptic Camera [Ng et al 2005][Ng et al 2005]
4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens
Contax medium format camera Kodak 16-megapixel sensor
Adaptive Optics microlens array 125μ square-sided microlenses
Digital RefocusingDigital Refocusing
[Ng et al 2005][Ng et al 2005]
Can we achieve this with a Can we achieve this with a MaskMask alone? alone?
Mask based Light Field CameraMask Sensor
[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]
How to Capture 4D Light Field with 2D
Sensor ?
What should be the pattern of the mask ?
Radio Frequency HeterodyningRadio Frequency Heterodyning
Baseband Audio Signal
Receiver: DemodulationHigh Freq Carrier 100 MHz
ReferenceCarrier
Incoming Signal
99 MHz
Optical HeterodyningOptical Heterodyning
Photographic Signal
(Light Field)
Carrier Incident Modulated
SignalReference
Carrier
Main LensObject Mask Sensor
RecoveredLight Field
Software Demodulation
Baseband Audio Signal
Receiver: DemodulationHigh Freq Carrier 100 MHz
ReferenceCarrier
Incoming Signal
99 MHz
Captured 2D Photo
Encoding due to Mask
2D FFT
Traditional Camera Photo
Heterodyne Camera Photo
Magnitude of 2D FFT
2D FFT
Magnitude of 2D FFT
Computing 4D Light Field2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800
2D FFT
Rearrange 2D tiles into 4D planes200*200*9*94D IFFT
4D Light Field
9*9=81 spectral copies
200*200*9*9
A Theory of Mask-Enhanced CameraA Theory of Mask-Enhanced Camera
Main LensObject Mask Sensor
•Mask == Light Field Modulator
•Intensity of ray gets multiplied by Mask
•Convolution in Frequency domain
fθ
fx
fθ0
fx0
Band-limited Light Field
Sensor Slice – Fourier Slice Theorem
Photo = Slice of Light Field in Fourier Domain
[Ren Ng, SIGGRAPH 2005]
How to Capture 2D Light Field with 1D Sensor ?
fθ
fx
fθ0
fx0
Band-limited Light Field
Sensor Slice
Fourier Light Field Space
Extra sensor bandwidth cannot capture extra dimension of the light field
fθ
fx
fθ0
fx0
Sensor Slice
Extra sensor bandwidth
fθ
fx
??????
??? ???
Solution: Modulation Theorem
Make spectral copies of 2D light field
fθ
fx
fθ0
fx0
Modulation Function
fθ
Modulated Light Field
fx
fθ0
fx0
Modulation Function
Sensor Slice captures entire Light Field
Demodulation to recover Light Field
fθ
fx
Reshape 1D Fourier Transform into 2D
1D Fourier Transform of Sensor Signal
fθ
fx
fθ0
fx0
Modulation Function == Sum of Impulses
Physical Mask = Sum of Cosines
1/f0
Mask Tile
Cosine Mask Used
Where to place the Mask?
Mask
Sensor
Mask
Sensor
Mask Modulation Function Mask Modulation
Function
fx
fθ
Mask Sensor
Where to place the Mask?
Mask Modulation Functionfx
fθ
Captured 2D Photo
Encoding due to Cosine Mask
Computing 4D Light Field2D Sensor Photo, 1800*1800 2D Fourier Transform
2D FFT
Rearrange 2D tiles into 4D planes200*200*9*94D IFFT
4D Light Field
9*9=81 spectral copies
200*200*9*9
Digital Refocusing
Only cone in focus
Captured Photo
Full resolution 2D image of Focused Scene Parts
Captured 2D Photo
Image of White Lambertian Plane
divide
Coding and Modulation in Camera Using MasksCoding and Modulation in Camera Using MasksMask? Sensor
Mask SensorMask
Sensor
Coded Aperture for Full Resolution
Digital RefocusingHeterodyne Light
Field Camera
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Discussion Topics
• What is the info content of a 3D scene?– Encoding appearance and geometric complexity
• Two approaches for multi-view capture or display, – Lenslet (multiscopic), pin-hole array (parallax barrier)
• Third choice – Multiplexing– coding: can we build a display on this principle
• Mask can go anywhere, what else can we achieve?
• Should we think about multi-camera arr like this
Mask-based Approaches
• Coded Illumination– Motion Capture [2007]
• 6D Display– Lighting aware [2008]
• Optical Heterodyning– Light Field Capture [2007]
http://raskar.info
P rojec tor
T ags
P os=0
P os=255
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Discussion Topics• What is the info content of a 3D scene?
– Encoding appearance and geometric complexity
• What are the dimensions beyond viewpt?– Lighting?
• What other optical sensors we can use?
• What are other display technologies?– Materials, configuration
Top Related