MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Amit Agrawal, Ashok...
-
Upload
silas-thompson -
Category
Documents
-
view
221 -
download
0
Transcript of MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Amit Agrawal, Ashok...
MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar
Amit Agrawal, Ashok Veeraraghavan and Ramesh Raskar
Mitsubishi Electric Research Labs (MERL)MIT Media Lab
Cambridge, MA, USA
Reinterpretable Imager: Towards Variable Post-Capture Space, Angle & Time Resolution
in Photography
Captured Photo
Video from Single-Shot (Temporal Frames)
Captured Photo
Rotating Doll in Focus
Captured 2D Photo
Captured 2D Photo
In-Focus
High Resolution 2D Image
Static Scene Parts
Captured 2D Photo
In-Focus Out of Focus
High Resolution 2D Image
4D Light Field
Static Scene Parts
Captured 2D Photo
In-Focus Out of Focus In-Focus
High Resolution 2D Image
4D Light Field
Video
Static Scene Parts Dynamic Scene Parts
Captured 2D Photo
In-Focus Out of Focus In-Focus
High Resolution 2D Image
4D Light Field
Video
Static Scene Parts Dynamic Scene Parts
1D Parallax + Motion
Out of Focus
MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar
Key Idea
• Resolution tradeoff for Conventional Imaging
– Fixed before capture
• video camera, lightfield camera– Scene independent
• Resolution tradeoff for Reinterpretable Imager– Variable in post-capture– Scene dependent– Different for different parts of the scene/captured photo
MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar
Reinterpretable Imager
• Capture Time-Varying Light Field• Spatial dimensions = 2• Angular dimensions = 2• Temporal dimensions = 1
• We capture 4D subsets in single-shot
• Single Camera Design can act as– Still camera, Video camera, lightfield camera
Optical Design
Dynamic Aperture
Mask
Sensor
Static Mask
Sensor
Static Aperture
Mask
Sensor
Coded Aperture Optical Heterodyning Reinterpretable Imager
Veeraraghavan et al. SIGGRAPH 2007
Veeraraghavan et al. SIGGRAPH 2007
This Paper
Static Mask
Digital Refocusing
Dynamic Aperture
Mask
Sensor
Static Mask
Sensor
Static Aperture
Mask
Sensor
Coded Aperture Optical Heterodyning Reinterpretable Imager
Veeraraghavan et al. SIGGRAPH 2007
Veeraraghavan et al. SIGGRAPH 2007
This Paper
Static Mask
Digital Refocusing
Light Field Capture
Static Mask
Sensor
Static Aperture
Mask
Sensor
Coded Aperture Optical Heterodyning
Veeraraghavan et al. SIGGRAPH 2007
Veeraraghavan et al. SIGGRAPH 2007
This Paper
Dynamic Aperture
Mask
Sensor
Reinterpretable Imager
Static Mask
Digital Refocusing
Light Field Capture
Post-Capture Resolution Control
MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar
Reinterpretable Imager
• Dynamic Aperture Mask
– Pinhole moved across the aperture during exposure time• Single shot video, lightfield, high res image
– Vertical slit moved across the aperture• 1D parallax + motion
• Near-Sensor Mask– Similar to Veeraraghvan et al. SIGGRAPH 2007
MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar
Implementation
Camera Motor Wheel
Aperture Mask on Wheel
Shutter
Near-Sensor Mask
MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar
Related Work
• Hand-held Light Field Camera• Single shot• Micro-lens Ng et al. 2005• Prims+lens Georgiev et al. 2006• Mask at sensor Veeraraghavan et al. 2007
• Light Field camera + Aperture Modulation– Horstmeyer ICCP 09– Polarization, Spectral
• Multiplexing techniques– Assorted Pixels– Illumination multiplexing, Schechner et al. ICCV 2003
MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar
Recovering Full Resolution 2D ImageRecovering Full Resolution 2D Image
Sensor
p
Mask
No Mask: pixel value = I(p)
With Mask: pixel value = β(p)I(p)
In-focus static scene
Mask
For Static In-Focus Scene
Captured 2D Photo
Divide by Calibration Photo
High Resolution Image
MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar
Recovering Full Resolution 2D ImageRecovering Full Resolution 2D Image
• For static in-focus scene
– Inserting Masks == Spatially Varying Image Attenuation
– Compensate using normalization photo β(p)
In Focus Out of Focus
In Focus Out of Focus
Captured Photo Normalization PhotoFull Resolution Image
MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar
Recovering Light Fields for Static SceneRecovering Light Fields for Static Scene
• Capture Light Fields:– Map angular variations to spatial dimension– Angle To Space Mapping
• For static scenes
– Mask close to sensor == captures light field
• Heterodyning, Veeraraghavan et al. SIGGRAPH 2007
– Mask in aperture == no impact, only lose light
For Static Scenes
Captured 2D Photo
Compute 4DLight Field
Digital Refocusing
Sub-Aperture Views == Angular Samples
Captured Photo
(Static Scene)
Reconstructed Sub-Aperture Views (3 by 3 Light Field)
Angle
Angle
MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar
Recovering Video from Single ShotRecovering Video from Single Shot
• In-focus dynamic scene
– Mask in aperture
• maps temporal variations to angular variations
– Mask close to sensor
• captures angular variations as a light field
• Mapping Time to Space Via– Time to Angle + Angle to Space
Static Mask
Moving Pinhole
K
K
Scene patch (t1)
Capturing In-focus Dynamic Scenes
Static Mask
K
K
Scene patch (t2)
Capturing In-focus Dynamic Scenes
Capturing In-focus Dynamic Scenes
Sensor
Static Mask
d
K
KspotScene patch (t3)
For Dynamic In-focus Scene
Captured 2D Photo
Compute 4DLight Field
Sub-Aperture Views == Temporal Frames
Captured Photo
Reconstructed Sub-Aperture Views (3 by 3 Light Field)
Time
Time
Rotating Doll
Reconstructed Sub-Aperture Views (3 by 3 Light Field)
Time
Time
For Rotating Doll
Angle
Angle
For Static Scene Parts
MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar
Recovering 1D Parallax + MotionRecovering 1D Parallax + Motion
• Vertical slit moved across the aperture
– Map angular variations to vertical dimension
– Maps temporal variations to horizontal dimension
• Capture dynamic out-of-focus scene
– However, only 1D out-of-focus blur (bokeh)
For Dynamic Out-of-focus Scene
Captured 2D Photo
Compute 4DLight Field
Sub-Aperture Views == Temporal Frames (Horizontally)
Sub-Aperture Views == Angular Samples (Vertically)
Refocus Using Vertical Views
Captured Photo
Reconstructed Sub-Aperture Views (3 by 3 Light Field)
Time
Angle
Digital Refocusing on Moving Rubik’s Cube
Digital Refocusing on Moving Rubik’s Cube
Digital Refocusing on Moving Rubik’s Cube
Digital Refocusing on Moving Rubik’s Cube
Digital Refocusing on Moving Rubik’s Cube
Digital Refocusing on Moving Rubik’s Cube
Keeping Playing Card in Focus
Keeping Playing Card in Focus
Keeping Playing Card in Focus
Keeping Playing Card in Focus
Keeping Playing Card in Focus
Keeping Playing Card in Focus
Captured Photo
Static Object (in-focus)
Static Objects (Out-of-focus)
Moving Object (in depth)
Rotating Object (in focus)
Reconstructed Sub-Aperture Views (3 by 3 Light Field)
All Static and Dynamic Objects are sharp
(No focus blur, no motion blur)
Angle
Angle
For Static Objects
Time
Angle
For Moving Toy in Middle
Time
Time
For Rotating Toy on Right
Refocused on Static Toy
High Resolution Image
Digital Refocusing on Static Objects
Digital Refocusing on Static Objects
Digital Refocusing on Static Objects
Digital Refocusing on Static Objects
Digital Refocusing on Static Objects
Digital Refocusing on Static Objects
Digital Refocusing on Toy Moving in Depth
Digital Refocusing on Toy Moving in Depth
Digital Refocusing on Toy Moving in Depth
Digital Refocusing on Toy Moving in Depth
Digital Refocusing on Toy Moving in Depth
Digital Refocusing on Toy Moving in Depth
Video frames of Rotating ToyVideo for Rotating Toy in-focus
MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar
Limitations
• Light Loss– To get extra information– Both at aperture and sensor– Micro-lens at sensor (Ng et al.) for lightfield capture
• Temporal Frames– No. of frames = Max angular resolution– Not independent as in a video camera– Large motions cause motion blur– Viewpoint shift– Ghosting artifacts across sub-aperture views
• Does not capture full 5D information– Video light field camera
MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar
Future Work
• LCD’s for modulation– Benefit: Faster modulation– Issues: Contrast, Diffraction
• Using Computer Vision– No high/mid-level processing at present
• Adaptive (Active) Sampling
MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar
AcknowledgementsAcknowledgements
• Anonymous Reviewers
• MERL – Jay Thornton, Kojima Keisuke, Joseph Katz, John Barnwell,
Brandon Taylor, Clifton Forlines and Yuichi Taguchi
• Mitsubishi Electric, Japan– Haruhisa Okuda & Kazuhiko Sumi
MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar
Google: ‘Reinterpretable Imager’
Captured 2D Photo
In-Focus Out of Focus In-Focus Out of Focus
High Resolution 2D Image
4D Light Field Video1D Parallax
+ Motion
Static Scene Parts Dynamic Scene Parts Dynamic Aperture
Mask
Sensor
Reinterpretable Imager
Static Mask