Post on 11-Sep-2021
Ashok Veeraraghavan, Ramesh Raskar, Amit Agrawal,
Ankit Mohan & Jack Tumblin
Mitsubishi Electric Research Labs (MERL), Cambridge, MA, USA
Northwestern University, IL
Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing
Coded Exposure [Raskar, Agrawal, Tumblin SIGGRAPH 2006]
Coded Exposure (Flutter Shutter) Camera Raskar, Agrawal, Tumblin [Siggraph2006]
Coding in Time: Shutter is opened and closed
Blurring == Convolution
Traditional Camera: Shutter is OPEN: Box Filter
PSF == Sinc Function
ω
Sharp Photo Blurred Photo
Flutter Shutter: Shutter is OPEN and CLOSED
Preserves High Spatial Frequencies
Sharp Photo Blurred Photo
PSF == Broadband Function
Coded Exposure
Traditional Coded Exposure
Image of Static Object
Deblurred Image
Deblurred Image
How to handle focus blur?
Coded Exposure Coded Aperture
Temporal 1-D broadband code: Motion Deblurring
Spatial 2-D broadband mask: Focus Deblurring
In Focus Photo
Point light source (LED)
Out of Focus Photo: Open Aperture
Lens Camera
sensor
Point
spread
function
Image of a point
light source
Lens and defocus
Focal plane
Lens’ aperture
Slide Credit: Levin et. al
Lens Object Camera
sensor
Point
spread
function
Image of a
defocused point
light source
Lens and defocus
Lens’ aperture
Focal plane
Slide Credit: Levin et. al
Lens Camera
sensor
Point
spread
function
Image of a
defocused point
light source
Lens and defocus
Object
Lens’ aperture
Focal plane
Slide Credit: Levin et. al
Lens Camera
sensor
Point
spread
function
Image of a
defocused point
light source
Lens’ aperture
Lens and defocus
Object
Focal plane
Slide Credit: Levin et. al
Lens and defocus
Lens Camera
sensor
Point
spread
function
Image of a
defocused point
light source
Lens’ aperture
Object
Focal plane
Slide Credit: Levin et. al
Out of Focus Photo: Coded Aperture
Blurred Photos
Coded Aperture, 7 * 7 Mask Open Aperture
Deblurred Photos
Coded Aperture, 7 * 7 Mask Open Aperture
Captured Blurred Photo
Full Resolution Digital Refocusing
Blur Estimation & Segmentation
Captured Blurred Photo
Deblur at different blur sizes k
k = 10
k = 1
•Defocus blur dependent on depth •Assumptions
•Layered Lambertian Scene •Constant blur within each layer
k = 10
k = 1
Deblurred Images
Define Cost Function
Cost Function Images
Likelihood Error: (Blurred image - Sharp Image * PSFk)2
Gradient Error: Natural Image Statistics, Gradient Kurtosis
k = 1
k = 10
Blur Estimation & Segmentation == Labeling
• Graph cuts for labeling
Error Images K = 7
K = 1
k = 10
k = 1
Captured
Photo
Deblur, k = 7
Reblur
Fusion
Less is More Blocking Light == More Information
Coded Exposure Coding in Time
Coded Aperture Coding in Space
Flexible Depth of Field Photography
Nagahara, Kuthirammal, Zhou, and Nayar
ECCV 2008 Slide-deck credit: Nagahara et al.
Hardware Setup
Captured Image
Aperture f/1.4, Exposure 0.36 sec
Deblurred EDOF image
Single “traditional” Image
Aperture f/1.4, Exposure 0.36 sec
Single image with same EDOF
Aperture f/8, Exposure 0.36 sec
Captured Image
Aperture f/1.4, Exposure 0.36 sec
Deblurred EDOF image
Single “traditional” Image
Aperture f/1.4, Exposure 0.36 sec
Single image with same EDOF
Aperture f/8, Exposure 0.36 sec
Tunable focus ring
Discontinuous DOF
Discontinuous DOF
Aperture f/11
Discontinuous DOF
Aperture f/1.4
Tilted DOF
Image from normal camera
Aperture f/1.4
Tilted DOF
Aperture f/1.4
Non-planar DOF
Image from a normal camera
Aperture f/1.4
Non-planar DOF
Aperture f/1.4
Multi-Aperture Photography Paul Green – MIT CSAIL
Wenyang Sun – MERL
Wojciech Matusik – MERL
Frédo Durand – MIT CSAIL
Motivation
http://photographertips.net
Portrait Landscape
Small Aperture
Large Aperture
Depth of Field Control
Shallow Depth of Field
Large Depth of Field
plane of focus
Depth and Defocus Blur
sensor lens
defocus blur depends on distance from plane of focus
subject
rays from point in focus converge to single pixel
circle of confusion
Defocus Blur & Aperture
lens plane of focus
defocus blur depends on aperture size
aperture
http://photographertips.net
sensor
subject
circle of confusion
Goals
Aperture size is a critical parameter for photographers
■ post-exposure depth of field control
■ extrapolate shallow depth of field beyond physical aperture
Outline
Multi-Aperture Camera – New camera design
– Capture multiple aperture settings simultaneously
Applications – Depth of field control
– Depth of field extrapolation
– (Limited) refocusing
Related Work Computational Cameras
– Plenoptic Cameras • Adelson and Wang ‘92 • Ng et al ‘05 • Georgiev et al ‘06
– Split-Aperture Camera • Aggarwal and Ahuja ‘04
– Optical Splitting Trees • McGuire et al ‘07
– Coded Aperture • Levin et al ’07 • Veeraraghavan et al ’07
– Wavefront Coding • Dowski and Cathey ‘95
Depth from Defocus – Pentland ‘87
Georgiev et al‘06
Aggarwal and Ahuja ‘04 McGuire et al ‘07
Adelson and Wang ‘92
Levin et al ’07 Veeraraghavan et al ’07
Plenoptic Cameras
Capture 4D LightField – 2D Spatial (x,y)
– 2D Angular (u,v Aperture)
Trade resolution for flexibility after capture – Refocusing
– Depth of field control
– Improved Noise Characteristics
Lens Aperture
u
v
Sensor (x,y)
Lenslet Array
Subject
Lens (u,v)
1D vs 2D Aperture Sampling
u
v
Aperture
2D Grid
Sampling http://photographertips.net
4 Samples
u
v
Aperture
2D Grid
Sampling
1D vs. 2D Aperture Sampling
Aperture
1D “Ring”
Sampling
45 Samples
http://photographertips.net
Optical Design Principles
Aperture
3D sampling – 2D spatial
– 1D aperture size
– 1 image for each “ring”
Sensor
http://photographertips.net
Goal: Split aperture into 4 separate optical paths – concentric tilted mirrors
– at aperture plane
Aperture Splitting
Tilted Mirrors
Aperture Splitting
Incoming light
Sensor
Mirrors Focusing lenses
Tilted Mirrors
Aperture Splitting
Photographic Lens
Aperture Plane
Relay system Aperture splitting optics
New Aperture Plane
X
Ideally at aperture plane , but not physically possible! Solution: Relay Optics to create virtual aperture plane
Optical Prototype
Mirror Close-up
main lens relay optics
mirrors
tilted mirrors
lenses SLR Camera
Sample Data
Raw data from our camera
Ideally would be rings
Gaps are from occlusion
Point Spread Function Occlusion combined inner ring 1 ring 2 outer
Outline
Multi-Aperture Camera – New camera design
– Capture multiple aperture settings simultaneously
Applications – Depth of field control
– Depth of field extrapolation
– Refocusing
DOF Navigation
0I 2I
1I 3I
Approximate defocus blur as convolution
DOF Extrapolation?
0I
1I 2I 3I
? EI)(0 nn KII
)( nK - Circular aperture blurring kernel
Depends on depth and aperture size What is at each pixel in ? E
EI
Blur s
ize
Aperture Diameter
Largest physical aperture
DOF Extrapolation Roadmap
capture estimate blur fit model extrapolate blur
I E
I 1
I 2
I 0
I 3
Blur s
ize
Aperture Diameter D
I 1 I 2
I E
I 0
σ
I 3
Largest physical aperture
Defocus Gradient
Defocus blur
o
sos
fd
fddfdG
)(
o
sos
fd
fddfd )(DG
odsd
σ D
G is slope of this line
Defocus Gradient Map
Defocus Gradient
focal length
aperture diameter
sensor distance
object distance
Blur proportional to aperture diameter
Optimization
solve for discrete defocus gradient values G at each pixel
Data term
Graph Cuts with spatial regularization term
i
Ni iGKIIGD )()( 1
0
Defocus Gradient Map Smallest Aperture Image
Depth of Field Extrapolation
Synthetic Refocusing
Modify gradient labels and re-synthesize image
gradient map “refocused” map extrapolated f/1.8 “refocused” synthetic f/1.8
Discussion
■ Occlusion ■ Could help depth discrimination (coded aperture)
■ Difficult alignment process ■ Mostly because prototype
■ Refocusing limited by Depth of Field ■ helped by depth-guided deconvolution
■ Texture required for accurate defocus gradient map ■ Not critical for depth of field and refocus
74
4D Frequency Analysis of Computational Cameras for
Depth of Field Extension
Anat Levin1,2 Sam Hasinoff 1 Paul Green1
Frédo Durand 1 Bill Freeman1
1MIT CSAIL 2Weizmann Institute
75 Defocus blur in a standard lens
At focus depth,
sharp
Away from focus
depth, blurred
76 Small aperture – increased depth of field but noisy
Depth 1: sharp but
noisy
Depth 2: sharp but
noisy
77 Extended depth of field cameras
Modified optics
Extended DOF cameras: remove blur computationally
and design optics to make that easy
output input
Deconvolution
78 In this talk
• How much can depth of field be extended?
• New lens extending depth of field
Our design: assembly of subsquares with
different focal powers
each element focuses on a different depth
toy lattice-focal lens
with 4 elements
The lattice-focal lens 79
||)(E
,
3/13/4
3/82
,
yx
yxsS
A
Proof of concept
• 12 subsquares cut
from plano-convex
spherical lenses
• Attached to main lens
extra focal power
needed very low
• Modest DOF
extension with only
12 subsquares
Hardware construction 80
input depth map
• Defocus kernels vary with depth
• Depth estimation as for the coded aperture camera
[Levin et al. 07]
Depth estimation
defocus kernels at
different depths
81
Standard lens reference 82
Lattice-focal lens
Standard lens reference 84
Lattice-focal lens
Standard lens reference 86
Results Lattice-focal lens
Application: Refocusing from single captured image 88
89
Application: Refocusing from single captured image
90
Application: Refocusing from single captured image
The lattice-focal lens – limitations
• Depth estimation needed for deblurring
• Only capture part of the 4D light field
spectrum
• Subsquare size and focal power
optimized for a given focusing range
• Higher spectrum than previous designs,
but does not reach the upper bound
91