Motion Deblurring Using Hybrid Imaging
description
Transcript of Motion Deblurring Using Hybrid Imaging
Motion Deblurring Using Hybrid Imaging
Moshe Ben-Ezra and Shree K. Nayar Columbia University
IEEE CVPR ConferenceJune 2003, Madison, USA
Image Recording Requires Time
Niépce 18278 hours exposure
Daguerre 18291/2 hour exposure
Motion Blur is Everywhere
Object Motion
Camera Motion
Stabilized Lenses
1/250 second (< -1 stop) Stabilization drifts with time Rotation only
1/15 second (< -5 stops)
Canon Stabilized lens 400mm
Blind Image Deconvolution
Accurate Point Spread Function (PSF) Needed.
Motion Point Spread Function (PSF)
Motion PSF is a Function of:
1. Motion path2. Motion speed
X
Y
Ener
gy ~
1/ s
peed
Spatial spread
H
PSF Detector?
Camera
PSF Detector
Can the PSF detector be a small and simple
imaging device ?
Electron wells
Fundamental Limits of Imaging
Detector’s noise level
Photon flux
Detector
Pixel’s Signal Noise
Fundamental Resolution Tradeoff
Spatial resolution (pixels)
Tem
pora
l res
olut
ion
(fps
)
30
330K720x480
Conventional video camera
130
3
3M2048x1536
Hi-resolution camera
75K320x240
Low-resolution camera
Hybrid imaging system
A Hybrid camera enjoys both worlds
Overview of Approach
PSF Estimation
Low-Res. camera
Hi-Res. camera
Same time period
Deconvolution
Motion Analysis
x
y
Global Motion From Low Resolution Detector
uv1
cos sin x sin cos y
0 0 1
xy1
TranslationRotation
2
),(minarg
tI
yIv
xIu
vu
Objective function (Optical flow constraint)
Lucas Kanade
Simulations: Motion Accuracy from Low- Res. Images
NoiseResolution
= 3 = 9 = 27 = 81
640x640 (1:1) 0.01 0.01 0.02 0.04320x320 (1:4) 0.03 0.04 0.05 0.1
160x160 (1:16) 0.03 0.04 0.07 0.480x80 (1:64) 0.13 0.21 0.39 2.6
Average Motion Error in Pixels
Constraints on Continuous PSF
Energy conservation constraint:
yx
dydxyxh 1),(
h(x(t),y(t))dt ttend tstartt
t t
Path is continuous and twice differentiable
Constant flux assumption:
Smoothness constraint:
PSF Estimation from Computed Motion
x
f1f2 f3 f4
f5
f6y
Frame 2 … Frame 5
y f1f2 f3 f4
f5
f6
x
yh
h2
h3
h4h5
Frame 2 … Frame 5
yh
h2
h3
h4h5
Frame 2 … Frame 5
x
Deconvolution of High Resolution Image Standard iterative ratio-based algorithm*
,ˆ)()()(ˆ)(ˆ
)()(ˆ
)()()1(
)0(
k
kk
OSxIxSxOxO
xIxO
Guaranties non-negative pixel result
* Richardson [72] Lucy [74]
ErrorPSFImage estimate
Designs for Hybrid Imaging
A rig of two cameras
Using a special chip
Using a beam splitter
Our Prototype: Rig of Two Cameras
Primary detector(2048x1536)
Secondary detector(360x240)
Resolution ratio of 1 : 36
Example 1 - Blurred Hi-Res Image
f = 633mm, Exp. Time 1 Sec (> -9 stops)
PSF Estimation from Motion
Low resolution sequence.
X (Pixels)10 130
10
90
Y (P
ixel
s)0.001
0.06
Estimated PSF
f = 633mm, Exp. Time 1 Sec
Deblurred Image
f = 633mm, Exp. Time 1 Sec
Example 1 - Comparison
Deblurred imageBlurred image f = 633mm, Exp. Time 1 Sec
Tripod image (Ground Truth)
Example 2 - Blurred Night Image
f = 884mm, Exp. Time 4 Sec (> -11 stops)
PSF Estimation from Motion
X (Pixels)10 60
10
30
Y (P
ixel
s)0.001
0.003
f = 884mm, Exp. Time 4 Sec
Low resolution sequence.
Deblurred Night Image
f = 884mm, Exp. Time 4 Sec
Example 3 - Comparison
Deblurred imageBlurred image
Tripod image (Ground Truth)
f = 884mm, Exp. Time 4 Sec
Object Deblurring Problem
Moving objects blend into the background
Hybrid Imaging Solution (simulated)
Requires clear high-resolution background image
Quantifying The Affect of Motion Blur
Empirical tests: RMS error. Volume of Solutions (Linear Model):
y Ax z x A 1y A 1z
High-ResolutionImage
Uncertainty (Quantization)
InputImages
Volume of Solutions 1/det(A)
BlurDecimation