High-quality Motion Deblurring from a Single Image...
86
High-quality Motion Deblurring from a Single Image High-quality Motion Deblurring from a Single Image Qi Qi Shan Shan Leo Leo Jiaya Jiaya Jia Jia Aseem Aseem Agarwala Agarwala The Chinese University of Hong Kong The Chinese University of Hong Kong Adobe Systems, Inc.
Transcript of High-quality Motion Deblurring from a Single Image...
High-quality Motion Deblurringfrom a Single Image
High-quality Motion Deblurringfrom a Single Image
QiQi ShanShanLeo Leo JiayaJiaya JiaJiaAseemAseem AgarwalaAgarwala
The Chinese University of Hong KongThe Chinese University of Hong KongAdobe Systems, Inc.
2
The Problem
3
Blur Model
Blurred image I Sharp image L
Camera Noise n
Blur kernel f
4
Previous Work (1)
Hardware solutions:
[Raskar et al. 2006]
[Ben-Ezra and Nayar 2004]
[Levin et al. 2008]
5
Previous Work (2)
Multi-frame solutions:
[Petschnigg et al. 2004]
[Jia et al. 2004] [Rav-Acha and Peleg 2005]
[Yuan et al. 2007]
6
Previous Work (3)
Single image solutions:
[Jia 2007][Fergus et al. 2006]
[Levin et al. 2007] [Yuan et al. 2008]
7
The Problems
Non-blind deconvolution
Blind deconvolution
An Example
9
Challenges (1)
Assuming no noise
10
Challenges (2)
With noise
An Example
An Example
An Example
An Example
15
n
16
n
17
Noise constraint
n
18
0( | 0, )iiN n ζ∏
1( | 0, )iiN n ζ∇∏
0 1( | 0, ) ( | 0, )i ii iN n N nζ ζ∇∏ ∏
Noise constraint
n
Possible noise models:
(1)
(2)
19
0 1
2
( | 0, ) ( | 0, )
( | 0, )i ii i
ii
N n N n
N n
ζ ζ
ζ
∇
∇∇
∏ ∏∏
Noise constraint
n
0( | 0, )iiN n ζ∏
1( | 0, )iiN n ζ∇∏
Possible noise models:
(1)
(2)
20
0 1
2
( | 0, ) ( | 0, )
( ( )
(
| 0, )
) i ii i
ii
P n N n N n
N n
ζ ζ
ζ
= ∇
∇ ∇
∏ ∏∏
A random variable following an independent Gaussian distribution also has its any order derivative following it. [Simon 2002]
n
21
n
22
n
23
n
24
exponentially distributed
, ( 0) jfj
i
ep f fτ−= ≥∏
Kernel Statistics
f
25
L
26
L
Prior p(L) trabajando en dos escalas:Global: para ayudar a mitigar la
ill-possedness del problema (a nivel estadístico)
Local: para corregir los ringingartifacts
27
Image Global Statistics
L
…
28
Image Global Statistics
L
…
29
Image Global Statistics
…
L
30
Image Global Statistics
L
21
| |( ( )
( ))
k L x ca L
P Lb x c
− ∇ ≤⎧= ⎨− ∇ + >
∇⎩
))(log( 1 LP ∇
31
Image Global Statistics
L
21
| |( ( )
( ))
k L x ca L
P Lb x c
− ∇ ≤⎧= ⎨− ∇ + >
∇⎩
))(log( 1 LP ∇
32
Image Local Constraint
L
33
Image Local Constraint
L
34
LI
Image Local Constraint
L
35
LI
Image Local Constraint
L
36
LI
Image Local Constraint
L
37
LI
Image Local Constraint
12 ( | 0,) )(i white
p N L IL σ∈
= ∇ −∇∏
L
∏∈
∇−∇whitei
ii ILN ),0|( 1σ
38
LI
Image Local Constraint
12 ( | 0,) )(i white
p N L IL σ∈
= ∇ −∇∏
L
∏∈
∇−∇whitei
ii ILN ),0|( 1σ
39
Combining All constraints
1 2min ( , ) min log[ ( ) ( ) ( ) ( )]E L f p n p L p L p f= ∇
L f n
Two-step iterative optimization• Optimize L• Optimize f
40
1 2( ) log[ ( ) ( ) ( )]E L p n p L p L= ∇
**
* * 22( ) || ||E L w L f I
∇∇
⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑
Convolution is time-consuming
log ( )p n
1log ( )p L∇ 2log ( )p L1 1 1|| log ( ) ||p Lλ ∇ 2
2 2( || || )i ii white
L Iλ∈
+ ∇ −∇∑
Optimize L
41
Idea: separate convolution
**
* * 22( ) || ||E L w L f I
∇∇
⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑
log ( )p n
Optimize L
1log ( )p L∇ 2log ( )p L1 1 1|| log ( ) ||p Lλ ∇ 2
2 2( || || )i ii white
L Iλ∈
+ ∇ −∇∑
42
Idea: separate convolution
**
* * 22'( ) || ||E L w L f I
∇∇
⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑
log ( )p n
Optimize L
1log ( )p L∇ 2log ( )p L1 1 1|| log ( ) ||pλ Ψ 2
2 2( || || )i ii white
L Iλ∈
+ ∇ −∇∑
43
Idea: separate convolution
**
* * 22'( ) || ||E L w L f I
∇∇
⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑
log ( )p n
Optimize L
1log ( )p L∇ 2log ( )p L1 1 1|| log ( ) ||pλ Ψ 2
2 2( || || )ii white
i Iλ∈
+ −∇Ψ∑
44
**
* * 22'( ) || ||E L w L f I
∇∇
⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑
log ( )p n
Idea: separate convolution
22(|| || )Lγ+ Ψ −∇
Optimize L
1log ( )p L∇ 2log ( )p L1 1 1|| log ( ) ||pλ Ψ 2
2 2( || || )ii white
i Iλ∈
+ Ψ −∇∑
45
**
* * 22'( ) || ||E L w L f I
∇∇
⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑
1 1 1|| log ( ) ||pλ Ψ 22(|| || )Lγ+ Ψ −∇2
2 2( || || )i ii white
Iλ∈
+ Ψ −∇∑
Iteratively optimize L:Update LUpdate Ψ
**
* * 22|| ||w L f I
∇∇
⎛ ⎞= ∇ ⊗ −∇⎜ ⎟⎝ ⎠∑ 2
2(|| || )Lγ Ψ −∇+
46
**
* * 22'( ) || ||E L w L f I
∇∇
⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑ 2
2(|| || )Lγ+ Ψ −∇
Plancherel’s theorem implies:the sum of the square of a function equals the sum of the square of its Fourier transform
Iteratively optimize L:Update LUpdate Ψ
47
**
* * 22'( ) || ||E L w L f I
∇∇
⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑
Iteratively optimize L:Update LUpdate Ψ
22(|| || )Lγ+ Ψ −∇
We minimize E’(F(L)) instead, with a closed-formsolution.
**
* * 22'( ( )) || ( ) ( ) ( ) ||E F L w F L F f F I
∇∇
⎛ ⎞= ∇ − ∇ +⎜⎝
• ⎟⎠
∑22(|| ( ) ( ) || )F F Lγ Ψ − ∇
48
Iteratively optimize L:Update LUpdate Ψ
**
* * 22|| ||w L f I
∇∇
⎛ ⎞∇ ⊗ −∇ +⎜ ⎟
⎝ ⎠∑
1 1 1|| log ( ) ||pλ Ψ 22(|| || )Lγ+ Ψ −∇2
2 2( || || )i ii white
Iλ∈
+ Ψ −∇∑
'( )E Ψ =
49
Iteratively optimize L:Update LUpdate Ψ
1 1 1|| log ( ) ||pλ Ψ 22(|| || )Lγ+ Ψ −∇2
2 2( || || )i ii white
Iλ∈
+ Ψ −∇∑'( )E Ψ =
'i
iE Ψ=∑
each only contains a single variable Ψi'i
E Ψ
50
Iteratively optimize L:Update LUpdate Ψ
Iteration 0Iteration 2Iteration 1Iteration 4Iteration 3Iteration 7Iteration 5Iteration 6Iteration 8 (converge)
51
Time: about 30 seconds for an 800x600 image
Iteration 8 (converge)
52
A comparison
RL deconvolution
53
A comparison
Our deconvolution
54
Two-step iterative optimization• Optimize L• Optimize f
**
* * 22 1( ) || || || ||E f w L f I f
∇∇
⎛ ⎞= ∇ ⊗ −∇ +⎜ ⎟⎝ ⎠∑
1 2min ( , ) min log[ ( ) ( ) ( ) ( )]E L f p n p L p L p f= ∇
A form of L1-norm regularized problem and is solved using an interior point method
55Iteration 0
56Iteration 1
57Iteration 6
58Iteration 10
59
Iteration 1
60
Iteration 2
61
Iteration 4
62
Iteration 6
63
Iteration 8
64
Convergence
Time: about 350 seconds for an 800x600 image
65
Results
66
Results
67
Results
68
Results
69
70
71
More results
72
More results
73
More results
74
More results
75
More results
76
More results
77
More results
78
More results
79
Running Time• Non-blind deconvolution (an 800 x 600 image)
• Blind deconvolution– Approximately 10 minutes for an 800 x 600 image
Approximately 30 seconds in our Matlab implementation
Less than 10 seconds in our C++ implementation
80
Program Available
81
Conclusion
• A well-constrained single-image deblurring model
• Advanced optimization
• Improvement is possible for– Spatially-variant blur– Saturated images
General Blur Images
83
General Blur Images
85
Initialization
• User specified initialization
• Multi-Scale approach– Initialize the clear image in each scale by the one
recovered in a upper layer
86
Color channel
• For a blur images with a large PSF, we take all the three channels (RGB) into computation
• For a blur image with a small PSF, only the luminance channel are taken into computation
![Gated Fusion Network for Joint Image Deblurring and Super ... · Motion deblurring. Conventional image deblurring approaches [2,24,30,31,33,39] assume that the blur is uniform and](https://static.fdocuments.us/doc/165x107/5f89f6087a76073aa41c9ade/gated-fusion-network-for-joint-image-deblurring-and-super-motion-deblurring.jpg)
