Introduction to nonlinear image processing Introduction to...
Transcript of Introduction to nonlinear image processing Introduction to...
![Page 1: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/1.jpg)
1 Introduction to nonlinear image processing
IPAM Summer School on Computer Vision
July 22, 2013
Introduction to Nonlinear Image Processing
Iasonas Kokkinos
Center for Visual Computing Ecole Centrale Paris / INRIA Saclay
![Page 2: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/2.jpg)
2 Introduction to nonlinear image processing
Mean and median
Observations in a 3x3 window:
Mean:
Median:
1) Sort:
2) Pick mid-point
robust to outliers
non-linear
outlier
![Page 3: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/3.jpg)
3 Introduction to nonlinear image processing
Mean
![Page 4: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/4.jpg)
4 Introduction to nonlinear image processing
Median
![Page 5: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/5.jpg)
5 Introduction to nonlinear image processing
Gaussian blur
![Page 6: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/6.jpg)
6 Introduction to nonlinear image processing
Image Processing
image image filter
Previous lecture:
This lecture: remove this constraint
freedom! at the expense of control
![Page 7: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/7.jpg)
7 Introduction to nonlinear image processing
7
Multi-scale Gaussian smoothing
![Page 8: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/8.jpg)
8 Introduction to nonlinear image processing
Rewrite Gaussian:
Scale space:
Gaussian scale space
‘time’
A. Witkin, Scale-space filtering, IJCAI, 1983.
J. Koenderink, The structure of images, Biological Cybernetics, 1984
J. Babaud, A. P. Witkin, M. Baudin, and R. O. Duda, ‘Uniqueness of the Gaussian kernel for
scale-space filtering’, PAMI, 1986.
A. Yuille, T.A. Poggio: Scaling theorems for zero crossings. PAMI, 1986.
T. Lindeberg, Scale-Space Theory in Computer Vision, Kluwer, 1994
L. Florack, Image Structure, Kluwer, 1997
B. Romeny, Front-End Vision and Multi-Scale Image Analysis, Kluwer, 2003.
J. Weickert, Linear scale space has first been proposed in Japan. JMIV, 1999.
![Page 9: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/9.jpg)
9 Introduction to nonlinear image processing
Heat diffusion and image processing
Gaussian satisfies:
Scale-space satisfies:
Associative property:
Scale-space satisfies:
Heat diffusion PDE (Partial Differential Equation)
![Page 10: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/10.jpg)
10 Introduction to nonlinear image processing
Heat diffusion and image processing
![Page 11: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/11.jpg)
11 Introduction to nonlinear image processing
Heat diffusion in 1D
![Page 12: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/12.jpg)
12 Introduction to nonlinear image processing
Heat diffusion in 1D – inhomogeneous material
12
conductivity
![Page 13: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/13.jpg)
13 Introduction to nonlinear image processing
Heat diffusion in 2D
Homogeneous
material
Inhomogeneous material
![Page 14: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/14.jpg)
14 Introduction to nonlinear image processing
Perona-Malik Diffusion
P. Perona and J. Malik, Scale-space and edge detection using anisotropic diffusion, PAMI 1990
F. Catte, P.L. Lions, J.M. Morel, T. Coll, Image selective smoothing and edge detection by
nonlinear diffusion, SIAM J. Numer. Analysis, 1992
Image-dependent conductivity
CLMC formulation:
Diffusion stops at strong image gradients (structure-preserving)
![Page 15: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/15.jpg)
15 Introduction to nonlinear image processing
(a) Linear diffusion at t=2 (b) Nonlinear diffusion at t=4.4
Nonlinear vs. linear diffusion
![Page 16: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/16.jpg)
16 Introduction to nonlinear image processing
Extension to vectorial images
• Extension of nonlinear diffusion to vectorial images:
generalization
where:
![Page 17: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/17.jpg)
17 Introduction to nonlinear image processing
(a) Color Image with Noise (b) Perona-Malik diffusion
Nonlinear diffusion for color image denoising
![Page 18: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/18.jpg)
18 Introduction to nonlinear image processing
• Cost functional:
• Euler-Lagrange:
• Heat diffusion: modifies temperature to decrease E quickly
Variational interpretation of heat diffusion
![Page 19: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/19.jpg)
19 Introduction to nonlinear image processing
Variational techniques
• Denoising as functional minimization – Functional: encodes undesirable properties
• Total Variation:
Minimization flow
u1(x) u2(x)
TV[u1] = TV[u2]
TV[f] = TV[g] = TV[h]
w1(x) w2(x)
TV[w1] > TV[w2]
Rudin, L. I.; Osher, S.; Fatemi, E.
"Nonlinear total variation based noise
removal algorithms". Physica D 60, 1992
![Page 20: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/20.jpg)
20 Introduction to nonlinear image processing
(a) Noisy image (b) Total Variation diffusion
Total Variation diffusion
![Page 21: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/21.jpg)
21 Introduction to nonlinear image processing
(a) Perona-Malik diffusion (b) Total-Variation diffusion
Perona-Malik versus Total Variation
![Page 22: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/22.jpg)
22 Introduction to nonlinear image processing
What is the ‘right’ cost functional?
Natural images: natural image statistics
Can we learn the cost function?
S.C. Zhu, Y. N. Wu and D. Mumford, ‘FRAME’, IJCV 1997
• Filters: Use Gabors, Difference-of-Gaussians, Gaussian filters,
• Random Fields: Construct distribution that reproduces their histograms
• And Maximum Entropy: while being as random as possible
Sample: Training:
![Page 23: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/23.jpg)
23 Introduction to nonlinear image processing
GRADE: Gibbs Reaction & Diffusion Equation
GRADE: maximize image probability using Euler-Largange PDEs
From FRAME to GRADE
S.C. Zhu, Y. N. Wu, D. Mumford, ‘Filters, Random Fields and Max. Ent.’, IJCV 1997.
S.C. Zhu and D. Mumford, ‘Gibbs Reaction and Diffusion Equation’, PAMI 1998.
S. Roth and M. Black, ‘Fields of Experts’, IJCV 2009
![Page 24: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/24.jpg)
24 Introduction to nonlinear image processing
Second Moment Matrix
Distribution of gradients:
![Page 25: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/25.jpg)
25 Introduction to nonlinear image processing
Second Moment Matrix
Distribution of gradients:
![Page 26: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/26.jpg)
26 Introduction to nonlinear image processing
Second Moment Matrix
Distribution of gradients:
![Page 27: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/27.jpg)
27 Introduction to nonlinear image processing
Second Moment Matrix
Distribution of gradients:
![Page 28: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/28.jpg)
28 Introduction to nonlinear image processing
• Eigenvectors : directions of maximal and minimal variation of u
• Eigenvalues: amounts of minimal and maximal variation u
Second Moment Matrix
![Page 29: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/29.jpg)
29 Introduction to nonlinear image processing
J. Weickert, Coherence-Enhancing Diffusion Filtering, Image and Vision Computing, 1999.
R. Kimmel, R. Malladi, and N. Sochen. Images as Embedded Maps and Minimal Surfaces, IJCV, 2001.
D. Tschumperlé, and R. Deriche (2005), Vector-valued image regularization with PDE’s. PAMI, 2005.
A. Roussos and P. Maragos, Reversible Interpolation of vectorial Images, IJCV 2009
M. Lindenbaum, M. Fischer and A. Bruckstein, "On Gabor's contribution to image enhancement”, 1994.
D. Gabor, ‘Information theory in electron microscopy’, 1965
Anisotropic diffusion
Nonlinear diffusion
diffusion tensor
structure tensor
Nonlinear Anisotropic diffusion
Slide credit: A. Roussos
![Page 30: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/30.jpg)
30 Introduction to nonlinear image processing
Anisotropic Diffusion example
Slide credits: A. Roussos
![Page 31: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/31.jpg)
31 Introduction to nonlinear image processing
Anisotropic Diffusion example
Slide credits: A. Roussos
![Page 32: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/32.jpg)
32 Introduction to nonlinear image processing
Nonlinear anisotropic diffusion
Extension to vectorial images
Structure tensor for vectorial image:
Slide credits: A. Roussos
![Page 33: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/33.jpg)
33 Introduction to nonlinear image processing
Nonlinear anisotropic diffusion
Nonlinear Anisotropic Diffusion Noisy input
Slide credits: A. Roussos
![Page 34: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/34.jpg)
34 Introduction to nonlinear image processing
Nonlinear vs. Nonlinear and anisotropic diffusion
Nonlinear Anisotropic Diffusion Nonlinear Diffusion
Slide credits: A. Roussos
![Page 35: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/35.jpg)
35 Introduction to nonlinear image processing
Nonlinear vs. Nonlinear and anisotropic diffusion
Nonlinear Anisotropic Diffusion Nonlinear Diffusion
Slide credits: A. Roussos
![Page 36: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/36.jpg)
36 Introduction to nonlinear image processing
(a) Image with missing region
(c1) 300 iterations
(b) Initial prediction (constant)
(c2) 600 iterations (c3) 1500 iterations (converged)
Inpainting region D
Inpainting problem
M. Bertalmío, G. Sapiro, V. Caselles and C. Ballester., "Image Inpainting", SIGGRAPH 2000
T. F. Chan and J. Shen, "Mathematical Models for Local Nontexture Inpainting", SIAM, 2001
![Page 37: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/37.jpg)
37 Introduction to nonlinear image processing
(a) Input (b) Total Variation Inainting
Application: text removal
Slide credits: A. Roussos
![Page 38: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/38.jpg)
38 Introduction to nonlinear image processing
Application: fake bravado
Slide credits: A. Roussos
![Page 39: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/39.jpg)
39 Introduction to nonlinear image processing
(a) Low resolution input image (b) Zero-Padding initialization (c) 4x4 PDE-based magnification
PDE-based interpolation Interpolation problem
A. Roussos and P. Maragos, Reversible Interpolation of vectorial Images, IJCV 2009
![Page 40: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/40.jpg)
40 Introduction to nonlinear image processing
(a) Input Signal (b) Bicubic interpolation (4 x 4)
(d) PDE-based interpolation(4 x 4)
PDE-based interpolation
A. Roussos and P. Maragos, Reversible Interpolation of vectorial Images, IJCV 2009
![Page 41: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/41.jpg)
41 Introduction to nonlinear image processing
(a) Low-resolution Input (b) Initialization: Zero-Padding (c) 4x4 PDE-based interpolation
PDE-based interpolation
A. Roussos and P. Maragos, Reversible Interpolation of vectorial Images, IJCV 2009
![Page 42: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/42.jpg)
42 Introduction to nonlinear image processing
PDE-based interpolation
(a) Input
(c) TV based Interpolation (d) Structure-tensor based interpolation
(b) Bilinear Interpolation
A. Roussos and P. Maragos, Reversible Interpolation of vectorial Images, IJCV
2009
![Page 43: Introduction to nonlinear image processing Introduction to ...helper.ipam.ucla.edu/publications/gss2013/gss2013_11351.pdf · Introduction to nonlinear image processing 43 Further](https://reader030.fdocuments.us/reader030/viewer/2022041010/5eb7fb5e4776c931745e40ba/html5/thumbnails/43.jpg)
43 Introduction to nonlinear image processing
Further study
G. Papandreou and P. Maragos, Multigrid Geometric Active Contour Models,TIP, 2007.
J. Weickert and B. H. Romeny, 'Efficient Schemes for Nonlinear Diffusion Filtering', TIP '98
A. Chambolle, 'An Algorithm for Total Variation Minimization and Applications', JMIV 2004
T. Goldstein, S. Osher The Split Bregman method for L1-regularized problems, SIAM 2009
Fast numerical solutions
Online software
http://www.ipol.im/
Not covered in this talk
Bilteral Filter, Non-Local Means
Buades, B. Coll, and J. Morel, “A non-local algorithm for image denoising,” CVPR, 2005
C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” ICCV 1998
P Milanfar, " A Tour of Modern Image Filtering ", IEEE Signal Processing Magazine, 2013