Super Resolution in Digital Image processing
-
Upload
shriram-desai -
Category
Education
-
view
637 -
download
7
description
Transcript of Super Resolution in Digital Image processing
![Page 1: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/1.jpg)
Super-Resolution
![Page 2: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/2.jpg)
3/7/2003 Super-Resolution 2
Super-resolution
• convolutions, blur, and de-blurring• Bayesian methods
• Wiener filtering and Markov Random Fields
• sampling, aliasing, and interpolation• multiple (shifted) images• prior-based methods
• MRFs• learned models• domain-specific models (faces)- Gary
![Page 3: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/3.jpg)
3/7/2003 Super-Resolution 3
Linear systems
Basic properties• homogeneity T[a X] = a T[X]
• additivity T[X1+X2] = T[X1]+T[X2]
• superposition T[aX1+bX2] = aT[X1]+bT[X2]
Linear system superposition
Examples:• matrix operations (additions, multiplication)• convolutions
![Page 4: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/4.jpg)
3/7/2003 Super-Resolution 4
Signals and linear operators
Continuous I(x)Discrete I[k] or Ik
Vector form I
Discrete linear operator y = A x
Continuous linear operator:convolution integral
g(x) = sh(,x) f() d, h(,x): impulse response
g(x) = s h(-x) f() d= [f * h](x) shift invariant
![Page 5: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/5.jpg)
3/7/2003 Super-Resolution 5
2-D signals and convolutions
Continuous I(x,y)Discrete I[k,l] or Ik,l
2-D convolutions (discrete)
g[k,l] = m,n f[m,n] h[k-m,l-n]
= m,n f[m,n] h1[k-m]h2[l-n] separable
Gaussian kernel is separable and radial
h(x,y) = (22)-1exp-(x2+y2)/2
![Page 6: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/6.jpg)
3/7/2003 Super-Resolution 6
Convolution and blurring
![Page 7: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/7.jpg)
3/7/2003 Super-Resolution 7
Separable binomial low-pass filter
![Page 8: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/8.jpg)
3/7/2003 Super-Resolution 8
Fourier transforms
Project onto a series of complex sinusoids
F[m,n] = k f[k,l] e-i 2(km+ln)
Properties:
• shifting g(x-x0) exp(-i 2fxx0)G(fx)
• differentiation dg(x)/dx i 2fxG(fx)
• convolution[f * g](x) [F G] (fx)
![Page 9: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/9.jpg)
3/7/2003 Super-Resolution 9
Blurring examples
Increasing amounts of blur + Fourier transform
![Page 10: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/10.jpg)
3/7/2003 Super-Resolution 10
Sharpening
Unsharp mask (darkroom photography):• remove some low-frequency content
y’ = y + s (y – g * y)
spatial (blur, sharp) freq (blur,sharp)
![Page 11: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/11.jpg)
3/7/2003 Super-Resolution 11
Sharpening - result
Unsharp mask: original, blur (σ=1), sharp(s=0, 1, 2)
![Page 12: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/12.jpg)
3/7/2003 Super-Resolution 12
Deconvolution
Filter by inverse of blur• easiest to do in the Fourier domain• problem: high-frequency noise amplification
![Page 13: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/13.jpg)
3/7/2003 Super-Resolution 13
Bayesian modeling
Use prior model for image and noise
• y = g * x + n, x is original, y is blurred• p(x|y) = p(y|x)p(x)
= exp(-|y – g*x|2/2σn-2) exp(-|x|2/2σx
-2)
• -log p(x|y) |y – g*x|2σn-2 + |x|2σx
-2
where the norm || is summed squares over all pixels
![Page 14: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/14.jpg)
3/7/2003 Super-Resolution 14
Parseval’s Theorem
Energy equivalence in spatial ↔ frequency domain
• |x|2 = |F(x)|2
• -log p(x|y) |Y(f) – G(f)X(f)|2σn-2 + |X(f)|2σx
-2
• least squares solution (∂/∂X = 0)X(f) = G(f)Y(f) / [G2(f) + σn
2/σx2]
![Page 15: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/15.jpg)
3/7/2003 Super-Resolution 15
Wiener filtering
Optimal linear filter given noise and signal statistics
• X(f) = G(f)Y(f) / [G2(f) + σn2/σx
2]
• low frequencies: X(f) ≈ G-1(f)Y(f)boost by inverse gain (blur)
• high frequencies:X(f) ≈ G(f) σn-2σx
2 Y(f)attenuate by blur (gain)
![Page 16: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/16.jpg)
3/7/2003 Super-Resolution 16
Wiener filtering – white noise prior
Assume all frequencies equally likely
• p(x) ~ N(0,σx2)
• X(f) = G(f)Y(f) / [G2(f) + σn2/σx
2]
• solution is too noisy in high frequencies
![Page 17: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/17.jpg)
3/7/2003 Super-Resolution 17
Wiener filtering – pink noise prior
Assume frequency falloff (“natural statistics”)
• p(X(f)) ~ N(0,|f|-βσx2)
• X(f) = G(f)Y(f) / [G2(f) + |f|βσn2/σx
2]
• greater attenuation at high frequencies
G(f) H(f)
![Page 18: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/18.jpg)
3/7/2003 Super-Resolution 18
Markov Random Field modeling
Use spatial neighborhood prior for image
• -log p(x) = ijCρ(xi-xj)where ρ(v) is a robust norm:• ρ(v) = v2: quadratic norm ↔ pink noise• ρ(v) = |v|: total variation (popular with maths)• ρ(v) = |v|β: natural statistics• ρ(v) = v2,|v|: Huber norm
[Schultz, R.R.; Stevenson, IEEE TIP, 1996]
i j
![Page 19: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/19.jpg)
3/7/2003 Super-Resolution 19
MRF estimation
Set up discrete energy (quadratic or non-)
• -log p(x|y) σn-2 |y – Gx|2 + ijCρ(xi-xj)
where G is sparse convolution matrix• quadratic: solve sparse linear system• non-quadratic: use sparse non-linear least
squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …)
![Page 20: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/20.jpg)
3/7/2003 Super-Resolution 20
Sampling a signal
• sampling:• creating a discrete signal from a continuous signal
• downsampling (decimation)• subsampling a discrete signal
• upsampling• introducing zeros between samples
• aliasing• two sampled signals that differ in their original
form (many → one mapping)
![Page 21: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/21.jpg)
3/7/2003 Super-Resolution 21
Sampling
interpolation
![Page 22: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/22.jpg)
3/7/2003 Super-Resolution 22
Nyquist sampling theorem
Signal to be (down-) sampled must have a bandwidth no larger than twice the sample frequency
s = 2 / ns > 2 0
![Page 23: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/23.jpg)
3/7/2003 Super-Resolution 23
Box filter (top hat)
![Page 24: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/24.jpg)
3/7/2003 Super-Resolution 24
Ideal low-pass filter
![Page 25: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/25.jpg)
3/7/2003 Super-Resolution 25
Simplified camera optics
1. Blur = pill-box*Bessel2 (diffr.) ≈ Gaussian
2. Integrate = box filter
3. Sample = produce single digital sample
4. Noise = additive white noise
![Page 26: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/26.jpg)
3/7/2003 Super-Resolution 26
Aliasing
Aliasing (“jaggies” and “crawl”) is present ifblur amount < sampling (σ = 1)
• shift each image in previous pipeline by 1
![Page 27: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/27.jpg)
3/7/2003 Super-Resolution 27
Aliasing - less
Less aliasing (“jaggies” and “crawl”) is present ifblur amount ~ sampling (σ = 2)
• shift each image in previous pipeline by 1
![Page 28: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/28.jpg)
3/7/2003 Super-Resolution 28
Multi-image super-resolution
Exploit aliasing to recover frequencies above Nyquist cutoff
kσn-2 |yk – Gkx|2 + ijCρ(xi-xj)
where Gk are sparse convolution matrices
• quadratic: solve sparse linear system• non-quadratic: use sparse non-linear least
squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …)
• projection onto convex sets (POCS)
![Page 29: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/29.jpg)
3/7/2003 Super-Resolution 29
Multi-image super-resolution
Need:• accurate (sub-pixel) motion estimates
(Wednesday’s lecture)• accurate models of blur (pre-filtering)• accurate photometry• no (or known) non-linear pre-processing
(Bayer mosaics)• sufficient images and low-noise relative to
amount of aliasing
![Page 30: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/30.jpg)
3/7/2003 Super-Resolution 30
Prior-based Super-Resolution
“Classical” non-Gaussian priors:
• robust or natural statistics• maximum entropy (least blurry)• constant colors (black & white images)
![Page 31: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/31.jpg)
3/7/2003 Super-Resolution 31
Example-based Super-Resolution
William T. Freeman, Thouis R. Jones, and Egon C. Pasztor,IEEE Computer Graphics and Applications, March/April, 2002
• learn the association between low-resolution patches and high-resolution patches
• use Markov Network Model (another name for Markov Random Field) to encourage adjacent patch coherence
![Page 32: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/32.jpg)
3/7/2003 Super-Resolution 32
Example-based Super-Resolution
William T. Freeman, Thouis R. Jones, and Egon C. Pasztor,IEEE Computer Graphics and Applications, March/April, 2002
![Page 33: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/33.jpg)
3/7/2003 Super-Resolution 33
References – “classic”Irani, M. and Peleg. Improving Resolution by Image Registration. Graphical Models and Image
Processing, 53(3), May 1991, 231-239.
Schultz, R.R.; Stevenson, R.L. Extraction of high-resolution frames from video sequences. IEEE Trans. Image Proc., 5(6), Jun 1996, 996-1011.
Elad, M.; Feuer, A.. Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Trans. Image Proc., 6(12) , Dec 1997, 1646-1658.
Elad, M.; Feuer, A.. Super-resolution reconstruction of image sequences. IEEE PAMI 21(9), Sep 1999, 817-834.
Capel, D.; Zisserman, A.. Super-resolution enhancement of text image sequences. CVPR 2000, I-600-605 vol. 1.
Chaudhuri, S. (editor). Super-Resolution Imaging. Kluwer Academic Publishers. 2001.
![Page 34: Super Resolution in Digital Image processing](https://reader034.fdocuments.us/reader034/viewer/2022042518/555caa6ad8b42aad358b4c82/html5/thumbnails/34.jpg)
08/03/2014 Super-Resolution 34
References – strong priorsFreeman, W.T.; Pasztor, E.C.. Learning low-level vision, CVPR 1999, 182-1189 vol.2
William T. Freeman, Thouis R. Jones, and Egon C. Pasztor, Example-based super-resolution, IEEE Computer Graphics and Applications, March/April, 2002
Baker, S.; Kanade, T. Hallucinating faces. Automatic Face Gesture Recognition, 2000, 83-88.
Ce Liu; Heung-Yeung Shum; Chang-Shui Zhang. A two-step approach to hallucinating faces: global parametric model and local nonparametric model. CVPR 2001. I-192-8.