Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to...
-
Upload
nguyenliem -
Category
Documents
-
view
225 -
download
2
Transcript of Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to...
![Page 1: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/1.jpg)
Introduction to Image Interpolation
and Super Resolution
Digital Image and Signal Processing Lab
National Taiwan University
Hsin-Hui Chen
![Page 2: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/2.jpg)
Outline
• Introduction
• Image interpolation
– Example1: New-edge directed interpolation (NEDI)
– Example2: Soft-decision adaptive interpolation (SAI)
– Example3: Adaptive Wiener Filter (AWF)
• Super resolution
– Example1: Iterative back-projection (IBP)
– Example2: Bilateral back-projection (BFIBP)
• Conclusions
• References
![Page 3: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/3.jpg)
Introduction
• Applications
• Why image interpolation and super resolution matters?
• The difference between image interpolation and super resolution
• Classification of the image interpolation and super resolution
![Page 4: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/4.jpg)
Applications
• HDTV
• Image/Video Coding
• Image/Video Resizing
• Image Manipulation
• Face Recognition
• View Synthesis
• Surveillance
![Page 5: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/5.jpg)
Why image interpolation and super
resolution matters?
• Storage limitation
• Limited computational power
• Cost of camera
• Insufficient bandwidth (Limited network bandwidth)
![Page 6: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/6.jpg)
The difference between interpolation
and super resolution
• Interpolation only involves upsampling the low-resolution
image, which is often assumed to be aliased due to direct
down-sampling.
• Super resolution aims to address undesirable effects,
including the resolution degradation, blur and noise effects.
Super resolution usually involves three major processes which
are upsampling (interpolation), deblurring, and denoising.
![Page 7: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/7.jpg)
Classification of Image Interpolation
![Page 8: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/8.jpg)
Classification of Super Resolution
![Page 9: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/9.jpg)
Basic Premise for Super Resolution
• How can we obtain an HR image from multiple LR images?
• Basic premise: The availability of multiple LR images
captured from the same scene.
![Page 10: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/10.jpg)
Basic Premise for Super Resolution
• Scene motions can occur due to the controlled or uncontrolled
motions in imaging systems.
• If these scene motions are known or can be estimated within
subpixel accuracy and if combine these LR images, SR image
reconstruction is possible.
![Page 11: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/11.jpg)
Common Image Acquisition System
![Page 12: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/12.jpg)
Observation Model
![Page 13: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/13.jpg)
New Edge-Directed Interpolation
• Geometric duality
![Page 14: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/14.jpg)
New Edge-Directed Interpolation
• Training window
![Page 15: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/15.jpg)
Lossless image coding system
• MED (JPEG-LS)
• GAP (CALIC)
• EDP
• Hybrid Predictor
Arithmetic coding
• Spatial domain
• Prediction error domain
• …
p(e|c1)
p(e|c2)
p(e|c3)
p(e|c4)
…
p(e) Arithmetic coding Arithmetic coding
Arithmetic coding
…
![Page 16: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/16.jpg)
New Edge-Directed Interpolation
• Consider the N nearest neighbors, which are the supports of the
predictor, the value of the current pixel X(n) can be predicted by
where ak is the prediction coefficient of the neighbor X(n-k).
1
ˆ ( ) ( )N
k
k
X n a X n k
![Page 17: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/17.jpg)
New Edge-Directed Interpolation
• To determine the coefficients ak , LS optimization is used for
minimizing
where
2
2y Ca
T
1 2[ , ,..., ]Na a a a
![Page 18: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/18.jpg)
New Edge-Directed Interpolation
• The optimal coefficient vector can be solved from
T 1 T( ) ( )a y C C C
![Page 19: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/19.jpg)
Soft-decision adaptive interpolation
• Sample relations in estimating model
![Page 20: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/20.jpg)
Soft-decision adaptive interpolation
• Existing interpolation methods estimate each missing pixel
independently from others, which is called “hard-decision”
• A new strategy of “soft-decision” estimation is adopted
![Page 21: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/21.jpg)
Soft-decision adaptive interpolation
• Existing interpolation methods estimate each missing pixel
independently from others, which is called “hard-decision”
• A new strategy of “soft-decision” estimation is adopted
![Page 22: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/22.jpg)
Soft-decision adaptive interpolation
• Include horizontal and vertical correlations
![Page 23: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/23.jpg)
Soft-decision adaptive interpolation
![Page 24: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/24.jpg)
Original image
NEDI
Bicubic
SAI
![Page 25: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/25.jpg)
Original image
NEDI
Bicubic
SAI
![Page 26: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/26.jpg)
Original image
NEDI
Bicubic
SAI
![Page 27: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/27.jpg)
Adaptive Wiener Filter
• Observation model relating a desired 2-D continuous scene,
d(x,y), with a set of corresponding LR frames
![Page 28: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/28.jpg)
Adaptive Wiener Filter
• Alternative observation model
![Page 29: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/29.jpg)
Adaptive Wiener Filter
• Overview of the proposed SR algorithm
![Page 30: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/30.jpg)
Adaptive Wiener Filter
![Page 31: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/31.jpg)
Adaptive Wiener Filter
![Page 32: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/32.jpg)
Iterative Back-Projection
• The formulation of an LR image from the unknown HR image
can be formulated as follows:
where D is the down-sampling matrix, and G is the point
spread function (PSF) which is generally a smoothing kernel.
![Page 33: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/33.jpg)
Iterative Back-Projection
• The underlying criterion is that the reconstructed HR image
should produce the same LR image if passing it through the
same image formation process.
![Page 34: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/34.jpg)
Iterative Back-Projection
• The reconstruction error is defined as
![Page 35: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/35.jpg)
Iterative Back-Projection
• Given an LR image, the updating procedure can be
summarized as follows
![Page 36: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/36.jpg)
Bilateral Back-Projection
![Page 37: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/37.jpg)
Bilateral Back-Projection
• Bilateral filtering
![Page 38: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/38.jpg)
Bilateral Back-Projection
• Bilateral filtering
![Page 39: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/39.jpg)
Bilateral Back-Projection
![Page 40: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/40.jpg)
Bilateral Back-Projection
![Page 41: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/41.jpg)
Bilateral Back-Projection
Bicubic IBP BFIBP
![Page 42: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/42.jpg)
Bilateral Back-Projection
Bicubic IBP BFIBP
![Page 43: Introduction to Image Interpolation and Super Resolutiondisp.ee.ntu.edu.tw/class/Introduction to Image...–Example3: Adaptive Wiener Filter (AWF) • Super resolution –Example1:](https://reader031.fdocuments.us/reader031/viewer/2022022504/5ab80f617f8b9ac1058c3c00/html5/thumbnails/43.jpg)
Thank You
Q & A
2010