ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas,...
-
date post
20-Dec-2015 -
Category
Documents
-
view
216 -
download
1
Transcript of ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas,...
![Page 1: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/1.jpg)
ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH
ACCURACY SUPER-RESOLUTION
Michalis Vrigkas, Christophoros Nikou, Lisimachos P. KondiUniversity of Ioannina
Department of Computer ScienceIoannina, Greece
![Page 2: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/2.jpg)
• Objective– Reconstruct a high-resolution image from a
sequence of low-resolution images.• Improve spatial resolution.
• Constraints on low-resolution images– Motion– Rotation– Blurring– Subsampling– Additive noise
MOTIVATION
![Page 3: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/3.jpg)
• MAP scheme for image super-resolution.
• Registration in two parts– At first, the LR images are registered by
establishing correspondences between robust SIFT (Scale-Invariant Feature Transform) features.
– In the second step, the estimation of the registration parameters is fine tuned along with the estimation of the HR image.• Mutual Information Criterion: maximize the mutual
information between HR image and each of the upscaled LR images.
APPROACH
![Page 4: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/4.jpg)
• Let the high-resolution image
where
• The set of LR images is described as
We consider p LR images each of size
FORMULATION MODEL
1 2 [ , , , ]TNz z zz 1 1 2 2N L N L N
1 2[ , , , ]T T T Tpy y yy
1 2M N N
![Page 5: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/5.jpg)
• Observation model:– All images are ordered lexicographically
– represents zero-mean additive Gaussian noise,
– is the degradation matrix, performing the operations of:• motion• blur• down-sampling
FORMULATION MODEL (cont.)
y = Wz +n
nW
( )k k kW = DB M s
![Page 6: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/6.jpg)
• The Gaussian prior for the HR image is:
– is the Laplacian of the image z– controls the precision and the shape of the
distribution
• The likelihood of the LR images is Gaussian:
MAP ESTIMATOR
/2
/2
( | |) 1( ) exp ( ) ( )
(2 ) 2
T NT
Np
Q Qz Qz Qz
Qz
22
1 ( ) ( )( | ) exp
2(2 )
T
pMpM
p
y Wz y Wzy z
![Page 7: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/7.jpg)
• MAP approach– Maximize – Which leads to a MAP functional to be minimized with
respect to HR image z and the transformation parameters s:
• Use gradient descent method– The update equation is given by:
where εn is the step size at the n-th iteration.
MAP ESTIMATOR (cont.)
( | ) ( | ) ( )p p pz y y z z
2 2 2
1
( , ) | | ( ) || || where =||p
k k kk
L
z s y W s z Qz
1( , ) | n
n nn L
z z z
z z z s
![Page 8: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/8.jpg)
• Objective: independently detect corresponding keypoints in scaled versions of the same image.
• Idea: Given a keypoint in two images, determine if the surrounding neighborhoods contain the same structure up to scale.
• SIFT features are invariant to:– Image scale and rotation– Affine transformations– Changes in illumination and noise
[D. G. Lowe. "Distinctive image features from scale invariant
keypoints.”International Journal of Computer Vision 60 (2), pp. 91-110, 2004.]
SCALE INVARIANT FEATUTE TRANSFORM - SIFT
![Page 9: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/9.jpg)
• Basics: the mutual information is maximized when the two images are correctly registered.
• The mutual information between two images A and B is:
– H(A) and H(B) are the marginal entropies of the random variables A and B.
– H(A,B) is the joint entropy.
MUTUAL INFORMATION CRITERION
( , ) ( ) ( ) ( , )
( , ) ( , ) log
( )· ( )AB
ABa b A B
I A B H A H B H A B
p a bp a b
p a p b
![Page 10: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/10.jpg)
• Normalized Mutual Information:– Robust measure in order to provide invariance to the
overlapping areas between the two images.
• Problem: – If mutual information is not initialized close to the
optimal solution it is trapped by local maxima.• Good initialization is important.
• Solution:– Initialization using SIFT descriptors.
MUTUAL INFORMATION CRITERION (cont.)
( ) ( )( , )
( , )
H A H BNMI A B
H A B
![Page 11: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/11.jpg)
• Estimation of registration parameters in two steps.– First step, LR images are registered by
employing SIFT features.• Minimization of mean square error between the
locations of features between the reference image and the LR images.
• Provides good initialization.
IMAGE REGISTRATION
![Page 12: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/12.jpg)
– Second step, the estimation of the registration parameters is fine-tuned along with the estimation of the HR image, by maximization of mutual information criterion.• Iterative scheme.
• Contribution:– The registration accuracy is improved at each
iteration step.– Refinement of the mutual information
registration.
IMAGE REGISTRATION (cont.)
![Page 13: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/13.jpg)
• Synthetic data sets.
• LR images were created by rotating, translating, blurring, down-sampling and degrading by noise.– Translation: uniformly selected in [-3, 3] (in pixels)– Rotation: uniformly selected in [-5, 5] (in degrees)– Down-sampling factor: 2 (4 pixels to 1)– Blurring: 5x5 Gaussian kernel, standard deviation
of 1– Additive noise: AWGN to obtain SNR of 30 dB and
20 dB
EXPERIMENTAL PARAMETERS
![Page 14: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/14.jpg)
• First estimate of the HR image– Bicubic interpolation
• Total number of realizations for each case: 10
• Convergence: or 70 iterations reached.
• Quantitative evaluation: peak signal to noise ratio
EXPERIMENTAL PARAMETERS (cont.)
15/ 10
n n n z z z‖ ‖ ‖ ‖
2
10 2
255PSNR 10log
ˆ|| ||z z
![Page 15: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/15.jpg)
COMPARE METHODS
![Page 16: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/16.jpg)
• Books (PSNR = 26.06 dB)
• 4 LR images used
EXPERIMETAL RESULTS
LR image
Reconstructed HR image
![Page 17: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/17.jpg)
• Front page (PSNR = 26.14 dB)
• 6 LR images used
EXPERIMETAL RESULTS (cont.)
LR image
Reconstructed HR image
![Page 18: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/18.jpg)
• Car (PSNR = 28.13 dB)
• 5 LR images used
EXPERIMETAL RESULTS (cont.)
LR image
Reconstructed HR image
![Page 19: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/19.jpg)
• Eye chart (PSNR = 27.33 dB)
• 4 LR images used
EXPERIMETAL RESULTS (cont.)
LR image
Reconstructed HR image
![Page 20: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/20.jpg)
EXPERIMETAL RESULTS (cont.)
• Statistics for the compared SR methods
+1.5 dB on average better results than SIFT.
![Page 21: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/21.jpg)
• Hybrid registration approach– SIFT-based image registration combined with
the maximization of mutual information.– High precision registration
• High accuracy super-resolved image.– Improvement is 1.5 dB on average higher for
both 30 dB and 20 dB.
• Proposed algorithm converges faster than the standard solution.
CONCLUTIONS
![Page 22: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/22.jpg)
QUESTIONS?
?
![Page 23: ON THE IMPROVEMENT OF IMAGE REGISTRATION FOR HIGH ACCURACY SUPER-RESOLUTION Michalis Vrigkas, Christophoros Nikou, Lisimachos P. Kondi University of Ioannina.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649d4b5503460f94a29411/html5/thumbnails/23.jpg)
THANK YOU ALL FOR YOUR PARTICIPATION AND
PATIENCE!