Image Fusion Using Principal Component Analysis Method

Vinit Anil Patil Vijaykumar Vilas Kulkarni13MCE0088, Communication Engineering, 13MCE0062, Communication EngineeringSENSE, VIT University, Vellore, India SENSE, VIT University, Vellore, Indiavinit.anilmalini2013@vit.ac.in kulkarni.vijaykumar2013@vit.ac.in

Abstract— Information fusion can be achieved at any level of the image information representation- Signal (pixel), feature (object) and decision (symbol). In this project, we are concentrating on the Pixel-level image fusion which defines the process of fusing visual information from a number of registered images into a single fused image. There are many image fusion methods such as intensity-hue-saturation, Brovey transform, principal component analysis (PCA), high-pass filtering, high-pass modulation, wavelet transform etc. In this paper, Principal Component Analysis method is used for pixel-level Image Fusion which basically transforms a number of Correlated variables into a number of uncorrelated variables. Performance metrics are used to evaluate the performance of PCA method with simple average based and wavelet transform based image fusion algorithm.

Index Terms—Multi-sensor Image Fusion (MIF),Principal Component Analysis (PCA)

I. INTRODUCTION

Multi-sensor image fusion (MIF) is a technique to combine the registered images to increase the spatial resolution of acquired low detail multi-sensor images and preserving their spectral information. The benefiting fields from MIF are: Military, remote sensing, machine vision, robotic and medical imaging, etc. Main objectives of Image Fusion using Principal Component Analysis are:

(a) The fusion process should preserve all relevant information contained in the source images, (b) The fusion process should not introduce any inconsistencies, and (c) Irrelevant features and noise should be suppressed to a maximum extent.

The problem that MIF tries to solve is to merge the information content from several images (or acquired from different imaging sensors) taken from the same scene in order to accomplish a fused image that contains the finest information coming from the original images. Hence, the fused image would provide enhanced superiority image than any of the original source images.

II. LITERATURE REVIEW

A. Image Fusion

Pixel-level image fusion defines the process of fusing visual information from a number of registered images into a single fused image. It is part of the much broader subject of multi sensor information fusion which has attracted a considerable amount of research attention in the last two decades.

Multi-sensor information fusion utilizes information obtained from a number of different sensors surveying an environment. The aim is to achieve better situation assessment and/or more rapid and accurate completion of a pre-defined task than would be possible using any of the sensors individually. In the literature, it has been defined as the synergistic combination of different sources of sensory information into a single representational format The only formal definition of information fusion (data fusion) to date, is that given by the U.S. Department of Defence, Joint Directors of Laboratories Data Fusion Subpanel which represents the first formal body explicitly dealing with the process of data fusion. Their definition can be found in as: a multilevel, multifaceted process dealing with the automatic detection, association, correlation, estimation and combination of data and information from multiple sources.

Image fusion represents a specific case of multi sensor information fusion in which all the information sources used represent imaging sensors. Information fusion can be achieved at any level of the image information representation. Analogous to other forms of information fusion, image fusion is usually performed at one of the three different processing levels: signal, feature and decision. Signal level image fusion, also known as pixel-level image fusion, represents fusion at the lowest level, where a number of raw input image signals are combined to produce a single fused image signal. Object level image fusion, also called feature level image fusion, fuses feature and object labels and property descriptor information that have already been extracted from individual input images. Finally, the highest level, decision or symbol level image fusion represents fusion of probabilistic decision information obtained by local decision makers operating on the results of feature level processing on image data produced from individual sensors.

B.Principal Component Analysis

B.1.Theory:The PCA involves a mathematical procedure that

transforms a number of correlated variables into a number of uncorrelated variables called principal components. It computes a compact and optimal description of the data set. The first principal component accounts for as much of the variance in the data as possible and each succeeding component accounts for as much of the remaining variance as possible. First principal component is taken to be along the direction with the maximum variance. The second principal component is constrained to lie in the subspace perpendicular of the first. Within this subspace, this component points the direction of maximum variance. The third principal component is taken in the maximum variance direction in the subspace perpendicular to the first two and so on. The PCA is also called as Karhunen-Loeve transform or the Hotelling transform .The PCA does not have a fixed set of basis vectors like FFT, DCT and wavelet etc. and its basis vectors depend on the data set. Let X be a d-dimensional random vector and assume it to have zero empirical mean. Orthonormal projection matrix V would be such that Y =VTX with the following constraints. The covariance of Y, i.e., cov(Y) is a diagonal and inverse of V is equivalent to its transpose ( V-1=VT ). Using matrix algebra:

Cov (Y) = E {YYT} = E {(VTX) (VTX)T} = E {(VTX) (XTV)}……………….(1) = VT E {XXT} V = VTcov(X) V Multiplying both sides of Equation (1) by V, one gets,

V cov(Y) =VVT cov(X ) V = cov(X )V…………………. (2)

One could write V as V= [V1,V2,………..Vd] and

cov(y)

1

1

0 00 0

0 00 0

d

d

Substituting Equation (1) into the Equation (2) gives

[λ1 V1 , λ2V2 ……. λd Vd ] =[cov(X)V1 , cov(X)V2, ……. cov(X)Vd ]

This could be rewritten asλVi =cov(X) Vi

Where i =1 ,2,...,d and Vi is an eigenvector of cov(X ).

B.2.PCA Algorithm:

Let the source images (images to be fused) be arranged in two-column vectors. The steps followed to project this data into a 2-D subspaces are:

1. Organize the data into column vectors. The resulting matrix Z is of dimension 2 x n.

2. Compute the empirical mean along each column .The empirical mean vector Me has a dimension of 1 x 2.

3. Subtract the empirical mean vector Me from each column ofthe data matrix Z. The resulting matrix X is of dimension 2 x n.

4. Find the covariance matrix C of X i.e. C=XX T mean of expectation = cov(X)

5. Compute the eigenvectors V and eigenvalue D of C and sort them by decreasing eigenvalue. Both V and D are of dimension 2 x 2.

6. Consider the first column of V which corresponds to larger eigenvalue to compute P1 and P2 as: P1 = V(1) and P2 = V(2) ∑V ∑V

B.3.Image Fusion Using PCA

The information flow diagram of PCA-based image fusion algorithm is shown in Fig. 1. The input images (images to be fused) I1 (x, y) and I2 (x, y) are arranged in two column vectors and their empirical means are subtracted. The resulting vector has a dimension of n x 2, where n is length of the each image vector.

Compute the eigenvalues and eigenvectors for this resulting vector are computed and the eigenvectors corresponding to the larger eigenvalue are obtained. The normalized components P1 and P2 (i.e., P1 + P2 = 1) are computed from the obtained eigenvector. The fused image is:

If(x,y) = P1I1(x,y)+ P2I2(x,y).

Fig.1 Block Diagram of Image Fusion using PCA

III.PERFORMANCE ANALYSIS

Metrices for performance evaluation:

1.Standard deviation-

a) Features-It is known that standard deviation is composed of the signal and noise parts. This metric would be more efficient in the absence of noise. It measures the contrast in the fused image. An image with high contrast would have a high standard deviation.

b) Formula-

20

(i)f

L

Ii

i i h

,0

f

L

Ii

i ih

where (i)

fIh is the normalized histogram of the Fused image.

(x, y)fI and L number of frequency bins in the histogram.

2.Entropy-

a) Features- Entropy is used to measure the information content of an image. Entropy is sensitive to noise and other unwanted rapid fluctuations. An image with high information content would have high entropy.

b) Formula- Using the entropy, the information content of a fused image is:

20

(i)log (i)f f

L

I Ii

He h h

3. Cross entropy-

a)Features-Cross-entropy evaluate the similarity in information content between input images and fused image. Fused and reference images containing the same information would have a low cross entropy.

b)Formula is given by-

1

1

2

2

1 21 2

10

20

(I ; I ) CE(I ; I )(I , I ; I )

2

(i)(I ; I ) (i) log

(i)

(i)(I ; I ) (i) log

(i)

f

f

f ff

LI

IfIi

LI

IfIi

CECE

hCE h

h

hCE h

h

4. Spatial frequency-

a) Features- This frequency in spatial domain indicates the overall activity level in the fused image.

b) Formula-

Spatial Frequency criterion SF is: 2 2SF RF CF Where Row Frequency of the image:

2

1 1

1 (x, y) I (x, y 1)M N

f fx y

RF IMN

Column Frequency of the image:

2

1 1

1 (x, y) I (x 1, y)N M

f fy x

CF IMN

5. Fusion mutual information-

a) Features-It measures the degree of dependence of the two images. A larger measure implies better quality.

b) Formula-

If the joint histogram between 1(x, y)I and (x, y)fI is defined

as 1

(i, j)fI Ih and 2 (x, y)I and (x, y)fI as

2(i, j)

fI Ih . Then

the mutual information between source and fused images are:

1 1f fI I I IFMI MI MI where

1

1 11

2

2 22

21 1

21 1

(i, j)(i, j) log

(i, j) (i, j)

(i, j)(i, j) log

(i, j) (i, j)

f

f ff

f

f ff

M N I II I I I

I Ii j

M N I II I I I

I Ii j

hM I h

h h

hM I h

h h

6. Fusion quality index-

a) Features-The range of this metric is 0 to 1. One indicates the fused image contains all the information from the source images.

) Formula-

1 2(w) (w) QI(I , I | w) 1 (w) (I , I | w)f fw W

FQI c QI

where 1

1 2

2

2 2(w)

I

I I

computed over a window;

1 2

2 2(w) MAX( , )I IC over a window (w)c is a

normalized version of (w)C and 1QI(I ,I | w)f is the quality

index over a window for a given source image and fused image.

7. Fusion similarity metric-

a) Feature-It takes into account the similarity between the source and fused image block within the same spatial position. The range of this metric is zero to one. The value one indicates that the fused image contains all the information from the source images.

b)Formula-

11 2 2

2

QI(I , I | w)(I , I , I | w) (I , I | w)

(I , I | w)f

f ffw W

FSM sim QIQI

where

1 2(I ,I ,I | w) 0fsim if 1

1 2

0f

f f

I I

I I I I

= 1

1 2

f

f f

I I

I I I I

if

1

1 2

0 1f

f f

I I

I I I I

= 1 if 1

1 2

1f

f f

I I

I I I I

IV.RESULT

The aircraft, shown in Fig.2 is considered as a reference image to evaluate the performance of the fusion algorithm. The complementary pair input images and are taken to evaluate the fusion algorithm and these images are shown in Fig. 3 and 4. The images are complementary in the sense thatthe blurring occurs at the left-half and the right halfrespectively. The error (difference) image (Fig.6) is computed by taking the corresponding pixel differenceof reference mage and fused image ,i.e., IE(x, y) = IR (x, y) – IF (x, y)The fused and error images by PCA algorithm areshown in Fig.5 & 6.

Figure 2. Reference image (x, y)rI

Figure 3. Source image1 1(x, y)I

Figure 4. Source image2 2 (x, y)I

Figure 5. Fused image IF (x, y)

Figure 6. Error image IE(x, y)

V.OBSERVATION & CONCLUSION

Pixel-level image fusion using principal component analysis is implemented in PC MATLAB. Different image fusion performance metrics have been evaluated. The simple averaging fusion algorithm shows degraded performance. Image fusion using the PCA shows better performance.

Algorithm used He SD SFPCA 6.2897 45.4530 7.6082DWT 6.2925 45.6415 11.0283

VI. FUTURE SCOPE

Future of image fusion will be to develop such techniques that will automatically combine images of a scene captured under different illumination. Beyond providing digital tools for artists for creating surrealist images and videos, the methods can also be used for practical applications. For example, the non-realistic appearance can be used to enhance the context of night time traffic videos so that they are easier to understand. The context is automatically captured from a fixed camera and inserted from a day-time image (of the same scene). Our approach is based on a gradient domain technique that preserves important local perceptual cues while avoiding traditional problems such as aliasing, ghosting and haloing.

VII. REFERENCES

[1] Zhijun Wang, Djemel Ziou, Costas Armenakis, Deren Li, and Qingquan Li,”A Comparative Analysis of Image Fusion Methods” IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 43, NO. 6, JUNE 2005.

[2] Digital Image Processing by Gonzalez & Woods.

[3] Digital Image Processing using MATLAB by Gonzalez & Woods.