IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 51, NO. 7, JULY 2003 1941

Hypercomplex Correlation Techniquesfor Vector Images

C. Eddie Moxey, Member, IEEE, Stephen J. Sangwine, Senior Member, IEEE, and Todd A. Ell, Member, IEEE

Abstract—Correlation techniques have been applied to almostevery area of signal processing over the past century, yet their usehas, in general, been limited to scalar signals. While there havebeen implementations in multichannel applications, these can becharacterized as a combination of single channel processes. Truevector correlation techniques, with global and interchannel mea-sures, have only recently been demonstrated and are still in theirinfancy by comparison.

This paper describes our work on vector correlation based onthe use of hypercomplex Fourier transforms and presents, for thefirst time, a unified theory behind the information contained in thepeak of a vector correlation response. By using example applica-tions for color images, we also demonstrate some of the practicalimplications, together with our latest results.

Index Terms—Color image processing, correlation techniques,hypercomplex correlation, quaternions, vector signal processing.

I. INTRODUCTION

FOR MANY engineers and scientists, an initial impressionmight be that the processing of vector signals must have

been largely explored by now. However, this is far from beingthe case. Until recently, there were very few examples oftrue vector signal processing. This apparent contradictionowes much to the numerous applications of “so-called” vectorprocessing in multichannel systems. In general, the techniquesemployed in multichannel applications tend to ignore thetrue nature of vector signals and amount to no more than thecombination of single channel processes. The fact that thecomputation may have been performed in parallel should notjustify the use of the term “vector,” especially when applied tothe data being processed.

Another distraction is the concept of an-dimensionalproblem space. Examples occur in pattern recognition prob-lems where the pattern is given a signature comprising anumber of features or parameters. Identifying such patterns ina large database of examples is often considered to be a cor-relation-based problem and solved using principal component

Manuscript received June 25, 2002; revised February 3, 2003. This workwas supported in part by the UK Engineering and Physical Sciences ResearchCouncil under Grant GR/M 45764. The associate editor coordinating the reviewof this paper and approving it for publication was Dr. Masaaki Ikehara.

C. E. Moxey was with the Multimedia Architectures and Applications Group,Department of Electronic Systems Engineering, University of Essex, ColchesterCO4 3SQ, U.K. He is now with University of Surrey, Surrey, U.K. (e-mail:eddie.moxey@ieee.org).

S. J. Sangwine is with the Multimedia Architectures and Applications Group,Department of Electronic Systems Engineering, University of Essex, ColchesterCO4 3SQ, U.K. (e-mail: s.sangwine@ieee.org).

T. A. Ell is with Goodrich Sensor Systems, Minneapolis, MN USA (e-mail:t.ell@ieee.org).

Digital Object Identifier 10.1109/TSP.2003.812734

analysis (PCA) or similar techniques. While these analyze thesubspace to find the basis vectors, or principal components, inthe -dimensional data, usually, to reduce the dimensions ofthe problem space to simplify the matching process, the natureof the “vector” data is induced by the number of examples ofthe patterns and not as a direct consequence of the individualpattern or signal. In essence, these techniques construct direc-tions, or vectors, in data, and they do not actually process truevector signals.

In order to help clarify the concept of the dimensionality ofsignals and their samples, Table I offers examples for particularcases and demonstrates where the techniques presented in thispaper apply. The number of components per sample is given inthe first column, and the dimension of the signal is given in thefirst row. The boxes within the table give examples of signalsin each category. The particular case of interest in this paperis that of a color image that has three components per sample(e.g., red, green, and blue amplitudes). Each sample value istherefore avector, and since the signal is an image, the samplesare also known aspixels. We therefore refer to the samples asvector pixels. Apart from the application presented in this paper,another example of vector signal processing has recently beenapplied to the analysis of three-dimensional (3-D) seismic datacollected from vector geophones [1], [2]. In this work, the sam-ples are also vector-valued and are collected over two dimen-sions: one of space and one of time. The signals are thereforevector images and mathematically are similar to color images.

An alternative view of signal dimensionality is that ofSommeret al.[3], [4]. In their work, the number of componentsper sample does not define the dimensionality of the signal.Rather, the intrinsic symmetries of the signal determine itsdimensionality. Their work has been exclusively with grayscaleimages (that is signals with scalar samples) and is thereforesomewhat orthogonal to the work presented in this paper. It istoo early to say whether the two aspects of dimensionality canbe combined into a more general theoretical framework, butthis is undoubtedly an interesting issue for future research.

What is a vector signal? Contrary to the interpretations givenin the first two paragraphs, a vector signal is a signal that de-pends on all its components in order to exist. While each ofthese components may be considered in isolation, either mea-sured by individual sensors or decomposed by processing, theoriginal signal cannot be deduced from combinations of someof the components. True vector signals (in the sense used inthis paper) have three components, and although this may beregarded as a special case of more generalized signals with anarbitrary number of components, it is an important special casebecause mathematically, vectors with three components have a

1053-587X/03$17.00 © 2003 IEEE

1942 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 51, NO. 7, JULY 2003

TABLE ICLARIFICATION OF SIGNAL AND /OR SAMPLE DIMENSIONALITY

special geometrical significance in three dimensions. It happensthat human color vision is trichromatic and that, therefore, thereis a natural match between vector mathematics and the mathe-matics required to manipulate color images. There are numerousexamples of the requirement for vector signal processing. Un-fortunately, the vast majority are implemented using a combi-nation of single channel techniques.

The motivation for the work presented in this paper is theanalysis and manipulation of color images. Digital imageprocessing has made a considerable impact over the last fewdecades, and applications can be found in the home as wellas in the industrial environment. Most of the image data thatwe encounter is now in color, and monochrome, or gray-scale,images are the exception. The processing of color imageshas therefore become increasingly important. However, whilelow-level image processing techniques for monochromeimages have matured, the same is not true for color images.Historically, the reluctance to process color images was dueto the lack of computing power and storage. However, theserestrictions are no longer significant.

In general, the techniques derived for processing colorimages are a haphazard collection that are more applicationspecific than based on unified theory. Although there area number of different formats for color image data, basedon internationally agreed standards, each can be derivedfrom another by known conversions, and all of them can berepresented by vectors within their respective color spaces.Often, gray-scale techniques are applied without modificationto each of the three color components, and the overall resultis derived from the sum of the three individual responses.Such acomponent-wiseapproach to image processing mayproduce the desired results but rarely appreciates or addressesthe vectorial nature of a color image. In some applications,where changing lighting conditions, which affect color much

more than luminance perception, prevail, the color informationis completely ignored, and a gray-scale image is derivedfor processing. However, an accurate measure of such colorchanges could be useful in many applications of image analysisand understanding. It should be noted that color is aperceivedphenomenonand not a physical property of light [5]. Formore information on color perception, standards, formats, andprocessing, see [6].

The basic use of correlation is to determine how similar,or dissimilar, one signal is from another. This knowledge ofsimilarity has numerous uses in signal and image processing.In gray-scale images, the similarity can be considered to bestructural or spatial, and this can be over whole images orjust small regions within them. Example applications are theregistration of two images of the same scene or the correspon-dence matching of the same object, or feature, within a scenecaptured from differing view points. Other examples of the useof correlation can be found in pattern recognition problemswhere the identification of particular objects in an image isperformed using a reference image of the target object. The useof such techniques is so widespread that they can be found inalmost every image processing and computer vision text.

Since the correlation of signals with scalar components yieldsa measure of structural, or one-dimensional (1-D), similarity inboth 1-D and two-dimensional (2-D) signals, the correlation of

-dimensional samples, or vector data, in-dimensional sig-nals should yield similarity measures across-dimensionalspace. For color images, this suggests an ability to measure colorchanges between images as well as the structural similarity. Ig-noring aspects of gamma and luminance correction, vector cor-rection of color images offers solutions for applications becauseof, or in spite of, color-space distortions. Examples of such ap-plications are given herein, but these are not intended to be thefocus of this paper.

MOXEY et al.: HYPERCOMPLEX CORRELATION TECHNIQUES 1943

This paper consolidates our recent work, some of which hasbeen published in conference proceedings [7]–[9], on hyper-complex correlation of vector, or color, images and presents,for the first time, a technique for the correction of global colorspace distortions. Hypercomplex algebras provide an elegantmathematical way to deal with vector signals in the same waythat complex algebra provides a natural way to handle signalswith two orthogonal components (or equivalently,amplitudeandphase). There are many hypercomplex algebras, but one inparticular stands out as suitable for the applications presentedin this paper. Thequaternion algebrawas the first hypercom-plex number system to be discovered, and of all the hypercom-plex algebras, it is the closest in its mathematical properties tothe familiar systems of the real and complex numbers. Indeed,the only way in which the quaternions depart from the familiarproperties of the real and complex numbers is in lacking com-mutative multiplication, and this is a significant enough advan-tage for us to avoid the use of other hypercomplex algebras forthe time being. The mathematical literature on hypercomplexnumbers is not extensive, but for those who wish to understandmore, we recommend the book by Kantor and Solodnikov [10].

The paper begins, in Section II, with a review of our use ofquaternions, or hypercomplex numbers, to represent and ma-nipulate color images. Sections III–VI demonstrate the variousvector correlation techniques we have applied to color imagesand present the relevant theory together with example applica-tions.

II. HYPERCOMPLEXNUMBERS AND COLOR IMAGES

Throughout this paper, we represent the pixels of a colorimage using quaternions. We also use quaternions to representthe coefficients of the Fourier transform of a color image(the spectrum). Quaternions (which are also referred to ashypercomplex numbers) are an extension of complex numbersto four dimensions. Originally proposed by Hamilton in 1843[11], quaternions have been used to encode and computetransformation and particularly rotation1 in 3-D space formany years. They can be considered as a complex numberwith a vector imaginary part consisting of three mutuallyorthogonal components. In Cartesian form, a quaternion isusually represented as , where , ,, and are all real, and, , and are orthogonal imaginary

operators. Apurequaternion has a zero real part ( ), anda full quaternion has a nonzero real part. It is often useful toconsider a quaternion as composed of ascalar and avectorpart, which is represented as

(1)

where and . At this point, readersunfamiliar with quaternions should see Appendixes A–C, whichreview all the main concepts used throughout this paper.

A red, green, and blue (RGB) color image may be representedusing quaternions by encoding the red, green, and blue channels

1Although quaternions implicitly encode rotation, they should not be consid-ered as simple rotation operators.

of the image as a pure quaternion such that the image functionis given by

(2)

where is the red component at the coordinatein the image and similarly for the green and blue. This represen-tation effectively maps the RGB color space to the right-handcoordinate frame imposed by the imaginary part of quaternion-space. However, this is not an exclusive representation. An alter-native could be used to define the origin of the coordinate frameto be coincident with the center of a unit RGB color cube, or anyother position, in any other color-space with appropriate scaling.

A. Rotation Operator

Quaternions provide a convenient representation for rotationsin 3-D space. A rotation through an angleabout an axis ,where is a unit pure quaternion, is represented by the quater-nion operator , where , is the quaternionconjugate, and the square brackets indicate space for the purequaternion that is to be operated on. The use of the rotation op-erator was demonstrated in [12], where a number of quaternionfilters, based on traditional gray-scale filters, were derived usingdouble-sided convolution masks.

B. Decomposition and Reordering

Many of the functions described in this paper require the useof quaternionic decomposition and quaternion and/or vector re-ordering.

Given two pure quaternionsand , may beresolvedintocomponentsparallel andperpendicularto using

(3)

(4)

It can also be shown that afull quaternion can be resolvedabout a vector as

and (5)

In either case, the quaternion can be recovered by adding thetwo components back together. The proof for these equations isgiven in Appendix C.

In general, quaternion multiplication is noncommutative.However, parallel quaternions (strictly co-planar quaternionsor quaternions with parallel vector parts) commute. Ifis afull quaternion and is a pure quaternion, or a vector, where

, they can be reordered such that . Again,proof is given in Appendix C.

The resolution of a quaternion into components parallel andperpendicular to a chosen vector is essential to the derivation ofthe operational formulae used for computing correlation, as isthe reordering of a quaternion perpendicular to a vector. Suchalgebraic techniques have been largely neglected in the mathe-matical literature, yet they are vital to the successful applicationof the quaternion algebra to a practical problem like correlation.

1944 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 51, NO. 7, JULY 2003

III. V ECTOR CORRELATION AND THE HYPERCOMPLEX

FOURIER TRANSFORM

The cross-correlation of two images andwas originally extended to hypercomplex images using basicquaternion arithmetic

(6)where the shift operation on was implemented cycli-cally using modulo arithmetic [13]. is the correlationfunction of the images, and is the resulting correla-tion surface as an image. Equation (6) effectively treatsthe images as one tile sliding over an infinite plane of tiles.This is a useful analogy, which will be used later. If the images

and are the same image the autocorrelation ofthe image is computed. If the mean, orDClevel, of each image issubtracted first, the cross-covariance is obtained. In the case ofvector image data, the mean is more accurately defined by theterm centroid. However, the use of the latter does not alwaysconvey the concept of the “mean-level” in a 1-D, or 2-D, signalprocessing sense.

Direct evaluation of the cross-correlation function is imprac-tical for all but the smallest images due to the high computa-tional cost ( for a image). This necessitates theuse of fast Fourier transforms, the hypercomplex form of whichwas first published in [14]. There is no space in this paper togive a detailed exposition of the quaternion Fourier transform;therefore, see [15] for further details and references. It shouldbe noted here however, that many formulations of quaternionicFourier transforms are possible. The ones used in this paperhave proved to be the best so far discovered for yielding rela-tively straightforward operational formulae for the computationof correlation. Other work on quaternionic Fourier transformshas been published by Peiet al. [16].

The complete set of transform pairs used in this paper is re-produced here for convenience:

(7)

(8)

where , , and is an arbitrary unit vectorthat defines the axis of the transformation. For processing themajority of natural RGB images, the choice is

, which is aligned with the luminance, orgrayline, axis ofthe unit RGB color cube. However, it should be noted that this isnot exclusive. Due to the noncommutative nature of quaternionmultiplication, the ordering of the modulating term is significantand introduces the concept ofhandedtransforms. Reference tothe opposite hand is termed itstransposetransform, and each

forward and reverse pair is denoted by, , , and .Decomposing the transform pairs yields

(9)

where denotes the decomposition function about theaxis , as defined by (3) and (4).

The standard Wiener–Khintchine theorem relates the auto-correlation and the power spectral density of a scalar imageby , where is the auto-correlation, and is the Fourier transform of the image

. Using the transforms given in (7), together with thetheory of decompositions, the Wiener–Khintchine theorem wasextended in [17] to give a generalized hypercomplex form

(10)

where , , , and. Computing the cross-covariance in the spatial-fre-

quency domain consists of simply setting theDC, or first, spec-tral component to zero before performing the reverse transform.

A. Hypercomplex Operators and Valid Alternate Forms

It should be noted that the operational formula given in (10)is not unique. There are, in fact, a number of equivalent formsfor the hypercomplex cross-correlation of two images. There aretwo reasons for this. First, there are equivalent forms of standardcross-correlation in the spatial domain, and second, there are, ingeneral, multiple implementations of any particular hypercom-plex function. Consider the route traversed in 3-D space betweentwo arbitrary points using only one rotation and one translation.There are two distinct routes; the rotation followed by the trans-lation or vice versa. The following is intended to offer readersthe supporting concepts necessary to prove, for themselves, thevalidity of equivalent forms of correlation, or other hypercom-plex, functions.

1) Review of Correlation:Unlike the convolution of twofunctions, the correlation of two functions does not commute,even in the nonvector cases. Consider

(11)where substituting and yields the secondform. In [18], the convolution of two functions is illustrated asthe integral of the product of the two functions with the secondreversed. This is distinctly different from the correlation of twofunctions that does not involve a reversal.

For the correlation of real functions, (11) is entirely suffi-cient. However, for complex functions, a conjugate operationis required to ensure the relation of the autocorrelation function( ) to the power spectrum ( ) asrequired by the Wiener–Khintchine theorem; see [19, Sec. 2.2].This effectively ensures that the power spectrum of a complexautocorrelation is real-valued.

MOXEY et al.: HYPERCOMPLEX CORRELATION TECHNIQUES 1945

The extension to the standard forms of cross-correlation func-tion in (11) is given in [18], [19]

(12)where the assumption made for the latter is that, since

and , the reversal cancels the conjugate.However, the latter is not always valid since the assumption isnot guaranteed. For complex numbersand , .When images are represented by complex, or hypercomplex,numbers, they are often purely imaginary. In such cases,

, and the assumption is valid. In general, the literature doesnot give significance to this since the direction of the shiftedsignal ( ) is immaterial for most correlation problems. How-ever, in the case of image registration and global color correctionusing vector phase-correlation (see Sections V and VI), direc-tion is fundamental.

The equivalent form of (12) should therefore be

(13)

2) Relationship of Hypercomplex Operators:The hy-percomplex Fourier transform can be implemented as two,standard, complex transforms usingsymplecticdecomposition;see Appendix A. The relationship between thesimplexandperplexparts and the four transform pairs is fundamental to theidea of hypercomplex operator identities. By decomposing thehypercomplex Fourier transform into symplectic form, it canbe shown that while the parallel components can be exchangeddirectly between transform kernels, the perpendicular compo-nents can only be exchanged for the transposed kernel in thereverse direction. This leads to the identities

(14)

This can be verified experimentally by decomposing an imageabout a given axis then transforming the parallel and perpen-dicular components using a chosen transform pair with trans-form kernel of the perpendicular component exchanged for thereversed transpose. Add the two spectra, and apply the reversetransformation for the original choice of transform pair. The re-sult is the original image, even though the forward transformof the two components was computed with the opposite handedtransforms in the opposite direction.

The product of applying two of the same transforms, one afterthe other, yields an operator relationship between the transform,the identity operator , the conjugate, and an image reversal

(15)

This relationship can be rewritten in operator notation as

and (16)

Fig. 1. Hypercomplex operator graph.

where is the conjugate operator, andis image reversal. Given that , a further set of operator

identities can be derived from (16) as

(17)

From (16) and (17), it can be shown that and.

The complete relationship of all of these operators can bedemonstrated by an operator graph, as shown in Fig. 1. The di-rected lines indicate the forward direction of the particular trans-form. Traversing the line in the opposite direction would yieldthe reverse transform. The conjugate and reversal operators arenot directed since and . Any closed-loop patharound the graph results in.

3) Alternate Forms of : Using the discrete methodof derivation described in [17] or a continuous method describedin [18], a set of equivalent hypercomplex forms for (12) and (13)can be derived. These are

(18)

(19)

and

(20)

(21)

Note that the discrete derivation yields an identical form. Thesehave all been proven experimentally, yielding identical spectrafor the same image set.

Using the hypercomplex identities and the operator graphgiven in Fig. 1, the algebraic validation of these equivalent formscan be proven, as described in [8].

1946 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 51, NO. 7, JULY 2003

IV. CROSS-CORRELATION AND COLOR-SPACEROTATIONS

In order to present a complete picture of hypercomplex corre-lation techniques and to simplify the discussion of later theory,we begin by considering a pathological case for the correlationof two uniform single color images. The following proof wasfirst published in [7].

Theorem 1: The hypercomplex cross-correlation of two uni-form, single-color images yields the mapping or rotation in colorspace between the two images.

Proof: From (6), consider only a single point, say the cen-tral pixel, from the correlation between two uniform, single-color, images such that

(22)

Since the image data are the same at each pixel location, thecorrelation of this pathological case yields the same result acrossthe whole image. Therefore, ignoring the size of the image, thecorrelation can be considered as the product of two quaternions.Since any quaternion can be expressed as the product of twopure quaternions, the correlation function can be represented by

(23)

where the simplification is given by the generalized Euler’s for-mula in (38) with .

Image can be considered as an appropriately scaled, color-space rotated version of, given by a rotation about an axis,where and , by some “unknown” angle suchthat

(24)

Substituting in (23) yields

(25)

which, by removing the conjugate and canceling terms, simpli-fies to

(26)

Since , we can apply the reordering rule ,where is a full quaternion, and is an axis vector, orpure quaternion, such that

(27)

where .The axis of rotation is therefore given by the axis of the

quaternion from the correlation surfaceat the position of reg-istration where the maximum modulus occurs. In this case, thecorrelation surface is uniform, and there is no maximum mod-ulus. However, the significance of this will become apparentlater. The rotation angle is given by the negated phase of thecorrelation peak .

Fig. 2 shows the relationship of the phase and axis quantitiesfor correlating a uniform cyan image with a uniform magentaimage. The values quoted were derived directly from the corre-lation function (applied to this experiment) and are equivalent to

Fig. 2. Phase and axis relationship in the correlation of cyan to magenta.

a 60 rotation about an axis of . Inde-pendent verification for these results can be achieved by calcu-lating the angle and cross product between the vectorsand . The resulting axis will be the conjugate of thatgiven here since the vector calculation cannot implement the ac-tion of the quaternion product that ensures that the axis (not thephase) is reversed. The hypercomplex correlation of vectorA tovectorB yields the vector mapping that encodes the rotation ofvectorB back to vectorA.

V. CROSS-COVARIANCE AND GLOBAL COLOR CORRECTION

Extending the theory of color-space mapping given above tonatural images, Fig. 3 shows three versions of the Lena imagewith Fig. 3(b) rotated by 60about the red/cyan axis.2 The color-space rotation is produced by applying to each pixel the rotationoperator described in Section II-A, where the unit axis is definedfrom the center of RGB space, in this case being set to

. Computing the cross-correlation ofthese natural images does not yield the color-space mappingdescribed in Section IV since both the phase and axis valuesare dominated by the accumulation of the mean, or centroid,component. The centroid of an imageis defined as

(28)

where is a pure quaternion or the average vector for the color-space.

Theorem 2: The hypercomplex cross-covariance betweentwo color images effectively computes the average rotationquaternion over every corresponding pixel between the twoimages.

Proof: Consider the correlation of two images describedby the addition of their centroids and theirvarying3 parts, thatis an image with the centroid subtracted from each pixel so thatthe image has zero mean. The cross-correlation, using either (6)or (10), then becomes

(29)

where is the cross-correlation of the varying partsand . This is equivalent to the cross-covariance of the

2The images appearing in this paper are grayscale versions of the originalcolor images.

3This is colloquially referred to as theACpart of an image.

MOXEY et al.: HYPERCOMPLEX CORRELATION TECHNIQUES 1947

(a)

(b)

(c)

Fig. 3. Example of color-space rotation and correction.

original images and . Computing the cross-co-variance and extracting the phase and axis from the same posi-tion as the peak modulus yields the color-space mapping be-tween the images. It should be noted that a perfect mapping forevery pixel across the whole image can only be achieved underspecific conditions, for example, when the contents of both im-

ages are uniform, as demonstrated in the previous section. Sincethe rotation between individual pixels is summed over the wholeimage, the vector mapping effectively encodes the best globalcolor-space transformation between the images.

Using the phase and axis of the quaternion from the cross-covariance surface at the position coincident with maximummodulus, a quaternion rotation operator, as described in Sec-tion II-A, can be constructed and a close “approximation” of theoriginal image can be recovered as demonstrated in Fig. 3(c).The phase and axis for this correction were computed as

and . These are very close to theoriginal values used to create the rotated image in Fig. 3(b), eventhough the image was in fact offset to the center of RGB space toperform the rotation and then returned to normal RGB space inorder to display the distortion here as an image. Effectively, wedestroyed some information in order to create the example, thatbeing the origin of the center of rotation. The significance of thiswill become more apparent in Section V-A. Later, in Section VI,an example of recovering a color rotated image in the presenceof considerable additive noise demonstrates further the robustnature of the hypercomplex correlation measure. Next, the ex-tent of global color space correction is explored and another,more practical, example of this correlation measure is given inthe following section.

A. Correcting for Global Color-Space Distortions

In practice, of course, the distortion of the color space of animage could consist of more than a rotation about some axis.Consider the content of two images as two individual color-sub-spaces existing in some abstract three-space, subject to arbitraryscaling and displaced from each other by some unknown rota-tion and translation. A visualization of this is given in Fig. 4,where the color subspaces are depicted as cubes, the centroidsof the image content are shown asand , is the displace-ment of the origin of one subspace relative to the other, andisa scaling factor between one subspace and the other. The rota-tion between the two subspaces is represented by the quaternionoperation . Given two images and ,the problem of correcting global color-space distortions can bemodeled by

(30)

This is analogous to solid body transformations used in com-puter graphics or in 3-D computer vision. There are numerousexamples of this. However, an excellent treatment of the mathmay be found in [20].

Theorem 3: The global color-space distortion between twoimages and can be measured from the vectormapping encoded in either the hypercomplex cross-covarianceof the images or the hypercomplex cross-correlation ofthe varying parts and the mean modulus of the varyingparts and the centroids of the images.

Proof: Consider the color-space distortion applied to animage represented by its centroid and varying parts

(31)

1948 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 51, NO. 7, JULY 2003

Fig. 4. Spatial relationship between arbitrary images in some abstract colorsubspace.

where

(32)

and is the mean modulus of the varying part of the ref-erence image, and likewise for the object or distorted image.Equating the terms yields

and (33)

from which we can derive

(34)

Since all the geometric operations that constitute a globalcolor-space distortion have been defined, a global correction canbe performed using the addition of the three terms given in (31).However, there is a simplification.

Theorem 4: The global color-space correction of a distortedimage back toward a reference image can becomputed from the vector mapping encoded in either the hyper-complex cross-covariance of the images or the hyper-complex cross-correlation of the varying parts and thecentroid of the reference image.

Proof: From (31) and (33), since

(35)

The algorithm for performing global color-space correctionusing a reference image of the same scene, correctly registeredas described in Section VI, consists of the following steps.

1) Compute the centroids and using (28) and subtractfrom the respective images, thereby creating the varyingparts and .

2) Compute the scale factorfrom the mean moduli of thevarying parts as in (31).

3) Compute the rotation operator from the phase and axis ofthe peak from the cross-correlation, or covariance, of thevarying parts.

4) Apply the rotation operator to the varying part of the dis-torted image and multiply by the scale factor.

5) Add the centroid of the reference image to the rotated andscaled image as in (35).

An example application demonstrating the practical aspectsof global color correction is given in Fig. 5. A difficult problemin computer vision is to segment any features that have changedbetween images of the same scene. The simplest method toachieve this is to compute the difference image. However, theproblem, especially in color images, has been that the differ-ence image will contain structural changes (the appearance ofsomething new, which we desire) and changes due to illumina-tion, which can affect the color and create shadows. Fig. 5(a)and (b) are of the same scene in the morning and afternoon of asunny day. They have been correctly registered so that the cor-responding pixels are at the same position in each image. Thetechnique used to achieve this image registration is discussedin Section VI. Notice the difference in the shadows and the ap-pearance of a person on the path in Fig. 5(a). Fig. 5(c) is theimage recovered from Fig. 5(b), using Fig. 5(a) as the reference.The recovered image should appear as a compromise betweenthe two. The difference image between the original images isgiven in Fig. 5(d). Notice the highlighted changes, the personon the path, the illuminated wall on the right-hand side, and, toa lesser but significant extent, the shadows beneath the trees.Ideally, the latter should be suppressed allowing for a broadthreshold to be used for the segmentation of the desired fea-tures. Fig. 5(e) demonstrates one possible application of globalcolor correction. In this case, the difference image between theoriginal, reference, image, and the color corrected object imageenhances the structural changes and suppresses everything else.This example is not presented as a method for removing un-wanted features or for the segmentation of structural change inimages from, for instance, security cameras. It simply serves asan extreme example of color-space distortion, where no consid-eration has been given to support techniques like balancing theillumination levels, yet it demonstrates the ability of the hyper-complex cross-covariance to compute an accurate global mea-surement of the distortion and using the algorithm given aboveto correct for it.

VI. PHASE CORRELATIONS AND IMAGE REGISTRATION

Phase correlation [21] is a standard image processing tech-nique for image registration and motion estimation, where thespatial shift between two similar grayscale images is given bycomputing the normalized cross-power spectrum of the two im-ages. The inverse Fourier transform of the normalized cross-power spectrum yields a complex image, the modulus of whichdefines a 2-D surface with a delta function (colloquially referredto as “the peak”) at the position corresponding to the spatial shiftbetween the two images. Mathematically, phase correlation isdefined as

(36)

where and are the Fourier transforms of the two imagesand , respectively. Here, the “overbar” denotes a complex con-jugate, and denotes a standard inverse Fourier transform.

MOXEY et al.: HYPERCOMPLEX CORRELATION TECHNIQUES 1949

(a) (b)

(c) (d)

(e)

Fig. 5. Example of the application of global color correction. (a) Image 1 (am). (b) Image 2 (pm). (c) Recovered image. (d) Standard difference image. (e)Difference image after correction.

The hypercomplex generalization of phase correlation, foruse with vector, or color, images was first presented in [22].Subsequent work on vector correlation [8] presented an equiv-alent form of hypercomplex cross-correlation, as described in

Section III-A. This has now become the preferred option sinceit yields a covariance response with the correct direction forthe vector mapping [7] between the reference and object im-ages and has been used here for global color correction. This

1950 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 51, NO. 7, JULY 2003

form also has the added bonus that computing the modulus ofthe cross-correlation in the frequency domain maintains the al-gebraic similarity with the original, nonhypercomplex, form.However, it should be noted that there is no difference in com-putational cost.

The hypercomplex phase correlation between two imagesand is given as

where

(37)

is the correlation of the images and andwhere , , , and

.Since the vector mapping between the images only has any

practical use when recovered from the point of registration,where the maximum modulus occurs, it is sensible to recoverboth the image registration from hypercomplex phase-corre-lation and the mapping from the cross-covariance in a singleprocess. Since the computation of the cross-correlation isfundamental to both techniques, reporting both the position ofimage registration and the axis and phase from the coincidentquaternion on the cross-covariance surface is easily achievedin practice by storing a copy of the original correlation priorto applying the normalization. Fig. 6 demonstrates the useof phase correlation for image registration and the ability torecover a close approximation of the original image, using theglobal color correction technique described in Section V-A, inthe presence of considerable noise. Cross-covariance should notbe used for image registration since the lack of normalizationleads to a broad peak and less accurate registration.

A. Color to Gray-Scale Image Registration

The use of phase correlation was extended in [9] to the reg-istration of color with gray-scale images. Fig. 7 shows anotherpair of Lena subimages with a known spatial shift between them.The reference image Fig. 7(a) is 24-bit true color, and the objectimage Fig. 7(b) is an 8-bit gray-scale. Again, both images werecorrupted with zero mean Gaussian noise with different vari-ances: 0.1 for the color image and 0.3 for the gray-scale image.Both images were converted to our quaternion image format forprocessing with the gray-scale image encoded in both the realpart and then the imaginary, or vector, part of the quaternionpixels. The reasoning behind this was to investigate any per-formance difference between vector–vector and vector–scalarphase-correlation. In both vector–vector or vector–scalar cases,the signal-to-noise ratio was found to be the same (16.7 dB). Re-peating these experiments with increasing noise content in theimages found a similar relationship in the results, confirmingthe valid use of both the grayscale encodings. However, closeinspection of the spectral components revealed the underlyingcause of this apparently arbitrary choice of processing. In thecase of vector–vector phase correlation, most of the power iscontained in the real part of the full quaternion. Alternatively,in the scalar–vector case, most of the power is contained in thevector part of the quaternion. Although the modulus response

(a)

(b)

(c)

Fig. 6. (a) Subimage plus zero mean Gaussian noise with 0.1 variance. (b)Spatially shifted, color-space rotated subimage plus zero mean Gaussian noisewith 0.2 variance. (c) Recovered sub-image.

of the scalar–vector case is identical to that of the vector–vectorcase, it is created by the orthogonal geometry, or phase differ-ence, of the scalar-grayscale image with each of the RGB com-ponents in the color image, in accordance with the definition ofa quaternion in Appendix A.

MOXEY et al.: HYPERCOMPLEX CORRELATION TECHNIQUES 1951

(a)

(b)

(c)

Fig. 7. (a) Color and (b) gray-scale images with added noise and (c) modulusof phase correlation indicating spatial shift.

As stated in Section III the vector–vector cross-covarianceyields a vector-space mapping between the images. In thiscase the mapping is between the grayline representation of thevector-grayscale image and the centroid of the color image.The vector mapping has no significance in the scalar–vectorcross-covariance since the processing is effectively the addition

of the three separate vector-to-vector correlations, those beingeach of the three red, green and blue basis vectors with the cen-troid of the color image, which sum to 90.

VII. CONCLUDING REMARKS

This paper has presented all of the vector correlation tech-niques that we have published to date and formulated the theorybehind the vector, or color-space, mapping that is encoded inthe peak of a correlation surface computed between two colorimages represented by hypercomplex pixels. We have shownthat while the position of maximum modulus of the hypercom-plex phase correlation surface conveys the spatial displacementbetween the structure of two images, the vector mapping fromthe cross-covariance of the images can be used to measure theglobal color-space distortions, and we have shown how to makeuse of the phase and axis information, together with the imagecentroids, to correct for them. Specifically, we have shown thatthe hypercomplex cross-covariance effectively computes the av-erage rotation quaternion over every corresponding pixel be-tween the two images. Perfect restoration of the image can there-fore only be achieved under specific conditions, for example,when the axis of rotation is perpendicular to the entire contentsof both images, as demonstrated in Section IV. However, wehave demonstrated that by using a reference image of the samescene, a “close approximation” can be recovered in the pres-ence of noise, spatial shifts, and global color-space distortionsconsisting of rotation, translation, and scaling. The scope of thework presented in this paper has not yet extended to an analysisof the errors in this determination of color-space rotation, norto its robustness against different confusing factors, and thesecould be useful topics for further study.

APPENDIX AQUATERNIONS ORHYPERCOMPLEXNUMBERS

A quaternion can be represented in Cartesian form as, where , , , and are all real, and, ,

and are the imaginary operators that obey

and

Since the imaginary operators are mutually orthogonal, they de-fine a 3-D imaginary space. Therefore, quaternions can be moreaccurately considered ashypercomplexnumbers.

Given a quaternion , its quaternionconjugate is , and its modulus is given by

A pure4 quaternion has a zero real part ( ), and aunitquaternion has a unit modulus. The inverse of a quaternion,such that , is given by

4The terminology “pure” is due to Hamilton, who viewed quaternions as amethod of solving the ratio of two vectors but later realized that this requiresat least four parameters, even though he sought a solution that only requiredthree. Hence, the four-component quaternion was less “pure” than the desiredthree-part vector.

1952 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 51, NO. 7, JULY 2003

A quaternion may be regarded as a complex number withcomplex real and imaginary parts. The two parts (and ) use adifferent, orthogonal complex operator than the full quaternion

versus , respectively [23, p. 7]. Therefore

where and

which combine to form . Mul-tiplying out the imaginary part using the rules for combiningthe orthogonal operatorsand yields a simple proof. This isknown as the Cayley–Dickson form, named after its 19th cen-tury discoverers, and is the basis for asymplecticdecompositionthat is used to implement hypercomplex Fourier transforma-tion using standard fast Fourier transform libraries such as the“FFTW” library at the Massachusetts Institute of Technology(see http://www.fftw.org/). We refer to the two parts (and ) ofthe quaternion in this form as thesimplexand theperplexparts,respectively. Note that this decomposition may be extended tothe general case

where and are perpendicular unit pure quaternions ().

APPENDIX BPOLAR FORM

Euler’s formula for the complex exponential generalizes tothe hypercomplex form

(38)

where is a unit pure quaternion ( ). Any quaternionmay be represented in polar form by

where and are referred to as theeigenaxisand theeige-nangle, respectively. The eigenaxis, or axis, is computed as

with the only exception being when , inwhich case, is undefined. The eigenangle, or phase, is com-puted as

and is always positive in the range . If the quaternionis zero, the phase is undefined.

APPENDIX CQUATERNIONIC DECOMPOSITION ANDREORDERING

Given twopurequaternions and , may be resolved intocomponents parallel and perpendicular tousing

, where and with [17].This follows from Coxeter [24], where , and

, .Geometrically, represents a reflection of in the plane

normal to . Writing , we have

Therefore, fromwhich follows and similarly for , from which

follows.

This concept extends to a full quaternionand a pure quater-nion, or three-vector, such that the decomposition ofaboutyields and . Note that the scalarcomponent of is part of the parallel component. This is some-what counterintuitive. Parallel quaternions (strictly co-planarquaternions or quaternions with parallel vector parts) commute.If is a full quaternion and is a vector where , we canreorder them such that . The proof for this is given by

but

Two perpendicular vectors [hereand ] reverse their signon reordering since the product of any two vectorsand isgiven by , and when , the dot productis zero. The product of two quaternionsand yields

ACKNOWLEDGMENT

The authors thank the anonymous referees for their commentson the first version of this paper and for making suggestions thatthey have included in the discussion of possible further work.

REFERENCES

[1] N. Le Bihan and J. Mars, “Blind wave separation using vector-sensors,”in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process.Istanbul,Turkey, June 5–9, 2000.

[2] , “Quaternion subspace method for vector-sensor wave separation,”in Proc. XI Eur. Signal Process. Conf.Toulouse, France, Sept. 3–6,2002, pp. 637–640.

[3] T. Bülow and G. Sommer, “Hypercomplex signals—A novel extensionof the analytic signal to the multidimensional case,”IEEE Trans. SignalProcessing, vol. 49, pp. 2844–2852, Nov. 2001.

[4] M. Felsberg and G. Sommer, “The monogenic signal,”IEEE Trans.Signal Processing, vol. 49, pp. 3136–3144, Dec. 2001.

[5] S. J. Sangwine, “Color in image processing,”Electron. Commun. Eng.J., vol. 12, no. 5, pp. 211–219, Oct. 2000.

[6] S. J. Sangwine and R. E. N. Horne, Eds.,The Color Image ProcessingHandbook. London, U.K.: Chapman and Hall, 1998.

[7] C. E. Moxey, T. A. Ell, and S. J. Sangwine, “Vector correlationof color images,” inProc. First Eur. Conf. Color Graphics, Imag.Vision. Poitiers, France: Univ. Poitiers, Apr. 2–5, 2002, pp. 343–7.

[8] , “Hypercomplex operators and vector correlation,” inProc. XI Eur.Signal Process. Conf.Toulouse, France, Sept. 3–6, 2002, pp. 247–250.

[9] C. E. Moxey, S. J. Sangwine, and T. A. Ell, “Color-grayscale imageregistration using hypercomplex phase-correlation,” inProc. IEEE Int.Conf. Image Process.Rochester, NY, Sept. 22–25, 2002, vol. II, pp.385–388.

[10] I. L. Kantor and A. S. Solodnikov,Hypercomplex Numbers, an Elemen-tary Introduction to Algebras. New York: Springer-Verlag, 1989.

[11] W. R. Hamilton,Elements of Quaternions. London, U.K.: Longmans,Green, 1866.

[12] S. J. Sangwine and T. A. Ell, “Color image filters based on hypercomplexconvolution,”Proc. Inst. Elect. Eng. Vision, Image Signal Process., vol.147, no. 2, pp. 89–93, Apr. 2000.

[13] , “Hypercomplex auto- and cross-correlation of color images,” inIEEE Int. Conf. Image Process., Kobe, Japan, Oct. 24–28, 1999, pp.319–322 (missing one page).

[14] , “The discrete Fourier transform of a color image,” inImage Pro-cessing II Mathematical Methods, Algorithms, and Applications, J. M.Blackledge and M. J. Turner, Eds. Chichester, U.K.: Ellis Horwood,2000, pp. 430–441.

[15] , “Hypercomplex Fourier transforms of color images,” inProc.IEEE Int. Conf. Image Process.Thessaloniki, Greece, Oct. 7–10,2001, vol. I, pp. 137–140.

MOXEY et al.: HYPERCOMPLEX CORRELATION TECHNIQUES 1953

[16] S.-C. Pei, J.-J. Ding, and J.-H. Chang, “Efficient implementation ofquaternion Fourier transform, convolution, and correlation by 2-Dcomplex FFT,”IEEE Trans. Signal Processing, vol. 49, pp. 2783–2797,Nov. 2001.

[17] T. A. Ell and S. J. Sangwine, “Hypercomplex Wiener–Khintchine the-orem with application to color image correlation,” inProc. IEEE Int.Conf. Image Process.Vancouver, BC, Canada, Sept. 11–14, 2000, vol.II, pp. 792–795.

[18] R. N. Bracewell,The Fourier Transform and Its Applications, 2nded. New York: McGraw-Hill, 1978.

[19] F. H. Lange,Correlation Techniques: Foundations and Applications ofCorrelation Analysis in Modern Communications, Measurements andControl, 1st ed. London, U.K.: Iliffe, 1967. English translation by P.B. Johns.

[20] B. K. P. Horn, H. M. Hilden, and S. Negahdaripour, “Closed-form so-lution of absolute orientation using orthonormal matrices,”J. Opt. Soc.Amer. Series A, Optics Image Sci., vol. 5, no. 7, pp. 1127–1135, July1988.

[21] C. D. Kuglin and D. C. Hines, “The phase correlation image alignmentmethod,” inProc. IEEE Conf. Cybern. Soc., 1975, pp. 163–165.

[22] S. J. Sangwine, T. A. Ell, and C. E. Moxey, “Vector phase correlation,”Electron. Lett., vol. 37, no. 25, pp. 1513–5, Dec. 6, 2001.

[23] T. A. Ell, “Hypercomplex spectral transformations,” Ph.D. dissertation,Univ. Minnesota, Minneapolis, MN, 1992.

[24] H. S. M. Coxeter, “Quaternions and reflections,”Amer. Math. Monthly,vol. 53, pp. 136–146, Mar. 1946.

C. Eddie Moxey (M’02) received the first classB.Eng. honors degree in electronic systems en-gineering in 1997 and the Ph.D. degree from theUniversity of Essex, Colchester, U.K., in July 2002.

He was a Research Officer with the MultimediaArchitectures and Applications Group, Departmentof Electronic Systems Engineering, University ofEssex. He is currently at the University of Surrey,Surrey, UK. His research interests are hypercom-plex processing of color images and large-scale,3-D scene reconstruction from extended image

sequences.

Stephen J. Sangwine(SM’90) received the B.Sc.degree in electronic engineering from the Universityof Southampton, Southampton, U.K., in 1979 andthe Ph.D. degree from the University of Reading,Reading, U.K., in 1991.

He is a Senior Lecturer with the Department ofElectronic Systems Engineering, University of Essex,Colchester, U.K. His interests include linear vectorfiltering and transforms of color images, nonlinearvector image filtering, and digital hardware design.He is an IEE Member and a Chartered Engineer in

the United Kingdom.

Todd A. Ell (M’93) received the B.S. (magna cumlaude) and M.S. degrees in electrical engineeringfrom the University of North Dakota, Grand Forks,in 1982 and 1983, respectively, and the Ph.D. degreein control sciences and dynamics systems from theUniversity of Minnesota, Minneapolis, in 1992.

He is a Senior Principal Engineer at GoodrichSensor Systems, Minneapolis. His personal interestsinclude the study and application of the hypercom-plex algebras to dynamic systems analysis.

Dr. Ell is a member of the Eta Kappa Nu and TauBeta Pi honorary societies.