POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING

Mohammad Amin Sadeghi, Seyyed Mohammad Mohsen Hejrati and Niloofar GheissariComputer Vision Group, School of Mathematics

Institute for studies in theoretical Physics and Mathematics, Tehran, Iranm.a.sadeghi@gmail.com, mhejrati@gmail.com, niloofar@mail.ipm.ir

Keywords: Image stitching, Poisson equation, Panorama, Color correction, Registration, Image mosaicing.

Abstract: A new method for seamless image stitching is presented. The proposed algorithm is a hybrid method whichuses optimal seam methods and smoothes the intensity transition between two images by color correction. Adynamic programming algorithm that finds an optimal seam along which gradient disparities are minimizedis used. A modification of Poisson image editing is utilized to correct color differences between two images.Different boundary conditions for the Poisson equation were investigated and tested, and mixed boundaryconditions generated the most accurate results. To evaluate and compare the proposed method with competingones, a large image database consisting of more than two hundred image pairs was created. The test imagepairs are taken at different lighting conditions, scene geometries and camera positions. On this database theproposed approach tested favorably as compared to standard methods and has shown to be very effective inproducing visually acceptable images.

1 INTRODUCTION

Since two registered images usually have local andglobal intensity differences, in order to create visu-ally acceptable images it is essential to complementthe registration process with seamless image stitch-ing. As noted in (Hasler and Susstrunk, 2000), sev-eral factors reflect the intensities recorded by cameras.They include: exposure variances, white balancing,gamma correction, vignetting and digitizer parame-ters. Seamless stitching has important applications inmany high level tasks such as building panoramic im-ages, virtual reality, super resolution and texture syn-thesis. Recently there have been many breakthroughattempts in seamless image stitching such as (Levinet al., 2003), (Uyttendaele et al., 2001), (Jia and Tang,2005a), (Jia and Tang, 2005b).

There are two main approaches for the eliminationof seams and artifacts in the overlapping images. Thefirst approach is to find an optimal seam along whichsome energy function is minimized (Jia and Tang,2005a), (Agarwala et al., 2004). Jia and Tang (Jia andTang, 2005a) have described different possible energyfunctions (weighting functions) that might be use tofind the optimal seam. These energy functions maybe based on intensity or gradient differences of thetwo images. Dynamic programming or graph cut is

used to minimize the energy function. One advantageof optimal seam methods is that they are applicableto dynamic scenes. This is because the optimal seamdoes not pass through areas of high gradient disparity.However, optimal seam algorithms are less suitablewhen the illuminations of the input images are not thesame or their overlapping area consists of thin strips.

The second approach is based on enhancing theintensity values of one or both images so that the re-sulting stitched image looks natural. This task can beperformed either globally or locally.

Szeliski and Shum (Shum and Szeliski, 2001) pro-pose a method for building panoramic images capa-ble of correcting local and global intensity misalign-ments. Their feathering algorithm results in more nat-ural blending effects. Still in many cases it producesunnatural seams under different illumination condi-tions. Belt and Adelson (Burt and Adelson, 1983)use multiresolution splines for color correction. Uyt-tendaele et al. (Uyttendaele et al., 2001) proposed amethod capable of stitching multiple images taken atvarious exposures and illuminance conditions. Theyallow for large scene changes or misregistrations be-tween two images. Levin et al. (Levin et al., 2003)introduced Gradient Domain Image Stitching (GIST)method. GIST utilizes on an optimization algorithmto minimize a cost function over image derivatives.

275

This cost function is a dissimilarity measure betweenthe derivatives of the resulting stitched image and thederivatives of the input images. Jia and Tang (Jia andTang, 2005b) proposed a tensor voting method forglobal and local intensity alignment. They use a sym-metric tensor representation to model the joint imagespace (source image and target image intensities) as atoken. They extract curves in the image and let tokenson a curve communicate mutually to reinforce eachother. A dense tensor field postulates smooth connec-tions among tokens and ensures natural color transi-tions. This method can successfully handle occlusionand motion blur. Their local intensity alignment ap-proaches is devised to correct vignetting effects.

We have observed that a hybrid method that cantake advantage of the strengths of both main ap-proaches, generates more reliable results. Thereforethe proposed hybrid method first finds an optimalseam to stitch two images. Then it uses the Pois-son equation with mixed boundary conditions in thecolor domain. As a result, the local color differencesbetween two images are corrected. Poisson equationhas been already used for image editing (Perez et al.,2003) or correcting structure misalignments producedby stitching (Jia and Tang, 2005a). An advantage ofthe proposed method is its low computational timethat makes it particularly desirable for applications tohigh resolution images. The proposed method relieson a model based registration algorithm to align twoimages (Gheissari et al., 2008). In fact, the registra-tion algorithm computes the transformation parame-ters after recovering the transformation model.

This paper is organized as follows: Section 2,gives a brief summery of the registration methodused. Methods for finding optimal cutting curve aredescribed in section 3. In section 4, the proposedmethod, Poisson Local Color Correction (PLCC) isexplained. Different boundary conditions for the so-lution of the Poisson equation are described in thefollowing section (4). Some challenging images forlocal color correction and the results of applying theproposed algorithm on them are shown in the section5.

2 MODEL-BASED IMAGEREGISTRATION

In this work, a registration method introduced in(Gheissari et al., 2008) is utilized. In summary thisregistration method first finds a set of SIFT inter-est points (Lowe, 1999) and the correspondences be-tween them. The outliers are eliminated to gener-ate an outlier-free set of correspondences. This is

achieved by robustly fitting a translation model to thecorrespondences. Next all models in the model libraryare fitted to the above set of inliers.

(a) (b)

(c) (d)

(e)

Figure 1: Different transformation models for registration:(a) Translation, (b) Euclidian (c) Similarity, (d) Affine and(e) Projective transformation.

CAIC is applied to select the correct transforma-tion model from a library of candidate models (Boz-dogan, 1987). Figure 1 shows the set of candidatemodels. The final set of inliers is determined by ap-plying the chosen model to the data set. The algorithmestimates the parameters of the chosen model (basedon the set of inliers) to generate the transformationmatrix.

Before finding the optimal seam and using thePoisson equation to correct color difference, we ap-ply the transformation matrix to the source image.This results in two registered images with an over-lapping area which can be a triangle, a rectangle, ora higher order polygon depending on the transforma-tion model.

3 OPTIMAL SEAM METHODS

Optimal seam algorithms are used widely in appli-cations such as pattern synthesis and image stitching(Kwatra et al., 2003), (Jia and Tang, 2005a), (Uyt-tendaele et al., 2001). These methods find a cuttingcurve in the overlapping area and stitch the images

VISAPP 2008 - International Conference on Computer Vision Theory and Applications

276

along the cutting curve as their common boundary.The main approach in optimal seam algorithms is tospecify a weight for each pixel belonging to the over-lapping area. Then a curve that minimizes this weightfunction is selected as the optimal cutting seam.

(a)

(b)

(e)

Figure 2: (a) Two registered images and their overlappingarea, (b) optimal cutting curve, (c) final stitched images.

Assume we have registered images IT and IS. Theoverlapping area is IovT in image IT and IovS in image IS(as shown in figure 2). We would like to cut and stitchIT and IS along an optimal curve C.

Let C be a cutting curve, then the cost of having Cas cutting curve is defined as:

W (C) = pC

W (p), (1)

where W (p) is the value of the weight function asso-ciated with pixel p. The optimal curve(seam) is thecurve which minimizes the cost W (C).

Different candidates for weight function are

Wg(p) = ||IS(p)IT (p)||nn, (2)and

Wi(p) = ||IS(p) IT (p)||nn, (3)where ||.||nn is the nnorm. A linear combination ofWg(p) and Wi(p) could also be considered as weightfunction (Jia and Tang, 2005a). We have tested differ-ent energy functions and observed that the function

W (p) = ||IS(p)IT (p)||11 (4)

(a) (b)

(c) (d) (e)

( f ) (g) (h)Figure 3: (a) And (b) are the input images. The images areborrowed from figure 6 of (Jia and Tang, 2005a). The im-ages, used in our experiments, are of our size 337 334.(c) cutting curve using intensity based weight functionWi(p) = ||IS(p) IT (p)||11. (d) Stitching using intensitybased weight function without color correction (withoutPLCC) (e) Stitching using intensity based weight functionwith color correction (PLCC) (f) cutting curve using gra-dient based weight function Wg(p) = ||IS(p)IT (p)||11.(g) Stitching using gradient based weight function withoutcolor correction (without PLCC) (h) Final result after usinggradient based weight function and applying PLCC method.

is more reliable than the other candidates speciallywhen the illuminations of IT and IS are different. Af-ter finding the optimal seam PLCC is applied to cor-rect color and illumination differences between twoinput images. Setting n = 1 (using 1norm) leadsto better results because small intensity differencesare ignored in higher norms. We use a 3-dimensionalRGB color vector for each pixel as I(p), and observedthat using HSV color space has the same performanceas the RGB color space.

The result of using intensity based weight func-tion and gradient based weight function is shown inFigures 3 and 5.

POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING

277

There are various algorithms with different com-plexities for finding the optimal cutting curve. Min-Cut(Max-flow) algorithm (Kwatra et al., 2003) worksaccurately. However due to its high complexity, it im-poses significant computational burden on images ofhigh resolution. We used a combination of dynamicprogramming and BFS (Breadth First Search) algo-rithm in our experiments to find optimal cutting curve.Its complexity is O(n) where n is the number of pixelsin the overlapping area.

Figure 4: In some cases the optimal seam is not a simplecurve.

In some cases the optimal seam gets more com-plex as shown in Figure 4. Therefore an optimal seamis not necessarily one simple curve.

4 COLOR CORRECTION BASEDON POISSON EQUATION

Poisson equation has been used for image editingand image alignment applications, for example (Perezet al., 2003), (Jia and Tang, 2005a) and (Agarwalaet al., 2004). Here we use Poisson equation to cor-rect color differences between two already registeredimages.

Lets be the region (in IS) that lies on one side ofthe optimal seam (the side belonging to the source im-age). The boundary of is denoted by . We sep-arate into two parts: the common boundary withIT (referred to as C), and the other boundaries (de-fined as B). Our notations are illustrated in Figure6.

To eliminate artifacts and color differences overthe seam C, we define a correction function on as

IS = IS + (5)where IS is the corrected image. Then the Poissonequation is solved on and the resulting correctionfunction , is added to IS to compensate for color dif-ferences between two images. For colored images,we apply the same method on each color channel ofthe input image. According to our experiments theperformance of the method is the same in RGB and

(a) (b) (c)

(d) (e) ( f )

(g) (h) (i)Figure 5: Illustration of applying PLCC on two images.(a) and (b) input images of size 800 600. (c) the resultof putting two registered images together without applyingany corrections. (d) a cutting curve found by using inten-sity based weight function. (e) the solution of the poissonequation solved on the corresponding boundary. (f) the re-sult of stitching the images using an intensity based seamline. (g) shows a cutting curve found using Gradient basedweight function. (h) the solution of the poisson equationassociated with the cutting curve in (g) and (i) the final re-sult of stitching using intensity based seam line. As youcan see, the color is not corrected perfectly in (f) becauseits corresponding cutting curve has passed dark areas andthe boundary conditions of the poisson equation in (e) is notvalid.

Figure 6: Notation of the region : is the region of thevisible part of IS. The common boundary with IT is definedas C, and the other boundaries are defined as B.

HSV color spaces. For solving Poisson equation on several boundary conditions could be assumed. The

VISAPP 2008 - International Conference on Computer Vision Theory and Applications

278

(a) (b)

(c) (d)

(e) ( f )Figure 7: Comparison between the results of using Drich-let boundary conditions or Neumann boundary conditionson B. (a) and (b) source and target images, both of size800600. (c) the solution of Poisson equation with Drich-let boundary conditions on B as color correction factor.(d) the respective result of color correction. It is equivalentto applying Poisson image editing (Perez et al., 2003) (e)the solution of Poisson equation with Neumann boundaryconditions on B and (f) the the result of color correctionusing the correction factor (e). The correction factor (c) and(e) are solved on a rectangular domain before applying thetransformation and added to source images after transfor-mation.

Dirichlet boundary condition equal to the differencebetween intensity of two input images along C wasimposed on C. For B the Neumann boundarycondition was used.Imposing the proper boundary condition is critical inour application. One might use Dirichlet boundarycondition on B (equation 6). However, the prob-lem here is the choice of the boundary function onB. For the Dirichlet boundary condition, an ap-proximation to the value on B is necessary. Thetrivial approximation zero for the boundary value re-sults in imperfect color correction as shown in Fig.7.

= 0 over with

= IT IS, On C

= Constant, On B,(6)

where denotes the Laplace operator.

The Neumann boundary condition on B inPLCC (equation 7) produced significantly better re-sults. The superior performance of the Neumannboundary condition on B is explicable in view ofthe fact that it does not fix intensity values along Bbut constraints the gradient values along B to bezero. Fixing the intensity on B results in tobe constant over B, while the ideal values of over the are not necessarily constant. Further-more, there is no straightforward theoretical way toset the values of over B and an approximation isinevitable. This problem is aggravated where the il-lumination condition between the input images differsignificantly.

= 0 over with

= IT IS, On Cn = 0, On B,

(7)

where n is normal vector at . This Poisson equa-tion is the result of the following minimization prob-lem (equation 8).

(a)IT (b)IS

(a) (b)

(c) (d)Figure 8: Applying PLCC and optimal seam to dynamicscene. (a) and (b) input images of size 800600. (c) Opti-mal seam based on gradient disparities. (d) Stitching with-out using optimal seam. The shadow of pedestrians walk-ing behind is visible, while the pedestrians body are elimi-nated from the stitching result. The color difference is veryprominent. (e) Stitching using optimal seam. The shadowof pedestrians are no longer visible. The color differences isstill prominent (f) Final result of applying color correction(PLCC) and optimal seam.

POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING

279

(a) (b)

(c) (d)

(e)

( f )Figure 9: Creating super resolution images using imagesof poor quality. (a), (b), (c) and (d) are input images (of apuzzle hanged to a wall) taken at four different viewpointsand illumination conditions. input image (a) is taken un-der yellow light source, (b) taken under yellow and whitelight sources, (c) taken at low exposure time under whitelight source and (d) taken under white light source. (e) thestitching result without PLCC. (f) the stitching result afterPLCC.

min

||2 with

= IT IS, On Cn = 0, On B,

(8)

where the gradient operator [ x ,y ] is represented by

. This minimization problem results in visually nat-ural output images because it minimizes the 2normof the gradient of intensities near the boundary.

We used the method of finite differences and thecentral difference formula to solve the Poisson equa-tion and the resulting system of linear equations.

5 EXPERIMENTAL RESULTS

A database for testing and evaluating image stitchingalgorithms was created. This database contains overtwo hundred pairs of images from over forty scenes.All pairs are available in three different standard sizes.The scenes vary in color, camera properties, cameralocation and lighting. The database will be publiclyavailable at math.ipm.ac.ir/vision. The transforma-tion matrix for each pair of images was determinedusing (Gheissari et al., 2008). This database containschallenging images for registration, color correction,and building panoramic images.

As mentioned before the optimal seam method isapplicable to dynamic scenes. This is because the op-timal seam does not pass through areas of high gra-dient disparity. This capability is shown in figure 8.As can be seen from this figure, two different groupsof pedestrians are present in the input images. Theoptimal seam passes through the narrow gap betweenthese two groups and therefore both groups are pre-sented in the resulting stitched image. If the opti-mal seam has not been used, then the shadow of thepedestrian group behind would be visible without thepedestrians themselves.

Performance of the proposed method is comparedwith some standard competing methods. As shown infigure 10, a combination of PLCC and optimal seamoutperforms feathering (Uyttendaele et al., 2001),pyramid blending (Adelson et al., 1984), GIST (Levinet al., 2003) and Poisson image editing (Perez et al.,2003).

The proposed algorithm is desirable for makingsuper resolution images due to its high accuracy andlow time complexity. To show this capability wechose four images of a puzzle hanged to a wall. Theimages were taken at four different view points andsignificantly various illumination conditions as shownin figure 9. The results of applying the proposed al-gorithm shows its success .

VISAPP 2008 - International Conference on Computer Vision Theory and Applications

280

(a)IT (b)IS

(c)

(d)

(e) ( f ) (g) (h) (i) ( j)Figure 10: Comparing PLCC with some competing meth-ods: (a) and (b) are input images borrowed from figure 4of (Levin et al., 2003). The images are of size 348 255and 290257 respectively. (c) Stitched image after apply-ing optimal seam , (d) The color correction result generatedby PLCC after finding optimal seam, (e) Feathering (Uyt-tendaele et al., 2001), (f) Pyramid blending (Adelson et al.,1984), (g) Feathering on the gradients (Uyttendaele et al.,2001), (h) Poisson editing (Perez et al., 2003), (i) GIST(Levin et al., 2003) (j) The result of PLCC method.

6 CONCLUSIONS

A new method for seamless image stitching was pre-sented. The proposed algorithm is a hybrid method,that combines optimal seam approaches and methods

which smooth the intensity transition between twoimages (by color correction). Using a dynamic pro-gramming algorithm, an optimal seam along whichgradient disparities are minimized is determined. Alocal color correction method based on Poisson equa-tion is used to enhance color variations between im-ages. Different boundary conditions to solve the Pois-son equation were investigated and tested. We con-cluded that using mixed boundary conditions gen-erates the most accurate results. To evaluate andcompare the proposed method (with some competingmethods), a large image database consisting of overtwo hundred image pairs was utilized. The test im-age pairs are taken at different lightening conditions,scene geometry and camera positions. Compared tostandard methods, proposed approach performed fa-vorably on this database and has shown to be veryeffective in producing visually acceptable images.

ACKNOWLEDGEMENTS

We would like to acknowledge Dr. Mohammad RezaMokhtarzade for his valuable advices on solvingPDEs. In addition, we would like to acknowledgeDr. Mehrdad Shahshahani for his kind helps in thisproject and in writing this paper. Mr. Mostafa Ka-mali Tabrizi helped us in modifying the source codefor registration. We appreciate the valuable discus-sions he had with us. Finally we thank Zohre Sharafi,Parisa Mirshams, Paria Mehrani, Sheyda Beigpoor forproviding us with beautiful test images. This projectwas supported by Institute for studies in theoreticalPhysics and Mathematics (IPM).

REFERENCESAdelson, E. H., Anderson, C. H., Bergen, J. R., Burt, P. J.,

and Ogden, J. M. (1984). Pyramid methods in imageprocessing. RCA Engineer, 29(6).

Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S.,Colburn, A., Curless, B., Salesin, D., and Cohen, M.(2004). Interactive digital photomontage. ACM Trans.Graph., 23(3):294302.

Bozdogan, H. (1987). Model selection and akaikes infor-mation criterion (aic): The general theory and its ana-lytical extensions.

Burt, P. J. and Adelson, E. H. (1983). A multiresolutionspline with application to image mosaics. ACM Trans.Graph., 2(4):217236.

Gheissari, N., Tabrizi, M. K., Sharafi, Z., and Mirshams, P.(2008). Model-based global image registration. InProc. of the 3rd International Conference on Com-puter Vision Theory and Applications, page to appear.

POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING

281

Hasler, D. and Susstrunk, S. (2000). Colour handling inpanoramic photography.

Jia, J. and Tang, C.-K. (2005a). Eliminating structure andintensity misalignment in image stitching. In ICCV05: Proceedings of the Tenth IEEE InternationalConference on Computer Vision, pages 16511658,Washington, DC, USA. IEEE Computer Society.

Jia, J. and Tang, M.-C.-K. (2005b). Tensor voting for im-age correction by global and local intensity alignment.IEEE Trans. Pattern Anal. Mach. Intell., 27(1):3650.

Kwatra, V., Schodl, A., Essa, I., Turk, G., and Bobick, A.(2003). Graphcut textures: image and video synthe-sis using graph cuts. In SIGGRAPH 03: ACM SIG-GRAPH 2003 Papers, pages 277286, New York, NY,USA. ACM Press.

Levin, A., Zomet, A., Peleg, S., and Weiss, Y. (2003).Seamless image stitching in the gradient domain.

Lowe, D. G. (1999). Object recognition from local scale-invariant features. In Proc. of the InternationalConference on Computer Vision ICCV, Corfu, pages11501157.

Perez, P., Gangnet, M., and Blake, A. (2003). Poisson im-age editing. In SIGGRAPH 03: ACM SIGGRAPH2003 Papers, pages 313318, New York, NY, USA.ACM Press.

Shum, H.-Y. and Szeliski, R. (2001). Construction ofpanoramic image mosaics with global and local align-ment. pages 227268.

Uyttendaele, M., Eden, A., and Skeliski, R. (2001). Elim-inating ghosting and exposure artifacts in image mo-saics. pages II509 II516 vol.2.

VISAPP 2008 - International Conference on Computer Vision Theory and Applications

282