ANIMPROVEDIMAGEMATCHINGMETHODBASEDONSURFALGORITHM · (a)surf+ransac (b)cc_surf The International...

AN IMPROVED IMAGE MATCHING METHOD BASED ON SURF ALGORITHM

S. J. Chen1,*, S. Z. Zheng2, Z. G. Xu1, C. C. Guo2, X. L. Ma3

1 School of Resource Engineering, Longyan University, Longyan, China - [email protected], [email protected] College of Surveying and Geo-Informatics, Tongji University, Shanghai, China - (zhengshouzhu0, candyguo_fly)@163.com

3 Chinese Academy of Surveying and Mapping, Beijing, China [email protected]

Commission III, WG III/6

KEYWORDS: Image matching, Information entropy, SURF, Delaunay Triangulation

ABSTRACT:

Many state-of-the-art image matching methods, based on the feature matching, have been widely studied in the remote sensing field.These methods of feature matching which get highly operating efficiency, have a disadvantage of low accuracy and robustness. Thispaper proposes an improved image matching method which based on the SURF algorithm. The proposed method introduces colorinvariant transformation, information entropy theory and a series of constraint conditions to increase feature points detection andmatching accuracy. First, the model of color invariant transformation is introduced for two matching images aiming at obtainingmore color information during the matching process and information entropy theory is used to obtain the most information of twomatching images. Then SURF algorithm is applied to detect and describe points from the images. Finally, constraint conditionswhich including Delaunay triangulation construction, similarity function and projective invariant are employed to eliminate themismatches so as to improve matching precision. The proposed method has been validated on the remote sensing images and theresult benefits from its high precision and robustness.

1. INTRODUCTION

Image matching is a process of a geometrically matching up twoimages of a same scene, which widely used in variousapplications (Zitova and Flusser, 2003; Arévalo and González,2008), including image mosaicing, change detection, 3Dconstruction, etc. Generally speaking, image matchingalgorithms can be classified to feature-based and intensity-basedmethods (Zitova and Flusser, 2003; Wu et al., 2015). Comparedwith intensity-based algorithms, feature-based algorithms notonly have advantages of illumination changes and complexdistortion, but also take less time consuming due to the wholeimage information replaced by a finite number of points. Sofeature-based algorithms have been a research hotspot of imagematching. Feature-based algorithms firstly extract feature pointsfrom two images, and the matching method is performed togenerate the matching result from the feature points pair.

In recent years, feature-based algorithms have been extensivelystudied. Lowe proposed scale invariant features transform (SIFT)algorithm (Lowe, 2004) to solve some problems such asillumination, speckle, rotation, scale, translation, etc. However,the algorithm is not fit for dealing with a large number ofimages because of SIFT itself complexity. Then manyresearchers have already been attracted to improve SIFT(Mikolajczyk and Schmid, 2005; Juan and Gwun, 2009; Teke etal., 2011). Speeded Up Robust Features (SURF) algorithm (Bayand Van, 2006) is the improved version of SIFT, having anadvantage of time efficiency which is attributed to introduceintegral image and box filter to reduce time-consuming. But ithas a disadvantage of lower accuracy. Lee et al. (Lee et al.,2010) presents a Coarse-to-Fine approach of image matchingbased on Haar Wavelet Transform and SURF algorithm. TheCoarse-to-Fine strategy from Harris operator to normalizedcross-correlation and RANdom SAmple Consensus (RANSAC)algorithm are used to achieve the fine points. Bouchiha et al.

developed a matching method which separated the detectorfrom the descriptor and proposed an extension to the SURFdescriptor. Zheng et al. (Zheng et al., 2015) proposed animproved SURF method by combining color invariant modeland a series of constraint conditions (CC_SURF), this methodhas a good matching rate and feature points are well-distributed.Anzid et al. (Anzid et al., 2017) proposed an improvement ofthe SURF algorithm and it could automatically remove theoutliers by means of both Distance and Orientation filteringstrategy (DO_SURF). However, in the aforementioned methodsthe number of the detected feature points is not at a high levelowing to the loss of color information caused by color imagesoften converting to grayscale images in the matching process.

Therefore, this paper proposes an improved SURF mean whichintroduces color invariant transformation, information entropyand constraint conditions aiming at improving a number offeature points and matching precision. First, orthogonal colortransformation model is introduced to generate the orthogonalcolor space of two images from RGB color space of bothimages, and the information entropy is used to selected the mostrich color channel as the subsequent matching image. ThenSURF algorithm is applied to extract and describe feature pointsfrom the image, and constraint conditions which includingconstructing Delaunay triangulation, triangle similarity functionand projective invariant are employed to eliminate the mismatchmatch and increase image match precision and robustness.

The rest of this paper is organized as follows: Section 2describes the proposed method. In Section 3, experimentalresults obtained on remote sensing images by the proposedmethod is compared with other related SURF algorithms.Section 4, the conclusion is drawn.

The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-3, 2018 ISPRS TC III Mid-term Symposium “Developments, Technologies and Applications in Remote Sensing”, 7–10 May, Beijing, China

This contribution has been peer-reviewed. https://doi.org/10.5194/isprs-archives-XLII-3-179-2018 | © Authors 2018. CC BY 4.0 License.

179

mailto:[email protected]

mailto:[email protected],

2. THE PROPOSED METHOD

2.1 Overview of the proposed method

In this paper, the overall flowchart of the proposed method isshown in Fig. 1. It can be divided into 3 main steps: (1) Colortransformation and Information entropy are introduced forobtaining the more information data of two matching images; (2)SURF algorithm are used to detect and extract the feature points;(3) methods of mismatch points removal (Li and Zhang, 2009;Zheng et al., 2015) which including Delaunay triangulation,triangle similarity function and projective invariant, are appliedto eliminate the mismatch feature points.

Fig. 1. The overall flowchart of the proposed method

2.2 Color transformation and Information entropy

Due to the RGB color space of images without invariant space,we transform RGB color space into orthogonal color space withcolor invariant characteristic. So each color channel of imagewhich including R, G and B band, is done by calculated themean and the standard deviation of pixel value. The orthogonalcolor space calculation expression can be defined as:

BB

GG

RR

BGR

BGR

/)(/)(/)(

0

0

0

(1)

where R, G and B are the mean of pixel value in each channel,respectively. R, G and B are the standard deviation value ofeach band respectively. R, G and B respectively represent thepixel value of every band in original image. R0, G0 and B0 arethe pixel value of every band via orthogonal transformation.

Information entropy is calculated to select the most abundantinformation for each channel. Information entropy is a measureof image information. The greater the information entropy of animage, the more the information content has. Thereforeinformation entropy is introduced to calculate the information ofeach channel and select the best channel as import informationdata to be processed. More feature points would be detected bySURF detector in the both images. Information entropyexpression can be described as:

000max

min , , ,)(ln)()( BGRkjjjE

jk

(2)

where k represents the three corresponding color channels; jrepresents pixel value level of color channels; (j) representsoccurrence probability of j-level; min and max are the pixelvalue of minimum and maximum, respectively. E(j) representsinformation entropy of color channels.

As the above mentioned by equation 1 and 2, the most entropyof color channels is used for the feature detection and extractionof the next step.

2.3 SURF detector and descriptor

2.3.1 SURF detector: For the two matching images, wecalculate the information entropy for all the channels and selectthe channel with the maximum entropy as input data. SURFdetector is used to detect the feature points of two image.Integral image is set up and multi-scale space is built by boxfilter. Hessian matrix is applied as it has good performance andaccuracy. The Hessian matrix can be defined as:

),(),(),(),(

),,(

xLxLxLxL

yxHyyxy

xyxx (3)

where is a scale. Lxx(x,) is the convolution result of thesecond order derivative of Gaussian filter and similarly forLxy(x,) and Lyy(x,).

In order to improve the calculation efficiency, we construct aFast-Hessian matrix in SURF detector. We equate the initial9×9 box filter with =1.2 of second order derivative ofGaussian filter and Dxx is a result which received byconvolution operation of box filter template and input image,represents Lxx, similarity for Dxy and Dyy. The Fast-Hessianmatrix can be expressed as follows:

2)9.0()det( xyyyxx DDDHessian (4)

Finally, the threshold is appropriately selected and a Non-maximum suppression in a 3×3×3 neighborhood of each pointis applied to detect extremum points. The point is regarded as acandidate only if its response value is larger or smaller than allthe 26 neighbors at the current and adjacent scales. Then, thesteady feature point location and its scale are obtained byinterpolating in scale space and image space.

2.3.2 SURF descriptor: To realize the rotation invariant offeature point, the dominant orientation for each feature pointshould be determined. First, Harr-wavelet responses arecalculated for the pixels within a circular neighbourhood ofradius 6s around the feature points, where refers to the scale ofpoint detected. Then weighted Gaussian function is performedfor the feature points, and we give greater weights only if thefeature point is close to the circle center and contributing a lot inorientation. The dominant orientation is estimated bycalculating the sum of all responses within a sliding orientationwindow covering an angle of 60° . The x- and y- responseswithin the window are summed up to produce a new vector. Sothe longest vector’s orientation is selected as the feature pointdominant orientation via scanning the entire circularneighbourhood.



180

After generating feature descriptor, the rectangular region withthe side length 20s×20s is selected, and the dominantorientation of the area are rotated to the dominant orientation ofthe feature points. The rectangular region is divided into 16smaller 4×4 square sub-regions and 5×5 sampling points areselected in each sub-region to calculate the corresponding Harr-wavelet responses dx and dy. Furthermore, summate Harr-wavelet response values and their absolute values of the 4 sub-regions separately. So we get a four-dimension descriptor vectorv=(dx, dy, |dx|, |dy|) for each sub-regions and form a 64(4×4×4)-dimension feature vector.

2.4 Mismatch points removal

SURF detector and descriptor only detects and extracts featurepoints, but it can hardly eliminate a lot of mismatch points. Forthis reason, this paper introduces the Delaunay triangulation,triangle similarity function and projective invariant under thematching process.

2.4.1 Delaunay triangulation construction: In this paper,incremental insertion algorithm (Lawson, 1977) which has someadvantages of simple thought, easy implementation and highefficiency, is used to build triangulation network for all featurepoints in both images. Then LOP (Local OptimizationProcedure) is used to optimize the quality and performance ofDelaunay triangulation. The procedure of triangulation buildingand LOP optimizing are defined as follows:

(1) Incremental insertion algorithm: As shown in Fig. 2, thebasic steps of building triangulation which use the incrementalinsertion algorithm, is described in Table 1:

Step1 build a large triangle as the initial triangle whichcontains all of the points.

Step2 freely select one from the points as interpolation pointin the large triangle.

Step3search the triangle which includes this point, then linkthis point with other three point of the triangle togenerate three new small triangle.

Step4 call the LOP optimization approach to update all thetriangle generated by Step2.

Step5 repeat Step2 to Step4 until all other points are toprocess.

Step6delete the triangles which contains initial trianglevertices to generate Delaunay triangulation network(Fig. 2(f)).

Table 1 Basic steps of incremental insertion algorithm

(a) build an initial triangle (b) generate an initial triangulation network

(c) insert the second feature point (d) insert the third feature point

(e) insert the n-th point by iteration (f) Delaunay triangulation networkFig. 2. Incremental insertion algorithm schematic diagram

(2) LOP (Local Optimization Procedure): LOP is based on thenature of maximum-minimum angle, which means that theminimum of the six angles won’t be increased by exchangingtwo diagonals of a convex quadrilateral. So we can usually usethe related properties of angle of circumference to judgewhether the minimum of the six angles changes or not. Asshown in Fig. 3(a), when the point D locates at thecircumscribed circle of the triangle ABC, it can meet thecondition of ∠A+∠D=π and sin(∠A+∠D)=0. So we couldnot exchange two diagonals of a convex quadrilateral. In Fig.3(b), when the point D lies outside of the circumscribed circleof the triangle ABC, two diagonals would not be alsoexchanged due to meet the condition of ∠A+∠D<π andsin(∠A+∠D)>0. When the point D locates in thecircumscribed circle as shown in Fig. 3(c), we can exchangetwo diagonals to meet this nature of minimum angle.

(a) Point D locates at the circle (b) Point D locates outside of the circle

(c) Point D locates in the circleFig. 3. LOP judgment of three cases

2.4.2 Triangle similarity function: Triangles generated bythe incremental insertion algorithm in triangulation of bothimages, have the great geometric similarity. Then we caneliminate the mismatched points via this similarity of triangle.

For the two similar triangles of ABC and A’B’C’ (A correspondsto A’, similarly for B and C), the similarity Ia of ∠A (its valueis a) and ∠A’ (its value is x) is defined as:

)))(1(2

(cos3 xdIa (7)

where d(x) equals to exp{-(x-a)2/22}, equals to a/6.

For a pair of triangles, the similarity Ii of other two angles canalso be calculated by equation (7). So the similarity of bothtriangles can be indicated as equation (8).

3/)( cba IIII (8)



181

In this paper, we search and select the triangles which aregreater than 0.75.

2.4.3 Projective invariant: According to the projectionrelation of the two images, projective invariant can be used tojudge whether points remain the same nature and quantity afterthe projective transformation. Based on the above process,projective invariant process can make the two images to obtainfine matching. In this paper, cross-ratio is used to analyze a pairof triangles from two images. As stated above, the cross-ratio ofstraight line to point A and the corresponding A’ is taken as anexample, as shown in Fig. 4.

(a) Point A triangle and its collinear adjacent triangle

(b) Point A’ triangle and its collinear adjacent triangleFig. 4. The triangle and the collinear adjacent triangle

From Fig. 4(a) and Fig. 4(b), the cross-ratio of point Arepresents IA=(sin∠FAC*sin∠BAE)/(sin∠FAE*sin∠BAC),and the cross-ratio of point A’ expressesIA’=(sin∠F’A’C’*sin∠B’A’E’)/(sin∠F’A’E’*sin∠B’A’C’). Thesestraight lines which consist triangles are regarded as the rightmatch, only if IA equals to IA’. Finally, the feature points whichcompose of these straight lines, are accurately extracted fromboth images.

3. EXPERIMENTAL RESULTS

In order to verify the reliability and advantage of the proposedmethod, three reference methods (SURF+RANSAC, CC_SURF,DO_SURF) and the proposed approach are compared to analyzetheir performance and the obtained results. Then evaluationcriterion which consists of correct matches number (N) (Li et al.,2015; Ma et al., 2017) and root mean square error (Gong et al.,2014; Kupfer et al., 2015; Ma et al., 2017), is used to verify theaccuracy and robustness of the proposed method. In the paper,we selected the two images with different perspectives that bothimages as shown Fig. 5 (Zheng et al., 2015). These fourmethods stated above are used to perform and operate matchingprocess of the both image, and the corresponding results offeature points detection, feature points matching and matchingresults are shown in Fig. 6, Fig. 7 and Fig. 8, respectively. The

corresponding value of N and RMSE are shown in Table 2,where Fig. 2 only lists the results of feature points detectionwith Image I22.

(a) Image I11 (b) Image I22Fig. 5. Original images (Zheng et al., 2015)

(a) SURF+RANSAC (b) CC_SURF

(c) DO_SURF (d) Proposed approachFig. 6. Feature points detection of three methods




182

(c) DO_SURF (d) Proposed approachFig. 7. Feature matching of three methods


(c) DO_SURF (d) Proposed approachFig. 8. Matching results of three methods

Methods N RMSE (pixel)SURF+RANSAC 424 0.9253

CC_SURF 166 0.8647DO_SURF 323 08983

Proposed method 893 0.7507Table 2 Comparison of N and RMSE

From Fig. 6, we can see that the proposed method detectedabout ten times feature points more than other three methodsdue to the more rich color information obtained by colorinvariant transformation and information entropy. Fig. 7 and Fig.8 show that all four methods greatly removed many outliers andremain correct matches point pairs. However, the number ofcorrect matches obtained by the proposed method, are morethan those by three comparison methods, and the evenness ofmatch points obtained by the proposed approach, are better thanthe others. In addition, as shown in Table 2, the RMSE value of

SURF+RANSAC, CC_SURF and DO_SURF are larger thanthe proposed approach. On the contrary, the Correct MatchesNumber (N) of SURF+RANSAC, CC_SURF and DO_SURFare less than the proposed approach. It can be concluded that theproposed approach outperforms the three comparison methodsin terms of the correct matches number and matching accuracy.

4. CONCLUSION

In the paper, we propose a method based on SURF to improvedetecting and matching performance. The proposed methodintroduces color invariant transformation and informationentropy to greatly maintain color information and detect morefeature points. Then a series of constraint conditions includingDelaunay triangulation, triangle similarity function andprojective invariant, are used to filter out the mismatch featurepoints, and are to ensure a high correct matching accuracy ofmatching results. The above experiments also showed that therobustness and precision of the proposed approach are superiorto the others three methods.

ACKNOWLEDGEMENTS

This paper was substantially supported by the Research on thekey techniques in Spatial Identity-Coding of GeographicalName and Address (Project No. 7771810), the Fujian ProvinceNatural Science Foundation of China (Project Nos. 2018J0106and 2017J01662) and Fujian Province Administrtion ofServeying, Mapping and Geoinformation Natural ScienceFoundation of China ( 2017JX06)..

REFERENCES

Anzid, H., Le Goic, G., Bekkari, A., Mansouri, A., andMammass, D., 2017. An automatic filtering algorithm forSURF-based registration of remote sensing images. In: 3rdInternational Conference on Advanced Technologies for Signaland Image Processing (ATSIP2017), Morroco, Venezuela, pp.1-7.

Arévalo, V., and González, J., 2008. An experimentalevaluation of non-rigid registration techniques on Quickbirdsatellite imagery. International Journal of Remote Sensing,29(2), pp. 513-527.

Bay, H., Tuytelaars, T. and Van G. L., 2006. Surf: Speeded uprobust features. In European conference on computer visionSpringer, Berlin, Heidelberg, pp. 404-417.

Besl, P. J., and McKay, N. D., 1992. Method for registration of3-D shapes. In: International Society for Optics and Photonics,Sensor Fusion IV: Control Paradigms and Data Structures, Vol.1611, pp. 586-607.

Gong, M., Zhao, S., Jiao, L., Tian, D., and Wang, S., 2014. Anovel coarse-to-fine scheme for automatic image registrationbased on SIFT and mutual information. IEEE Transactions onGeoscience and Remote Sensing, 52(7), pp. 4328-4338.

Juan, L., and Gwun, O., 2009. A comparison of sift, pca-sift andsurf. International Journal of Image Processing (IJIP), 3(4), pp.143-152.

Kupfer, B., Netanyahu, N. S., and Shimshoni, I., 2015. Anefficient SIFT-based mode-seeking algorithm for sub-pixel



183

registration of remotely sensed images. IEEE Geoscience andRemote Sensing Letters, 12(2), pp. 379-383.

Lawson, C. L., 1977. Software for C1 surface interpolation. InMathematical software, pp. 161-194.

Lee, S. R., 2010. A Coarse-to-Fine Approach for Remote-Sensing Image Registration Based on a Local Method.International Journal on Smart Sensing & Intelligent Systems,3(4), pp. 690-702.

Li, Q., Qi, S., Shen, Y., Ni, D., Zhang, H., and Wang, T., 2015.Multispectral image alignment with nonlinear scale-invariantkeypoint and enhanced local feature matrix. IEEE Geoscienceand Remote Sensing Letters, 12(7), pp. 1551-1555.

Li S. K. and Zhang P., 2009. Image matching algorithm basedon delaunay triangulation and projective invariant.Microcomputer Information, 25(28), pp. 115-117.

Lowe, D. G., 2004. Distinctive image features from scale-invariant keypoints. International journal of computer vision,60(2), pp. 91-110.

Ma, W., Wen, Z., Wu, Y., Jiao, L., Gong, M., Zheng, Y., andLiu, L., 2017. Remote sensing image registration with modifiedsift and enhanced feature matching. IEEE Geoscience andRemote Sensing Letters, 14(1), pp. 3-7.

Mikolajczyk, K., and Schmid, C., 2005. A performanceevaluation of local descriptors. IEEE transactions on patternanalysis and machine intelligence, 27(10), pp. 1615-1630.

Teke, M., Vural, M. F., Temizel, A., and Yardimci, Y., 2011.High-resolution multispectral satellite image matching usingscale invariant feature transform and speeded up robust features.Journal of Applied Remote Sensing, 5(1), pp. 053553:1-9.

Wu, Y., Ma, W., Gong, M., Su, L., and Jiao, L., 2015. A novelpoint-matching algorithm based on fast sample consensus forimage registration. IEEE Geoscience and Remote SensingLetters, 12(1), pp. 43-47.

Zheng, S. Z., Cheng, P. G., Chen, X. Y., and Chen, S. J., 2015.An approach of image matching combining with surf anddelaunay triangulation. Science of Surveying & Mapping. 40(2),pp. 155-159.

Zitova, B., and Flusser, J., 2003. Image registration methods: asurvey. Image and vision computing, 21(11), pp. 977-1000.



184

ANIMPROVEDIMAGEMATCHINGMETHODBASEDONSURFALGORITHM · (a)surf+ransac (b)cc_surf The International...

Documents

Transcript of ANIMPROVEDIMAGEMATCHINGMETHODBASEDONSURFALGORITHM · (a)surf+ransac (b)cc_surf The International...