Graph clustering using heat kernel embedding and spectral geometry

Hewayda ElGhawalby and Edwin R. Hancock

Department of Computer Science, University of York, YO10 5DD, UK{howaida, erh}@cs.york.ac.uk

Abstract. In this paper we study the manifold embedding of graphs resulting from the Young-Householder decomposition of the heat kernel. We aim to explore how the sectional curvature associated with the embedding can be used as feature for the purposes of gauging the similarity of graphs, and hence clustering them. The curvature is computed from the difference between the geodesic (edge weight) and the Euclidean embedding distance between pairs of nodes. To gauge the similarity of pairs of graphs, we require a means of comparing sets of such features without explicit correspondences between the nodes of the graphs being considered. To this end, the Hausdorff distance, and a robust modified variant of the Hausdorff distance are used. We experiment on sets of graphs representing the proximity image features in different views of different objects. By applying multidimensional scaling to the Hausdorff distances between the different object views, it is demonstrated that our sectional curvature representation is capable of clustering the different views of the same object together.

Keywords: Heat kernel, Hausdorff distance, Sectional curvature, Graph matching.

1 Introduction

Spectral graph theory ([1], [2], [11]) is a branch of mathematics that is concerned with characterizing the structural properties of graphs using the eigenvectors of the adjacency matrix or the closely related Laplacian matrix (the degree matrix minus the adjacency matrix). The use of spectral graph theory in computer vision and pattern recognition is a recent development, and has proved to be a powerful tool for image segmentation and object recognition ([9], [12], [14]). In fact the discovery of shape or object classes can be posed as one of clustering spectral features extracted from graph representations that abstract image structure ([15], [16]).

The starting point for most graph-spectral methods is the Laplacian matrix, i.e. the degree matrix minus the adjacency matrix. The Laplacian matrix is positive semi-definite and the multiplicity of the zero eigenvalue gives the number of connected components of the graph. The eigenvector associated with the second smallest eigenvalue can be used to bipartition the nodes of the graph into disjoint subsets of nodes, and his is the basis of a number of data clustering algorithms ([13]). Recently, the spectrum of the Laplacian matrix has been used to embed the nodes of a graph into a vector space ([7]). In this space the clustering of nodes can be effect using

standard geometric techniques such as k-means. Closely related to the Laplacian spectrum is the heat equation. According to the heat equation the heat kernel can be found by exponentiating the spectrum of the Laplacian matrix with time. The heat kernel is a compact representation of the path length distribution on a graph, and determines information diffusion along edges of the graph with time. An embedding of the nodes of a graph into a vector space may also be performed using a Young-Householder decomposition [17] of the heat kernel. This embedding offers the advantage that the time parameter can be used to control the condensation of clusters. If the nodes of a graph as viewed as residing on a manifold, then the Laplacian matrix may be regarded as the discrete approximation to the Laplace-Beltrami curvature operator for the manifold. In the mathematics literature the study of the eigenvalues and eigenvectors of the Laplace-Beltrami operator is referred to as spectral geometry. In the manifold learning literature [8] techniques from spectral geometry have recently been used to analyze the properties of the Laplacian embedding.

In this paper we aim to explore the geometry associated with the heat-kernel embedding, and in particular to investigate whether the geometric attributes of the edges can be used as features for the purposes of gauging the similarity of graphs, and hence clustering them. To this end we aim to associate with the edges of a graph a sectional curvature. Nodes connected by an edge have unit geodesic distance. The Euclidean distance between nodes, on the other hand, is given by the L2 norm associated with the heat kernel embedding. By computing the difference between the geodesic and Euclidean distances, we can estimate the sectional curvature associated with the heat kernel embedding of the graph. We treat the sectional curvatures associated with the edges of the graph as features or measurements.

To gauge the similarity of pairs of graphs, we require a means of comparing sets of such features without requiring explicit correspondences between the nodes of the graphs being considered. To this end we make use of the Hausdorff distance, and a robust modified variant of the Hausdorff distance. It avoids the need for correspondence information by associating elements that maximizes the sum of minimum distances between elements. With the Hausdorff distances to hand, we experiment on sets of graphs representing the proximity image features in different views of different objects. By applying multidimensional scaling to the Hausdorff distances between the different object views, we demonstrate that our sectional curvature representation is capable of clustering the different views of the same object together.

2 Heat kernel and curvature on graphs

We commence by letting the graph under study to be denoted by G= (V, E) where V is the set of nodes and E V V is the set of edges. The elements of the adjacency matrix A of the graph G is defined by:

()

To construct the Laplacian matrix we first establish a diagonal degree matrix D with elements

()

From the degree and the adjacency matrices we can construct the Laplacian matrix L=D-A, that is the degree matrix minus the adjacency matrix.

()

The normalized Laplacian is given by . The spectral decomposition of

the normalized Laplacian matrix is where is the

number of nodes and , is the

diagonal matrix with the ordered eigenvalues as elements and is the matrix with the eigenvectors as columns. The heat kernel plays an important role in spectral graph theory. It encapsulates the way in which information flows through the edges of graph over time under the heat equation, and is the solution of the partial differential equation

()

where is the heat kernel and t is the time. The solution is found by exponentiating the Laplacian eigenspectrum as follows

()The heat-kernel can be used to embed the nodes of a graph in a vector-space. The matrix of embedding co-ordinates Y (i.e. the matrix whose columns are the vectors of node co-ordinates), is found by performing the Young-Householder decomposition

()

As a result the matrix of node embedding co-ordinates is

()

where is the co-ordinate vector for the node . In the vector-space, the Euclidean distance between the nodes u and v of the graph is

()

Recall that the number of paths of length n between nodes u and v of the graph are given by the element of the matrix As a result the geodesic distance between nodes u and v is given by

()

When a pair of nodes are connected by an edge, then .We are now in position to compute the sectional curvature associated with an edge

using the difference in geodesic and Euclidean distance. To estimate the sectional curvature we assume that the geodesic between the pair of nodes can be approximated by a segment of circle whose arc-length is the geodesic distance and whose chord-length is the Euclidean distance. If the radius of the circle is and that the tangent-vector to the circle arc undergoes a change in direction of , then we have

()and

()Approximating the sine using the McClaurin series in (11) and substituting for

from (10), we obtain

()

Solving this equation for the radius of circle, the squared sectional curvature of the edge is

()

Since for an edge of the graph , we have

()

3 Hausdorff distance

The Hausdorff distance is one of the most studied similarity measures, and it is defined on a large class of shapes. It provides a means of computing the distance between sets of unordered observations when the correspondences between the individual items are unknown. In its most general setting, the Hausdorff distance is defined between compact sets in a metric space. Given two such sets, we consider for each point in one set the closest point in the other set, and the Hausdorff distance is the maximum over all these values. More formally, the classical Hausdorff distance (HD) [10] between two finite point sets A and B is given by

()where the directed Hausdorff distance from A to B is defined to be

()

and is some underlying norm on the points of A and B (e.g., the L2 or Euclidean norm). Dubuisson and Jain [5] proposed a robust modified variant of the Hausdorff distance called the modified Hausdorff distance (MHD) based on the average distance

value instead of the maximum value, in this sense they defined the directed distance of the MHD as

()

using these notations we can describe how Hausdorff distances can be extended to graph-based representations, to commence let us consider two graphs and , where , are the sets of nodes, , the sets of edges and , the matrices whose elements are the curvature defined in the previous section. We can now write the distances between the two graphs as follows: 1) The classic Hausdorff distance (HD)

()

2) The modified Hausdorff distance (MHD)

()

4 Multidimensional scaling

Having in hand the distances matrix, we need now a way to visualize it. We choose to use the classical Multidimensional scaling method [3] to embed the data specified in the matrix in a Euclidean space. Let H be the distance matrix with entries . The first step of MDS is to calculate a matrix T whose element with row r and column c is

given by where is the

average value over the rth row in the distance matrix, is the similarly defined

average value over the cth column and is the average value

over all rows and columns of the distance matrix. Then, we subject the matrix T to an eigenvector analysis to obtain a matrix of embedding co-ordinates X. If the rank of T is ; , then we will have non-zero eigenvalues. We arrange these k non-zero eigenvalues in descending order, i.e. . The corresponding ordered eigenvectors are denoted by where is the ith eigenvalue. The embedding co-

ordinate system for the graphs is for the graph

indexed i, the embedded vector of co-ordinates is .

5 Experiments

In our experiments we use the standard CMU, MOVI and chalet-house sequences as our dataset [10]. These data-sets contain different views of model houses from equally spaced viewing directions. From the house images, corner features are extracted, and Delaunay graphs representing the arrangement of feature points are constructed. Our data consists of ten graphs for each of the three houses. Example images and graphs are shown in Fig 1.

Fig. 1. Sample images from the house image sequences (from left to right CMU/VASC, INRIA MOVI and Swiss chalet) with the extracted graphs below.

To commence, we compute the Euclidean distances between the nodes in each graph based on the heat kernel. We then compute the curvature matrix with the values of t= 10.0, 1.0, 0.1 and 0.01. From the curvatures, we compute distance matrices for the thirty different graphs using the classical Hausdorff distance and the modified Hausdorff distance. Finally, we subject the distance matrices to the MDS procedure to embed them into a 2D space. Here each graph is represented by a single point.

Figure (2) shows the results obtained when embedding the distances matrix obtained using the classical Hausdorff distance. The subfigures are ordered from left to right, using t= 10, 1, 0.1, 0.01 in the heat kernel. Figure (3) shows the corresponding results obtained when the modified Hausdorff distance is used.

Fig. 2. MDS embedding obtained using HD.

Fig. 3. MDS embedding obtained using MHD.

When the Euclidean distances are computed directly from the Laplacian embedding (i.e. ), then the results shown in Figure (4) are obtained. Here the left-hand subfigure is obtained when using HD and the right-hand one when using MHD.

Fig. 4. MDS embedding obtained using curvatures computed from the Laplacian eigenmap and distances computed using HD (left) and MHD (right).

There are two conclusions that can be drawn from this experimental study. First, the heat-kernel embedding distinguishes the different classes better than the Laplacian eigenmap. Second, MHD gives better clusters than MD.

6 Conclusion and Future Work

In this paper we used the sectional curvatures associated with the edges of a graph under the heat-kernel embedding as features. By computing the modified Hausdorff distances between sets of edge curvatures we are able to gauge the similarity of graphs, and by applying MDS to these similarities we are able to cluster graphs into different object categories. A comparison shows that the curvature attributes computed from the heat-kernel embedding give better clusters than those obtained from the Laplacian eigenmap

Our future plans are as follows. First, we intend to investigate whether the spectrum of the edge curvature matrix can be used to compute permutation invariants. Second, we aim to use the sectional curvatures to specify the geometry of a manifold. Third, we aim to explore the use of additional curvature attributes, such as torsion, together with the Gauss-Bonnet theorem to characterize graphs.

7 References

1. Biggs, N. L.: Algebraic graph theory. Cambridge University Press (1993).

2. Chung, Fan R. K.: Spectral Graph Theory. American Mathematical Society. ISSN 0160-7642; no 92 (1997).

3. Cox, T., Cox, M.: Multidimensional Scaling. Chapman-Hall (1994).4. Do Carmo, M.: Differential Geometry of Curves and Surfaces. Prentice-Hall,

Englewood Cliffs, NJ (1976).5. Dubuisson, M., Jain, A.: A modified Hausdorff distance for object matching. In Proc.

12th Int. Conf. Pattern Recognition, pp. 566–568, Jerusalem, Israel (1994).6. Grigor’yan, A.: Heat kernels on manifolds, graphs and fractals. European Congress of

Mathematics, I, pp. 393–406 (2001).7. He, X., Yan, S., Hu, Y., Niyogi, P., Zhang, H.: Face Recognition Using

Laplacianfaces. IEEE Trans. Pattern Anal. Mach. Intell., 27(3), pp. 328-340 (2005).8. Hein, M., Audibert, J., Von Luxburg, U.: From graphs to manifolds—weak and

strong pointwise consistency of graph Laplacians. In Proceedings of the 18th Conference on Learning Theory, pp. 470–485 (2005).

9. Horaud, R., Sossa, H.: Polyhedral object recognition by indexing. Pattern Recognition, 28(12), pp. 1855–1870 (1995).

10. Huttenlocher, D., Klanderman, G. and Rucklidge, W.: Comparing images using the Hausdorff distance. IEEE Trans. Pattern Anal. Machine Intell., 15, pp. 850–863, (1993).

11. Sachs, H., Cvetkovic, D. M., Doob, M.: Spectra of graphs. Academic Press (1980).12. Sengupta, K., Boyer, K.: Modelbase partitioning using property matrix spectra.

Computer Vision and Imaging Understanding, 70(2), 177–196 (1998).13. Shi, J., Malik, J.: Normailzed cuts and image segmentation. IEEE PAMI, 22, pp. 888-

905 (2000).14. Shokoufandeh, A., Dickinson, S., Siddiqi, K., Zucker, S.: Indexing using a spectral

encoding of topological structure. International Conference on Computer Vision and Pattern Recognition, (1999)..

15. Luo, B, Wilson, R.C., Hancock, E.R..: Spectral embedding of graphs. Pattern recognition, 36(10), pp. 2213-2230 (2003).

16. Wilson, R. C., Hancock, E. R., Luo, B.: Pattern Vectors from Algebraic Graph Theory. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(7) pp. 1112-1124 (2005).

17. Xiao, B., Hancock, E.R..: Heat kernel, Riemannian manifolds and Graph Embedding. LNCS 3138, pp. 198–206 (2004).

18. Young, G., Householder, A. S.: Discussion of a set of points in terms of their mutual distances. Psychometrika, 3(1), pp. 19–22 (1938).