CLUSTERING UNCERTAIN DATA BASED ON PROBABILITY ...

CLUSTERING UNCERTAIN DATA BASED ON PROBABILITY DISTRIBUTION SIMILARITY 1

Clustering Uncertain DataBased on Probability Distribution Similarity

Bin Jiang, Jian Pei, Yufei Tao, and Xuemin Lin

Abstract—Clustering on uncertain data, one of the essential tasks in mining uncertain data, posts significant challenges onboth modeling similarity between uncertain objects and developing efficient computational methods. The previous methodsextend traditional partitioning clustering methods like k-means and density-based clustering methods like DBSCAN to uncertaindata, thus rely on geometric distances between objects. Such methods cannot handle uncertain objects that are geometricallyindistinguishable, such as products with the same mean but very different variances in customer ratings. Surprisingly, probabilitydistributions, which are essential characteristics of uncertain objects, have not been considered in measuring similarity betweenuncertain objects. In this paper, We systematically model uncertain objects in both continuous and discrete domains, wherean uncertain object is modeled as a continuous and discrete random variable, respectively. We use the well known Kullback-Leibler divergence to measure similarity between uncertain objects in both the continuous and discrete cases, and integrate itinto partitioning and density-based clustering methods to cluster uncertain objects. Nevertheless, a naıve implementation is verycostly. Particularly, computing exact KL divergence in the continuous case is very costly or even infeasible. To tackle the problem,we estimate KL divergence in the continuous case by kernel density estimation and employ the fast Gauss transform techniqueto further speed up the computation. Our extensive experiment results verify the effectiveness, efficiency, and scalability of ourapproaches.

Index Terms—Clustering, Uncertain data, Probabilistic distribution, Density estimation, Fast Gauss Transform

✦

1 INTRODUCTIONClustering uncertain data has been well recognizedas an important issue [21] [22] [27] [36]. Generally,an uncertain data object can be represented by aprobability distribution [7] [29] [35]. The problem ofclustering uncertain objects according to their proba-bility distributions happens in many scenarios.

For example, in marketing research, users are askedto evaluate digital cameras by scoring on variousaspects, such as image quality, battery performance,shotting performance, and user friendliness. Eachcamera may be scored by many users. Thus, theuser satisfaction to a camera can be modeled as anuncertain object on the user score space. There areoften a good number of cameras under a user study. Afrequent analysis task is to cluster the digital camerasunder study according to user satisfaction data.

One challenge in this clustering task is that weneed to consider the similarity between cameras notonly in terms of their score values, but also theirscore distributions. One camera receiving high scoresis different from one receiving low scores. At the same

• Bin Jiang and Jian Pei are with the School of Computing Science,Simon Fraser University, Canada.E-mail: {bjiang, jpei}@cs.sfu.ca

• Yufei Tao is with the Department of Computer Science and Engineer-ing, Chinese University of Hong Kong, China.E-mail: [email protected]

• Xuemin Lin is with the School of Computer Science and Engineering,The University of New South Wales, Australia and the School ofSoftware, East China Normal University, China.E-mail: [email protected]

time, two cameras, though with the same mean score,are substantially different if their score variances arevery different.

As another example, a weather station monitorsweather conditions including various measurementslike temperature, precipitation amount, humidity,wind speed and direction. The daily weather recordvaries from day to day, which can be modeled asan uncertain object represented by a distribution overthe space formed by several measurements. Can wegroup the weather conditions during the last monthfor stations in North America? Essentially we needto cluster the uncertain objects according to theirdistributions.

1.1 Limitations of Existing WorkThe previous studies on clustering uncertain data arelargely various extensions of the traditional clusteringalgorithms designed for certain data. As an object ina certain data set is a single point, the distribution re-garding the object itself is not considered in traditionalclustering algorithms. Thus, the studies that extendedtraditional algorithms to cluster uncertain data arelimited to using geometric distance-based similaritymeasures, and cannot capture the difference betweenuncertain objects with different distributions.

Specifically, three principal categories exist in liter-ature, namely partitioning clustering approaches [27][18] [24], density-based clustering approaches [21][22], and possible world approaches [36]. The first twoare along the line of the categorization of clustering

Digital Object Indentifier 10.1109/TKDE.2011.221 1041-4347/11/$26.00 © 2011 IEEE

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERINGThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication.


methods for certain data [14], the possible worldapproaches are specific for uncertain data followingthe popular possible world semantics for uncertaindata [8] [31] [17].

As these approaches only explore the geometricproperties of data objects and focus on instances ofuncertain objects. They do not consider the similaritybetween uncertain objects in terms of distributions.

Let us examine this problem in the three existingcategories of approaches in detail. Suppose we havetwo sets A and B of uncertain objects. The objects inA follow uniform distribution, and those in B followGaussian distribution. Suppose all objects in both setshave the same mean value (i.e., the same center).Consequently, their geometric locations (i.e., areas thatthey occupied) heavily overlap. Clearly, the two setsof objects form two clusters due to their differentdistributions.

Partitioning clustering approaches [27] [18] [24]extend the k-means method with the use of the ex-pected distance to measure the similarity between twouncertain objects. The expected distance between anobject P and a cluster center c (which is a certainpoint) is ED(P, c) =

∫PfP (x)dist(x, c)dx, where fP

denotes the probability density function of P andthe distance measure dist is the square of Euclideandistance. In [24], it is proved that ED(P, c) is equal tothe dist between the center (i.e., the mean) P.c of Pand c plus the variance of P . That is,

ED(P, c) = dist(P.c, c) + V ar(P ). (1)

Accordingly, P can be assigned to the cluster cen-ter argminc{ED(P, c)} = argminc{dist(P.c, c)}. Thus,only the centers of objects are taken into account inthese uncertain versions of the k-means method. Inour case, as every object has the same center, the ex-pected distance-based approaches cannot distinguishthe two sets of objects having different distributions.

Density-based clustering approaches [21] [22] ex-tend the DBSCAN method [10] and the OPTICSmethod [3] in a probabilistic way. The basic ideabehind the algorithms does not change – objects ingeometrically dense regions are grouped together asclusters and clusters are separated by sparse regions.However, in our case, objects heavily overlap. Thereis no clear sparse regions to separate objects into clus-ters. Therefore, the density-based approaches cannotwork well.

Possible world approaches [36] follow the possi-ble world semantics [8] [31] [17]. A set of possibleworlds are sampled from an uncertain data set. Eachpossible world consists of an instance from eachobject. Clustering is conducted individually on eachpossible world and the final clustering is obtainedby aggregating the clustering results on all possibleworlds into a single global clustering. The goal isto minimize the sum of the difference between theglobal clustering and the clustering of every possible

obj 1obj 2

(a)

obj 1obj 2

(b)

Fig. 1. Probability distributions and geometric loca-tions.

world. Clearly, a sampled possible world does notconsider the distribution of a data object since apossible world only contains one instance from eachobject. The clustering results from different possibleworlds can be drastically different. The most prob-able clusters calculated using possible worlds maystill carry a very low probability. Thus, the possibleworld approaches often cannot provide a stable andmeaningful clustering result at the object level, not tomention that it is computationally infeasible due tothe exponential number of possible worlds.

1.2 Our Ideas and ContributionsCan we cluster uncertain objects according to the sim-ilarity between their probability distributions? Sim-ilarity measurement between two probability distri-butions is not a new problem at all. In informationtheory, the similarity between two distributions canbe measured by the Kullback-Leibler divergence(KL divergence for short, also known as informationentropy or relative entropy) [23].

The distribution difference cannot be captured bygeometric distances. For example, in Figure 1 (a),the two objects (each one is represented by a set ofsampled points) have different geometric locations.Their probability density functions over the entiredata space are different and the difference can becaptured by KL divergence. In Figure 1 (b), althoughthe geometric locations of the two objects are heavilyoverlapping, they have different distributions (one isuniform and the other is Gaussian). The differencebetween their distributions can also be discovered byKL divergence, but cannot be captured by the existingmethods as elaborated in Section 1.1.

In this paper, we consider uncertain objects as ran-dom variables with certain distributions. We considerboth the discrete case and the continuous cases. Inthe discrete case, the domain has a finite numberof values, for example, the rating of a camera canonly take a value in {1, 2, 3, 4, 5}. In the continuouscase, the domain is a continuous range of values,for example, the temperatures recorded in a weatherstation are continuous real numbers.

Directly computing KL divergence between proba-bility distributions can be very costly or even infeasi-



ble if the distributions are complex, as will be shownin Section 3. Although KL divergence is meaningful,a significant challenge of clustering using KL diver-gence is how to evaluate KL divergence efficiently onmany uncertain objects.

To the best of our knowledge, this paper is thefirst to study clustering uncertain data objects usingKL divergence in a general setting. We make severalcontributions. We develop a general framework ofclustering uncertain objects considering the distribu-tion as the first class citizen in both discrete andcontinuous cases. Uncertain objects can have anydiscrete or continuous distribution. We show thatdistribution differences cannot be captured by theprevious methods based on geometric distances. Weuse KL divergence to measure the similarity betweendistributions, and demonstrate the effectiveness ofKL divergence in both partitioning and density-basedclustering methods. To tackle the challenge of eval-uating the KL divergence in the continuous case,we estimate KL divergence by kernel density esti-mation and apply the fast Gauss transform to boostthe computation. We conducted experiments on realand synthetic data sets to show clustering uncertaindata in probability distribution is meaningful and ourmethods are efficient and scalable.

The rest of the paper is organized as follows. Sec-tion 2 reviews the related work. In Section 3, wedefine uncertain objects and the similarity using KLdivergence, and show how to evaluate KL divergencein both discrete and continuous cases. In Section 4,we present the partitioning and density-based clus-tering methods using KL divergence. In Section 5, wedevelop implementation techniques to speed up theclustering and introduce the fast Gauss transform toboost the evaluation of KL divergence. We conduct anextensive empirical study in Section 6, and concludethe paper in Section 7.

2 RELATED WORKClustering is a fundamental data mining task. Clus-tering certain data has been studied for years indata mining, machine learning, pattern recognition,bioinformatics, and some other fields [14] [16] [19].However, there is only preliminary research on clus-tering uncertain data.

Data uncertainty brings new challenges to clus-tering, since clustering uncertain data demands ameasurement of similarity between uncertain dataobjects. Most studies of clustering uncertain data usedgeometric distance-based similarity measures, whichare reviewed in Section 2.1. A few theoretical studiesconsidered using divergences to measure the similar-ity between objects. We discuss them in Section 2.2.

2.1 Clustering Based on Geometric DistancesNgai et al. [27] proposed the UK-means method whichextends the k-means method. The UK-means method

measures the distance between an uncertain objectand the cluster center (which is a certain point) by theexpected distance. Recently, Lee et al. [24] showed thatthe UK-means method can be reduced to the k-meansmethod on certain data points due to Equation (1)described in Section 1.

Kriegel et al. [21] proposed the FDBSCAN algorithmwhich is a probabilistic extension of the deterministicDBSCAN algorithm [10] for clustering certain data. AsDBSCAN is extended to a hierarchical density basedclustering method referred to as OPTICS [3], Kriegelet al. [22] developed a probabilistic version of OPTICScalled FOPTICS for clustering uncertain data objects.FOPTICS outputs a hierarchical order in which dataobjects, instead of the determined clustering member-ship for each object, are clustered.

Volk et al. [36] followed the possible world seman-tics [1] [15] [8] [31] using Monte Carlo sampling [17].This approach finds the clustering of a set of sampledpossible worlds using existing clustering algorithmsfor certain data. Then the final clustering is aggregatedfrom those sample clusterings.

As discussed in Section 1, the existing techniqueson clustering uncertain data mainly focus on thegeometric characteristics of objects, and do not takeinto account the probability distributions of objects.In this paper, we propose to use KL divergence asthe similarity measure which can capture distributiondifference between objects. To the best of our knowl-edge, this paper is the first work to study clusteringuncertain objects using KL divergence.

2.2 Clustering Based on Distribution Similarity

We are aware that clustering distributions has ap-peared in the area of information retrieval whenclustering documents [37] [5]. The major difference ofour work is that we do not assume any knowledgeon the types of distributions of uncertain objects.When clustering documents, each document is mod-eled as a multinomial distribution in the languagemodel [30] [34]. For example, Xu et al. [37] discusseda k-means clustering method with KL divergenceas the similarity measurement between multinomialdistributions of documents. Assuming multinomialdistributions, KL divergence can be computed usingthe number of occurrences of terms in documents.Blei et al. [5] proposed a generative model approach –the Latent Dirichlet Allocation (LDA for short). LDAmodels each document and each topic (i.e., cluster)as a multinomial distribution, where a document isgenerated by several topics. However, such multino-mial distribution based methods cannot be applied togeneral cases where the type of distributions are notmultinomial.

There are also a few studies on clustering using KLdivergences.



Dhillon et al. [9] used KL divergence to measuresimilarity between words to cluster words in docu-ments in order to reduce the number of features indocument classification. They developed a k-meanslike clustering algorithm and showed that the algo-rithm monotonically decreases the objective functionas shown in Equation (9), and minimizes the intra-cluster Jensen-Shannon divergence while maximiz-ing inter-cluster Jensen-Shannon divergence. As theirapplication is on text data, each word is a discreterandom variable in the space of documents. Therefore,it is corresponding to the discrete case in our problem.

Banerjee et al. [4] theoretically analyzed the k-meanslike iterative relocation clustering algorithms based onBregman divergences which is a general case of KLdivergence. They summarized a generalized iterativerelocation clustering framework for various similaritymeasures from the previous work from an informa-tion theoretical viewpoint. They showed that findingthe optimal clustering is equivalent to minimizing theloss function in Bregman information correspondingto the selected Bregman divergence used as the under-lying similarity measure. In terms of efficiency, theiralgorithms have linear complexity in each iterationwith respect to the number of objects. However, theydid not provide methods for efficiently evaluatingBregman divergence nor calculating the mean of aset of distributions in a cluster. For uncertain objectsin our problem which can have arbitrary discrete orcontinuous distributions, it is essential to solve thetwo problems in order to scale on large data sets, aswe can see in our experiments.

Ackermann et al. [2] developed a probabilistic (1 +ε)-approximation algorithm with linear time complex-ity for the k-medoids problem with respect to an ar-bitrary similarity measure, such as squared Euclideandistance, KL divergence, Mahalanobis distance, etc., ifthe similarity measure allows the 1-medoid problembeing approximated within a factor of (1 + ε) bysolving it exactly on a random sample of constant size.They were motivated by the problem of compressingJava and C++ executable codes which are modeledbased on a large number of probability distributions.They solved the problem by identifying a good set ofrepresentatives for these distributions to achieve com-pression which involves non-metric similarity mea-sures like KL divergence. The major contribution oftheir work is on developing a probabilistic approxi-mation algorithm for the k-medoids problem.

The previous theoretical studies focused on thecorrectness of clustering using KL divergence [9], thecorrespondence of clustering using Bregman diver-gence in information theory [4], and the probabilis-tic approximation algorithm [2]. However, they didnot provide methods for efficiently evaluating KLdivergence in the clustering process, neither did theyexperimentally test the efficiency and scalability oftheir methods on large data sets. Different to them,

our work aims at introducing distribution differencesespecially KL divergence as a similarity measure forclustering uncertain data. We integrate KL divergenceinto the framework of k-medoids and DBSCAN todemonstrate the performance of clustering uncertaindata. More importantly, particular to the uncertainobjects in our problem, we focus on efficient compu-tation techniques for large data sets, and demonstratethe effectiveness, efficiency, and scalability of ourmethods on both synthetic and real data sets withthousands of objects, each of which has a sample ofhundreds of observations.

3 UNCERTAIN OBJECTS AND KL DIVER-GENCEThis section first models uncertain objects as randomvariables in probability distributions. We considerboth the discrete and continuous probability distribu-tions and show the evaluation of the correspondingprobability mass and density functions in the discreteand continuous cases, respectively. Then, we recallthe definition of KL divergence, and formalize thedistribution similarity between two uncertain objectsusing KL divergence.

3.1 Uncertain Objects and Probability Distribu-tionsWe consider an uncertain object as a random variablefollowing a probability distribution in a domain D.We consider both the discrete and continuous cases.

If the domain is discrete (e.g., categorical) witha finite or countably infinite number of values, theobject is a discrete random variable and its probabilitydistribution is described by a probability mass function(pmf for short). Otherwise if the domain is continuouswith a continuous range of values, the object is acontinuous random variable and its probability distribu-tion is described by a probability density function (pdffor short). For example, the domain of the ratings ofcameras is a discrete set {1, 2, 3, 4, 5}, and the domainof temperature is continuous real numbers.

In many case, the accurate probability distributionsof uncertain objects are not known beforehand inpractice. Instead, the probability distribution of an un-certain object is often derived from our observationsof the corresponding random variable. Therefore, weassociate each object with a sample of observations,and assume that the sample is finite and the obser-vations are independent and identically distributed(i.i.d. for short).

By overloading the notation, for an uncertain objectP , we still use P to denote the corresponding randomvariable, the probability mass/density function, andthe sample.

For discrete domains, the probability mass functionof an uncertain object can be directly estimated by



normalizing the number of observations against thesize of the sample. Formally, the pmf of object P is

P (x) =

∣∣{p ∈ P |p = x}∣∣|P | , (2)

where p ∈ P is an observation of P and | · | is thecardinality of a set.

For continuous domains, we estimate the probabil-ity density function of an uncertain object by kerneldensity estimation.

Kernel density estimation [33] [32] is a non-parametric way of estimating the probability densityfunction of a continuous random variable. Given asample of a continuous random variable P , the kerneldensity estimate of the probability density function isthe sum of |P | kernel functions. In this paper, we usethe popular Gaussian kernels.

Each Gaussian kernel function is centered at a sam-ple point p ∈ P with variance h. h is called the band-width, and is used to control the level of smoothing.A popular choice of the bandwidth is the Slivermanapproximation rule [33] for which h = 1.06× δ|P |− 1

5 ,where δ is the standard deviation of the sample pointsof P . In 1-dimensional case, the density estimator is

P (x) =1

|P |√2πh

∑p∈P

e−(x−p)2

2h2 .

For the d-dimensional (d ≥ 2) case, the kernel functionis the product of d Gaussian functions, each with itsown bandwidth hj (1 ≤ j ≤ d), and the densityestimator is

P (x) =1

|P |(2π)d/2 ∏dj=1 hj

∑p∈P

d∏j=1

e−

(x.Dj−p.Dj)2

2h2j , (3)

where we denote a d-dimensional point p by(p.D1, . . . , p.Dd).

3.2 KL DivergenceIn general, KL divergence between two probabilitydistributions is defined as follows,

Definition 1 (Kullback-Leibler divergence [23]): In thediscrete case, let f and g be two probability massfunctions in a discrete domain D with a finite orcountably infinite number of values. The Kullback-Leibler diverge (KL divergence for short) between fand g is

D(f ‖g) =∑x∈D

f(x) logf(x)

g(x). (4)

In the continuous case, let f and g be two probabil-ity density functions in a continuous domain D witha continuous range of values. The Kullback-Leiblerdivergence between f and g is

D(f ‖g) =∫D

f(x) logf(x)

g(x)dx. (5)

In both discrete and continuous cases, KL diver-gence is defined only in the case where for any x indomain D if f(x) > 0 then g(x) > 0. By convention,0 log 0

p = 0 for any p �= 0 and the base of log is 2.Note that, KL divergence is not symmetric in gen-

eral, that is, D(f ‖g) �= D(g‖f).

3.3 Using KL Divergence as SimilarityIt is natural to quantify the similarity between two un-certain objects by KL divergence. Given two uncertainobjects P and Q and their corresponding probabilitydistributions, D(P ‖ Q) evaluates the relative uncer-tainty of Q given the distribution of P . In fact, fromEquations (4) and (5), we have

D(P ‖Q) = E[log

P

Q

], (6)

which is the expected log-likelihood ratio of the twodistributions and tells how similar they are. TheKL divergence is always non-negative, and satisfiesGibbs’ inequality. That is, D(P ‖Q) ≥ 0 with equalityonly if P = Q. Therefore, the smaller the KL diver-gence, the more similar the two uncertain objects.

In the discrete case, it is straightforward to evaluateEquation (4) to calculate the KL divergence betweentwo uncertain objects P and Q from their probabilitymass functions calculated as Equation (2).

In the continuous case, given the samples of P andQ, by the law of large numbers and Equation (6), wehave

lims→∞

1

s

s∑i=1

logP (pi)

Q(pi)= D(P ‖Q),

where we assume the sample of P = {p1, . . . , ps}.Hence, we estimate the KL divergence D(P ‖Q) as

D(P ‖Q) =1

s

s∑i=1

logP (pi)

Q(pi). (7)

It is important to note that the definition of KLdivergence necessitates that for any x ∈ D if P (x) > 0then Q(x) > 0. To ensure that the KL divergenceis defined between every pair of uncertain objects,we smooth the probability mass/density function ofevery uncertain object P so that it has a positiveprobability to take any possible value in the domain.In other words, we assume that there is always a smallnon-zero probability for an uncertain object to observeany unseen value. As in the camera rating example,if we only observe ratings 2, 3, 4, and 5 of a camera,but not 1, we still assume that the camera has a smallprobability to be rated as 1, even we have no suchobservations.

The smoothing is based on the following two as-sumptions about the uncertain objects to be clusteredin our problem.

1) We assume that the probability distribution ofevery uncertain object to be clustered is definedin the same domain D.



2) We assume that the domain D is bounded.If D is discrete, we assume it has a finite number ofvalues. For the continuous domain, we only considerD as a bounded range of values. In most applications,possible values are within a sufficiently large boundedrange, as we can use the lowest and highest recordedvalues to define the range. For example, cameras canbe rated at 5 possible grades, and temperatures onEarth’s surface usually range between −90◦C to 60◦C.

We smooth a probability distribution P as follows,

P (x) =P (x) + δ

1 + δ |D| , (8)

where 0 < δ < 1, |D| is the number of possible valuesin D if D is discrete and the area of D (i.e., |D| = ∫

Ddx)

if D is continuous.Clearly, the sum/integral of P (x) over the entire

domain remains 1. For two uncertain objects P andQ, after smoothing, P (x) = Q(x) for any x ∈ D stillholds if and only if P (x) = Q(x).

The error incurred by such an approximation is

∣∣∣P (x)− P (x)∣∣∣ = ∣∣∣∣1− P (x) |D|

1/δ + |D|∣∣∣∣ ∈

[0,

max{1, ∣∣1− |D| ∣∣}1/δ + |D|

].

As the domain D is bounded, |D| is bounded. Bychoosing a sufficient small δ, we can provide anapproximation at arbitrary accuracy level.

In summary of this section, we model every un-certain object to be clustered as a random variablein the same bounded domain D and represent itwith a sample of i.i.d. observations. The probabil-ity distribution of an uncertain object is estimatedfrom its sample by Equation (2) in the discrete caseand Equation (3) using kernel density estimation inthe continuous case. In both cases, the probabilitymass/density function is smoothed by Equation (8).

We define the similarity between two uncertainobjects as the KL divergence between their probabil-ity distributions. The KL divergence is calculated byEquations (4) and (7) in the discrete and continuouscases, respectively.

4 CLUSTERING ALGORITHMSAs the previous geometric distance-based clusteringmethods for uncertain data mainly fall into two cate-gories, partitioning and density-based approaches. Inthis section, we present the clustering methods usingKL divergence to cluster uncertain objects in these twocategories. In Section 4.1, we present the uncertain k-medoids method which extends a popular partition-ing clustering method k-medoids [19] by using KL di-vergence. Then, we develop a randomized k-medoidsmethod based on the uncertain k-medoids methodto reduce the time complexity. Section 4.2 presentsthe uncertain DBSCAN method which integrates KLdivergence into the framework of a typical density-based clustering method DBSCAN [10]. We describe

the algorithms of the methods and how they use KLdivergence as the similarity measure.

4.1 Partitioning Clustering MethodsA partitioning clustering method organizes a set of nuncertain objects O into k clusters C1, · · · ,Ck, suchthat Ci ⊆ O (1 ≤ i ≤ k), Ci �= ∅,

⋃ki=1 Ci = O,

and Ci ∩ Cj = ∅ for any i �= j. We use Ci to denotethe representative of cluster Ci. Using KL divergenceas similarity, a partitioning clustering method tries topartition objects into k clusters and chooses the best krepresentatives, one for each cluster, to minimize thetotal KL divergence as below,

TKL =k∑

i=1

∑P∈Ci

D(P ‖Ci). (9)

For an object P in cluster Ci (1 ≤ i ≤ k), the KL diver-gence D(P ‖Ci) between P and the representative Ci

measures the extra information required to constructP given Ci. Therefore,

∑P∈Ci

D(P ‖Ci) captures thetotal extra information required to construct the wholecluster Ci using its representative Ci. Summing overall k clusters, the total KL divergence thus measuresthe quality of the partitioning clustering. The smallerthe value of TKL, the better the clustering.

k-means [25] [26] and k-medoids [19] are two classi-cal partitioning methods. The difference is that the k-means method represents each cluster by the mean ofall objects in this cluster, while the k-medoids methoduses an actual object in a cluster as its representative.In the context of uncertain data where objects areprobability distributions, it is inefficient to computethe mean of probability density functions. k-medoidsmethod avoids computing the means. For the sakeof efficiency, we adopt the k-medoids method todemonstrate the performance of partitioning cluster-ing methods using KL divergence to cluster uncertainobjects.

In Section 4.1.1, we first present the uncertain k-medoids method which integrates KL divergence intothe original k-medoids method. Then, we developa randomized k-medoids method in Section 4.1.2 toreduce the complexity of the uncertain one.

4.1.1 Uncertain K-Medoids MethodThe uncertain k-medoids method consists of twophases, the building phase and the swapping phase.

Building Phase: In the building phase, the uncer-tain k-medoids method obtains an initial clustering byselecting k representatives one after another. The firstrepresentatives C1 is the one which has the smallestsum of the KL divergence to all other objects in O.That is,

C1 = argminP∈O

⎛⎝ ∑

P ′∈O\{P}

D(P ′ ‖P )

⎞⎠ .



The rest k − 1 representatives are selected iteratively.In the i-th (2 ≤ i ≤ k) iteration, the algorithm selectsthe representative Ci which decreases the total KLdivergence (Equation (9)) as much as possible. Foreach object P which has not been selected, we testwhether it should be selected in the current round. Forany other non-selected object P ′, P ′ will be assignedto the new representative P if the divergence D(P ′ ‖P ) is smaller than the divergence between P ′ andany previously selected representatives. Therefore, wecalculate the contribution of P ′ to the decrease of thetotal KL divergence by selecting P as

max

(0,

i−1minj=1

(D(P ′ ‖Cj)

)−D(P ′ ‖P )

).

We calculate the total decrease of the total KL diver-gence by selecting P as the sum over the contributionof all non-selected object, denoted by DEC(P ). Then,the object to be selected in the i-th iteration is the onethat can incur the largest decrease, that is,

Ci = argmaxP∈O\{C1,··· ,Ci−1}

(DEC(P )

).

We end the building phase as long as k representativesare selected and proceed to the swapping phase.

Swapping Phase: In the swapping phase, the un-certain k-medoids method iteratively improves theclustering by swapping a non-representative objectwith the representative to which it is assigned. Fora non-representative object P , suppose it is assignedto cluster C whose representative is C. We considerthe effect of swapping P and C in two cases on everynon-selected object P ′ other than P ,• If P ′ currently belongs to C, when C is replaced

by P , we will reassign P ′ to P or one of the otherk − 1 existing representatives, to which P ′ is themost similar.

• If P ′ currently belongs to a representative C ′

other than C, and D(P ′ ‖ P ) < D(P ′ ‖ C ′), P ′

is reassigned to P .When a reassignment happens, the decrease of thetotal KL divergence by swapping P and C is recorded.After all non-representative objects are examined, weobtain the total decrease of swapping P and C. Then,we select the object Pmax which can make the largestdecrease. That is,

Pmax = argmaxP∈O\{C1,··· ,Ck}

(DEC(P )

).

We check that whether the swapping of Pmax canimprove the clusters, i.e., whether DEC(Pmax) > 0. Ifso, the swapping is carried into execution. Otherwise,the method terminates and report the final clustering.

Complexity: The building phase in the uncertaink-medoids method requires evaluating the KL diver-gences between O(kn2) pairs of objects, where n isthe number of objects. In the swapping phase, in eachiteration, it evaluates O(n2) KL divergences to find the

optimal swapping. In total, the uncertain k-medoidsmethod has complexity O((k+ r)n2E), where r is thenumber of iterations in the swapping phase and E isthe complexity of evaluating the KL divergence of twoobjects. The method cannot scale on big data sets dueto it quadratic complexity with respect to the numberof object. Next, we present a randomized k-medoidsmethod to bring down the complexity.

4.1.2 Randomized K-Medoids MethodThe randomized k-medoids method, instead of find-ing the optimal non-representative object for swap-ping, randomly selects a non-representative object forswapping if the clustering quality can be improved.We follow the simulated annealing technique [20] [6]to prevent the method from being stuck at a localoptimal result.

The randomized k-medoids method follows thebuilding-swapping framework. At the beginning, thebuilding phase is simplified by selecting the initialk representatives at random. Non-selected objectsare assigned to the most similar representative ac-cording to KL divergence. Then, in the swappingphase, we iteratively replace representatives by non-representative objects.

In each iteration, instead of finding the optimal non-representative object for swapping in the uncertaink-medoids method, a non-representative object P israndomly selected to replace the representative C towhich P is assigned. To determine whether P is agood replacement of C, we examine the two casesas described in the swapping phase in Section 4.1.1.After all non-representative objects are examined, thetotal decrease of the total KL divergence by swappingP and C is recorded as DEC.

Then, we compute the probability of swapping as

Prswap(DEC) =

{1 if DEC > 0

eDEC×log(rcur) if DEC ≤ 0,

where rcur is the current number of iterations. Ba-sically, if DEC > 0, the swapping can improve theclustering, then the swapping is put into practice. Inthe cases where DEC ≤ 0, the swapping may notimprove the clustering. However, the swapping maystill be carried out with probability eDEC×log(rcur) toprevent the algorithm from being stuck at a localoptimal result. We repeat the iterative process untila swapping is not executed.

The randomized k-medoids method has time com-plexity O(rnE) where r is the number of iterationsin the swapping phase and E is the complexity ofevaluating the KL divergence of two objects. The costof the building phase in the uncertain k-medoidsmethod is removed since the representatives are ran-domly initialized. The object selected for swappingin each iteration in the swapping phase is also ran-domly picked, so the randomized k-medoids method



evaluates O(n) KL divergences in each iteration. Wewill see in our experiments that the randomized k-medoids method can scale well on data sets with alarge number of objects.

4.2 Density-Based Clustering MethodsUnlike partitioning methods which organize similarobjects into the same partitions to discover clusters,density-based clustering methods regard clusters asdense regions of objects that are separated by regionsof low density.

DBSCAN [10] is the first and most representativedensity-based clustering method developed for cer-tain data. To demonstrate density-based clusteringmethods based on distribution similarity, we developthe uncertain DBSCAN method which integrates KLdivergence into DBSCAN. Different to the FDBSCANmethod [21] which is based on geometric distancesand finds dense regions in the original geometricspace, the uncertain DBSCAN method transformsobjects into a different space where the distributiondifferences are revealed.

The uncertain DBSCAN method finds dense regionsthrough core objects whose ε-neighborhood containsat least μ objects. Formally, P is a core object, if∣∣{Q ∈ O|D(Q‖P ) ≤ ε}∣∣ ≥ μ.

An object Q is said to be direct density-reachable froman object P if D(Q‖P ) ≤ ε and P is a core object.

Initially, every core object forms a cluster. Twoclusters are merged together if a core object of onecluster is density-reachable from a core object of theother cluster. A non-core object is assigned to theclosest core object if it is direct density reachable fromthis core object. The algorithm iteratively examinesobjects in the data set until no new object can beadded to any cluster.

Note that direct density-reachability describedabove is an asymmetric relationship. This is notcaused by the asymmetry of KL divergence. Therelationship is also asymmetry on certain data wheresimilarity between points are measured by the squareof Euclidean distance.

The quality of the clustering obtained by the uncer-tain DBSCAN method depends on the parameters εand μ. We will show the performance in Section 6.

The complexity of the uncertain DBSCAN methodis O(n2E) where n is the number of uncertain objectsand E is the cost of evaluating the KL divergence oftwo objects. Essentially, the KL divergence of any pairof objects is evaluated because we do not assume anyindex on the data set.

5 BOOSTING COMPUTATIONThe uncertain k-medoids method, the randomized k-medoids method, and the uncertain DBSCAN method

all require evaluation of KL divergences of many pairsof objects. As the number of uncertain objects andthe sample size of each object increase, it is costly toevaluate a large amount of KL divergence expressions.In this section, we first show the implementationtechnique to save the computation in Section 5.1.Then, Section 5.2 introduces fast Gauss transform toprovide a fast approximation of KL divergence in thecontinuous case.

5.1 Efficient ImplementationIn both the uncertain and randomized k-medoidsmethods but not DBSCAN, the most used operation iscomputing the difference between two KL divergenceexpressions, D(P ‖ C) − D(P ‖ C ′), for example, todecide whether P should be assigned to representa-tive C or C ′, and to compute the DEC of replacing arepresentative.

We can evaluate the divergence difference D(P ‖C) − D(P ‖ C ′) more efficiently than evaluating thetwo divergences separately and then computing thesubtraction.

In the discrete case, applying Equation (4), we have

D(P ‖C)−D(P ‖C ′)=

∑p∈P

P (p) logP (p)

C(p)−

∑p∈P

P (p) logP (p)

C ′(p)

=∑p∈P

P (p)(logC ′(p)− logC(p)

) (10)

In the continuous case, applying Equation (7), wehave

D(P ‖C)−D(P ‖C ′)=

1

|P |∑p∈P

logP (p)

C(p)− 1

|P |∑p∈P

logP (p)

C ′(p)

=1

|P |∑p∈P

(logC ′(p)− logC(p)

) (11)

By directly computing the difference between the twodivergence expressions, we simplify the computationin both the discrete and continuous cases. More im-portantly, we avoid evaluating the term logP (p) inthe continuous case, thus substantially reduce thecomputation in density estimation.

5.2 Fast Gauss Transform for Efficiently Calculat-ing Probability Density FunctionsThe probability mass functions of uncertain objects inthe discrete case can be directly calculated as shownin Equation (2). The complexity is linear to the samplesize of the uncertain object. Moreover, as the domainis finite, we can pre-compute the probability massfunction of every object and store them in a hash tableeither in-memory or on-disk.

However, in the continuous case, the complexityof calculating the probability density functions is



quadratic to the sample size of the uncertain object.Pre-computation is not feasible since the domain isuncountably infinite. No matter we evaluate KL diver-gences directly or evaluate the divergence differencesas described in Section 5.1, the major cost is onevaluating the following expression,

∑p∈P

log∑q∈C

d∏j=1

e−

(p.Dj−q.Dj)2

2h2C,j . (12)

It requires to evaluate the sum of N Gaussian func-tions at M points, where N is the sample size of Cand M is the sample size of P . Straightforwardly,N × M Gaussian functions are evaluated, and thecomputational complexity is O(N ×M). Clearly, it iscostly for large N and M . To make the computationpractical for large data sets, we resort to approximateEquation (12).

The main idea of approximating the sum of a seriesof Gaussian functions is to make use of the fact thatthe probability density of the Gaussian function de-cays exponentially and is negligible outside a certaindistance to the center of the Gaussian. The fast Gausstransform [13] is one of the best techniques to dothe job. It reduces the computational complexity fromO(N ×M) to O(N +M) using a divide-and-conquerstrategy and combined with the manipulation of Her-mite polynomial expansions and Taylor series. How-ever, as pointed out by Yang et al. [38], the constantfactor in O(N +M) grows exponentially with respectto the dimensionality d, which makes the techniqueimpractical for d > 3. Yang et al. [38] developed animproved fast Gauss transform to reduce the constantfactor to asymptotically polynomial order. We adopttheir improved Gauss transform to boost the efficiencyof evaluating Equation (11).

The improved fast Gauss transform takes advan-tages of the exponential decay of the Gaussian. An-other key technique is to expand the sum of Gaussianfunctions into a multivariate Taylor expansion andtruncate the series after degree p, where p is chosento be sufficiently large such that the bounded error isno more than the precision parameter ε.

To evaluate Equation (12), the algorithm works inthe following steps:

Step 1:Approximate the sample of C by clustering.Partition the N sample points of C into kclusters S1, · · · , Sk using the farthest-pointclustering algorithm [12] [11] such that themaximum radius of all clusters is less thanhρ1, here h is the bandwidth of the Gaussianfunction and ρ1 is a controlling parameter.

Step 2:Choose parameter p for truncating Taylorexpansion. Choose p sufficiently large suchthat the error estimate is less than the preci-sion ε. Here, the error at any point of P isbounded by N

(2p

p! ρ1ρ2 + e−ρ22

), where ρ2 is

a controlling parameter.

Step 3:Compute the coefficients of the Taylor ex-pansion. For each cluster Si (1 ≤ i ≤ k), letci denote the center found by the farthest-point clustering algorithm, compute the co-efficients below,

Ciα =

2|α|

α!

∑q∈Si

d∏j=1

e−(q.Dj−ci.Dj)

2

h2 (q − cih

)α.

Here, α = (α1, · · · , αd) is a multi-indexwhich is a d-dimensional vector of nonnega-tive integers. For any multi-index α and anyreal value vector x = (x.D1, · · · , x.Dd), wedefine the following operations.

xα = x.Dα11 · · ·x.Dαd

d ,|α| = α1 + · · ·+ αd,α! = α1! · · ·αd!.

Step 4:Compute the sum of approximated Gaussianfunctions. For each point s ∈ P , find theclusters whose centers lie within the rangehρ2. Then, the sum of Gaussian functions inEquation (12) is evaluated as

∑q∈C

d∏j=1

e−

(p.Dj−q.Dj)2

2h2C,j =

∑dist(s,ci)≤hρ2

∑|α|<p

Ciα

d∏j=1

e−(p.Dj−ci.Dj)

2

h2

(p− cih

)α

,

where dist(s, ci) is the distance between pand ci.

The tradeoff between the computational cost andthe precision is controlled by parameters p, ρ1, andρ2. The larger p and ρ2, and the smaller ρ1, thebetter precision, but the computation and storage costincreases. In our experiments, as suggested in [38], weset p = ρ2 = 10 and ρ1 is indirectly set by k (see Step1) which is set to

√N .

Our empirical study in Section 6 shows that thefast Gauss transform boosts the efficiency of ouralgorithms dramatically with only a small decreaseof the clustering quality.

6 EMPIRICAL STUDYWe conducted extensive experiments on both syn-thetic and real data sets to evaluate the effectiveness ofKL divergence as a similarity measure for clusteringuncertain data and the efficiency of the techniques forevaluating KL divergences.

Our programs were implemented in C++ and com-piled by GCC 4.3.3. The experiments were con-ducted on a computer with an Intel Core 2 DuoP8700 2.53GHz CPU and 4GB main memory runningUbuntu 9.04 Jaunty. All programs ran in-memory. TheI/O cost is not reported.



(a) Uniform (b) Gaussian (c) Inverse Gaussian

Fig. 2. Three types of distributions.

6.1 Synthetic Data

We generated data sets in both continuous and dis-crete domains. In the continuous case, an uncertainobject is a sample drawn from a continuous distri-bution. In the discrete case, a data set is generatedby converting a data set in the continuous case. Wediscretized the continuous domain by partitioning itinto a grid. Every dimension is equally divided into2 parts. So a d-dimensional space is divided into 2d

cells of equal size. We use the central points of cellsas values in the discrete domain. The probability ofan object in a cell is the sum of the probabilities of allits sample points in this cell.

The data space was restricted to [0, 1]d. We use threedifferent types of distributions shown in Figure 2,the uniform distribution, the Gaussian distribution,and the inverse Gaussian distribution. An inverseGaussian distribution was generated from a Gaussiandistribution. Given a point x in a Gaussian distribu-tion, we generated a point y for the inverse Gaussiandistribution according to the following formula for1 ≤ i ≤ d,

y.Di =

{1.5− x.Di if x.Di ∈ [0.5, 1];

0.5− x.Di if x.Di ∈ [0, 0.5).

We consider 4 major factors in our experiments, thedimensionality d of the data space (from 2 to 10), thecardinality (i.e., the number of objects) n of the dataset (from 50 to 10, 000), the sample size s (from 50 to900), and the number of clusters k (from 3 to 25).

Given the number of clusters k for a data set, wegenerated k probability density functions, in whichthere is one uniform distribution, (k − 1)/2 Gaussiandistributions with different variances, and (k−1)/2 in-verse Gaussian distributions with different variances.The variance of the i-th Gaussian/inverse Gaussiandistribution is 0.05i. For each distribution, we gener-ated a group of samples, each of which forms oneobject. Thus, objects in the same group are sampledfrom the same probability density function. The num-ber of objects of a group was randomly picked with anexpectation of n

k . We made sure that the total numberof objects is equal to n. As we generated a syntheticdata set in this way, we have the ground truth of theclustering in the data set. We used the ground truthto evaluate the clustering quality of our algorithms.

Our experiments include three parts. Section 6.1.1first evaluates the effect of KL divergence used inclustering methods presented in Section 4. Then, Sec-tion 6.1.2 shows the speedup by the implementationtechniques and fast Gauss transform introduced inSection 5. Last, Section 6.1.3 shows the scalability andfeasibility of our algorithms on large data sets.

6.1.1 Effectiveness of KL Divergence in ClusteringWe first compare the clustering quality of KL diver-gences with geometric distances in both partition-ing clustering methods and density-based clusteringmethods. For partitioning clustering methods, we im-plemented the UK-means method [24] (denoted byUK) using geometric distances as well as the uncertaink-medoids method (denoted by KM-KL) and the ran-domized k-medoids method (denoted by RKM-KL)using KL divergences. For density-based clusteringmethods, we implemented the FDBSCAN method [21](denoted by FD) using geometric distances and theuncertain DBSCAN method (denoted by DB-KL) us-ing KL divergences.

We use precision and recall as the quality mea-surements. Let G denote the ground truth clusteringgenerated by the synthetic data generator, C is theclustering obtained by a clustering method. Two ob-jects are called a pair if they appear in the same clusterin a clustering. We define

TP true positive, the set of common pairs ofobjects in both G and C;

FP false positive, the set of pairs of objects in C

but not G;FN false negative, the number of pairs of objects

in G but not C.Then, the precision and recall of a clustering C arecalculated respectively as

precision(C) = |TP |/(|TP |+ |FP |),recall(C) = |TP |/(|TP |+ |FN |).

By default a data set contains n = 100 objects, eachof which has s = 100 sample points in a d = 4dimensional space. The number of clusters is k = 6by default.

We note that density-based methods, FD and DB-KL, do not use k as a parameter. Instead, the cluster-ing is controlled by the minimum number μ of pointsrequired to form a cluster and the neighborhood



0

0.2

0.4

0.6

0.8

1

2 4 6 8 10

prec

isio

n

UKKM-KL

RKM-KLFD

DB-KL

(a) dimensionality d

0

0.2

0.4

0.6

0.8

1

50 100 150 200 250

prec

isio

n

UKKM-KL

RKM-KLFD

DB-KL

(b) cardinality n

0

0.2

0.4

0.6

0.8

1

50 100 150 200 250

prec

isio

n

UKKM-KL

RKM-KLFD

DB-KL

(c) sample size s

0

0.2

0.4

0.6

0.8

1

3 6 9 12 15

prec

isio

n

UKKM-KL

RKM-KLFD

DB-KL

(d) # of clusters k

Fig. 3. Comparison in precision between KL divergences and geometric distances in the continuous case.

0

0.2

0.4

0.6

0.8

1

2 4 6 8 10

reca

ll

UKKM-KL

RKM-KLFD

DB-KL


0

0.2

0.4

0.6

0.8

1

50 100 150 200 250

reca

ll

UKKM-KL

RKM-KLFD

DB-KL

(b) cardinality n

0

0.2

0.4

0.6

0.8

1

50 100 150 200 250

reca

ll

UKKM-KL

RKM-KLFD

DB-KL

(c) sample size s

0

0.2

0.4

0.6

0.8

1

3 6 9 12 15

reca

ll

UKKM-KL

RKM-KLFD

DB-KL

(d) # of clusters k

Fig. 4. Comparison in recall between KL divergences and geometric distances in the continuous case.

0

0.2

0.4

0.6

0.8

1

2 4 6 8 10

prec

isio

n

UKKM-KL

RKM-KLFD

DB-KL


0

0.2

0.4

0.6

0.8

1

50 100 150 200 250

prec

isio

n

UKKM-KL

RKM-KLFD

DB-KL

(b) cardinality n

0

0.2

0.4

0.6

0.8

1

50 100 150 200 250

prec

isio

nUK

KM-KLRKM-KL

FDDB-KL

(c) sample size s

0

0.2

0.4

0.6

0.8

1

3 6 9 12 15

prec

isio

n

UKKM-KL

RKM-KLFD

DB-KL

(d) # of clusters k

Fig. 5. Comparison in precision between KL divergences and geometric distances in the discrete case.

0

0.2

0.4

0.6

0.8

1

2 4 6 8 10

reca

ll

UKKM-KL

RKM-KLFD

DB-KL


0

0.2

0.4

0.6

0.8

1

50 100 150 200 250

reca

ll

UKKM-KL

RKM-KLFD

DB-KL

(b) cardinality n

0

0.2

0.4

0.6

0.8

1

50 100 150 200 250

reca

ll

UKKM-KL

RKM-KLFD

DB-KL

(c) sample size s

0

0.2

0.4

0.6

0.8

1

3 6 9 12 15

reca

ll

UKKM-KL

RKM-KLFD

DB-KL

(d) # of clusters k

Fig. 6. Comparison in recall between KL divergences and geometric distances in the discrete case.

distance/divergence. As suggested in [21], we setμ = 5 and varied ε so that FD and DB-KL can outputapproximately the specified number k of clusters.

Figures 3 and 4 compare the precision and re-call of UK, KM-KL, RKM-KL, FD, and DB-KL inthe continuous case, respectively. Clearly, geometricdistance-based methods UK and FD have very poorprecision and recall, while the KL divergence-basedmethods KM-KL, RKM-KL, and DB-KL can obtainmuch better clustering. KM-KL has higher precisionand recall than the randomized versions RKM-KL,since the uncertain k-medoids method tries to findthe optimal non-representative object for swappingin each iteration. In the discrete case, as shown inFigures 5 and 6, we see similar results as in thecontinuous case.

We also see that the precision and recall of all algo-

rithms follow similar trends. They are not sensitive tothe dimensionality. They increase as the sample sizeincreases or the number of clusters decreases. Also,they drop when there are more clusters in a data setsince the number of object pairs in a clustering de-creases linearly as k increases, thus the mis-clusteredpairs incur large penalties on the precision and recall.

6.1.2 Efficiency of Boosting Techniques

Figure 7 shows the speedup of the boosting tech-niques, especially the fast Gauss transform. We plotthe runtime of KM-KL, RKM-KL, and DB-KL andtheir corresponding methods implemented with thefast Gauss transform, annotated by KM-KL-FGT,RKM-KL-FGT, and DB-KL-FGT. Methods with the fastGauss transform are significantly faster than theircounterparts, since the fast Gauss transform brings



0

200

400

600

800

1000

1200

1400

50 100 150 200 250

time

(s)

KM-KLRKM-KL

DB-KL

KM-KL-FGTRKM-KL-FGT

DB-KL-FGT

(a) cardinality n

0 200 400 600 800

1000 1200 1400 1600 1800

50 100 150 200 250

time

(s)

KM-KLRKM-KL

DB-KL

KM-KL-FGTRKM-KL-FGT

DB-KL-FGT

(b) sample size s

Fig. 7. Comparison in runtime between methods w/ andw/o the fast Gauss transform in the continuous case.

0

0.2

0.4

0.6

0.8

1

50 100 150 200 250

prec

isio

n

KM-KLRKM-KL

DB-KL

KM-KL-FGTRKM-KL-FGT

DB-KL-FGT

(a) cardinality n

0

0.2

0.4

0.6

0.8

1

50 100 150 200 250

prec

isio

n

KM-KLRKM-KL

DB-KL

KM-KL-FGTRKM-KL-FGT

DB-KL-FGT

(b) sample size s

Fig. 8. Comparison in precision between methods w/and w/o the fast Gauss transform in the continuous case.

0

0.2

0.4

0.6

0.8

1

50 100 150 200 250

reca

ll

KM-KLRKM-KL

DB-KL

KM-KL-FGTRKM-KL-FGT

DB-KL-FGT

(a) cardinality n

0

0.2

0.4

0.6

0.8

1

50 100 150 200 250

reca

ll

KM-KLRKM-KL

DB-KL

KM-KL-FGTRKM-KL-FGT

DB-KL-FGT

(b) sample size s

Fig. 9. Comparison in recall between methods with andwithout the fast Gauss transform in the continuous case.

0

0.2

0.4

0.6

0.8

1

2000 4000 6000 8000 10000

prec

isio

n

RKM-KLRKM-KL-FGT

(a) cardinality n

0

0.2

0.4

0.6

0.8

1

100 300 500 700 900

prec

isio

n

RKM-KLRKM-KL-FGT

(b) sample size s

Fig. 10. Precision on large data sets in the continuouscase.

0

0.2

0.4

0.6

0.8

1

2000 4000 6000 8000 10000

reca

ll

RKM-KLRKM-KL-FGT

(a) cardinality n

0

0.2

0.4

0.6

0.8

1

100 300 500 700 900

reca

ll

RKM-KLRKM-KL-FGT

(b) sample size s

Fig. 11. Recall on large data sets in continuous case.

0 200 400 600 800

1000 1200 1400 1600

2000 4000 6000 8000 10000

time

(s)

RKM-KLRKM-KL-FGT

(a) cardinality n

0

500

1000

1500

2000

2500

3000

3500

100 300 500 700 900

time

(s)

RKM-KLRKM-KL-FGT

(b) sample size s

Fig. 12. Runtime on large data sets in continuous cases.

down the complexity of evaluating the sum of KLdivergences from quadratic to linear.

Figures 8 and 9 show that the precision and recallof the methods with the fast Gauss transform areslightly lower than their counterparts which computeKL divergences directly due to the error incurred inthe approximation.

In a word, the fast Gauss transform is a tradeoffbetween quality and time. The experiment resultsshow that it can speed up the clustering a lot withan acceptable accuracy tradeoff.

6.1.3 ScalabilityAs the uncertain k-medoids method and the uncertainDBSCAN method have quadratic complexity withrespect to the number of objects, they cannot scaleon data sets with a large number of objects. In thissection, we test the scalability of the randomizedk-medoids method on large data sets which con-sist of 4, 000 objects and 300 sample points in a 4-dimensional space by default. The number of clustersin a data set is k = 10 by default.

Figures 10 and 11 show similar trends of precisionand recall of RKM-KL and RKM-KL-FGT as on smalldata sets.

0

0.2

0.4

0.6

0.8

1

precision recall

RDUK

RKM-KLRKM-KL-FGT

(a) Precision and recall.

10-510-410-310-210-1100101102103

time

(s)

RDUK

RKM-KLRKM-KL-FGT

(b) Time.

Fig. 14. Results on weather data.

Figure 12 shows the runtime of RKM-KL and RKM-KL-FGT on large data sets. RKM-KL scales well as thenumber of objects increases, however, it suffers fromthe quadratic complexity of evaluating the sum of KLdivergences with respect to the sample size. Thanks tothe fast Gaussian transform, RKM-KL-FGT is scalableon data sets with both a large number of objects anda large sample per object.

6.2 Real DataWe obtained a weather data set from the NationalCenter for Atmospheric Research data archive (http://dss.ucar.edu/datasets/ds512.0/). The data set con-sists of 2, 936 stations around the world. Each station



0

5

10

15

20

25

30

-20 -10 0 10 20 30 40

prec

ipita

tion

temperature

(a) tropical

0

5

10

15

20

25

30

-20 -10 0 10 20 30 40

prec

ipita

tion

temperature

(b) dry

0

5

10

15

20

25

30

-20 -10 0 10 20 30 40

prec

ipita

tion

temperature

(c) temperate

0

5

10

15

20

25

30

-20 -10 0 10 20 30 40

prec

ipita

tion

temperature

(d) continental

0

5

10

15

20

25

30

-20 -10 0 10 20 30 40

prec

ipita

tion

temperature

(e) polar

Fig. 13. Example distributions of the 5 types of climates.

contains 366 daily records in the year of 2008. Eachrecord has 3 dimensions, average temperature, precip-itation, and average humidity. We labeled the climatetype of each station according to the Koppen-Geigerclimate classification [28]. For the ground truth, sta-tions with the same label are considered to be in thesame cluster. In total, we have 5 clusters, tropicalclimate, dry climate, temperate climate, continentalclimate, and polar climate.

Figure 13 shows an example distribution of eachof the 5 climates. We only plot dimensions of averagetemperature and precipitation for better visualization.Clearly, we observe the difference among the 5 typesof climates.

Figure 14 shows the results of the UK, RKM-KL,RKM-KL-FGT, and a random clustering methods (de-noted by RD) on the weather data set. The randomclustering method randomly assigns an object to oneof the k clusters, and it serves as the baseline ofother methods. We see that RKM-KL and RKM-KL-FGT have much higher precision and recall thanRD and UK. RKM-KL-FGT has feasible running timecomparing to RKM-KL.

6.3 SummaryIn summary, both partitioning and density-based clus-tering methods have better clustering quality whenusing KL divergences as similarity than using geo-metric distances. The results confirm that KL diver-gence can naturally capture the distribution differencewhich geometric distance cannot capture.

To boost the computation in the continuous caseto battle the costly kernel density estimation, the fastGauss transform can speed up the clustering a lot withan acceptable accuracy tradeoff. To scale on large datasets, the randomized k-medoids method equippedwith the Gauss transform has linear complexity withrespect to both the number of objects and the samplesize per object. It can perform scalable clustering taskson large data sets with moderate accuracy.

7 CONCLUSIONSIn this paper, we explore clustering uncertain databased on the similarity between their distributions.We advocate using the Kullback-Leibler divergenceas the similarity measurement, and systematically

define the KL divergence between objects in boththe continuous and discrete cases. We integrated KLdivergence into the partitioning and density-basedclustering methods to demonstrate the effectivenessof clustering using KL divergence. To tackle the com-putational challenge in the continuous case, we esti-mate KL divergence by kernel density estimation andemploy the fast Gauss transform technique to furtherspeed up the computation. The extensive experimentsconfirm that our methods are effective and efficient.

The most important contribution of this paper isto introduce distribution difference as the similaritymeasure for uncertain data. Besides clustering, sim-ilarity is also of fundamental significance to manyother applications, such as nearest neighbor search. Inthe future, we will study those problems on uncertaindata based on distribution similarity.

ACKNOWLEDGEMENTThe authors are grateful to the anonymous review-ers and the associate editor for their constructiveand critical comments on the paper. The research ofBin Jiang and Jian Pei is supported in part by anNSERC Discovery Grant and an NSERC DiscoveryAccelerator Supplement Grant. The research of YufeiTao is supported in part by grants GRF 4169/09,GRF 4166/10, and GRF 4165/11 from HKRGC. Theresearch of Xuemin Lin is supported in part by grantsARCDP110102937, ARCDP0987557, ARCDP0881035,and NSFC61021004. All opinions, findings, conclu-sions, and recommendations in this paper are those ofthe authors and do not necessarily reflect the viewsof the funding agencies.

REFERENCES[1] S. Abiteboul, P. C. Kanellakis, and G. Grahne. On the represen-

tation and querying of sets of possible worlds. In SIGMOD,1987.

[2] M. R. Ackermann, J. Blomer, and C. Sohler. Clustering formetric and non-metric distance measures. In SODA, 2008.

[3] M. Ankerst, M. M. Breunig, H.-P. Kriegel, and J. Sander.Optics: Ordering points to identify the clustering structure.In SIGMOD, 1999.

[4] A. Banerjee, S. Merugu, I. S. Dhillon, and J. Ghosh. Clus-tering with bregman divergences. Journal of Machine LearningResearch, 2005.

[5] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichletallocation. Journal of Machine Learning Research, 2003.


14

[6] V. Cerny. A thermodynamical approach to the travellingsalesman problem: an efficient simulation algorithm. Journalof Optimization Theory and Applications, 1985.

[7] R. Cheng, D. V. Kalashnikov, and S. Prabhakar. Evaluatingprobabilistic queries over imprecise data. In SIGMOD, 2003.

[8] N. N. Dalvi and D. Suciu. Management of probabilistic data:foundations and challenges. In PODS, 2007.

[9] I. S. Dhillon, S. Mallela, and R. Kumar. A divisive information-theoretic feature clustering algorithm for text classification.Journal of Machine Learning Research, 2003.

[10] M. Ester, H.-P. Kriegel, J. Sander, and X. Xu. A density-basedalgorithm for discovering clusters in large spatial databaseswith noise. In KDD, 1996.

[11] T. Feder and D. H. Greene. Optimal algorithms for approxi-mate clustering. In STOC, 1988.

[12] T. F. Gonzalez. Clustering to minimize the maximum inter-cluster distance. Theoretical Computer Science, 1985.

[13] L. Greengard and J. Strain. The fast gauss transform. In SIAMJournal on Scientific and Statistical Computing, 1991.

[14] J. Han and M. Kamber. Data Mining: Concepts and Techniques.2000.

[15] T. Imielinski and W. L. Jr. Incomplete information in relationaldatabases. Journal of ACM, 1984.

[16] A. K. Jain and R. C. Dubes. Algorithms for Clustering Data.1988.

[17] R. Jampani, F. Xu, M. Wu, L. L. Perez, C. M. Jermaine, and P. J.Haas. Mcdb: a monte carlo approach to managing uncertaindata. In SIGMOD, 2008.

[18] B. Kao, S. D. Lee, D. W. Cheung, W.-S. Ho, and K. F. Chan.Clustering uncertain data using voronoi diagrams. In ICDM,2008.

[19] L. Kaufman and P. J. Rousseeuw. Finding Groups in Data: AnIntroduction to Cluster Analysis. 1990.

[20] S. Kirkpatrick, D. G. Jr., and M. P. Vecchi. Optimization bysimmulated annealing. Science, 1983.

[21] H.-P. Kriegel and M. Pfeifle. Density-based clustering ofuncertain data. In KDD, 2005.

[22] H.-P. Kriegel and M. Pfeifle. Hierarchical density-based clus-tering of uncertain data. In ICDM, 2005.

[23] S. Kullback and R. A. Leibler. On information and sufficiency.In The Annals of Mathematical Statistics, 1951.

[24] S. D. Lee, B. Kao, and R. Cheng. Reducing uk-means to k-means. In ICDM Workshops, 2007.

[25] S. P. Lloyd. Least squares quantization in pcm. IEEE Transac-tions on Information Theory, 1982.

[26] J. B. MacQueen. Some methods for classification and analysisof multivariate observations. In Proceedings of the fifth BerkeleySymposium on Mathematical Statistics and Probability, 1967.

[27] W. K. Ngai, B. Kao, C. K. Chui, R. Cheng, M. Chau, and K. Y.Yip. Efficient clustering of uncertain data. In ICDM, 2006.

[28] M. C. Peel, B. L. Finlayson, and T. A. McMahon. Updatedworld map of the koppen-geiger climate classification. Hy-drology and Earth System Sciences, 2007.

[29] J. Pei, B. Jiang, X. Lin, and Y. Yuan. Probabilistic skylines onuncertain data. In VLDB, 2007.

[30] J. M. Ponte and W. B. Croft. A language modeling approachto information retrieval. In SIGIR, 1998.

[31] A. D. Sarma, O. Benjelloun, A. Y. Halevy, and J. Widom.Working models for uncertain data. In ICDE, 2006.

[32] D. W. Scott. Multivariate Density Estimation: Theory, Practical,and Visualization. 1992.

[33] B. W. Silverman. Density Estimation for Statistics and DataAnalysis. 1986.

[34] F. Song and W. B. Croft. A general language model forinformation retrieval. In CIKM, 1999.

[35] Y. Tao, R. Cheng, X. Xiao, W. K. Ngai, B. Kao, and S. Prab-hakar. Indexing multi-dimensional uncertain data with arbi-trary probability density functions. In VLDB, 2005.

[36] P. B. Volk, F. Rosenthal, M. Hahmann, D. Habich, andW. Lehner. Clustering uncertain data with possible worlds.In ICDE, 2009.

[37] J. Xu and W. B. Croft. Cluster-based language models fordistributed retrieval. In SIGIR, 1999.

[38] C. Yang, R. Duraiswami, N. A. Gumerov, and L. S. Davis.Improved fast gauss transform and efficient kernel densityestimation. In ICCV, 2003.

Bin Jiang is a Ph.D. candidate in Schoolof Computing Science at Simon Fraser Uni-versity, Canada. He received his B.Sc. andM.Sc. degrees in Computer Science fromPeking University, China and University ofNew South Wales, Australia, respectively. Hisresearch interests lie in mining uncertaindata.

Jian Pei is a Professor at the School ofComputing Science at Simon Fraser Univer-sity, Canada. His research interests can besummarized as developing effective and effi-cient data analysis techniques for novel dataintensive applications. He is currently inter-ested in various techniques of data mining,Web search, information retrieval, data ware-housing, online analytical processing, anddatabase systems, as well as their applica-tions in social networks, health-informatics,

business and bioinformatics. His research has been supported inpart by government funding agencies and industry partners. He haspublished prolifically and served regularly for the leading academicjournals and conferences in his fields. He is an associate editor ofACM Transactions on Knowledge Discovery from Data (TKDD) andan associate editor-in-chief of IEEE Transactions of Knowledge andData Engineering (TKDE). He is a senior member of the Associationfor Computing Machinery (ACM) and the Institute of Electrical andElectronics Engineers (IEEE). He is the recipient of several presti-gious awards.

Yufei Tao is a Professor in the Departmentof Computer Science and Engineering, Chi-nese University of Hong Kong (CUHK). Be-fore joining CUHK in 2006, he was a VisitingScientist at the Carnegie Mellon Universityduring 2002-2003, and an Assistant Profes-sor at the City University of Hong Kong dur-ing 2003-2006. Currently, he is an associateeditor of ACM Transactions on DatabaseSystems (TODS).

Xuemin Lin is a Professor in the School ofComputer Science and Engineering, the Uni-versity of New South Wales. He has been thehead of database research group at UNSWsince 2002. He is a concurrent professorin the School of Software, East China Nor-mal University. Before joining UNSW, Xueminheld various academic positions at the Uni-versity of Queensland and the University ofWestern Australia. Dr. Lin got his PhD inComputer Science from the University of

Queensland in 1992 and his BSc in Applied Math from FudanUniversity in 1984. During 1984-1988, he studied for Ph.D. in AppliedMath at Fudan University. He currently is an associate editor of ACMTransactions on Database Systems. His current research interestslie in data streams, approximate query processing, spatial dataanalysis, and graph visualization.


CLUSTERING UNCERTAIN DATA BASED ON PROBABILITY ...

Documents

Transcript of CLUSTERING UNCERTAIN DATA BASED ON PROBABILITY ...