Visualizing Global Manifold Based on Distributed Local Data Abstractions

Xiaofeng ZhangDepartment of Computer Science

Hong Kong Baptist UniversityKowloon Tong, Hong Kongxfzhang@comp.hkbu.edu.hk

William K. CheungDepartment of Computer Science

Hong Kong Baptist UniversityKowloon Tong, Hong Kongwilliam@comp.hkbu.edu.hk

Abstract

Mining distributed data for global knowledge is gettingmore attention recently. The problem is especially challeng-ing when data sharing is prohibited due to local constraintslike limited bandwidth and data privacy. In this paper, weinvestigate how to derive the embedded manifold (as a 2-D map) for a horizontally partitioned data set, where datacannot be shared among the partitions directly. We pro-pose a model-based approach which computes hierarchicallocal data abstractions, aggregates the abstractions, andfinally learns a global generative model – generative to-pographic mapping (GTM) based on the aggregated dataabstraction. We applied the proposed method to two bench-marking data sets and demonstrated that the accuracy of thederived manifold can effectively be controlled by adjustingthe data granularity level of the adopted local abstraction.

1. Introduction

Recent progress in automatic data collection, data stor-age and networking technologies has resulted in high ac-cessability of distributed and massive data for applicationdomains like e-Science and e-Commerce. This becomesimportant for data mining techniques, if needed, to be ap-plied in a distributed environment. Distributed data miningis challenging as data sharing is in many cases prohibiteddue to local constraints like limited bandwidth and data pri-vacy. The former constraint is faced as the distributed datacan be of high volume, e.g., in e-Science. The latter onehappens when the local data owners indicate high privacyconcern while they still prefer some degree of personalizede-services, like the situations in e-Commerce.

To avoid sharing data directly, one possible approach isto adopt some flexible statistical model for abstracting thelocal data so that the local data granularity (or the local pri-vacy level from the data privacy perspective) can be con-

trolled. A model with high complexity usually can retainmore details when compared with one of low complexity.The use of the model-based approach for distributed datamining can be found in [4, 3, 5].

In particular, Zhanget al. demonstrated in [5] how aglobal cluster model can be learned based on local data ab-stractions. In this paper, this distributed model-based ap-proach was extended to visualizing the embedded manifoldof a set of distributed data. Generative topographic map-ping (GTM) which is an effective nonlinear mapping toolfor visualizing high dimensional data was chosen to be theglobal model, and Gaussian mixture model (GMM) [2] waschosen for local data abstraction due to its representationflexibility. To learn the global GTM, we proposed a modi-fied EM-like algorithm for learning directly from the aggre-gated local data abstraction. The experimental results ob-tained based on two benchmarking data sets demonstratedthat the proposed distributed learning approach can achievecomparably good visualization results and at the same timesatisfy the limited bandwidth and data privacy requirementsof the local sources in a controlled manner.

2 Problem Formulation

2.1 Local Data Abstraction

Local data abstraction is here defined as the process ofrepresenting a given set of data by its statistics forming anabstraction. Via the abstractions, the statistical informationof the data can be shared, instead of the data themselves.The abstraction process is formulated as a parametric den-sity estimation problem and a hierarchical GMM with dif-ferent numbers of components at different levels of the hi-erarchy is adopted to support sharing local data details atdifferent data granularity levels.

Assume that there are totallyL distributed data sources.Let ti ∈ <d denote theith observed data item of dimen-sion d, θl denote the set of parameters of the local model(GMM) as the abstraction of thelth source,θlj denote the

jth component’s parameters of thelth local model (includ-ing the component’s meanµlj and covariance matrixΣlj),αlj denote the mixing proportion of thejth component inthe lth local model. The probability density function of thelth local modelplocal(ti|θl) with Kl components is givenas,

plocal(ti|θl) =∑Kl

j=1αljpj(ti|θlj)∑Kl

j=1αlj = 1

pj(ti|θlj) = (2π)−d2 |Σlj |−

12 exp{− 1

2(ti − µlj)

T Σ−1lj

(ti − µlj)}.

The local GMM parameters,i.e., {θ1, θ2, ..., θL}, arefirst derived as the abstractions of the distributed localdata by applying the agglomerative hierarchical algorithm(AGH). Given the dendrogram, illustrated in Figure 1, a hi-erarchy of GMMs with different number of components forrepresenting the local data at different granularity levels caneasily be computed. In particular, at a particular data gran-ularity level, a local Gaussian component can be derivedby computing the mean and covariance matrix of the datawithin a group at that level. Then the local GMM parame-ters can be sent to a global server for learning a global datamodel. In principle, the global model can be any type ofgenerative model.

Figure 1. A hierarchy of data abstractions. A higherlevel of abstraction is acquired by merging and com-puting the statistics of the two nearest data subgroupsat the next lower level.

2.2 Learning A Global GTM

Generative topographic mapping (GTM) [1] is a prob-abilistic non-linear latent variable model which can beused to explore the embedded manifold of a set of high-dimensional data. GTM assumes that the data are generateddue to a lattice of latent variables in a low-dimensional (usu-ally 2D) latent space. Via a non-linear mapping, a pointin the latent space is mapped to a data item in thedataspace. Visualizing the latent space with the original high-dimensional data projected back to it can result in an “un-folded” version of the embedded manifold which is usefulfor understanding the structure and organization of the data.

2.2.1 GTM Formulation

Let N denote the total number of data items,zk ∈ <H de-note thekth lattice point (altogetherM ) defined in the latentspace.y(z;W ) := WΨ(z) maps in an non-linear fashion apoint z in the latent space onto a corresponding pointy inthe data space, with the mapping governed by a generalizedlinear regression modelΨ weighted byW . A multivariateGaussian distribution in the data space is assumed in GTMfor ti givenzk, given as

p(ti|zk, W, β) = (2π)−d2 β

d2 exp{−β

2‖(ti − y(zk; W )‖2} (1)

whereβ is the reciprocal of the data variance.The EM algorithm is typically used for estimating the

parametersW andβ. The E-step for the original GTM isgiven as

Rik(Wold, βold) = P (zk|ti, W, β) =p(ti|zk, Wold, βold)∑M

j=1p(ti|zj , Wold, βold)

and the M-step is given as

N∑i=1

M∑k=1

Rik(Wold, βold){WnewΨ(zk)− ti}Ψ(zk)T = 0 (2)

1

βnew=

1

Nd

N∑i=1

M∑k=1

Rik(Wold, βold)‖WnewΨ(zk)− ti‖2. (3)

2.2.2 Learning from Local Data Abstraction

In order to learn the global GTM model parameters directlyfrom the local GMM parameters, we first approximate theoriginal estimated indicatorsRik by a uniform distributionover the data items corresponding to a particular GMMcomponent which is now to be shared instead of the data.

Assume thatRlk is now an indicator for thelth localcomponent1 with its underlying data to be generated by thekth global component. That is, the likelihood of the subsetof the data generated by thekth component of the globalmodel is assumed to be approximated by an overall estimateof the correspondinglth local component being generatedby the same component of the global model.Rik can thenbe approximated as

Rik ≈∑

i∈lthsourceRik

Nl(4)

whereNl denotes the number of data from thelth source.Rlk is hence equivalent to

Rlk =∑

i∈lthsource

Rik. (5)

To estimateRlk, the formulation adopted is given as

1Note that in the subsequent derivation, we abuse the index “l” to referto one of the local components of the aggregated local model.

Rlk =exp{−D(plocal(t|θl)||pgtm(t|zk, W, β))}∑M

j=1exp{−D(plocal(t|θl)||pgtm(t|zj , W, β))}

(6)

where the Kullback Leibler (KL) divergence between alocal component and a global component, denoted asD(plocal||pgtm), can be derived as2

lnβ−

d2

|Σl|12

+β

2tr(Σl) +

1

2(β(||y(zk; W )− µl)||2 − d). (7)

For the extreme case that one local GMM is used to repre-sent one data item, the first two terms of Eq.(7) will becomeconstant with respect to the local data. Thus, only the thirdterm will be in effect and Eq.(7) degenerates back to theoriginal GTM’s E-step (Eq.(1)). Accordingly, the new M-step can be derived as

L∑l=1

M∑k=1

Rlk(Wold, βold){WnewΨ(zk)− µl}Ψ(zk)T = 0 (8)

1

βnew=

1

Nd

M∑k=1

(

L∑l=1

Rlk(Wold, βold)(Σl + µlµTl ))

− 1

Nd

M∑k=1

((WnewΨ(zk))2L∑

l=1

Rlk). (9)

2.2.3 GTM Initialization Based on Local Abstractions

Given the aggregated local model, the initialization of theglobal GTM can be obtained as equivalent to that of theoriginal GTM. Original GTM uses principle componentanalysis (PCA) for initializingβ andW . For the proposedmethod, the original data are lacking for computing theglobal data covariance matrix, and thus the PCA. Fortu-nately, one can easily show that the global covariance ma-trix can analytically be derived based on the covariance ma-trices of the local data, given as

µglobal =

∑L

l=1Nlµl

N

Σglobal =

∑L

l=1Nl(Σl + µlµ

Tl )

N− µglobalµ

Tglobal.

3 Experiments on Visualizing DistributedData

To evaluate the effectiveness of the proposed approach ofvisualizing distributed data using GTM, experiments wereperformed based on two synthetic datasets — oil flow dataand S-curve data for benchmarking. In each experiment,the data set was first horizontally partitioned in a random

2Due to the space limitation, detailed derivation of Eq. (7) is omitted.

manner into three equal parts as local data sources. Then,global GTMs were to be learned under different settings forcomparison. Both the original GTM learned directly fromthe original dataset and the new GTM learned from the ag-gregated local model were tested. For all the experiments,1600 latent lattice points were chosen as the global GTMparameters.

−1 −0.5 0 0.5 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1new GTM: Document=300, noLts=1600, noBsn=81

(a) 100 components foreach local source.

−1 −0.5 0 0.5 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1new GTM: Document=600, noLts=1600, noBsn=81

(b) 200 components foreach local source.

−1 −0.5 0 0.5 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1new GTM: Document=900, noLts=1600, noBsn=81

(c) 300 components foreach local source.

−1 −0.5 0 0.5 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1orig GTM: Document=1000, noLts=40, noBsf=9

(d) The original GTM.

Figure 2. The visualization of the oil flow data us-ing GTMs. The posterior means of the projected dataof the three different configurations, namelyhomoge-neous, annular andstratified, are labelled as red cir-cles, blue triangles and green asterisks, respectively.Their posterior modes are all shown as crosses.

The oil flow dataset was originally used in [1] for mim-icking the measurements of oil flows mixed with gas andwater along multi-phase pipelines. The 12-dimensionaldata set consists of 1000 instances evenly distributed amongthree different geometrical configurations. In the related ex-periments, 100, 200 and 300 local components were testedfor the abstraction of each local data source. We expectthat if each local data item is to be represented by one localGaussian component (the extreme case), the performanceof the proposed approach will be equivalent to that of theoriginal GTM. If less local components are assumed, thevisualization results will start to degrade.

The visualization results obtained for the oil flow datawhere shown in Figure 2. It was observed that the visualiza-tion map using 300 local components for each source was

comparable to that of the original GTM, as shown in Fig-ure 2(d). The visualization result degraded gracefully whenthe number of local components was dropped to 200 andthen to 100. This is consistent to what being anticipated.

S-Curve data another commonly used benchmarkingdataset for testing nonlinear manifold learning algorithms.It is in a 3-D data space of which the embedded manifold’sshape of 2-D and is like the shape of alphabet ‘S’. For theease of visualization, the 2000 data items in the dataset werelabelled. We first divided the dataset into six continuousparts along the data embedded 2-D manifold and were thenlabelled as blue, red, green, yellow, magenta and cyan cir-cles respectively. 30, 60, 90, 120 and 1500 local compo-nents were chosen for the abstractions of each local source.The corresponding visualization results were shown in Fig-ure 3.

−0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8new GTM: Document=30, noLts=1600, noBsn=81

(a) 30 local com-ponents.

−0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8new GTM: Document=60, noLts=1600, noBsn=81

(b) 60 local com-ponents.

−1 −0.5 0 0.5 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1new GTM: Document=90, noLts=1600, noBsn=81

(c) 90 local com-ponents.

−1 −0.5 0 0.5 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1new GTM: Document=120, noLts=1600, noBsn=81

(d) 120 local com-ponents.

−1 −0.5 0 0.5 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1new GTM: Document=1500, noLts=1600, noBsn=81

(e) 1500 localcomponents.

−1 −0.5 0 0.5 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1new GTM: Document=1980, noLts=1600, noBsn=81

(f) The originalGTM.

Figure 3. The visualization of the S-Curve datawhich was partitioned into three equally weighteddistributed local sources for this experiment.

Figure 3(f) reveals the unfolded manifold obtained us-ing the original GTM learned directly from the data. Thoseobtained using the proposed GTM based on different num-bers of local components are shown in Figure 3(a-e). Theunfolded manifold in Figure 3(a) was obtained with only 30components per local source and found to be the worst whencompared with the others using more local components. Inparticular, in the top region of the map, it can been seen thatthe blue circles tangled up with the red ones which meansthat it failed to unfold the top part of the original S-curvedata well. A similar situation was observed for the bottompart. In Figure 3(b-d), the aforementioned two unfolded ar-

eas,i.e. the top and bottom parts, started to be unfoldedand were finally completely unfolded as the number of lo-cal components per source was increased from 30 to 60, 90and 120. When the number of local components was closeto the number of data items, as shown in Figure 3(e), the vi-sualization results was found to be almost equivalent to thatof the original one shown in Figure 3(f).

4 Conclusions

In this paper, we proposed the use of the model-basedapproach for visualizing distributed data with the constraintthat the distributed local data cannot be shared directly.Gaussian mixture models (GMM) was adopted for localdata abstraction and generative topological mapping (GTM)was chosen as the global model for high-dimensional datavisualization. A novel EM-like algorithm was proposed forlearning the global GTM solely based on the aggregatedlocal GMM. The effectiveness of the proposed methodwas rigorously evaluated using a number of datasets withpromising results. Gracefully degrading global visualiza-tion results were obtained as the granularity level of the lo-cal data became coarser. We believe that the positive resultsobtained and the formulation we introduced in this paperhint the potential of the proposed method to be a principledway of mining highly distributed high-dimensional data ina distributed environment with limited bandwidth or highdata privacy concern.

5 Acknowledgement

This work is jointly supported by RGC Central Alloca-tion Research Grant (HKBU 2/03/C) and Hong Kong Bap-tist University FRG Grant (FRG/05-06/I-16).

References

[1] C. M. Bishop, M. Svensen, and C. K. I. Williams. GTM:The generative topographic mapping.Neural Computation,10(1):215–235, 1998.

[2] G. J.McLachlan and K. E. Basford.Mixture Models - Infer-ence and Applications to Clustering. Marcel Dekker, NewYork, 1988.

[3] M. Klusch, S. Lodi, and G. L. Moro. Distributed ClusteringBased on Sampling Local Density Estimates. InProceedingsof International Joint Conference on Artificial Intelligence(IJCAI 2003), pages 485–490, Mexico, August 2003.

[4] S. Merugu and J. Ghosh. Privacy-preserving Distributed Clus-tering using Generative Models. InThe Third IEEE Interna-tional Conference on Data Mining (ICDM’03), Melbourne,FL, November 2003.

[5] X. Zhang and W. K. Cheung. Learning Global Models Basedon Distributed Data Abstractions. InProceedings of Inter-national Joint Conference on Artificial Intelligence (IJCAI2005), Edinburgh, August 2005.