Penalized Clustering of Large-Scale Functional Data...

Penalized Clustering of Large-Scale FunctionalData With Multiple Covariates

Ping MA and Wenxuan ZHONG

In this article we propose a penalized clustering method for large-scale data with multiple covariates through a functional data approach.In our proposed method, responses and covariates are linked together through nonparametric multivariate functions (fixed effects), whichhave great flexibility in modeling various function features, such as jump points, branching, and periodicity. Functional ANOVA is used tofurther decompose multivariate functions in a reproducing kernel Hilbert space and provide associated notions of main effect and interaction.Parsimonious random effects are used to capture various correlation structures. The mixed-effects models are nested under a general mixturemodel in which the heterogeneity of functional data is characterized. We propose a penalized Henderson’s likelihood approach for modelfitting and design a rejection-controlled EM algorithm for the estimation. Our method selects smoothing parameters through generalizedcross-validation. Furthermore, Bayesian confidence intervals are used to measure the clustering uncertainty. Simulation studies and real-dataexamples are presented to investigate the empirical performance of the proposed method. Open-source code is available in the R packageMFDA.

KEY WORDS: Clustering; EM algorithm; Functional data analysis; Mixed-effects model; Smoothing spline.

1. INTRODUCTION

With the rapid advancement in high-throughput technology,extensive repeated measurements have been taken to moni-tor the systemwide dynamics in many scientific investigations.A typical example is temporal gene expression studies, in whicha series of microarray experiments are conducted sequentiallyduring a biological process, for example, cell cycle microarrayexperiments (Spellman et al. 1998). At each time point, mRNAexpression levels of thousands of genes are measured simul-taneously. Collected over time, a gene’s “temporal expressionprofile” gives the scientist some clues as to what role this genemay play during the process. A group of genes with similar pro-files are often “co-regulated” or participants of a common andimportant biological function. Thus clustering genes into homo-geneous groups is a crucial first step in deciphering the underly-ing mechanism. The need to account for intrinsic temporal de-pendency of repeated observations within the same individualrenders traditional methods, such as K-means and hierarchi-cal clustering, inadequate. By casting repeated observations asmultivariate data with certain correlation structure, one ignoresthe time interval and time order of sampling. In addition, miss-ing observations in the measurements yield an unbalanced de-sign, necessitating imputation before applying multivariate ap-proaches, such as the multivariate Gaussian clustering method(MCLUST; Fraley and Raftery 1990).

In addition to the time factor, such repeated measurementsoften contain other covariates, for example, replicates at eachtime point, species in comparative genomics studies (McCar-roll et al. 2004), and treatment groups in case-control studies(Storey, Xiao, Leek, Tompkins, and Davis 2005), as well asmany factors in a factorial designed experiment. Incorporationof multiple covariates adds another layer of complexity. Clus-tering methods that take all of these factors into account remainlacking.

Ping Ma is Assistant Professor (E-mail: [email protected]) and WenxuanZhong is Assistant Professor (E-mail: [email protected]), Department of Sta-tistics, University of Illinois, Champaign, IL 61820. Ma’s research was sup-ported in part by National Science Foundation grant DMS-0723759. The au-thors are grateful to Jun S. Liu, Chong Gu, Yu Zhu, Xuming He, and StevePortnoy for many illuminating discussions on this article, and Kurt Kwast forproviding the yeast microarray data. The authors also thank the editor, the as-sociate editor, the two referees, John Marden, and Adam Martinsek for theirconstructive comments and suggestions that have led to significant improve-ment in this article.

Recently, nonparametric analysis of data in the form ofcurves (i.e., functional data) has been subject to active re-search. (See Ramsay and Silverman 2002, 2005 for a compre-hensive treatment of functional data analysis.) A curve-basedclustering method (FCM) was introduced by James and Sugar(2003) to cluster sparsely sampled functional data. Similar ap-proaches were developed by Luan and Li (2003, 2004) andHeard, Holmes, and Stephens (2006) to analyze temporal geneexpression data. Although these methods model the time fac-tor explicitly, none is designed to accommodate additional fac-tors. Moreover, smoothing-related parameters (e.g., knots anddegrees of freedom) in these methods are the same across allclusters and must be specified a priori. Consequently, they can-not model drastically different patterns among different clus-ters, leading to high false-classification rates. Finally, the com-putational costs of these methods are very high for large-scaledata.

Motivated by analysis of temporal gene expression data, wepropose a flexible functional data clustering method that over-comes the aforementioned obstacles. In our proposed method,responses and covariates are linked together through nonpara-metric multivariate functions (fixed effects), which have greatflexibility in modeling various function features, such as jumppoints, branching, and periodicity. Functional ANOVA is usedto further decompose multivariate functions (fixed effects) in areproducing kernel Hilbert space (RKHS) and provide associ-ated notions of main effect and interaction (Wahba 1990; Gu2002). Parsimonious random effects, complementing the fixedeffects, are used to capture various correlation structures. Themixed-effects models are nested under a general mixture model,in which the heterogeneity of the functional data is character-ized. We propose a penalized Henderson’s likelihood approachfor model fitting and design a rejection-controlled EM algo-rithm for estimation. In this EM algorithm, the E-step is fol-lowed by a rejection-controlled sampling step (Liu, Chen, andWong 1998) to eliminate a significant number of functionalobservations with negligible posterior probabilities of belong-ing to a particular cluster from calculation in the subsequentM-step. The M-step is decomposed into the simultaneous max-imization of penalized weighted least squares in each cluster.

© 2008 American Statistical AssociationJournal of the American Statistical Association

June 2008, Vol. 103, No. 482, Theory and MethodsDOI 10.1198/016214508000000247

625

626 Journal of the American Statistical Association, June 2008

The smoothing parameters associated with the penalty are se-lected by generalized cross-validation (GCV), which can beshown to track a squared error loss asymptotically. Thus ourmethod is data-adaptive and automatically captures some im-portant functional fluctuations. For model selection, we use theBayesian information criterion to select the number of clusters.Moreover, the proposed method provides not only subject-to-cluster assignment, but also the estimated mean function andassociated Bayesian confidence intervals for each cluster. TheBayesian confidence intervals are used to measure the cluster-ing uncertainty. These nice features make the proposed methodvery powerful for clustering large-scale functional data.

The remainder of the article is organized as follows. In Sec-tion 2 we present a nonparametric mixed-effects model repre-sentation for functional data. We consider a mixture model forclustering in Section 3, and simulation and real data analysisin Sections 4 and 5. A few remarks in Section 6 conclude thearticle. Proofs of the theorems are collected in Appendix.

2. NONPARAMETRIC MIXED–EFFECTSREPRESENTATION OF HOMOGENEOUS

FUNCTIONAL DATA

Assuming that the data are homogeneous (i.e., the numberof clusters is one), we present a mixed-effects representation offunctional observations.

2.1 Model Specification

We assume that the functional data of the ith individual, yi =(yi1, . . . , yini

)T , follows the mixed-effects model,

yi = μ(xi ) + Zibi + εi , (1)

where the population mean μ is assumed to be a smooth func-tion defined on a generic domain �, xi = (xi1, . . . , xini

)T isan ordered set of sampling points, bi ∼ N(0,B) is a p × 1random-effects vector associated with a ni × p design matrixZi , and random errors εi ∼ N(0, σ 2I) are independent of bi ’sand of each other. The random-effects covariance matrix B andrandom error variance σ 2 are to be estimated from the data.Model (1) has been studied extensively in the statistical litera-ture (see Wang 1998; Zhang, Lin, Raz, and Sowers 1998; Guand Ma 2005; and references therein).

For multivariate x, where x = (x〈1〉, x〈2〉, . . . , x〈d〉)T , eachentry x〈k〉 takes values in some fairly general domain �k , thatis, � = ⊗d

k=1 �k . Some examples are as follows.

Example 1. � = [0,T ] × {1, . . . , c} to model temporal vari-ation from time 0 to time T under multiple conditions, and� = circle × {1, . . . , s} to model periodicity of a biologicalprocess of multiple species.

The functional ANOVA decomposition of a multivariatefunction μ is

μ(x) = μ0 +d∑

j=1

μj

(x〈j〉

) +d∑

j=1

d∑k=j+1

μjk

(x〈j〉, x〈k〉

)

+ · · · + μ1,...,d

(x〈1〉, . . . , x〈d〉

), (2)

where μ0 is a constant, μj ’s are the main effects, μjk’s are thetwo-way interactions, and so on. The identifiability of the terms

in (2) is ensured by side conditions through averaging operators(see Wahba 1990; Gu 2002).

By using different specifications of the random effect bi andassociated design matrix Zi , model (1) can accommodate vari-ous correlation structures.

Example 2. If we let p = 1 (i.e., bi is a scalar, B = σ 2b ,

and Zi = 1), then we have the same correlation across time.

If we let p = 2 [i.e., bi = (bi1, bi2)T , B = (σ 2

b1σ 2

b1b2σ 2

b1b2σ 2

b2

), and

Zi = (1,xi )], then the difference between the ith subject profileand the mean profile is a linear function in time. The covariancebetween expression values at x1 and x2 for the same individualis σ 2

b1+ (x1 + x2)σ

2b1b2

+ x1x2σ2b2

.

2.2 Estimation

Model (1) is estimated using penalized least squares throughthe minimization of

n∑i=1

(yi − μ(xi ) − Zibi )T (yi − μ(xi ) − Zibi )

+n∑

i=1

σ 2bTi B−1bi + NλM(μ), (3)

for N = ∑i ni , where the first term measures the fidelity of the

model to the data, M(μ) = M(μ,μ) is a quadratic functionalthat quantifies the roughness of μ, and λ is the smoothing para-meter that controls the trade-off between the goodness of fit andthe smoothness of μ. Expression (3) also is referred to as the pe-nalized Henderson’s likelihood because the first two terms areproportional to the joint density (i.e., Henderson’s likelihood)of (yi ,bi ) (Robinson 1991).

To minimize (3), we need only consider smooth functionsin the space {μ :M(μ) < ∞} or a subspace therein. As a ab-stract generalization of the vector spaces used extensively inmultivariate analysis, Hilbert spaces inherit many nice proper-ties of the vector spaces; however, they are too loose to use forfunctional data analysis, because even the evaluation functional[x](f ) = f (x), the simplest functional that may be encoun-tered, is not guaranteed to be continuous in a general Hilbertspace. For example, in the Hilbert space of square-integrablefunctions defined on [0,1], evaluation is not even well de-fined. Consequently, we can focus on a constrained Hilbertspace for which the evaluation functional is continuous. Sucha Hilbert space is referred to as a reproducing kernel Hilbertspace (RKHS), for which Ramsay and Silverman (2005) sug-gested a nickname, continuous Hilbert space. For example, thespace of functions with square-integrable second derivativesis an RKHS if it is equipped with appropriate inner products(Gu 2002). For the evaluation functional [x](·), by the Rieszrepresentation theorem, there exists a nonnegative definite bi-variate function R(x, y), the reproducing kernel, that satisfies〈R(x, ·), f (·)〉 = f (x), called the “representer” of [x](·), in anRKHS. Given an RKHS, we may derive the reproducing ker-nel from the Green function associated with the quadratic func-tional M(μ). (The construction of the reproducing kernel is be-yond the scope of this article; see Wahba 1990 and Gu 2002 fordetails.)

Ma and Zhong: Penalized Clustering of Functional Data 627

The minimization of (3) is performed in a RKHS H ⊆{μ :M(μ) < ∞} in which M(μ) is a square seminorm. Toincorporate (2) in estimating multivariate functions, we con-sider μj ∈ H〈j〉, where H〈j〉 is a RKHS with tensor sum de-composition H〈j〉 = H0〈j〉 ⊕ H1〈j〉, where H0〈j〉 is the finite-dimensional “parametric” subspace consisting of parametricfunctions and H1〈j〉 is the “nonparametric” subspace consist-ing of smooth functions. The induced tensor product space is

H =d⊗

j=1

H〈j〉 =⊕S

[(⊗j∈S

H1〈j〉

)⊗

(⊗j /∈S

H0〈j〉

)]

=⊕S

HS ,

where the summation runs over all subsets S ⊆ {1, . . . , d}. Thesubspaces HS form two large subspaces, NM = {η :M(μ) =0}, which is the null space of M(μ), and H NM , with thereproducing kernel RM(·, ·). The solution of (3) has expression

μ(x) =m∑

ν=1

dνφν(x) +T∑

i=1

ciRM(si, x), (4)

where {φν}mν=1 is a basis of NM , dν and ci are the coefficients,and s = (s1, . . . , sT ) is a distinct combination of all xij (i =1, . . . , n, j = 1, . . . , ni ).

Example 3. Consider the temporal variation under a treat-ments. Take the fixed effect as μ(t, τ ), where τ ∈ {1, . . . , a}denotes the treatment levels. We can decompose

μ(t, τ ) = μ∅ + μ1(t) + μ2(τ ) + μ1,2(t, τ ),

where μ∅ is a constant, μ1(t) is a function of t satisfyingμ1(0) = 0, μ2(τ ) is a function of τ satisfying

∑aτ=1 μ2(τ ) = 0,

and μ1,2(t, τ ) satisfies μ1,2(0, τ ) = 0, ∀τ and∑a

τ=1 μ1,2(t,

τ ) = 0, ∀t . The term μ∅ + μ1(t) is the “average variation,” andthe term μ2(τ ) + μ1,2(t, τ ) is the “contrast variation.”

For flexible models, we can use

M(μ) = θ−11

∫ T

0(d2μ1/dt2)2 dt

+ θ−11,2

∫ T

0

a∑τ=1

(d2μ1,2/dt2)2 dt, (5)

which has a null space NM of dimension 2a. A set of φν isgiven by{

1, t, I{j}(τ ) − 1/a,(I{j}(τ ) − 1/a

)t, j = 1, . . . , a − 1

},

and the function RM is given by

RM(t1, τ1; t2, τ2)

= θ1

∫ T

0(t1 − u)+(t2 − u)+ du

+ θ1,2(I{τ1}(τ2) − 1/a

)∫ T

0(t1 − u)+(t2 − u)+ du

(see, e.g., Gu 2002, sec. 2.4.4). To force an additive model,

μ(t, τ ) = μ∅ + μ1(t) + μ2(τ ), (6)

which yields parallel curves at different treatments, we can setθ1,2 = 0 and remove (I{j}(τ ) − 1/a)t from the list of φν .

Substituting (4) into (3), we have

(y − Sd − Rc − Zb)T (y − Sd − Rc − Zb)

+ bT �b + NλcT Qc, (7)

where y = (yT1 , . . . ,yT

n )T , d = (d1, . . . , dm)T , c = (c1, . . . ,

cT )T , b = (bT1 , . . . ,bT

n )T , S = (ST1 , . . . ,ST

n )T with the (k, ν)thentry of the ni × m matrix Si equal to φν(tik), R = (RT

1 , . . . ,

RTn )T with the (l, j)th entry of the ni × T matrix Ri equal

to RM(til, sj ), the design matrix Z = diag(Z1, . . . ,Zn), � =σ 2 diag(B−1, . . . ,B−1), and Q is a T × T matrix with the(j, k)th entry equal to RM(sj , sk).

Differentiating (7) with respect to d, c, and b and setting thederivatives to 0, we have

( ST S ST R ST ZRT S RT R + (Nλ)Q RT ZZT S ZT R ZT Z + �

)⎛⎝ d

cb

⎞⎠ =

( ST yRT yZT y

). (8)

The system (8) can be solved through the pivoted Cholesky de-composition followed by backward and forward substitutions(see, e.g., Kim and Gu 2004 for details).

The fitted values y = Sd + Rc + Zb of (3) can be written asy = A(λ,�)y, where A(λ,�) is the smoothing matrix

A(λ,�)

= (S,R,Z)

×( ST S ST R ST Z

RT S RT R + (Nλ)Q RT ZZT S ZT R ZT Z + �

)+ ( ST

RT

ZT

), (9)

and C+ denotes the Moore–Penrose inverse of C satisfyingCC+C = C, C+CC+ = C+, (CC+)T = CC+, and (C+C)T =C+C.

With varying smoothing parameters λ (including θ ) and cor-relation parameters �, (8) defines an array of possible esti-mates, in which we need to choose a specific one in practice.A classic data-driven approach for selecting the smoothing pa-rameter λ is GCV, as proposed by Craven and Wahba (1979).Treating the correlation parameters � as extra smoothing para-meters, we adopt the approach of Gu and Ma (2005) to estimateλ and the correlation parameters � simultaneously by minimiz-ing the GCV score,

V (λ,�) = N−1yT (I − A(λ,�))2y{N−1 tr(I − A(λ,�))}2

. (10)

Because the GCV score V (λ,�) is nonquadratic in λ and�, we can use standard nonlinear optimization algorithms tominimize the GCV as a function of the tuning parameters. Inparticular, we used the modified Newton algorithm developedby Dennis and Schnabel (1996) to find the minimizer. The dis-tinguishing feature of GCV is that its asymptotic optimality canbe justified in a decision-theoretic framework. We can define aquadratic loss function as

L(λ,�) = 1

N

n∑i=1

(yi − μ(xi ) − Zibi )T (yi − μ(xi ) − Zibi ).


Under general conditions, Gu and Ma (2005) showed that theGCV tracks the loss function asymptotically,

V (λ,�) − L(λ,�) − 1

N

n∑i=1

εTi εi = op(L(λ,�)).

Note that εi does not depend on λ and �. It then followsthat the minimizer of the GCV score V (λ,�) approximatelyminimizes the loss function L(λ,�).

2.3 Bayesian Confidence Intervals

Unlike confidence estimates in parametric models, a rig-orously justified interval estimate is a rarity in nonparamet-ric functional estimation. An exception is the Bayesian confi-dence interval developed by Wahba (1983) from a Bayes model.A nice feature of Bayesian confidence intervals is that theyhave a certain across-the-function coverage property (see Ny-chka 1988). In this section we derive the posterior mean andvariance for constructing Bayesian confidence intervals in oursetting.

The regularization is equivalent to imposing a prior on thefunctional form of μ(x). To see this,we decompose μ = f0 +f1, where f0 has a diffuse prior in the space NM and f1 has anindependent Gaussian process prior with mean 0 and covariance

E[f1(sk)f1(sl)] = σ 2

NλRM(sk, sT )Q+RM(s, sl). (11)

The minimizer of (3) can be shown to be the posterior meanunder the foregoing prior by the following theorem.

Theorem 1. With the prior for μ specified earlier and ageneric np × 1 vector z, the posterior mean of μ(x) + zT b hasexpression

E[μ(x) + zT b|y] = φT d + ξT c + zT b, (12)

where φ is m × 1 with the νth entry φν (x), ξ is T × 1 with theith entry R(si, x), d, c, and b are the solutions of (8).

The posterior variance is given in the following theorem.

Theorem 2. Under the model specified in Theorem 1, theposterior variance has expression:

Nλ

σ 2var[μ(x) + zT b|y]

= ξT Q+ξ + NλzT �+z + φT (ST W−1S)−1φ

− 2φT (ST W−1S)−1ST W−1RQ+ξ

− 2NλφT (ST W−1S)−1ST W−1Z�+z

− (ξT Q+RT + NλzT �+Z)

× (W−1 − W−1S(ST W−1S)−1ST W−1)

× (RQ+ξ + NλZ�+z),

where W = RQ+RT + NλZ�+ZT + NλI.

The proofs of the foregoing two theorems are given in Ap-pendix. Using Theorems 1 and 2, we construct the 100(1 −α)% Bayesian confidence intervals as, E[μ(x) + zT b|y] ±�(1−α/2)−1

√var[μ(x) + zT b|y], where �(1−α/2)−1 is the

100(1 − α/2) percentile of the standard Gaussian distribution.Letting z = 0, we get Bayesian confidence intervals for μ(x).

Note that the construction of Bayesian confidence intervals ispointwise. Whether the across-the-function coverage propertyof Nychka (1988) holds in our case is unclear.

3. THE MIXTURE MODEL

Based on the mixed-effects representation of homogeneousfunctional data, we now present a mixture model for character-izing the heterogeneity.

3.1 The Model Specification

When the population is heterogeneous, we assume that theith functional observation can be modeled as

yi = μk(xi ) + Zibi + εi with probability pk, (13)

where k = 1, . . . ,K , the kth cluster’s mean μk is a smoothfunction defined on a generic domain �, bi ∼ N(0,Bk) is ap × 1 random-effects vector associated with a ni × p designmatrix Zi , εi ∼ N(0, σ 2I) are random errors independent ofthe bi ’s and of one another, cluster probabilities pk satisfy∑K

k=1 pk = 1, and K is the number of clusters in the popula-tion.

To ease the computation, we introduce a “latent” member-ship labeling variable Jik such that Jik = 1 indicates that indi-vidual i belongs to the kth cluster and Jik = 0 otherwise. Thuswe have the probability that Jik = 1 is pk . The mixture Hender-son’s likelihood is seen to be

n∑i=1

logK∑

k=1

[pkfy(yi;bi , Jik = 1)fb(bi;Jik = 1)],

where fy and fb are probability density functions for yi and bi .

3.2 Estimation

The negative penalized Henderson’s likelihood of completedata (yi , Jik) where i = 1, . . . , n, is seen to be

Lc = constant −n∑

i=1

K∑k=1

Jik logpk

+ 1

2σ 2

n∑i=1

K∑k=1

Jik

[(yi − μk(xi ) − Zibi )

T

× (yi − μk(xi ) − Zibi ) + σ 2bTi B−1

k bi

]+

K∑k=1

NλkM(μk), (14)

where λk is the smoothing parameter for μk .Once the penalized Henderson’s likelihood (14) is obtained,

the EM algorithm (Dempster, Laird, and Rubin 1977; Green1990) can be derived as follows: The E-step simply requirescalculation of

wik = pkϕ(yi;μk(xi ),�k)∑Kl=1 plϕ(yi;μl(xi ),�l)

, (15)

where �k = ZiBkZTi +σ 2I and ϕ is the Gaussian density func-

tion.


The M-step requires the conditional minimization of theequation

−K∑

k=1

n∑i=1

wik logpk (16)

+ 1

2σ 2

n∑i=1

K∑k=1

wik

[(yi − μk(xi ) − Zibik)

T

× (yi − μk(xi ) − Zibik) + σ 2bTikB−1

k bik

]+

K∑k=1

NλkM(μk), (17)

where bik is bi given the membership Jik . Thus the M-step isequivalent to minimizing (16) and (17) separately.

By minimizing (16), we have

pk = 1

n

n∑i=1

wik for k = 1, . . . ,K. (18)

By minimizing (17), we can minimize the following K equa-tions simultaneously

n∑i=1

wik

[(yi − μk(xi ) − Zibik)

T (yi − μk(xi ) − Zibik)

+ σ 2bTikB−1

k bik

] + NλkM(μk), k = 1, . . . ,K, (19)

where 1/2σ 2 is absorbed into λk . The minimization of (19) isperformed in the RKHS H ⊆ {η :M(μ) < ∞}. Substituting so-lution (4) into (19), we have

(y − Sdk − Rck − Zbk)T Wk(y − Sdk − Rck − Zbk)

+ bTk W1/2

k �kW1/2k bk + NλkcT

k Qck, (20)

where y = (yT1 , . . . ,yT

n )T , dk = (d1k, . . . , dmk)T , ck = (c1k,

. . . , cT k)T , bk = (bT

1k, . . . ,bTnk)

T , S = (ST1 , . . . ,ST

n )T with the(k, ν)th entry of the ni × m matrix Si equal to φν(tik), R =(RT

1 , . . . ,RTn )T with the (l, j)th entry of the ni × T matrix Ri

equal to RM(til, sj ), the design matrix Z = diag(Z1, . . . ,Zn),Wk = diag(w1kIn1 , . . . ,wnkInn), Wk = diag(w1kIp, . . . ,

wnkIp), �k = σ 2 diag(B−1k , . . . ,B−1

k ), and Q is a T ×T matrixwith the (j, k)th entry equal to RM(sj , sk).

Writing (20) in a more compact form, we have

(ywk − Swkdk − Rwkck − Zbwk)T

× (ywk − Swkdk − Rwkck − Zbwk)

+ bTwk�kbwk + NλkcT

k Qck, (21)

where ywk = W1/2k y, Swk = W1/2

k S, Rwk = W1/2k R, Zwk =

W1/2k ZW−1/2

k , and bwk = W1/2k bk . Then (21) can be minimized

using the techniques developed in Section 2.The variance of measurement error is estimated as

σ 2 = 1

N

n∑i=1

K∑k=1

wik(yi − μk(xi ) − Zibik)T

× (yi − μk(xi ) − Zibik). (22)

The algorithm iterates through (15), (18), (21), and (22) untilall the parameters converge.

The selection of the smoothing parameters �k and λk playsan important role in the proposed algorithm. When first run-ning our algorithm, in each iteration, we selected the optimalsmoothing parameters for each cluster using GCV in (21). Onceall parameters converge, we fixed the selected smoothing para-meters and ran our algorithm for fixed smoothing parameters.

Once we fit the mixture model to the data, we could give aprobabilistic (soft) clustering of each observation yi ; that is, foreach yi , wi1, . . . ,wiK give the estimated probabilities that thisobservation belongs to the first, second, . . . , and K th compo-nents of the mixture. But in many practical settings, it is highlydesirable to give hard clusterings of these observations by as-signing each observation to one component of the mixture. Inthe rest of the article, we adopt the hard clustering of McLach-lan and Peel (2001) by estimating the membership label

Jik ={

1 if k = arg maxh wih

0 otherwisewhere k = 1, . . . ,K and i = 1, . . . , n.

3.3 Efficient Computation With Rejection Control

With thousands of observations under consideration, theE-step (15) results in a huge number of wik’s, many of whichare extremely small. With the presence of these small wik’s, thecalculation of matrices involved in the M-step (21) is expensive,unstable, and sometimes even infeasible. To alleviate the com-putation and stabilize the algorithm, we propose adding a rejec-tion control step (Liu et al. 1998; Ma, Castillo-Davis, Zhong,and Liu 2006) in the EM algorithm and refer to the modifiedalgorithm as the rejection-controlled EM algorithm.

First, we set up a threshold value c (e.g., c = .05). Given thisthreshold value, we introduce the following rejection controlledstep:

w∗ik =

{wik if wik > c

c with probability wik/c if wik ≤ c

0 with probability 1 − wik/c if wik ≤ c.

The resulting w∗ik must be normalized, w∗∗

ik = w∗ik/

∑k w∗

ik .Then we replace wik by w∗∗

ik immediately after the E-step (15).Note that when c = 0, the proposed algorithm is exactly theoriginal EM algorithm, whereas when c = 1, the proposed al-gorithm reduces to a variant of the Monte Carlo EM algorithm(Wei and Tanner 1990). In this way, it is possible to make ac-curate approximations during the E-step while greatly reducingthe computation of the M-step.

Finally, to avoid local optima, we run the rejection-controlledEM with multiple chains. In practice, we first set the threshold c

close to 1 at an early stage of the iterations to expedite the calcu-lation, then gradually lower c so that the algorithm can achievea better approximation of the original EM.

A critical issue arising from the new algorithm is how tochoose an appropriate stopping rule. For the original EM algo-rithm, the likelihood function increases after each iteration, sowe can stop the iteration when the likelihood does not change.But for the rejection-controlled EM algorithm, the likelihoodfunctions fluctuates because of the sampling scheme, so a stop-ping rule like those used in the Gibbs sampler is used. Whenthe likelihood function is no longer increasing for several con-secutive iterations, we stop and choose the estimates with thehighest likelihood.


3.4 Selection of the Number of Clusters

The success of our proposed methods depends heavily onthe selection of the number of clusters K . A natural choice inmodel-based clustering is to use the BIC. The BIC imposes apenalty on the total number of parameters, scaled by the loga-rithm of sample size, so as to strike a balance between the good-ness of fit and the model complexity. A critical issue when us-ing BIC in nonparametric settings is to determine the effectivenumber of parameters. Here we use the trace of the smoothingmatrix to approximate the number of parameters in each cluster(Hastie and Tibshirani 1990; Gu 2002). Thus, under our model,the BIC is

BIC = −2n∑

i=1

logK∑

k=1

pkϕ(yi;μk(xi ),�k)

+(

K∑k=1

tr Ak(λk,�k) + P

)logN, (23)

where Ak is the smoothing matrix for the kth cluster as definedin (9) and P is the number of free parameters in pk , λk , and �k ,where k = 1, . . . ,K .

4. SIMULATION

To assess the performance of the proposed method, we car-ried out extensive analyses on simulated data sets.

4.1 Simulation 1

This simulation was designed to demonstrate the perfor-mance of the proposed method when the underlying clusters’mean functions are different for different clusters. First, we gen-erated 100 replicates of samples according to

y1ijτ = 3 sin(6πtj )(1 − tj ) + 2I{1}(τ ) − 1 + ε1ijτ ,

i = 1, . . . ,30;y2ijτ = 3 sin(6πtj )(1 − tj ) + ε2ijτ ,

i = 1, . . . ,40;y3ijτ = 1,980t7

j (1 − tj )3 + 858t2

j (1 − tj )10 − 2 + ε3ijτ ,

i = 1, . . . ,50;and

y4ijτ = 3 sin(2πtj ) + 2I{1}(τ ) − 1 + ε4ijτ , i = 1, . . . ,30,

where tj = 1/15,2/15, . . . ,1, τ = 0,1, indicator functionI{1}(τ ) = 1 if τ = 1 and 0 otherwise, and random errors ε weregenerated from a Gaussian distribution with mean 0 and covari-ance matrix as

var[εlijτ ] = 1, cov(εlijτ1, εlikτ2

) = .2, for l = 1,3,

and

var[εlijτ ] = 1.2, cov(εlijτ1, εlikτ2

) = .4, for l = 2,4.

We analyzed the simulated data using the proposed methodwith the mixture model

yi = μk(t, τ ) + bi1 + εi with probability pk,

where k = 1, . . . ,K , τ = 1,2 for two groups and bi ∼ N(0, σ 2b )

is the individual specific random effect. The important featureof the simulated data is that the true mean curves in two groups,indexed by τ , are either identical or parallel. We built this infor-mation into our method by enforcing the additive model (6). Weused the penalized Henderson’s likelihood for estimation withroughness penalty M(μ) = ∫ 1

0 (d2μ1/dt2)2 dt .We compared our method with MCLUST (Fraley and Raftery

1990), FCM classification likelihood (FCMc), and FCM mix-ture likelihood (FCMm) (James and Sugar 2003). Because thenumber of clusters must be specified a priori in the partiallyimplemented FCM software, we gave a significant starting ad-vantage to the FCM algorithm by letting the number of clustersbe the true number of clusters (four). For MCLUST, we reportthe clustering result with optimal BIC, which was estimatedfrom eight models with different covariance structures. The es-timated mean curves using the proposed method for each clusterand the true curves of one sample are plotted in Figure 1.

For comparison, we need a measure of the agreement of theclustering results with the true cluster membership. A popularone is the Rand index, the percentage of concordance pairs overall possible data pairs. Hubert and Arabie (1985) proposed anadjusted Rand index that takes 1 as the maximum value whentwo clustering results are the same and is expected to be 0when two clustering results are independent. The boxplots ofthe adjusted Rand indices for clustering results using the differ-ent methods are presented in Figure 2(a). We found that across100 samples, the average of the adjusted Rand indices for theproposed method is .9676 (median is .9838), whereas thoseof MCLUST, FCMm, and FCMc are .7553, .8936, and .8896.Moreover, the interquartile ranges of the adjusted Rand indicesof the proposed method are .0565 (.0972, .2189, and .2262 forMCLUST, FCMm, and FCMc). These results suggest that theproposed method outperforms FCMc and FCMm (even underthe ideal scenario where the true number of clusters is providedto FCMc and FCMm a priori), as well as MCLUST.

4.2 Simulation 2

This simulation is designed to demonstrate the performanceof the proposed method when the underlying clusters’ covari-ance structures are different for different clusters. We generatedreplicates of samples according to

y1ijτ = 3 sin(6πtj )(1 − tj ) + 2I{1}(τ ) − 1 + ε1ijτ ,

i = 1, . . . ,30;y2ijτ = 3 sin(6πtj )(1 − tj ) + 2I{1}(τ ) − 1 + ε2ijτ ,

i = 1, . . . ,40;y3ijτ = 1,980t7

j (1 − tj )3 + 858t2

j (1 − tj )10 − 2 + ε3ijτ ,

i = 1, . . . ,50;and

y4ijτ = 1,980t7j (1 − tj )

3 + 858t2j (1 − tj )

10 − 2 + ε4ijτ ,

i = 1, . . . ,30,

where tj = 1/15,2/15, . . . ,1, τ = 0,1, and indicator functionI{1}(τ ) = 1 if τ = 1 and 0 otherwise. We generated random er-rors ε from a Gaussian distribution with mean 0 and covariance


Figure 1. The estimated mean curves (- - - - -) and 95% Bayesian confidence intervals for one simulated data set. The true functions aresuperimposed as solid lines.

matrix

var[εlijτ ] = 1,

cov(εlijτ1, εlikτ2

) ={−.5 if τ1 �= τ2, for l = 1,3

.5 otherwise

and

var[εlijτ ] = 1, cov(εlijτ1, εlikτ2

) = .5 for l = 2,4.

We analyzed the simulated data using the proposed methodwith the mixture model

yi = μk(t, τ ) + Zibi + εi with probability pk,

where k = 1, . . . ,K , τ = 1,2 for two groups; bi = (bi1, bi2) isthe random effect with var(bi1) = var(bi2) = σ 2

b ; and cov(bi1,

bi2) = σ 212, Zi = diag(115,115) is a 30 × 2 random-effects de-

sign matrix. We enforced additive model (6). We use the pe-

(a) (b)

Figure 2. Boxplots of the adjusted Rand indices for clustering results of 100 simulated samples against the true clustering membership forall methods for simulations 1 (a) and 2 (b).


nalized Henderson’s likelihood for estimation with roughnesspenalty M(μ) = ∫ 1

0 (d2μ1/dt2)2 dt .The boxplots of the adjusted Rand indices for clustering

results using different methods are presented in Figure 2(b).We found that across 100 samples, the average of the ad-justed Rand indices for the proposed method is .4410, whereasthose of MCLUST, FCMm, FCMc are .3476, .3520, and .3530.These results suggest that the proposed method still outper-forms FCMc and FCMm as well as MCLUST.

5. REAL DATA EXAMPLES

5.1 Comparative Genomic Study of Fruitfly andWorm Gene Expressions

Development is an important biological process that sharesmany common features among different organisms. It is wellknown that D. melanogaster (fruitfly) and C. elegance (worm)are two highly diverged species, the last common ancestor ofwhich existed about 1 billion years ago. Their development isan area of active research. Arbeitman et al. (2002) measuredthe mRNA levels of 4,028 genes in D. melanogaster usingcDNA microarrays during 62 time points starting at fertilizationand spanning embryonic, larval, and pupal (metamorphosis)stages and the first 30 days of adulthood. mRNA was extractedfrom mixed male and female populations until adulthood, whenmales and females were sampled separately. Jiang et al. (2001)reported a cDNA microarray experiment for 17,871 genes overthe life cycle of C. elegans at 6 time points, including eggs,larval stages (L1, L2, L3, and L4), and young adults.

To study the genomic connections in expression patternsacross the two species, we combined the gene expression datasets of Arbeitman et al. (2002) and Jiang et al. (2001) using theorthologous genes provided by McCarroll et al. (2004), whichresulted in a merged expression data set containing 808 orthol-ogous genes. We analyzed the data using the proposed methodwith the mixture model

yi = μk(t, τ ) + bi1 + εi ,

with probability pk , where k = 1, . . . ,K , τ = 1 for fruitflyand τ = 2 for worm and bi ∼ N(0, σ 2

b ) is the gene-specificrandom effect. We used the penalized Henderson’s likelihoodwith roughness penalty M of the form (5). We modeled sexdifferentiation of the fruitfly by a branching spline (Silvermanand Wood 1987), the general analytic form of which with twobranches on the right is

μ(t) =

⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩

m∑ν=1

dνφν(t) +k∑

i=1

ciRM(si, t) if t ≤ sk

m∑ν=1

dνφν(t) +k∑

i=1

ciRM(si, t)

+T∑

i=k+1

c1iRM(si − sk, t − sk) if t > sk

m∑ν=1

dνφν(t) +k∑

i=1

ciRM(si, t)

+T∑

i=k+1

c2iRM(si − sk, t − sk) if t > sk ,

(24)

where sk is the branching point and the second and third rowsare expressions of the two branches. A cubic smoothing splinewas used. The 808 genes were clustered by our method into 34clusters. Biological functions of genes in each cluster were an-notated using Gene Ontology, and Bonferroni-corrected p val-ues of biological function enrichment were calculated basedon the hypergeometric distribution (Castillo-Davis and Hartl2003). Of the 34 clusters discovered, 21 clusters exhibited sig-nificant biological function overrepresentation (p value < .05).The estimated mean gene expression curves of three clustersand their 95% Bayesian confidence intervals are given in Fig-ure 3.

In cluster A, which consists of 31 genes, gene expressionsof worms have peaks at eggs, larvae, and young adults. In thesame cluster, we observed that fruitfly gene expressions thatare up-regulated during embryogenesis are also up-regulatedduring metamorphosis, suggesting that many genes used forpattern formation during embryogenesis (the transition fromegg to larva) are redeployed during metamorphosis (the transi-tion from larva to adult). Consistently, this cluster is enrichedfor genes involved in embryonic development (p = .0003),postembryonic body morphogenesis (p = .007), and mRNAprocessing (p = .002), among others.

In cluster B, consisting of 24 genes, gene expressions ofworms increase starting at eggs until they reach a peak at alate larval stage, then decrease during adulthood. But we ob-served that fruitfly gene expressions that are down-regulatedduring embryogenesis are up-regulated during metamorpho-sis and adulthood, suggesting that many genes are involvedin development. The enriched gene functions are embryonic(p = .02), larval development (p = .008), and growth regula-tion (p < 10−5).

Cluster C contains 25 genes. For worms, gene expressionshave peaks at larva and adult stages. An overrepresentation ofsuch gene functions as reproduction (p < 10−6), larval devel-opment (p < 10−7) are present in this cluster. Fruitflies showpeaks in gene expression in the early embryo and in older fe-males (but not males). An overrepresentation of such gene func-tions as reproduction (p < 10−6) and embryonic development(p < 10−5) are present in this cluster. Among related functions,this cluster also contains functions of female gamete genera-tion, growth, and positive regulation of growth rate. Thus genesof this cluster are believed to participate in sex determination,female egg production and growth regulation.

5.2 Budding Yeast Gene Expression Under Aerobicand Anaerobic Conditions

To study the oxygen-responsive gene networks, Lai,Kosorukoff, Burke, and Kwast (2006) used cDNA microarrayto monitor the gene expression changes of wild-type buddingyeast (Saccharomyces cerevisiae) under aerobic and anaerobicconditions in a galactose medium. Under aerobic conditions,the oxygen concentration was gradually decreased until oxygenwas exhausted over a 10-minute period. After 24 hours of anaer-obiosis, the oxygen concentration was progressively increasedback to the normal level during another of 10-minute periodknown as the anaerobic conditions. Microarray experimentswere conducted at 14 time points under aerobic conditions and


Figure 3. Estimated mean expression curves and 95% Bayesian confidence intervals (gray bands) for clusters A (a), B (b), and C (c) discov-ered in the worm-fly temporal expression data. Vertical solid lines separate worms (eggs, larva, and young adult are represented separately bydashed lines in the left frame) and fruitflies (embryogenesis, larva, pupa, and adult stages are separated by dash lines in the right frame). Adultfruitfly male and female mean expression curves are labeled M and F.

10 time points under anaerobic conditions. A reference samplepooled from all time points was used for hybridization.

For their analysis, Lai et al. (2006) normalized gene expres-sions to gene expressions of time 0 and filtered out differentiallyexpressed genes. We used the normalized expressions at 23 timepoints of 2,388 differentially expressed genes for our clusteringanalysis. We modeled normalized gene expression yi of the ithgene using the mixture model,

yi = μk(t, τ ) + bi1 + εi with probability pk,

where k = 1, . . . ,K , τ = 1 for aerobic conditions and τ = 2for anaerobic conditions, and bi ∼ N(0, σ 2

b ) is the gene-specificrandom effect. We fit the model using the penalty (5) with

a = 2. In total, 2,388 genes were clustered into 28 clusters usingour method. We used FunSpec (Robinson, Grigull, Mohammad,and Hughes 2002) for gene annotation and biological functionenrichment analysis. We found 26 out of 28 clusters with over-represented biological functions. The estimated mean gene ex-pression profiles and associated Bayesian confidence intervalsof three clusters are given in Figure 4.

In cluster A, which consists of 57 genes, the estimated meanexpression dropped progressively as oxygen level dropped, sug-gesting that the genes in this cluster were transiently down-regulated in response to anaerobiosis. Furthermore, the es-timated mean expression increased as oxygen concentrationshifted back to the normal level. Accordingly, genes involved


Figure 4. Estimated mean expression curves and 95% Bayesian confidence intervals (gray bands) for cluster A (a), B (b), and C (c) discoveredin the yeast aerobic and anaerobic expression data. The aerobic (left) and anaerobic (right) conditions are separated by two vertical lines.

in respiration, lipid fatty acid and isoprenoid biosynthesis, andcell defense were overrepresented in this cluster (p ≤ 10−5).

In contrast to cluster A, cluster B (85 genes) consistedof genes involved in various biosyntheses, metabolism, andcatabolism, such as glucose metabolism (p ≤ 10−6). Thesebiological processes are necessary to maintain the basic liv-ing needs of yeast cells. Interestingly, the alcohol biosynthesisand metabolism also were enriched in this cluster. Consistentwith biological function overrepresentation, the estimated meanexpression was up-regulated in aerobic conditions and down-regulated in anaerobic conditions.

We had 70 genes in cluster C, where the estimated meangene expression increased at the beginning and then droppedrapidly under aerobic conditions. Under anaerobic conditions,the estimated mean gene expression was up-regulated. In this

cluster, respiratory deficiency and carbon utilization also wereoverrepresented (p ≤ 10−8). The initial up-regulation of geneexpression under aerobic conditions can be explained in partby the fact that the cell increases energy up-take through otherbiological processes, such as carbon utilization, when oxygendecreases. But as the oxygen level continues to drop, theseprocesses are replaced by more energy-efficient processes, suchas glucose metabolism. Under the anaerobic conditions, theseprocesses are revitalized as oxygen level increases.

6. DISCUSSION

In this article we have proposed a clustering method forlarge-scale functional data with multiple covariates. We havebuilt nonparametric mixed-effects models that were nested un-der a mixture model. We used the penalized Henderson’s like-


lihood for estimation and data-driven smoothing parameters,selected through GCV validation, to automatically capture thefunctional features. The rejection-controlled EM algorithm wasdesigned to reduce the computational cost for large-scale data.The simulation analyses suggest that the proposed methodoutperforms the existing clustering methods. Moreover, theBayesian interpretation of the proposed method allows the de-velopment of an equivalent fully Bayesian functional data clus-tering method that can accommodate additional genomic andproteomic information for gene expression studies. Althoughit was motivated for clustering temporal expression data, ourproposed method has a wide spectrum of applications, includ-ing those involving seismic wave data arising from geophysi-cal research (Wang, de Hoop, van der Hilst, Ma, and Tenorio2006; Ma, Wang, Tenorio, de Hoop, and van der Hilst 2007).The calculations reported in this article were performed in R;open-source code is available in the R package MFDA.

As a sequel to this work, a clustering method for discretedata, especially those arising from temporal text mining, is un-der active development.

APPENDIX: PROOFS

Proof of Theorem 1

Note that if we specify the prior for f0 as a Gaussian process withmean 0 and covariances E[f0(sk)f0(sl)] = τ2 ∑m

ν=1 φν(sk)φν(sl),

then when τ2 → ∞, the prior for f0 becomes a diffuse prior (seeWahba 1983; Gu 2002). Assuming that f0(t) has a Gaussian processprior specified earlier, f1(x) has a Gaussian process prior specified asin Theorem 1, and b follows a normal distribution with mean 0 andvariance–covariance matrix B, we can derive that the joint distributionof y and f0(x) + f1(x) + zT b follows a Gaussian distribution withmean 0 and covariance matrix(

bFV+FT + τ2SST + σ 2I bFV+ξ + τ2Sφ

bξT

V+FT + τ2φT ST bξT

V+ξ + τ2φT φ

), (A.1)

where ξ = (R1(s1, x), . . . ,R1(sT , x), z)T is (T + p) × 1, φ is m × 1with the νth entry φν(t), F = (R,Z), and V + is the Moore–Penroseinverse of V = diag(Q, 1

Nλ�) satisfying VV+FT = FT .

Standard calculation yields

E[μ(x) + zT b|y]= (bξT V+FT + τ2φT ST )(bFV+FT + τ2SST + σ 2I)−1y

= ρφT ST (W + ρSST )−1y + ξT V+FT (W + ρSST )−1y,

where ρ = τ2/b, Nλ = σ 2/b, and W = FV+FT + NλI. Now, lettingρ → ∞, we have

limρ→∞(ρSST + W)−1 = W−1 − W−1S(ST W−1S)−1ST W−1 (A.2)

and

limρ→∞ρST (ρSST + W)−1 = (ST W−1S)−1ST W−1. (A.3)

(See Wahba 1983 and Gu 2002 for the proof.) Therefore,limτ 2→∞ E[μ(x) + zT b|y] = φT d + ξT c, where

d = (ST W−1S)−1ST W−1y,(A.4)

c = V+FT(W−1 − W−1S(ST W−1S)−1ST W−1)

y.

It is straightforward to verify that the d and c given in (A.4) sat-isfy (8).

Proof of Theorem 2

The posterior variance can be easily calculated using expres-sion (A.1) as follows:

var[μ(x) + zT b|y] = ξT V+ξ + ρφT φ − (ξT V+FT + ρφT ST )

× (W + ρSST )−1(FV+ξ + ρSφ).

Note that limρ→∞ ρI − ρ2ST (ρSST + W)−1S = (ST W−1S)−1

and VV+FT = FT . Therefore, as ρ → ∞, we have

limτ 2→∞

var[μ(x) + zT b|y]/b

= ξT V+ξ + φT (ST W−1S)−1φ

− 2φT (ST W−1S)−1ST W−1FV+ξ

− ξT V+FT(W−1 − W−1S(ST W−1S)−1ST W−1)

FV+ξ .

[Received June 2007. Revised January 2008.]

REFERENCES

Arbeitman, M., Furlong, E., Imam, F., Johnson, E., Null, B. H., Baker, B. S.,Krasnow, M., Scott, M. P., Davis, R. W., and White, K. P. (2002), “GeneExpression During the Life Cycle of Drosophila melanogaster,” Science, 297,2270–2275.

Castillo-Davis, C., and Hartl, D. (2003), “Genemerge: Post-Genomic Analysis,Data-Mining and Hypothesis,” Bioinformatics, 19, 891–892.

Craven, P., and Wahba, G. (1979), “Smoothing Noisy Data With Spline Func-tions: Estimating the Correct Degree of Smoothing by the Method of Gener-alized Cross-Validation,” Numerische Mathematik, 31, 377–403.

Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977), “Maximum LikelihoodFrom Incomplete Data via the EM Algorithm” (with discussion), Journal ofthe Royal Statistical Society, Ser. B, 39, 1–37.

Dennis, J. E., and Schnabel, R. B. (1996), Numerical Methods for Uncon-strained Optimization and Nonlinear Equations (corrected reprint of the 1983original), Philadelphia: SIAM.

Fraley, C., and Raftery, A. E. (1990), “Model-Based Clustering, DiscriminantAnalysis, and Density Estimation,” Journal of the American Statistical Asso-ciation, 97, 611–631.

Green, P. J. (1990), “On the Use of the EM Algorithm for Penalized LikelihoodEstimation,” Journal of the Royal Statistical Society, Ser. B, 52, 443–452.

Gu, C. (2002), Smoothing Spline ANOVA Models, New York: Springer-Verlag.Gu, C., and Ma, P. (2005), “Optimal Smoothing in Nonparametric Mixed Effect

Models,” The Annals of Statistics, 33, 1357–1379.Hastie, T., and Tibshirani, R. (1990), Generalized Additive Models, London:

Chapman & Hall.Heard, N. A., Holmes, C. C., and Stephens, D. A. (2006), “A Quantitative

Study of Gene Regulation Involved in the Immune Response of AnophelineMosquitoes: An Application of Bayesian Hierarchical Clustering of Curves,”Journal of the American Statistical Association, 101, 18–29.

Hubert, L., and Arabie, P. (1985), “Comparing Partitions,” Journal of Classifi-cation, 2, 193–218.

James, G. M., and Sugar, C. A. (2003), “Clustering for Sparsely Sampled Func-tional Data,” Journal of the American Statistical Association, 98, 397–408.

Jiang, M., Ryu, J., Kiraly, M., Duke, K., Reinke, V., and Kim, S. K. (2001),“Genome-Wide Analysis of Developmental and Sex-Regulated Gene Expres-sion Profiles in Caenorhabditis elegans,” Proceedings of the National Acad-emy of Sciences, 98, 218–223.

Kim, Y.-J., and Gu, C. (2004), “Smoothing Spline Gaussian Regression: MoreScalable Computation via Efficient Approximation,” Journal of the Royal Sta-tistical Society, Ser. B, 66, 337–356.

Lai, L. C., Kosorukoff, A. L., Burke, P., and Kwast, K. E. (2006), “MetabolicState–Dependent Remodeling of the Transcriptome in Response to Anoxiaand Subsequent Reoxygenation in Saccharomyces cerevisiae,” EukayoticCell, 5, 1468–1489.

Liu, J. S., Chen, R., and Wong, W. H. (1998), “Rejection Control and SequentialImportance Sampling,” Journal of the American Statistical Association, 93,1022–1031.

Luan, Y., and Li, H. (2003), “Clustering of Time-Course Gene Expression DataUsing a Mixed-Effects Models With B–Spline,” Bioinformatics, 19, 474–282.

(2004), “Model-Based Methods for Identifying Periodically RegulatedGenes Based on the Time Course Microarray Gene Expression Data,” Bioin-formatics, 20, 332–339.


Ma, P., Castillo-Davis, C., Zhong, W., and Liu, J. S. (2006), “A Data-DrivenClustering Method for Time Course Gene Expression Data,” Nucleic AcidsResearch, 34, 1261–1269.

Ma, P., Wang, P., Tenorio, L., de Hoop, M. V., and van der Hilst, R. D. (2007),“Imaging of Structure at and Near the Core Mantle Boundary Using a Gener-alized Radon Transform: 2. Statistical Inference of Singularities,” Journal ofGeophysical Research, 112, B08303.

McCarroll, S. A., Murphy, C. T., Zou, S., Pletcher, S. D., Chin, C. S., Jan, Y. N.,Kenyon, C., Bargmann, C. I., and Li, H. (2004), “Comparing Genomic Ex-pression Patterns Across Species Identifies Shared Transcriptional Programin Aging,” Nature Genetics, 36, 197–204.

McLachlan, G. J., and Peel, D. (2001), Finite Mixture Models, New York: Wi-ley.

Nychka, D. (1988), “Bayesian Confidence Intervals for Smoothing Splines,”Journal of the American Statistical Association, 83, 1134–1143.

Ramsay, J. O., and Silverman, B. W. (2002), Applied Functional Data Analysis:Methods and Case Studies, New York: Springer-Verlag.

(2005), Functional Data Analysis, New York: Springer-Verlag.Robinson, G. K. (1991), “That BLUP Is a Good Thing: The Estimation of the

Random Effects” (with discussion), Statistical Science, 6, 15–51.Robinson, M. D., Grigull, J., Mohammad, N., and Hughes, T. R. (2002), “Fun-

spec: A Web-Based Cluster Interpreter for Yeast,” BMC Bioinformatics, 3,3–35.

Silverman, B. W., and Wood, J. T. (1987), “The Nonparametric Estimation ofBranching Curves,” Journal of the American Statistical Association, 82, 551–558.

Spellman, P. T., Sherlock, G., Zhang, M. Q., Iyer, V. R., Anders, K., Eisen,M. B., Brown, P. O., Botstein, B., and Futcher, D. (1998), “ComprehensiveIdentification of Cell Cycle–Regulated Genes of the Yeast Saccharomycescerevisiae by Microarray Hybridization,” Molecular Biology of the Cell, 9,3273–3297.

Storey, J. D., Xiao, W., Leek, J. T., Tompkins, R., and Davis, R. G. (2005),“Significance of Time Course Microarray Experiments,” Proceedings of theNational Academy of Sciences, 102, 12837–12842.

Wahba, G. (1983), “Bayesian “Confidence Intervals” for the Cross-ValidatedSmoothing Spline,” Journal of the Royal Statistical Society, Ser. B, 45, 133–150.

(1990), Spline Models for Observational Data, Philadelphia: SIAM.Wang, P., de Hoop, M. V., van der Hilst, R. D., Ma, P., and Tenorio, L. (2006),

“Imaging of Structure at and Near the Core Mantle Boundary Using a Gener-alized Radon Transform: 1. Construction of Image Gathers,” Journal of Geo-physical Research, 111, B12304.

Wang, Y. (1998), “Mixed-Effects Smoothing Spline ANOVA,” Journal of theRoyal Statistical Society, Ser. B, 60, 159–174.

Wei, G. C., and Tanner, M. A. (1990), “A Monte Carlo Implementation of theEM Algorithm and the Poor Man’s Data Augmentation Algorithms,” Journalof the American Statistical Association, 85, 699–704.

Zhang, D., Lin, X., Raz, J., and Sowers, M. (1998), “Semiparametric StochasticMixed Models for Longitudinal Data,” Journal of the American StatisticalAssociation, 93, 710–719.

Penalized Clustering of Large-Scale Functional Data...

Documents

Transcript of Penalized Clustering of Large-Scale Functional Data...