SIMoNe: Statistical Iference for MOdular NEtworks
-
Upload
laboratoire-statistique-et-genome -
Category
Technology
-
view
1.518 -
download
2
Transcript of SIMoNe: Statistical Iference for MOdular NEtworks
SIMoNeAn R package for inferring Gausssian networks with latent
clustering
Julien Chiquet (and Camille, Christophe, Gilles, Catherine, Yves)
Laboratoire Statistique et Genome,La genopole - Universite d’Evry
SSB – 13 avril 2010
SIMoNe: inferring Gaussian networks with latent clustering 1
Problem
n ≈ 10s/100s of slides
g ≈ 1000s of genes
O(g2) parameters (edges) !
Inference
Which interactions?
The main statistical issue is the high dimensional setting
SIMoNe: inferring Gaussian networks with latent clustering 2
Handling the scarcity of data (1)By reducing the number of parameters
AssumptionConnections will only appear between informative genes
differential analysis
select p key genes P
p “reasonable” compared to n
typically, n ∈ [p/5; 5p]
the learning dataset
n size–p vectors of expression
(X1, . . . , Xn) with Xi ∈ Rp
inference
SIMoNe: inferring Gaussian networks with latent clustering 3
Handling the scarcity of data (2)By collecting as many observations as possible
Multitask learning Go to learning
How should we merge the data?organism
drug 1drug 2
drug 3
SIMoNe: inferring Gaussian networks with latent clustering 4
Handling the scarcity of data (2)By collecting as many observations as possible
Multitask learning Go to learning
by inferring each network independentlyorganism
drug 1drug 2
drug 3
(X(1)1 , . . . , X
(1)n1
), X(1)i ∈ Rp1 (X
(2)1 , . . . , X
(2)n2
), X(2)i ∈ Rp2 (X
(3)1 , . . . , X
(3)n3
), X(3)i ∈ Rp3
inference inference inference
SIMoNe: inferring Gaussian networks with latent clustering 4
Handling the scarcity of data (2)By collecting as many observations as possible
Multitask learning Go to learning
by pooling all the available dataorganism
drug 1drug 2
drug 3
(X1, . . . , Xn), Xi ∈ Rp , with n = n1 + n2 + n3 .
inference
SIMoNe: inferring Gaussian networks with latent clustering 4
Handling the scarcity of data (2)By collecting as many observations as possible
Multitask learning Go to learning
by breaking the separabilityorganism
drug 1drug 2
drug 3
(X(1)1 , . . . , X
(1)n1
), X(1)i ∈ Rp1 (X
(2)1 , . . . , X
(2)n2
), X(2)i ∈ Rp2 (X
(3)1 , . . . , X
(3)n3
), X(3)i ∈ Rp3
inference
SIMoNe: inferring Gaussian networks with latent clustering 4
Handling the scarcity of data (3)By introducing some prior
Priors should be biologically grounded
1. few genes effectively interact (sparsity),2. networks are organized (latent clustering),3. steady-state or time-course data
(directedness relies on the modelling).
G0 G1
G2
G3
G4
G5
G6
G7
G8
G9
SIMoNe: inferring Gaussian networks with latent clustering 5
Handling the scarcity of data (3)By introducing some prior
Priors should be biologically grounded
1. few genes effectively interact (sparsity),2. networks are organized (latent clustering),3. steady-state or time-course data
(directedness relies on the modelling).
G0 G1
G2
G3
G4
G5
G6
G7
G8
G9
SIMoNe: inferring Gaussian networks with latent clustering 5
Handling the scarcity of data (3)By introducing some prior
Priors should be biologically grounded
1. few genes effectively interact (sparsity),2. networks are organized (latent clustering),3. steady-state or time-course data
(directedness relies on the modelling).
A1 A2
A3
B1
B2
B3
B4
B5
C1
C2
SIMoNe: inferring Gaussian networks with latent clustering 5
Handling the scarcity of data (3)By introducing some prior
Priors should be biologically grounded
1. few genes effectively interact (sparsity),2. networks are organized (latent clustering),3. steady-state or time-course data
(directedness relies on the modelling).
A1 A2
A3
B1
B2
B3
B4
B5
C1
C2
SIMoNe: inferring Gaussian networks with latent clustering 5
Handling the scarcity of data (3)By introducing some prior
Priors should be biologically grounded
1. few genes effectively interact (sparsity),2. networks are organized (latent clustering),3. steady-state or time-course data
(directedness relies on the modelling).
A1 A2
A3
B1
B2
B3
B4
B5
C1
C2
SIMoNe: inferring Gaussian networks with latent clustering 5
Outline
Statistical modelsSteady-state dataTime-course dataMultitask learning
Algorithms and methodsOverall viewNetwork inferenceModel selectionLatent structure
Numerical experimentsPerformance on simulated dataR package demo: the breast cancer data set
SIMoNe: inferring Gaussian networks with latent clustering 6
Outline
Statistical modelsSteady-state dataTime-course dataMultitask learning
Algorithms and methodsOverall viewNetwork inferenceModel selectionLatent structure
Numerical experimentsPerformance on simulated dataR package demo: the breast cancer data set
SIMoNe: inferring Gaussian networks with latent clustering 6
Outline
Statistical modelsSteady-state dataTime-course dataMultitask learning
Algorithms and methodsOverall viewNetwork inferenceModel selectionLatent structure
Numerical experimentsPerformance on simulated dataR package demo: the breast cancer data set
SIMoNe: inferring Gaussian networks with latent clustering 6
The graphical models: general settings
AssumptionA microarray can be represented as a multivariate Gaussianvector X = (X(1), . . . , X(p)) ∈ Rp.
Collecting gene expression
1. Steady-state data leads to an i.i.d. sample.2. Time-course data gives a time series.
Graphical interpretationconditional dependency between X(i) and X(j)
ornon null partial correlation between X(i) and X(j)
if and only ifj
i
SIMoNe: inferring Gaussian networks with latent clustering 7
The graphical models: general settings
AssumptionA microarray can be represented as a multivariate Gaussianvector X = (X(1), . . . , X(p)) ∈ Rp.
Collecting gene expression
1. Steady-state data leads to an i.i.d. sample.2. Time-course data gives a time series.
Graphical interpretationconditional dependency between X(i) and X(j)
ornon null partial correlation between X(i) and X(j)
if and only ifj
i
?
SIMoNe: inferring Gaussian networks with latent clustering 7
The graphical models: general settings
AssumptionA microarray can be represented as a multivariate Gaussianvector X = (X(1), . . . , X(p)) ∈ Rp.
Collecting gene expression
1. Steady-state data leads to an i.i.d. sample.2. Time-course data gives a time series.
Graphical interpretationconditional dependency between Xt(i) and Xt−1(j)
ornon null partial correlation between Xt(i) and Xt−1(j)
if and only ifj
i
?
SIMoNe: inferring Gaussian networks with latent clustering 7
The general statistical approach
Let Θ be the parameters to infer (the edges).
A penalized likelihood approach
Θλ = arg maxΘL(Θ; data)− λ pen`1(Θ,Z),
I L is the model log-likelihood,I Z is a latent clustering of the network,I pen`1 is a penalty function tuned by λ > 0.
It performs1. regularization (needed when n� p),2. selection (sparsity induced by the `1-norm),3. model-driven inference (penalty adapted according to Z).
SIMoNe: inferring Gaussian networks with latent clustering 8
The general statistical approach
Let Θ be the parameters to infer (the edges).
A penalized likelihood approach
Θλ = arg maxΘL(Θ; data)− λ pen`1(Θ,Z),
I L is the model log-likelihood,I Z is a latent clustering of the network,I pen`1 is a penalty function tuned by λ > 0.
It performs1. regularization (needed when n� p),2. selection (sparsity induced by the `1-norm),3. model-driven inference (penalty adapted according to Z).
SIMoNe: inferring Gaussian networks with latent clustering 8
Outline
Statistical modelsSteady-state dataTime-course dataMultitask learning
Algorithms and methodsOverall viewNetwork inferenceModel selectionLatent structure
Numerical experimentsPerformance on simulated dataR package demo: the breast cancer data set
SIMoNe: inferring Gaussian networks with latent clustering 9
The Gaussian model for an i.i.d. sample
LetI X ∼ N (0p,Σ) with X1, . . . , Xn i.i.d. copies of X,I X be the n× p matrix whose kth row is Xk,I Θ = (θij)i,j∈P , Σ−1 be the concentration matrix.
Graphical interpretationSince corij|P\{i,j} = −θij/
√θiiθjj for i 6= j,
X(i) ⊥⊥ X(j)|X(P\{i, j})⇔
θij = 0
oredge (i, j) /∈ network.
Θ describes the undirected graph of conditionaldependencies.
SIMoNe: inferring Gaussian networks with latent clustering 10
Neighborhood selection (1)
LetI Xi be the ith column of X,I X\i be X deprived of Xi.
Xi = X\iβ + ε, where βj = −θijθii.
Meinshausen and Bulhman, 2006Since sign(corij|P\{i,j}) = sign(βj), select the neighbors of i with
arg minβ
1
n
∥∥Xi −X\iβ∥∥22
+ λ ‖β‖`1 .
The sign pattern of Θλ is inferred after a symmetrization step.
SIMoNe: inferring Gaussian networks with latent clustering 11
Neighborhood selection (2)
The pseudo log-likelihood of the i.i.d Gaussian sample is
Liid(Θ; S) =
p∑i=1
(n∑k=1
logP(Xk(i)|Xk(P\i); Θi)
),
=n
2log det(D)− n
2Trace
(D−1/2ΘSΘD−1/2
)− n
2log(2π),
where D = diag(Θ).
Proposition
Θpseudoλ = arg max
Θ:θij 6=θiiLiid(Θ; S)− λ ‖Θ‖`1
has the same null entries as inferred by neighborhood selection.
SIMoNe: inferring Gaussian networks with latent clustering 12
The Gaussian likelihood for an i.i.d. sample
Let S = n−1XᵀX be the empirical variance-covariance matrix: Sis a sufficient statistic of Θ.
The log-likelihood
Liid(Θ; S) =n
2log det(Θ)− n
2Trace(SΘ) +
n
2log(2π).
The MLE = S−1 of Θ is not defined for n < p and neversparse.
The need for regularization is huge.
SIMoNe: inferring Gaussian networks with latent clustering 13
Penalized log-likelihood
Banerjee et al., JMLR 2008
Θλ = arg maxΘ
Liid(Θ; S)− λ‖Θ‖`1 ,
efficiently solved by the graphical LASSO of Friedman et al, 2008.
Ambroise, Chiquet, Matias, EJS 2009Use adaptive penalty parameters for different coefficients
Liid(Θ; S)− λ‖PZ ?Θ‖`1 ,
where PZ is a matrix of weights depending on the underlyingclustering Z.
Works with the pseudo log-likelihood (computationallyefficient).
SIMoNe: inferring Gaussian networks with latent clustering 14
Penalized log-likelihood
Banerjee et al., JMLR 2008
Θλ = arg maxΘ
Liid(Θ; S)− λ‖Θ‖`1 ,
efficiently solved by the graphical LASSO of Friedman et al, 2008.
Ambroise, Chiquet, Matias, EJS 2009Use adaptive penalty parameters for different coefficients
Liid(Θ; S)− λ‖PZ ?Θ‖`1 ,
where PZ is a matrix of weights depending on the underlyingclustering Z.
Works with the pseudo log-likelihood (computationallyefficient).
SIMoNe: inferring Gaussian networks with latent clustering 14
Outline
Statistical modelsSteady-state dataTime-course dataMultitask learning
Algorithms and methodsOverall viewNetwork inferenceModel selectionLatent structure
Numerical experimentsPerformance on simulated dataR package demo: the breast cancer data set
SIMoNe: inferring Gaussian networks with latent clustering 15
The Gaussian model for time-course data (1)
Let X1, . . . , Xn be a first order vector autoregressive process
Xt = ΘXt−1 + b + εt, t ∈ [1, n]
where we are looking for Θ = (θij)i,j∈P andI X0 ∼ N (0p,Σ0),I εt is a Gaussian white noise with covariance σ2Ip,I cov(Xt, εs) = 0 for s > t, so that Xt is markovian.
Graphical interpretationsince
θij =cov (Xt(i), Xt−1(j)|Xt−1(P\j))
var (Xt−1(j)|Xt−1(P\j)),
Xt(i) ⊥⊥ Xt−1(j)|Xt−1(P\j)⇔
θij = 0
oredge (j � i) /∈ network
SIMoNe: inferring Gaussian networks with latent clustering 16
The Gaussian model for time-course data (2)
LetI X be the n× p matrix whose kth row is Xk,I S = n−1Xᵀ\nX\n be the within time covariance matrix,
I V = n−1Xᵀ\nX\0 be the across time covariance matrix.
The log-likelihood
Ltime(Θ; S,V) = n Trace (VΘ)− n
2Trace (ΘᵀSΘ) + c.
The MLE = S−1V of Θ is still not defined for n < p.
SIMoNe: inferring Gaussian networks with latent clustering 17
Penalized log-likelihood
Charbonnier, Chiquet, Ambroise, SAGMB 2010
Θλ = arg maxΘ
Ltime(Θ; S,V)− λ‖PZ ?Θ‖`1
where PZ is a (non-symmetric) matrix of weights depending onthe underlying clustering Z.
Major difference with the i.i.d. caseThe graph is directed:
θij =cov (Xt(i), Xt−1(j)|Xt−1(P\j))
var (Xt−1(j)|Xt−1(P\j))
6= cov (Xt(j), Xt−1(i)|Xt−1(P\i))var (Xt−1(i)|Xt−1(P\i))
.
SIMoNe: inferring Gaussian networks with latent clustering 18
Outline
Statistical modelsSteady-state dataTime-course dataMultitask learning
Algorithms and methodsOverall viewNetwork inferenceModel selectionLatent structure
Numerical experimentsPerformance on simulated dataR package demo: the breast cancer data set
SIMoNe: inferring Gaussian networks with latent clustering 19
Coupling related problems
ConsiderI T samples concerning the expressions of the same p genes,
I X(t)1 , . . . , X
(t)nt is the tth sample drawn from N (0p,Σ
(t)), withcovariance matrix S(t).
Multiple samples setup Go to scheme
Ignoring the relationships between the tasks leads to
arg maxΘ(t),t=1...,T
T∑t=1
L(Θ(t); S(t))− λ pen`1(Θ(t),Z).
Breaking the separability
I Either by modifying the objective functionI or the constraints.
SIMoNe: inferring Gaussian networks with latent clustering 20
Coupling related problems
ConsiderI T samples concerning the expressions of the same p genes,
I X(t)1 , . . . , X
(t)nt is the tth sample drawn from N (0p,Σ
(t)), withcovariance matrix S(t).
Multiple samples setup Go to scheme
Ignoring the relationships between the tasks leads to
arg maxΘ(t),t=1...,T
T∑t=1
L(Θ(t); S(t))− λ pen`1(Θ(t),Z).
Breaking the separability
I Either by modifying the objective functionI or the constraints.
SIMoNe: inferring Gaussian networks with latent clustering 20
Coupling related problems
ConsiderI T samples concerning the expressions of the same p genes,
I X(t)1 , . . . , X
(t)nt is the tth sample drawn from N (0p,Σ
(t)), withcovariance matrix S(t).
Multiple samples setup Go to scheme
Ignoring the relationships between the tasks leads to
arg maxΘ(t),t=1...,T
T∑t=1
L(Θ(t); S(t))− λ pen`1(Θ(t),Z).
Breaking the separability
I Either by modifying the objective functionI or the constraints.
RemarksI In the sequel, the Z is eluded for clarity (no loss of
generality).I Multitask learning is easily adapted to time-course data yet
only steady state version is presented here.
SIMoNe: inferring Gaussian networks with latent clustering 20
Coupling problems through the objective function
The Intertwined LASSO
maxΘ(t),t...,T
T∑t=1
L(Θ(t); S(t))− λ‖Θ(t)‖`1
I S = 1n
∑Tt=1 ntS
(t) is an “across-task” covariance matrix.I S(t) = αS(t) + (1− α)S is a mixture between inner/over-tasks
covariance matrices.
setting α = 0 is equivalent to pooling all the data and inferone common network,
setting α = 1 is equivalent to treating T independentproblems.
SIMoNe: inferring Gaussian networks with latent clustering 21
Coupling problems by grouping variables (1)
Groups definition
I Groups are the T -tuple composed by the (i, j) entries ofeach Θ(t), t = 1, . . . , T .
I Most relationships between the genes are kept or removedacross all tasks simultaneously.
The graphical group-LASSO
maxΘ(t),t...,T
T∑t=1
L(Θ(t); S(t)
)− λ
∑i,j∈Pi 6=j
(T∑t=1
(θ(t)ij
)2)1/2
.
SIMoNe: inferring Gaussian networks with latent clustering 22
1
1
−1
−1
β(1)2
β(1)1 1
1
−1
−1
β(1)2
β(1)1
1
1
−1
−1
β(1)2
β(1)1 1
1
−1
−1
β(1)2
β(1)1
β(2)
1=
0β(2)
1=
0.3
β(2)2 = 0 β
(2)2 = 0.3
Group-LASSO penaltyAssume
I 2 tasks (T = 2)
I 2 coefficients (p = 2)
Let represent the unit ball
2∑i=1
(2∑
t=1
β(t)i
2
)1/2
≤ 1
SIMoNe: inferring Gaussian networks with latent clustering 23
Coupling problems by grouping variables (2)
Graphical group-LASSO modification
I Inside a group, value are most likeliky sign consistent.
The graphical cooperative-LASSO
maxΘ(t),t...,T
T∑t=1
L(S(t); Θ(t)
)
− λ∑i,j∈Pi 6=j
(
T∑t=1
[θ(t)ij
]2+
)1/2
+
(T∑t=1
[θ(t)ij
]2−
)1/2 ,
where [u]+ = max(0, u) and [u]− = min(0, u).
SIMoNe: inferring Gaussian networks with latent clustering 24
1
1
−1
−1
β(1)2
β(1)1 1
1
−1
−1
β(1)2
β(1)1
1
1
−1
−1
β(1)2
β(1)1 1
1
−1
−1
β(1)2
β(1)1
β(2)
1=
0β(2)
1=
0.3
β(2)2 = 0 β
(2)2 = 0.3
Coop-LASSO penaltyAssume
I 2 tasks (T = 2)
I 2 coefficients (p = 2)
Let represent the unit ball
2∑i=1
(2∑
t=1
[β(t)i
]2+
)1/2
+
2∑i=1
(2∑
t=1
[−β(t)
i
]+
)1/2
≤ 1
SIMoNe: inferring Gaussian networks with latent clustering 25
Outline
Statistical modelsSteady-state dataTime-course dataMultitask learning
Algorithms and methodsOverall viewNetwork inferenceModel selectionLatent structure
Numerical experimentsPerformance on simulated dataR package demo: the breast cancer data set
SIMoNe: inferring Gaussian networks with latent clustering 26
The overall strategy
Our basic criteria is of the form
L(Θ; data)− λ ‖PZ ?Θ‖`1 .
What we are looking for
I the edges, through Θ,I the correct level of sparsity λ,I the underlying clustering Z with connectivity matrix πZ.
What SIMoNe does
1. Infer a family of networks G = {Θλ : λ ∈ [λmax, 0]}2. Select G? that maximizes an information criteria3. Learn Z on the selected network G?
4. Infer a family of networks with PZ ∝ 1− πZ
5. Select G?Z that maximizes an information criteria
SIMoNe: inferring Gaussian networks with latent clustering 27
The overall strategy
Our basic criteria is of the form
L(Θ; data)− λ ‖PZ ?Θ‖`1 .
What we are looking for
I the edges, through Θ,I the correct level of sparsity λ,I the underlying clustering Z with connectivity matrix πZ.
What SIMoNe does
1. Infer a family of networks G = {Θλ : λ ∈ [λmax, 0]}2. Select G? that maximizes an information criteria3. Learn Z on the selected network G?
4. Infer a family of networks with PZ ∝ 1− πZ
5. Select G?Z that maximizes an information criteria
SIMoNe: inferring Gaussian networks with latent clustering 27
SIMoNe
SIMoNESuppose you want to recover a clustered network:
Target Adjacency Matrix
Graph
Target Network
SIMoNe: inferring Gaussian networks with latent clustering 28
SIMoNe
SIMoNEStart with microarray data
Data
SIMoNe: inferring Gaussian networks with latent clustering 28
SIMoNe
SIMoNE
DataAdjacency Matrix
corresponding to G?
SIMoNE without prior
SIMoNe: inferring Gaussian networks with latent clustering 28
SIMoNe
SIMoNE
DataAdjacency Matrix
corresponding to G?
SIMoNE without prior
πZ
Connectivity matrix
Mixer
Penalty matrix PZ
Decreasing transformation
SIMoNe: inferring Gaussian networks with latent clustering 28
SIMoNe
SIMoNE
DataAdjacency Matrix
corresponding to G?
SIMoNE without prior
πZ
Connectivity matrix
Mixer
Penalty matrix PZ
Decreasing transformation
Adjacency Matrixcorresponding to G?Z
+
SIMoNE
SIMoNe: inferring Gaussian networks with latent clustering 28
Outline
Statistical modelsSteady-state dataTime-course dataMultitask learning
Algorithms and methodsOverall viewNetwork inferenceModel selectionLatent structure
Numerical experimentsPerformance on simulated dataR package demo: the breast cancer data set
SIMoNe: inferring Gaussian networks with latent clustering 29
Monotask framework: problem decomposition
Consider the following reordering of Θ
Θ =
[Θ\i\i Θi\iΘᵀi\i θii
], Θi =
[Θi\iθii
].
Block coordinate descent algorithm
arg maxΘ
L(Θ; data)− λ pen`1(Θ)
relies on p penalized, convex-optimization problems
arg minβ∈Rp−1
f(β; S) + λ pen`1(β), (1)
where f is convex and β = Θi\i for steady-state data.
SIMoNe: inferring Gaussian networks with latent clustering 30
Monotask framework: problem decomposition
Consider the following reordering of Θ
Θ =
[Θ\i\i Θi\iΘᵀi\i θii
], Θi =
[Θi\iθii
].
Block coordinate descent algorithm
arg maxΘ
L(Θ; data)− λ pen`1(Θ)
relies on p penalized, convex-optimization problems
arg minβ∈Rp
f(β; S,V) + λ pen`1(β), (1)
where f is convex and β = Θi for time-course data.
SIMoNe: inferring Gaussian networks with latent clustering 30
Monotask framework: algorithms
1. steady-state: Covsel/GLasso (Liid(Θ)− λ‖Θ‖`1)I starts from S + λIp positive definite,I iterates on the columns of Θ−1 until stabilization,I both estimation and selection of Θ.
2. steady-state: neighborhood selection (Liid(Θ)− λ‖Θ‖`1)I select signs patterns of Θi\i with the LASSO,I only one pass per column required,I post-symmetrization needed.
3. time-course: VAR(1) inference (Ltime(Θ)− λ‖Θ‖`1)I select and estimate Θi with the LASSO,I only one pass per column required,I both estimation and selection.
SIMoNe: inferring Gaussian networks with latent clustering 31
Monotask framework: algorithms
1. steady-state: Covsel/GLasso (Liid(Θ)− λ‖Θ‖`1)I starts from S + λIp positive definite,I iterates on the columns of Θ−1 until stabilization,I both estimation and selection of Θ.
2. steady-state: neighborhood selection (Liid(Θ)− λ‖Θ‖`1)I select signs patterns of Θi\i with the LASSO,I only one pass per column required,I post-symmetrization needed.
3. time-course: VAR(1) inference (Ltime(Θ)− λ‖Θ‖`1)I select and estimate Θi with the LASSO,I only one pass per column required,I both estimation and selection.
SIMoNe: inferring Gaussian networks with latent clustering 31
Monotask framework: algorithms
1. steady-state: Covsel/GLasso (Liid(Θ)− λ‖Θ‖`1)I starts from S + λIp positive definite,I iterates on the columns of Θ−1 until stabilization,I both estimation and selection of Θ.
2. steady-state: neighborhood selection (Liid(Θ)− λ‖Θ‖`1)I select signs patterns of Θi\i with the LASSO,I only one pass per column required,I post-symmetrization needed.
3. time-course: VAR(1) inference (Ltime(Θ)− λ‖Θ‖`1)I select and estimate Θi with the LASSO,I only one pass per column required,I both estimation and selection.
SIMoNe: inferring Gaussian networks with latent clustering 31
Multitask framework: problem decomposition (1)
Consider the (p T )× (p T ) block-diagonal matrix C composed bythe empirical covariance matrices of each tasks
C =
S(1) 0. . .
0 S(T )
,
and define
C\i\i =
S(1)\i\i 0
. . .
0 S(T )\i\i
, Ci\i =
S(1)i\i...
S(T )i\i
.
The (p − 1)T × (p − 1)T matrix C\i\i is the matrix C where weremoved each line and each column pertaining to variable i.
RemarkLet us consider multitask algorithms in the steady-state frame-work (easily adapted to time-course data)
SIMoNe: inferring Gaussian networks with latent clustering 32
Multitask framework: problem decomposition (1)
Consider the (p T )× (p T ) block-diagonal matrix C composed bythe empirical covariance matrices of each tasks
C =
S(1) 0. . .
0 S(T )
,
and define
C\i\i =
S(1)\i\i 0
. . .
0 S(T )\i\i
, Ci\i =
S(1)i\i...
S(T )i\i
.
The (p − 1)T × (p − 1)T matrix C\i\i is the matrix C where weremoved each line and each column pertaining to variable i.
SIMoNe: inferring Gaussian networks with latent clustering 32
Multitask framework: problem decomposition (2)
Estimate the ith-columns of the T tasks bind together
arg maxΘ(t),t=1...,T
T∑t=1
L(Θ(t); S(t))− λ pen`1(Θ(t))
is decomposed into p convex optimization problems
arg minβ∈RT×(p−1)
f(β; C) + λ pen`1(β),
where we set β(t) = Θ(t)i\i and
β =
β(1)
...β(T )
∈ RT×(p−1).
SIMoNe: inferring Gaussian networks with latent clustering 33
Solving the sub-problem
Subdifferential approach
minβ∈RT×(p−1)
L(β) = f(β) + pen`1(β) ,
β is a minimizer iif 0p ∈ ∂βL(β), with
∂βL(β) = ∇βf(β) + λ∂βpen`1(β).
SIMoNe: inferring Gaussian networks with latent clustering 34
Solving the sub-problem
Subdifferential approach
minβ∈RT×(p−1)
L(β) = f(β) + pen`1(β) ,
β is a minimizer iif 0p ∈ ∂βL(β), with
∂βL(β) = ∇βf(β) + λ∂βpen`1(β).
For the graphical Intertwined LASSO
pen`1(β) =
T∑t=1
∥∥∥β(t)∥∥∥1,
where the grouping effect is managed by the function f .
SIMoNe: inferring Gaussian networks with latent clustering 34
Solving the sub-problem
Subdifferential approach
minβ∈RT×(p−1)
L(β) = f(β) + pen`1(β) ,
β is a minimizer iif 0p ∈ ∂βL(β), with
∂βL(β) = ∇βf(β) + λ∂βpen`1(β).
For the graphical Group-LASSO
pen`1(β) =
p−1∑i=1
∥∥∥β[1:T ]i
∥∥∥2,
where β[1:T ]i =
(β(1)i , . . . , β
(T )i
)ᵀ∈ RT is the vector of the ith com-
ponent across tasks.
SIMoNe: inferring Gaussian networks with latent clustering 34
Solving the sub-problem
Subdifferential approach
minβ∈RT×(p−1)
L(β) = f(β) + pen`1(β) ,
β is a minimizer iif 0p ∈ ∂βL(β), with
∂βL(β) = ∇βf(β) + λ∂βpen`1(β).
For the graphical Coop-LASSO
pen`1(β) =
p−1∑i=1
(∥∥∥∥(β[1:T ]i
)+
∥∥∥∥2
+
∥∥∥∥(−β[1:T ]i
)+
∥∥∥∥2
),
where β[1:T ]i =
(β(1)i , . . . , β
(T )i
)ᵀ∈ RT is the vector of the ith com-
ponent across tasks.
SIMoNe: inferring Gaussian networks with latent clustering 34
General active set algorithm: , yellow belt// 0. INITIALIZATION
β ← 0,A ← ∅while 0 /∈ ∂βL(β) do
// 1. MASTER PROBLEM: OPTIMIZATION WITH RESPECT TO βAFind a solution h to the smooth problem
∇hf(βA + h) + λ∂hpen`1 (βA + h) = 0, where ∂hpen`1 ={∇hpen`1
}.
βA ← βA + h// 2. IDENTIFY NEWLY ZEROED VARIABLES
A ← A\{i}
// 3. IDENTIFY NEW NON-ZERO VARIABLES
// Select a candidate i ∈ Ac
i← arg maxj∈Ac
vj , where vj = minν∈∂βj gk
∣∣∣ ∂f(β)∂βj
+ λν∣∣∣
endSIMoNe: inferring Gaussian networks with latent clustering 35
General active set algorithm: / orange belt// 0. INITIALIZATION
β ← 0,A ← ∅while 0 /∈ ∂βL(β) do
// 1. MASTER PROBLEM: OPTIMIZATION WITH RESPECT TO βAFind a solution h to the smooth problem
∇hf(βA + h) + λ∂hpen`1 (βA + h) = 0, where ∂hpen`1 ={∇hpen`1
}.
βA ← βA + h// 2. IDENTIFY NEWLY ZEROED VARIABLES
A ← A\{i}
// 3. IDENTIFY NEW NON-ZERO VARIABLES
// Select a candidate i ∈ Ac which violates the more the optimality
conditions
i← arg maxj∈Ac
vj , where vj = minν∈∂βj gk
∣∣∣ ∂f(β)∂βj
+ λν∣∣∣
if it exists such an i thenA ← A∪ {i}
elseStop and return β, which is optimal
endend
SIMoNe: inferring Gaussian networks with latent clustering 35
General active set algorithm: M green belt// 0. INITIALIZATION
β ← 0,A ← ∅while 0 /∈ ∂βL(β) do
// 1. MASTER PROBLEM: OPTIMIZATION WITH RESPECT TO βAFind a solution h to the smooth problem
∇hf(βA + h) + λ∂hpen`1 (βA + h) = 0, where ∂hpen`1 ={∇hpen`1
}.
βA ← βA + h// 2. IDENTIFY NEWLY ZEROED VARIABLES
while ∃i ∈ A : βi = 0 and minν∈∂βigk
∣∣∣ ∂f(β)∂βi
+ λν∣∣∣ = 0 do
A ← A\{i}end// 3. IDENTIFY NEW NON-ZERO VARIABLES
// Select a candidate i ∈ Ac such that an infinitesimal change of βiprovides the highest reduction of L
i← arg maxj∈Ac
vj , where vj = minν∈∂βj gk
∣∣∣ ∂f(β)∂βj
+ λν∣∣∣
if vi 6= 0 thenA ← A∪ {i}
elseStop and return β, which is optimal
endend
SIMoNe: inferring Gaussian networks with latent clustering 35
Outline
Statistical modelsSteady-state dataTime-course dataMultitask learning
Algorithms and methodsOverall viewNetwork inferenceModel selectionLatent structure
Numerical experimentsPerformance on simulated dataR package demo: the breast cancer data set
SIMoNe: inferring Gaussian networks with latent clustering 36
Tuning the penalty parameterWhat does the literature say?
Theory based penalty choices
1. Optimal order of penalty in the p� n framework:√n log p
Bunea et al. 2007, Bickel et al. 2009
2. Control on the probability of connecting two distinctconnectivity sets
Meinshausen et al. 2006, Banerjee et al. 2008, Ambroise et al. 2009
practically much too conservative
Cross-validationI Optimal in terms of prediction, not in terms of selectionI Problematic with small samples:
changes the sparsity constraint due to sample size
SIMoNe: inferring Gaussian networks with latent clustering 37
Tuning the penalty parameterBIC / AIC
Theorem (Zou et al. 2008)
df(β lassoλ ) =
∥∥∥β lassoλ
∥∥∥0
Straightforward extensions to the graphical framework
BIC(λ) = L(Θλ; X)− df(Θλ)log n
2
AIC(λ) = L(Θλ; X)− df(Θλ)
I Rely on asymptotic approximations, but still relevant forsmall data set
I Easily adapted to Liid, Liid,Ltime and multitask framework.
SIMoNe: inferring Gaussian networks with latent clustering 38
Outline
Statistical modelsSteady-state dataTime-course dataMultitask learning
Algorithms and methodsOverall viewNetwork inferenceModel selectionLatent structure
Numerical experimentsPerformance on simulated dataR package demo: the breast cancer data set
SIMoNe: inferring Gaussian networks with latent clustering 39
MixNetErdos-Renyi Mixture for Networks
The data is now the network itselfConsider A = (aij)i,j∈P , the adjacency matrix associated to Θ:
aij = 1{θij 6=0}.
Latent structure modeling (Daudin et al., 2008)Spread the nodes on a set Q = {1, . . . , q, . . . , Q} of classes withI α a Q–size vector giving αi = P(i ∈ q),I ziq = 1{i∈q} are independent hidden variables Zi ∼M(1,α),I π a Q×Q matrix giving πq` = P(aij = 1|i ∈ q, j ∈ `).
Connexion probabilities depends on the node class belonging:
aij |{ZiqZj` = 1} ∼ B(πq`).
SIMoNe: inferring Gaussian networks with latent clustering 40
Estimation strategy
LikelihoodsI the observed data: P(A|α,π) =
∑Z P(A,Z|α,π).
I the complete data: P(A,Z|α,π).
The EM criteria
E(
logP(A,Z|α,π)|A′).
M requires P(Z|A,α,π) which is not tractable!
SIMoNe: inferring Gaussian networks with latent clustering 41
Variational inference
PrincipleApproximate P(Z|A,α,π) by Rτ (Z) chosen to minimize
KL(Rτ (Z);P(Z|A,α,π)),
where Rτ is such as logRτ (Z) =∑
iq Ziq log τiq and τ are thevariational parameters to optimize.
Variational Bayes (Latouche et al.)
I Put appropriate priors on α and π,I Give good performances especially for the choice of Q
and is thus relevant in the SIMoNe context.
SIMoNe: inferring Gaussian networks with latent clustering 42
Outline
Statistical modelsSteady-state dataTime-course dataMultitask learning
Algorithms and methodsOverall viewNetwork inferenceModel selectionLatent structure
Numerical experimentsPerformance on simulated dataR package demo: the breast cancer data set
SIMoNe: inferring Gaussian networks with latent clustering 43
Network generation
Let fixI the number p = card(P) of nodes,I if the graph is directed or not.
Affiliation matrix A = (aij)i,j∈P
1. usual MixNet frameworkI the Q×Q matrix Π, with πq` = P(aij = 1|i ∈ q, j ∈ `),I the Q-size vector α with αq = P(i ∈ q).
2. constraint MixNet versionI the Q×Q matrix Π, with πq` = card{(i, j) ∈ P × P : i ∈ q, j ∈ `},I the Q-size vector α with αq = card({i ∈ P : i ∈ q})/p.
SIMoNe: inferring Gaussian networks with latent clustering 44
Gaussian data generation
The Θ? matrix1. for undirected case, Θ? is the concentration matrix
I compute the normalized Laplacian of A,I generate a symmetric pattern of random signs.
2. for directed case, Θ? represents the VAR(1) parametersI generate random correlations for aij 6= 0,I normalized by the eigen-value with greatest modulus,I generate a pattern of random signs.
The Gaussian sample X
1. for undirected case,I compute Σ? by pseudo-inversion of Θ?,I generate the multivariate Gaussian sample with Cholesky
decomposition of Σ?.2. for directed case,
I Θ? permits to generate a stable VAR(1) process.
SIMoNe: inferring Gaussian networks with latent clustering 45
Gaussian data generation
The Θ? matrix1. for undirected case, Θ? is the concentration matrix
I compute the normalized Laplacian of A,I generate a symmetric pattern of random signs.
2. for directed case, Θ? represents the VAR(1) parametersI generate random correlations for aij 6= 0,I normalized by the eigen-value with greatest modulus,I generate a pattern of random signs.
The Gaussian sample X
1. for undirected case,I compute Σ? by pseudo-inversion of Θ?,I generate the multivariate Gaussian sample with Cholesky
decomposition of Σ?.2. for directed case,
I Θ? permits to generate a stable VAR(1) process.
SIMoNe: inferring Gaussian networks with latent clustering 45
Example 1: time-course data with star-pattern
Simulation settings
1. 50 networks with p = 100 edges, time series of length n = 100,2. two classes, hubs and leaves, with proportions α = (0.1, 0.9),3. P(hub to leaf) = 0.3, P(hub to hub) = 0.1, 0 otherwise.
SIMoNe: inferring Gaussian networks with latent clustering 46
Example 1: time-course data with star-pattern
Simulation settings
1. 50 networks with p = 100 edges, time series of length n = 100,2. two classes, hubs and leaves, with proportions α = (0.1, 0.9),3. P(hub to leaf) = 0.3, P(hub to hub) = 0.1, 0 otherwise.
precision wocl.BIC precision wocl.AIC
0.2
0.4
0.6
0.8
Boxplot of Precision values, without and with structure inference
precision = TP/(TP+FP)
SIMoNe: inferring Gaussian networks with latent clustering 46
Example 1: time-course data with star-pattern
Simulation settings
1. 50 networks with p = 100 edges, time series of length n = 100,2. two classes, hubs and leaves, with proportions α = (0.1, 0.9),3. P(hub to leaf) = 0.3, P(hub to hub) = 0.1, 0 otherwise.
recall wocl.BIC recall wcl.BIC recall wocl.AIC recall wcl.AIC
0.2
0.4
0.6
0.8
1.0
Boxplot of Recall values, without and with structure inference
recall = TP/P (power)
SIMoNe: inferring Gaussian networks with latent clustering 46
Example 1: time-course data with star-pattern
Simulation settings
1. 50 networks with p = 100 edges, time series of length n = 100,2. two classes, hubs and leaves, with proportions α = (0.1, 0.9),3. P(hub to leaf) = 0.3, P(hub to hub) = 0.1, 0 otherwise.
●
●
●
●
●
●
●
●
●
●
●
●
fallout wocl.BIC fallout wcl.BIC fallout wocl.AIC fallout wcl.AIC
0.00
0.01
0.02
0.03
0.04
Boxplot of Fallout values, without and with structure inference
fallout = FP/N (type I error)
SIMoNe: inferring Gaussian networks with latent clustering 46
Example 2: steady-state, multitask framework
Simulating the tasks
1. generate a “ancestor” with p = 20 node and K = 20 edges,2. generate T = 4 children by adding and deleting δ edges,3. generate T = 4 Gaussian samples.
Figure: ancestor and children with δ perturbations
SIMoNe: inferring Gaussian networks with latent clustering 47
Example 2: steady-state, multitask framework
Simulating the tasks
1. generate a “ancestor” with p = 20 node and K = 20 edges,2. generate T = 4 children by adding and deleting δ edges,3. generate T = 4 Gaussian samples.
Figure: ancestor and children with δ perturbations
SIMoNe: inferring Gaussian networks with latent clustering 47
Example 2: steady-state, multitask framework
Simulating the tasks
1. generate a “ancestor” with p = 20 node and K = 20 edges,2. generate T = 4 children by adding and deleting δ edges,3. generate T = 4 Gaussian samples.
Figure: ancestor and children with δ perturbations
SIMoNe: inferring Gaussian networks with latent clustering 47
Example 2: steady-state, multitask framework
Simulating the tasks
1. generate a “ancestor” with p = 20 node and K = 20 edges,2. generate T = 4 children by adding and deleting δ edges,3. generate T = 4 Gaussian samples.
Figure: ancestor and children with δ perturbations
SIMoNe: inferring Gaussian networks with latent clustering 47
Multitask: simulation results
Precision/Recall curveprecision = TP/(TP+FP)
recall = TP/P (power)
ROC curvefallout = FP/N (type I error)
recall = TP/P (power)
SIMoNe: inferring Gaussian networks with latent clustering 48
Multitask: simulation results
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
recall
pre
cisi
on
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
falloutre
ca
ll
Figure: nt = 25, δ = 1
SIMoNe: inferring Gaussian networks with latent clustering 48
Multitask: simulation results
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
recall
pre
cisi
on
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
falloutre
ca
ll
Figure: nt = 25, δ = 3
SIMoNe: inferring Gaussian networks with latent clustering 48
Multitask: simulation results
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
recall
pre
cisi
on
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
falloutre
ca
ll
Figure: nt = 25, δ = 5
SIMoNe: inferring Gaussian networks with latent clustering 48
Multitask: simulation results
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
recall
pre
cisi
on
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
falloutre
ca
ll
Figure: nt = 50, δ = 1
SIMoNe: inferring Gaussian networks with latent clustering 48
Multitask: simulation results
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
recall
pre
cisi
on
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
falloutre
ca
ll
Figure: nt = 50, δ = 3
SIMoNe: inferring Gaussian networks with latent clustering 48
Multitask: simulation results
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
recall
pre
cisi
on
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
falloutre
ca
ll
Figure: nt = 50, δ = 5
SIMoNe: inferring Gaussian networks with latent clustering 48
Multitask: simulation results
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
recall
pre
cisi
on
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
falloutre
ca
ll
Figure: nt = 100, δ = 1
SIMoNe: inferring Gaussian networks with latent clustering 48
Multitask: simulation results
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
recall
pre
cisi
on
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
falloutre
ca
ll
Figure: nt = 100, δ = 3
SIMoNe: inferring Gaussian networks with latent clustering 48
Multitask: simulation results
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
recall
pre
cisi
on
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
CoopLassoGroupLassoIntertwinedIndependentPooled
penalty: λmax −→ 0
falloutre
ca
ll
Figure: nt = 100, δ = 5
SIMoNe: inferring Gaussian networks with latent clustering 48
Outline
Statistical modelsSteady-state dataTime-course dataMultitask learning
Algorithms and methodsOverall viewNetwork inferenceModel selectionLatent structure
Numerical experimentsPerformance on simulated dataR package demo: the breast cancer data set
SIMoNe: inferring Gaussian networks with latent clustering 49
Breast cancerPrediction of the outcome of preoperative chemotherapy
Two types of patientsPatient response can be classified as
1. either a pathologic complete response (PCR),2. or residual disease (not PCR).
Gene expression data
I 133 patients (99 not PCR, 34 PCR)I 26 identified genes (differential analysis)
SIMoNe: inferring Gaussian networks with latent clustering 50
Pooling the data
cancer data: pooling approach
demo/cancer_pooled.swf
SIMoNe: inferring Gaussian networks with latent clustering 51
Multitask approach: PCR / not PCR
cancer data: graphical cooperative Lasso
demo/cancer_mtasks.swf
SIMoNe: inferring Gaussian networks with latent clustering 52
Conclusions
To sum-up
I SIMoNe embeds most state-of-the-art statistical methods forGGM inference based upon `1-penalization,
I both steady-state and time course data can be dealt with,I (hopefully) biologist-friendly R package.
PerspectivesAdding transversal tools such asI network comparison,I bootstrap to limit the number of false positives,I more critieria to choose the penalty parameter,I interface to Gene Ontology.
SIMoNe: inferring Gaussian networks with latent clustering 53
Publications
Ambroise, Chiquet, Matias, 2009.Inferring sparse Gaussian graphical models with latent structureElectronic Journal of Statistics, 3, 205-238.
Chiquet, Smith, Grasseau, Matias, Ambroise, 2009.SIMoNe: Statistical Inference for MOdular NEtworks Bioinformatics,25(3), 417-418.
Charbonnier, Chiquet, Ambroise, 2010.Weighted-Lasso for Structured Network Inference from TimeCourse Data., SAGMB, 9.
Chiquet, Grandvalet, Ambroise, arXiv preprint.Inferring multiple Gaussian graphical models.
SIMoNe: inferring Gaussian networks with latent clustering 54
Publications
Ambroise, Chiquet, Matias, 2009.Inferring sparse Gaussian graphical models with latent structureElectronic Journal of Statistics, 3, 205-238.
Chiquet, Smith, Grasseau, Matias, Ambroise, 2009.SIMoNe: Statistical Inference for MOdular NEtworks Bioinformatics,25(3), 417-418.
Charbonnier, Chiquet, Ambroise, 2010.Weighted-Lasso for Structured Network Inference from TimeCourse Data., SAGMB, 9.
Chiquet, Grandvalet, Ambroise, arXiv preprint.Inferring multiple Gaussian graphical models.
Working paper: Chiquet, Charbonnier, Ambroise, Grasseau.SIMoNe: An R package for inferring Gausssian networks withlatent structure, Journal of Statistical Softwares.
Working paper: Chiquet, Grandvalet, Ambroise, Jeanmougin.Biological analysis of breast cancer by multitasks learning.
SIMoNe: inferring Gaussian networks with latent clustering 54