Image segmentation combining Markov Random Fields and ... · Inference : Swendsen-Wang algorithm...
Transcript of Image segmentation combining Markov Random Fields and ... · Inference : Swendsen-Wang algorithm...
ANR meeting
Image segmentation combining MarkovRandom Fields and Dirichlet Processes
Jessica SODJO
IMS, Groupe Signal Image, TalenceEncadrants : A. Giremus, J.-F. Giovannelli, F. Caron, N. Dobigeon
Jessica SODJO ANR meeting 1 / 28
ANR meeting
Plan
1 Introduction
2 Segmentation using DP modelsMixed MRF / DP modelInference : Swendsen-Wang algorithm
3 Hierarchical segmentation with shared classesPrincipleHDP theory
4 Conclusion and perspective
Jessica SODJO ANR meeting 2 / 28
ANR meeting
Introduction
Segmentation
– partition of an image in K homogeneous regions calledclasses
– label the pixels : pixel i ↔ zi ∈ 1, . . . ,K
Bayesian approach
– prior on the distribution of the pixels– all the pixels in a class have the same distribution
characterized by a parameter vector Uk
– Markov Random Fields (MRF) : exploit the similarity ofpixels in the same neighbourhood
Constraint : K must be fixed a priori
Idea : use the BNP models to directly estimate K
Jessica SODJO ANR meeting 3 / 28
ANR meeting
Segmentation using DP models
Plan
1 Introduction
2 Segmentation using DP modelsMixed MRF / DP modelInference : Swendsen-Wang algorithm
3 Hierarchical segmentation with shared classesPrincipleHDP theory
4 Conclusion and perspective
Jessica SODJO ANR meeting 4 / 28
ANR meeting
Segmentation using DP models
Notations
– N is the number of pixels– Y is the observed image– Z = z1, . . . , zN– Π = A1, . . . ,AK is a partition and m = m1, . . . ,mK with
mk = |Ak |
A1
A3
A2
AK
m1 = 1m2 = 5m3 = 6mK = 4
FIGURE: Example of partitionJessica SODJO ANR meeting 5 / 28
ANR meeting
Segmentation using DP models
Mixed MRF / DP model
Markov Random Fields (MRF)
– Description of the image by a neighbouring system
4-neighbours 8-neighbours
ConsideredpixelNeighbours
FIGURE: Examples of neighbouring system
– A clique c is either a singleton either a set of pixels in thesame neighbourhood
Jessica SODJO ANR meeting 6 / 28
ANR meeting
Segmentation using DP models
Mixed MRF / DP model
Markov Random Fields
Let θi ∈ U1, . . . ,UK be the parameter vector associated to thei-th pixel
MRF⇔ p(θi | θ−i) = p(θi | θV(i))where V(i) is the set of neighbours of pixel i
Hammersley-Clifford theorem⇒ Gibbs field
p(θ) =1
ZΦexp (−Φ(θ)) =
1ZΦ
exp
(−∑
c
Φc(θc)
)(1)
with Φc(θc) the local potential and Φ(θ) the global one
Limitation : K is assumed to be known
Jessica SODJO ANR meeting 7 / 28
ANR meeting
Segmentation using DP models
Mixed MRF / DP model
Potts model
The Potts model is a special MRF defined by :
M(Π) ∝ exp
∑i↔j
βij1zi =zj
(2)
where– i ↔ j means that the pixels i and j are neighbours– βij > 0 if i and j are neighbours and βij = 0 otherwise
Jessica SODJO ANR meeting 8 / 28
ANR meeting
Segmentation using DP models
Mixed MRF / DP model
The DP model
τ ′k | γ,H ∼ Beta(1, γ) τk = τ ′k
k−1∏l=1
(1− τ ′l ) (3)
where Beta(.) is the Beta distribution
Let us write τ ∼ Stick(γ), τ = τ1, τ2, . . . and∑∞
k=1 τk = 1
G | γ,H ∼ DP(γ,H) G =∞∑
k=1
τkδUk (4)
withUk | H
iid∼ H (5)
The distribution of the observations is f , defined as :
yi | θi ∼ f (. | θi) and θi | G ∼ G (6)
Jessica SODJO ANR meeting 9 / 28
ANR meeting
Segmentation using DP models
Mixed MRF / DP model
The DP model
The Chinese Restaurant Process says,
θi | θ−i ∼K−i∑k=1
m−ik
N − 1 + γδUk +
γ
N − 1 + γH
– m−ik is the size of cluster k if we remove pixel i from the
partition– K−i is the number of clusters in the image with the i-th
pixel removed– Uk is the parameter vector associated to the k -th cluster
Limitation : the spatial interactions are not taken into account
Jessica SODJO ANR meeting 10 / 28
ANR meeting
Segmentation using DP models
Mixed MRF / DP model
Principle of the segmentation using DP models
Define a distribution on the partitions using :– a model that allows that pixels in the same neighbourhood
are likely to be in the same cluster (MRF)– DP model to deduce automatically the number of clusters
(and if needed their parameters)
Jessica SODJO ANR meeting 11 / 28
ANR meeting
Segmentation using DP models
Mixed MRF / DP model
Prior distribution mixing DP and MRF
p(θ) ∝ 1ZG
exp(−∑
i
Φi(θi))︸ ︷︷ ︸Ψ(θ) DP model
1ZM
exp(−∑c∈C2
Φc(θc))
︸ ︷︷ ︸M(θ) MRF model
where– C2 means |c| > 2 and |.| is the size.– Φi(.) is defined as :
Φi(θi) = − logG(θi) and ZG =
∫ N∏i=1
exp(− logG(θi))dθ1 . . . dθN
⇒ Ψ(θ) =N∏
i=1
G(θi)
P. Orbanz & J. M. BuhmannNonparametric Bayesian image segmentation, International Journal of Computer Vision, 2007
Jessica SODJO ANR meeting 12 / 28
ANR meeting
Segmentation using DP models
Mixed MRF / DP model
Prior distribution mixing DP and MRF
We can deduce :
P(θi | θ−i) ∝K∑
k=1
M(θi | θ−i)m−ik δUk +
γ
ZΦH (7)
Probability of assignment to a new cluster :
qi0 ∝∫
Ωθ
f (yi | θ)H(θ)dθ (8)
Probability of assignment to an existing cluster :
qik ∝ m−ik exp(−Φ(Uk | θ−i))f (yi | Uk ) (9)
Parameter update :
Uk ∼ G0(Uk )∏
i|i∈Ak
f (yi | Uk ) (10)
Jessica SODJO ANR meeting 13 / 28
ANR meeting
Segmentation using DP models
Inference : Swendsen-Wang algorithm
Swendsen-Wang algorithm : principle
* Estimation based on the joint posterior p(θ,Z | Y )
* Intractable⇒ Markov Chain Monte Carlo (MCMC)
Problem : very slow convergence
Goal : Sample faster the partition of the image– Introduction of a new set of latent variables r such that :
p(Π, r) = p(Π)p(r | Π)
p(r | Π) =∏
1<i<j<Np(rij | Π)
p(rij = 1 | Π) = 1− exp(βijδij1zi =zj )
The marginal posterior p(θ,Z | Y ) is unchanged– The links define the "so-called" spin-clusters
Jessica SODJO ANR meeting 14 / 28
ANR meeting
Segmentation using DP models
Inference : Swendsen-Wang algorithm
Swendsen-Wang algorithm : principle
– Update the labels of the spin-clustersThis operation update simultaneously the labels of all thepixels in a spin-cluster
FIGURE: Example of label update for spin-clusters
Jessica SODJO ANR meeting 15 / 28
ANR meeting
Segmentation using DP models
Inference : Swendsen-Wang algorithm
Swendsen-Wang algorithm : principle
– rij ∼ Ber(1− exp(βijδij1zi =zj ))with Ber(.) is the Bernouilli distribution
Let S = S1, . . . ,Sp be the set of spin-clusters.
– While removing the spin-cluster Sl ,
Π−l = A−l1 , . . . ,A−l
K−l is the partition obtained while
removing all pixels in spin-cluster Sl
m−lk = |A−l
k |
Jessica SODJO ANR meeting 16 / 28
ANR meeting
Segmentation using DP models
Inference : Swendsen-Wang algorithm
Swendsen-Wang algorithm : principle
For l = 1 : p* The probability to assign pixels in spin-cluster Sl to cluster
k is :
qlk ∝ Ψ(m−l1 , . . . ,m−l
k + |Sl |, . . . ,m−lK−l
)p(ySl | yA−lk
)∏(i,j)|i∈Sl ,rij =0
exp(βij (1− δij )1zi =zj )
* The probability to assign pixels in spin-cluster Sl to a newcluster is :
ql0 = Ψ(m−l1 , . . . ,m−l
K−l, |Sl |)p(ySl )
with p(yAk ) =∫ ∏
i∈Ak
f (yi | Uk )H(Uk )dUk
Jessica SODJO ANR meeting 17 / 28
ANR meeting
Hierarchical segmentation with shared classes
Plan
1 Introduction
2 Segmentation using DP modelsMixed MRF / DP modelInference : Swendsen-Wang algorithm
3 Hierarchical segmentation with shared classesPrincipleHDP theory
4 Conclusion and perspective
Jessica SODJO ANR meeting 18 / 28
ANR meeting
Hierarchical segmentation with shared classes
Principle
Proposed idea
– Different levels of classification can be considered– Coarse categories : urban, sub-urban, forest, etc.– Sub-classes shared between the categories : trees, roads,
buildings
Taking into account the fact that the classes are sharedbetween different categories can help estimating theirparameters and thereby improve the segmentation
Jessica SODJO ANR meeting 19 / 28
ANR meeting
Hierarchical segmentation with shared classes
HDP theory
Solution : Hierarchical DP
Let J be the number of categories
G0 | γ,H ∼ DP(γ,H)
Gj | α0,G0 ∼ DP(α0,G0) for j = 1, . . . , J
α0 ∈ R∗+G0 is a discrete distribution
Discreteness of G0 ⇒ clusters shared among categories
Jessica SODJO ANR meeting 20 / 28
ANR meeting
Hierarchical segmentation with shared classes
HDP theory
G0 =∞∑
k=1
τkδUk (11)
where τ |γ ∼ Stick(γ), τ = τ1, τ2, . . . and Uk | H ∼ H
Gj =∞∑
k=1
πjkδUk (12)
with πj | α0, τ ∼ DP(α0, τ ) and πj = πj1, πj2, . . .
ϕji | Gj ∼ Gj (13)
So, samples of the processes G0 and Gj can be seen as infinitecountable mixtures of Dirac measures with respectivecoefficients τ and πj .
Jessica SODJO ANR meeting 21 / 28
ANR meeting
Hierarchical segmentation with shared classes
HDP theory
Principle - Chinese Restaurant Franchise
NOTATIONS
– J restaurants– Same menu for all restaurants - U1,U2, . . .
– Tj is the number of tables in restaurant j– θjt is the t-th table of restaurant j– ϕji is the i-th client in restaurant j– njt is the number of clients at a table t– ηjk is the number of tables in restaurant j
which have chosen dish Uk and ηk =∑
k ηjk
Jessica SODJO ANR meeting 22 / 28
ANR meeting
Hierarchical segmentation with shared classes
HDP theory
Principle - Chinese Restaurant FranchiseRestaurant 1
. . .ϕ11ϕ13
ϕ12 ϕ14
Restaurant 2
. . .ϕ21ϕ23
ϕ22ϕ25
Restaurant 3
. . .ϕ31 ϕ32 ϕ33
MenuU1U2U3...
θ 11=
U 1
θ 12=
U 2
θ 13=
U 2
θ 21=
U 2
θ 22=
U 1
θ 31=
U 1
θ 32=
U 2
θ 33=
U 3
Jessica SODJO ANR meeting 23 / 28
ANR meeting
Hierarchical segmentation with shared classes
HDP theory
Principle - Chinese Restaurant Franchise
Exemple : Restaurant 1 ϕ13ϕ11
ϕ12
ϕ14
θ 11=
U 1
θ 12=
U 2
θ 13=
U 2
ϕ15
n11 =
2
n12 = 1
n13=
1
α 0
θ14
MenuU1
U2
U3
η1 = 3η2 = 4
η3 = 1
γU4Jessica SODJO ANR meeting 24 / 28
ANR meeting
Hierarchical segmentation with shared classes
HDP theory
Principle - Chinese Restaurant Franchise
Exemple : Restaurant 1 ϕ13ϕ11
ϕ12
ϕ14
θ 11=
U 1
θ 12=
U 2
θ 13=
U 2
ϕ15
n11 =
2
n12 = 1
n13=
1
α 0
θ14
MenuU1
U2
U3
η1 = 3η2 = 4
η3 = 1
γU4Jessica SODJO ANR meeting 24 / 28
ANR meeting
Hierarchical segmentation with shared classes
HDP theory
Principle - Chinese Restaurant Franchise
Exemple : Restaurant 1 ϕ13ϕ11
ϕ12
ϕ14
θ 11=
U 1
θ 12=
U 2
θ 13=
U 2
ϕ15
n11 =
2
n12 = 1
n13=
1
α 0
θ14
MenuU1
U2
U3
η1 = 3η2 = 4
η3 = 1
γU4Jessica SODJO ANR meeting 24 / 28
ANR meeting
Hierarchical segmentation with shared classes
HDP theory
Principle - Chinese Restaurant Franchise
Exemple : Restaurant 1 ϕ13ϕ11
ϕ12
ϕ14
θ 11=
U 1
θ 12=
U 2
θ 13=
U 2
ϕ15
n11 =
2
n12 = 1
n13=
1
α 0
θ14
MenuU1
U2
U3
η1 = 3η2 = 4
η3 = 1
γU4Jessica SODJO ANR meeting 24 / 28
ANR meeting
Hierarchical segmentation with shared classes
HDP theory
Chinese Restaurant Franchise
ϕji | ϕj1, . . . , ϕji−1, α0,G0 ∼Tj∑
t=1
njt
i − 1 + α0δθjt +
α0
i − 1 + α0G0 (14)
θjt | θj1, . . . , θ21, . . . , θjt−1, γ,H ∼K∑
k=1
ηk∑k ηk + γ
δUk +γ∑
k ηk + γH (15)
Y. W. Teh, M. I. Jordan, M. J. Beal & D. M. BleiHierarchical Dirichlet Processes, JASA, 2006
Jessica SODJO ANR meeting 25 / 28
ANR meeting
Conclusion and perspective
Plan
1 Introduction
2 Segmentation using DP modelsMixed MRF / DP modelInference : Swendsen-Wang algorithm
3 Hierarchical segmentation with shared classesPrincipleHDP theory
4 Conclusion and perspective
Jessica SODJO ANR meeting 26 / 28
ANR meeting
Conclusion and perspective
Conclusion– Spatial constraints : Potts model– Flexibility : DP model– Rapidity : Swendsen-Wang algorithm– Sharing : HDP
Perspective– Efficient sampling algorithm
Jessica SODJO ANR meeting 27 / 28
ANR meeting
Thank
Thank you for your attention
Jessica SODJO ANR meeting 28 / 28