Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14...
Transcript of Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14...
![Page 1: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/1.jpg)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 1
Statistical performance analysis by loopy belief propagation in
probabilistic image processing Kazuyuki Tanaka
Graduate School of Information Sciences,Tohoku University, Japan
http://www.smapip.is.tohoku.ac.jp/~kazu/
CollaboratorsD. M. Titterington (University of Glasgow, UK)
M. Yasuda (Tohoku University, Japan)S. Kataoka (Tohoku University, Japan)
![Page 2: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/2.jpg)
2
IntroductionBayesian network is originally one of the methods for probabilistic inferences in artificial intelligence. Some probabilistic models for information processing are also regarded as Bayesian networks. Bayesian networks are expressed in terms of products of functions with a couple of random variables and can be associated with graphical representations. Such probabilistic models for Bayesian networks are referred to as Graphical Model.
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
![Page 3: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/3.jpg)
3
Probabilistic Image Processing by Bayesian Network
Probabilistic image processing systems are formulated on square grid graphs. Averages, variances and covariances of the Bayesian network are approximately computed by using the belief propagation on the square grid graph.
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
![Page 4: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/4.jpg)
10 March, 2010 IW-SMI2010 (Kyoto) 4
MRF, Belief Propagation and Statistical Performance
Nishimori and Wong (1999): Physical Review EStatistical Performance Estimation for MRF
(Infinite Range Ising Model and Replica Theory)
Is it possible to estimate the performance of belief propagation statistically?
Tanaka and Morita (1995): Physics Letters ACluster Variation Method for MRF in Image Processing
CVM= Generalized Belief Propagation (GBP)
Geman and Geman (1986): IEEE Transactions on PAMIImage Processing by Markov Random Fields (MRF)
![Page 5: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/5.jpg)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 5
Outline
1. Introduction2. Bayesian Image Analysis by Gauss Markov
Random Fields3. Statistical Performance Analysis for Gauss
Markov Random Fields4. Statistical Performance Analysis in Binary
Markov Random Fields by Loopy Belief Propagation
5. Concluding Remarks
![Page 6: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/6.jpg)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 6
Outline
1. Introduction2. Bayesian Image Analysis by Gauss Markov
Random Fields3. Statistical Performance Analysis for Gauss
Markov Random Fields4. Statistical Performance Analysis in Binary
Markov Random Fields by Loopy Belief Propagation
5. Concluding Remarks
![Page 7: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/7.jpg)
Image Restoration by Bayesian Statistics
Original Image
14 October, 2010 7LRI Seminar 2010 (Univ. Paris-Sud)
![Page 8: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/8.jpg)
Image Restoration by Bayesian Statistics
Original Image
Degraded Image
Transmission
Noise
14 October, 2010 8LRI Seminar 2010 (Univ. Paris-Sud)
![Page 9: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/9.jpg)
Image Restoration by Bayesian Statistics
Original Image
Degraded Image
Transmission
Noise
Estimate
14 October, 2010 9LRI Seminar 2010 (Univ. Paris-Sud)
![Page 10: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/10.jpg)
10
Image Restoration by Bayesian Statistics
PriorProcessn Degradatio
Posterior
}Image OriginalPr{}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
∝
Bayes Formula
Original Image
Degraded Image
Transmission
Noise
Estimate
Posterior
14 October, 2010 10LRI Seminar 2010 (Univ. Paris-Sud)
![Page 11: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/11.jpg)
11
Image Restoration by Bayesian Statistics
PriorProcessn Degradatio
Posterior
}Image OriginalPr{}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
∝
Assumption 1: Original images are randomly generated by according to a prior probability.
Bayes Formula
Original Image
Degraded Image
Transmission
Noise
Estimate
Posterior
14 October, 2010 11LRI Seminar 2010 (Univ. Paris-Sud)
![Page 12: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/12.jpg)
7 July, 2010 12
Image Restoration by Bayesian Statistics
PriorProcessn Degradatio
Posterior
}Image OriginalPr{}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
∝
Bayes Formula
Assumption 2: Degraded images are randomly generated from the original image by according to a conditional probability of degradation process.
Original Image
Degraded Image
Transmission
Noise
Estimate
Posterior
12LRI Seminar 2010 (Univ. Paris-Sud)14 October, 2010
![Page 13: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/13.jpg)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 13
Bayesian Image Analysis
Prior Probability
−−=
−−∝= ∑∏
∈∈ Ejiji
Ejiji fffffF
},{
2
},{
2 )(21exp)(
21exp}|Pr{ ααα
Assumption: Prior Probability consists of a product of functions defined on the neighbouring pixels.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
×
∝
0>α
![Page 14: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/14.jpg)
14
Bayesian Image Analysis
Prior Probability
−−=
−−∝= ∑∏
∈∈ Ejiji
Ejiji fffffF
},{
2
},{
2 )(21exp)(
21exp}|Pr{ ααα
Assumption: Prior Probability consists of a product of functions defined on the neighbouring pixels.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
×
∝
0>α
14 October, 2010 14LRI Seminar 2010 (Univ. Paris-Sud)
![Page 15: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/15.jpg)
15
Bayesian Image Analysis
0005.0=α 0030.0=α0001.0=α
Patterns by MCMC.
Prior Probability
−−=
−−∝= ∑∏
∈∈ Ejiji
Ejiji fffffF
},{
2
},{
2 )(21exp)(
21exp}|Pr{ ααα
Assumption: Prior Probability consists of a product of functions defined on the neighbouring pixels.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
×
∝
0>α
14 October, 2010 15LRI Seminar 2010 (Univ. Paris-Sud)
![Page 16: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/16.jpg)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 16
Bayesian Image Analysis
Assumption: Degraded image is generated from the original image by Additive White Gaussian Noise.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
×
∝
∏∈
−−∝
==
Viii fg
fFgG
22 )(
21exp
},|Pr{
σ
σ
0>σV:Set of all the pixels
![Page 17: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/17.jpg)
17
Bayesian Image Analysis
Assumption: Degraded image is generated from the original image by Additive White Gaussian Noise.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
×
∝
14 October, 2010 17LRI Seminar 2010 (Univ. Paris-Sud)
∏∈
−−∝
==
Viii fg
fFgG
22 )(
21exp
},|Pr{
σ
σ
![Page 18: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/18.jpg)
18
Bayesian Image AnalysisAssumption: Degraded image is generated from the original image by Additive White Gaussian Noise.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
×
∝
∏∈
−−∝
==
Viii fg
fFgG
22 )(
21exp
},|Pr{
σ
σ
0>σ
14 October, 2010 18LRI Seminar 2010 (Univ. Paris-Sud)
![Page 19: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/19.jpg)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 19
Bayesian Image Analysis
f
g
}|Pr{ αfF
= },|Pr{ σfFgG
== gOriginalImage
Degraded Image
Prior ProbabilityPosterior Probability
Degradation Process
−−−−∝
====
=
==
∑∑∈∈ Eji
jiVi
ii ffgf
gGfFfFgG
gGfF
},{
222 )(
21)(
21exp
},|Pr{},Pr{},|Pr{
},,|Pr{
ασ
σαασ
σα
Estimate
![Page 20: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/20.jpg)
20
Bayesian Image Analysis
f
g
}|Pr{ αfF
= },|Pr{ σfFgG
== gOriginalImage
Degraded Image
Prior ProbabilityPosterior Probability
Degradation Process
−−−−∝
====
=
==
∑∑∈∈ Eji
jiVi
ii ffgf
gGfFfFgG
gGfF
},{
222 )(
21)(
21exp
},|Pr{},Pr{},|Pr{
},,|Pr{
ασ
σαασ
σα
SmoothingData Dominant
Estimate
14 October, 2010 20LRI Seminar 2010 (Univ. Paris-Sud)
![Page 21: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/21.jpg)
11 March, 2010 IW-SMI2010 (Kyoto) 21
Bayesian Image Analysis
f
g
}|Pr{ αfF
= },|Pr{ σfFgG
== gOriginalImage
Degraded Image
Prior ProbabilityPosterior Probability
Degradation Process
−−−−∝
====
=
==
∑∑∈∈ Eji
jiVi
ii ffgf
gGfFfFgG
gGfF
},{
222 )(
21)(
21exp
},|Pr{},Pr{},|Pr{
},,|Pr{
ασ
σαασ
σα
SmoothingData Dominant
Bayesian Network Estimate
14 October, 2010 21LRI Seminar 2010 (Univ. Paris-Sud)
![Page 22: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/22.jpg)
( ) g
zdgGzFz
ghf
12
},,|Pr{
),,(ˆ
−+=
===
=
∫CI ασ
σα
σα
22
Bayesian Image Analysis
f
g
}|Pr{ αfF
= },|Pr{ σfFgG
== gOriginalImage
Degraded Image
Prior ProbabilityPosterior Probability
Degradation Process
−−−−∝
====
=
==
∑∑∈∈ Eji
jiVi
ii ffgf
gGfFfFgG
gGfF
},{
222 )(
21)(
21exp
},|Pr{},Pr{},|Pr{
},,|Pr{
ασ
σαασ
σα
Field Random Markov Gauss),(
⇒
+∞−∞∈iF
SmoothingData Dominant
Bayesian Network Estimate
14 October, 2010 22LRI Seminar 2010 (Univ. Paris-Sud)
∈−∈=∂
=otherwise,0
},{,1|,|
EjiVjii
ji C
![Page 23: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/23.jpg)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 23
Image Restorations by Gaussian Markov Random Fields
and Conventional Filters
2||ˆ||
||1MSE ∑
∈−=
Viff
V
MSEGauss Markov Random Field 315
Lowpass Filter
(3x3) 388
(5x5) 413
Median Filter
(3x3) 486
(5x5) 445
(3x3) Lowpass (5x5) MedianGauss Markov Random Field
Original Image Degraded Image
RestoredImage V: Set of all
the pixels
Field Random Markov Gauss),(
⇒
+∞−∞∈iF
![Page 24: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/24.jpg)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 24
Outline
1. Introduction2. Bayesian Image Analysis by Gauss Markov
Random Fields3. Statistical Performance Analysis for Gauss
Markov Random Fields4. Statistical Performance Analysis in Binary
Markov Random Fields by Loopy Belief Propagation
5. Concluding Remarks
![Page 25: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/25.jpg)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 25
Statistical Performance by Sample Average of Numerical Experiments
f
Original Images
![Page 26: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/26.jpg)
26
Statistical Performance by Sample Average of Numerical Experiments
1g
f
2g
3g
4g
5g
Original ImagesN
oise
Pr
{G|F
=f,σ
}
ObservedData
14 October, 2010 26LRI Seminar 2010 (Univ. Paris-Sud)
![Page 27: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/27.jpg)
11 March, 2010 IW-SMI2010 (Kyoto) 27
Statistical Performance by Sample Average of Numerical Experiments
1g
f
2g
3g
4g
5g
1h
2h
3h
4h
4h
Post
erio
r Pr
obab
ility
Pr{F
|G=g
, α,σ
}
Original ImagesN
oise
Pr
{G|F
=f,σ
}
Estimated ResultsObserved
Data
14 October, 2010 27LRI Seminar 2010 (Univ. Paris-Sud)
![Page 28: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/28.jpg)
28
Statistical Performance by Sample Average of Numerical Experiments
1g
f
2g
3g
4g
5g
1h
2h
3h
4h
4h
Post
erio
r Pr
obab
ility
Pr{F
|G=g
,α,σ
}
Sample Average of Mean Square Error
Original ImagesN
oise
Pr
{G|F
=f,σ
}
∑=
−≅5
1
2||||51),|,(
nnhffD
σσα
Estimated ResultsObserved
Data
14 October, 2010 28LRI Seminar 2010 (Univ. Paris-Sud)
![Page 29: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/29.jpg)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 29
Statistical Performance Estimation
gdfFgGfghV
fD },|Pr{),,(1),|,(
2σσασσα ==−= ∫
( ) g
gh
12
),,(−
+= CI ασ
σα
gg
Additive White Gaussian Noise
},|Pr{ σfFgG
==f
},,|Pr{ σαgGfF ==
Posterior Probability
RestoredImage
Original Image Degraded Image
},|Pr{ σfFgG
==
Additive White Gaussian Noise
∈−∈=∂
=otherwise,0
},{,1|,|
EjiVjii
ji C
![Page 30: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/30.jpg)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 30
Statistical Performance Estimation for Gauss Markov Random Fields
T22
242
22
2
22
||2
2
2
)(||1
)(Tr1
21exp
211
},|Pr{),,(1),|,(
fI
fVV
gdfggfV
gdfFgGghfV
fD
V
CC
CII
CII
ασσα
ασσ
σσπασ
σσασσα
++
+=
−−
+
−=
==−=
∫
∫
∈−∈=∂
=otherwise
EjiVjii
ji,0
},{,1|,|
C
0
200
400
600
0 0.001 0.002 0.003α
σ=40
0
200
400
600
0 0.001 0.002 0.003
σ=40
α
),|,( σσα fD
),|,( σσα fD
![Page 31: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/31.jpg)
Outline
1. Introduction2. Bayesian Image Analysis by Gauss Markov
Random Fields3. Statistical Performance Analysis for Gauss
Markov Random Fields4. Statistical Performance Analysis in Binary
Markov Random Fields by Loopy Belief Propagation
5. Concluding Remarks
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 31
![Page 32: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/32.jpg)
32
Marginal Probability in Belief Propagation
{ } { }∑∑∑ ∑ ===1 3 4
,,,,,PrPr 43212F F F F
NN
FFFFFF gGgG
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
In order to compute the marginal probability Pr{F2|G=g},we take summations over all the pixels except the pixel 2.
![Page 33: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/33.jpg)
33
Marginal Probability in Belief Propagation
∑∑∑ ∑=1 3 4F F F FN
2
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
{ } { }∑∑∑ ∑ ===1 3 4
,,,,,PrPr 43212F F F F
NN
FFFFFF gGgG
![Page 34: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/34.jpg)
34
Marginal Probability in Belief Propagation
∑∑∑ ∑=1 3 4F F F FN
2 2≅
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
{ } { }∑∑∑ ∑ ===1 3 4
,,,,,PrPr 43212F F F F
NN
FFFFFF gGgG
In the belief propagation, the marginal probability Pr{F2|G=g} is approximately expressed in terms of the messages from the neighbouring region of the pixel 2.
![Page 35: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/35.jpg)
35
Marginal Probability in Belief Propagation
∑∑ ∑=3 4F F FN
1 2
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
In order to compute the marginal probability Pr{F1,F2|G=g},we take summations over all the pixels except the pixels 1 and2.
{ } { }∑∑ ∑ ===3 4
,,,,,Pr,Pr 432121F F F
NN
FFFFFFF gGgG
![Page 36: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/36.jpg)
36
Marginal Probability in Belief Propagation
∑∑ ∑=3 4F F FN
1 2 1 2≅
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
{ } { }∑∑ ∑ ===3 4
,,,,,Pr,Pr 432121F F F
NN
FFFFFFF gGgG
In the belief propagation, the marginal probability Pr{F1,F2|G=g} is approximately expressed in terms of the messages from the neighbouring region of the pixels 1 and 2.
![Page 37: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/37.jpg)
Belief Propagation in Probabilistic Image Processing
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 37
![Page 38: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/38.jpg)
Belief Propagation in Probabilistic Image Processing
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 38
![Page 39: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/39.jpg)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 39
Image Restorations by Gaussian Markov Random Fields and Conventional Filters
MSE
Gauss MRF (Exact) 315
Gauss MRF (Belief Propagation) 327
Lowpass Filter
(3x3) 388
(5x5) 413
Median Filter
(3x3) 486
(5x5) 445
Belief PropagationExact
Original Image Degraded Image
RestoredImage
V: Set of all the pixels
Field Random Markov Gauss),(
⇒
+∞−∞∈iF
( )2ˆ||
1MSE ∑∈
−=Vi
ii ffV
![Page 40: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/40.jpg)
40
Gray-Level Image Restoration(Spike Noise)
Original Image
MSE:135MSE: 217
MSE: 371 MSE: 523
MSE: 244
MSE: 3469
MSE: 2075
Degraded Image
Belief Propagation Lowpass Filter Median Filter
MSE: 395
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
![Page 41: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/41.jpg)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 41
Binary Image Restoration byLoopy Belief Propagation
α∗=0.465
![Page 42: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/42.jpg)
42
Statistical Performance by Sample Average of Numerical Experiments
1g
f
2g
3g
4g
5g
1h
2h
3h
4h
4h
Post
erio
r Pr
obab
ility
Pr{F
|G=g
,α,σ
}
Sample Average of Mean Square Error
Original ImagesN
oise
Pr
{G|F
=f,σ
}
∑=
−≅5
1
2||||51),|,(
nnhffD
σσα
Estimated ResultsObserved
Data
14 October, 2010 42LRI Seminar 2010 (Univ. Paris-Sud)
![Page 43: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/43.jpg)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 43
Statistical Performance Estimation for Binary Markov Random Fields
gdfFgGfghV
fD },|Pr{),,(1),|,(
2σσασσα ==−= ∫
||211 1 1 },{
222
22
1 2 ||
)(21)(
21explog
)(2
1exp
Vz z z Eji
jiVi
ii
Viii
dgdgdgzzgz
fg
V
−−−−×
−−
∑ ∑ ∑ ∑∑
∫ ∫ ∫ ∑
±= ±= ±= ∈∈
∞+
∞−
∞+
∞−
∞+
∞−∈
ασ
σ
It can be reduced to the calculation of the average of free energy with respect to locally non-uniform external fields g1, g2,…,g|V|.
Free Energy of Ising Model with Random External Fields
1±=ifLight intensities of the original image can be regarded as spin states of ferromagnetic system.
== >
== >
Eji ∈},{
−−−−∝
==
∑∑∈∈ Eji
jiVi
ii ffgf
gGfF
},{
222 )(
21)(
21exp
},,|Pr{
ασ
σα
![Page 44: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/44.jpg)
Statistical Performance Estimation for Binary Markov Random Fields
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 44
( ) },|Pr{)(),,(||
1
},|Pr{),,(1),|,(
2
2
σλρσαλ
σσασσα
iiiiij
ijijVi
iiij
iji fFgGghfddgV
gdfFgGfghV
fD
==
−
≅
==−=
∏∑∫ ∏∫
∫
∂∈→→
∈
∞+
∞−∂∈
∞+
∞− →
},|Pr{)(
tanh)tanh(tanh
)(
*
}/{
}/{2
1
}{\
σλρ
λσ
αλδ
λλρ
jjjjijk
jkjk
ijljl
iij
Vi ijkjkiijij
fFgG
g
ddg
==
×
+−×
=
∏
∑
∑∫ ∏ ∫
∂∈→→
∂∈→
−→
∈
∞+
∞−∂∈
∞+
∞− →→→
+≅ ∑
∂∈→
ijij
ii
ggh λ
σσα 2sgn),,(
![Page 45: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/45.jpg)
gdfFgGfghV
fD },|Pr{),,(1),|,(
2σσασσα ==−= ∫
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 45
0
0.1
0.2
0.3
0.4
0 0.2 0.4 0.6 0.8 1
Statistical Performance Estimation for Markov Random Fields
0
200
400
600
0 0.001 0.002 0.003α
σ=40σ=1
α
),|,( σσα fD
Spin Glass Theory in Statistical MechanicsLoopy Belief Propagation
0.8
0.6
0.40.2
),|,( σσα fD
Multi-dimensional Gauss Integral Formulas
![Page 46: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/46.jpg)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 46
Statistical Performance Estimation for Markov Random Fields
gdfFgGghfV
fD },|Pr{),,(1),|,(
2σσασσα ==−= ∫
![Page 47: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/47.jpg)
Outline
1. Introduction2. Bayesian Image Analysis by Gauss Markov
Random Fields3. Statistical Performance Analysis for Gauss
Markov Random Fields4. Statistical Performance Analysis in Binary
Markov Random Fields by Loopy Belief Propagation
5. Concluding Remarks
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 47
![Page 48: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/48.jpg)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 48
Summary
Formulation of probabilistic model for image processing by means of conventional statistical schemes has been summarized.Statistical performance analysis of probabilistic image processing by using Gauss Markov Random Fields has been shown.One of extensions of statistical performance estimation to probabilistic image processing with discrete states has been demonstrated.
![Page 49: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/49.jpg)
Image Impainting by Gauss MRF and LBP
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 49
Gauss MRFand LBP
Our framework can be extended to erase a scribbling.
![Page 50: Statistical performance analysis by loopy belief ...kazu/tutorial-lecture-note/ParisSudLRI... · 14 October, 2010 LRI Seminar 2010 (Univ. Paris -Sud) 1. Statistical performance analysis](https://reader031.fdocuments.us/reader031/viewer/2022022609/5b932ad209d3f206218ccd43/html5/thumbnails/50.jpg)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 50
References1. K. Tanaka and D. M. Titterington: Statistical Trajectory of Approximate EM
Algorithm for Probabilistic Image Processing, Journal of Physics A: Mathematical and Theoretical, vol.40, no.37, pp.11285-11300, 2007.
2. M. Yasuda and K. Tanaka: The Mathematical Structure of the Approximate Linear Response Relation, Journal of Physics A: Mathematical and Theoretical, vol.40, no.33, pp.9993-10007, 2007.
3. K. Tanaka and K. Tsuda: A Quantum-Statistical-Mechanical Extension of Gaussian Mixture Model, Journal of Physics: Conference Series, vol.95, article no.012023, pp.1-9, January 2008
4. K. Tanaka: Mathematical Structures of Loopy Belief Propagation and Cluster Variation Method, Journal of Physics: Conference Series, vol.143, article no.012023, pp.1-18, 2009
5. M. Yasuda and K. Tanaka: Approximate Learning Algorithm in Boltzmann Machines, Neural Computation, vol.21, no.11, pp.3130-3178, 2009.
6. S. Kataoka, M. Yasuda and K. Tanaka: Statistical Performance Analysis in Probabilistic Image Processing, Journal of the Physical Society of Japan, vol.79, no.2, article no.025001, 2010.