Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical...
Transcript of Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical...
![Page 1: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/1.jpg)
Probabilistic Graphical Models and Their Applications
Image Processing
Bernt Schiele
http://www.d2.mpi-inf.mpg.de/gm
![Page 2: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/2.jpg)
Graphical Models and Their Applications - December 9, 2015
Image Processing & Stereo
• Today we shift gears and look at another problem domain: Image processing
• 4 applications of interest ‣ Image denoising.
‣ Image inpainting.
‣ Super-resolution.
‣ Stereo
• Acknowledgement ‣ Majority of Slides (adapted) from Stefan Roth @ TU Darmstadt
2
![Page 3: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/3.jpg)
Graphical Models and Their Applications - December 9, 2015
Image Denoising
• Essentially all digital images exhibit image noise to some degree. ‣ Even high quality images contain some noise.
‣ Really noisy images may look like they are dominated by noise.
• Image noise is both: ‣ Visually disturbing for the human viewer.
‣ Distracting for vision algorithms.
• Image denoising aims at removing or reducing the amount of noise in the image.
3
![Page 4: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/4.jpg)
Graphical Models and Their Applications - December 9, 2015
Image Noise
• Image noise is a very common phenomenon that can come from a variety of sources: ‣ Shot noise or photon noise due to the stochastic nature of the photons arriving at
the sensor. - cameras actually count photons!
‣ Thermal noise (“fake” photon detections).
‣ Processing noise within CMOS or CCD, or in camera electronics.
‣ If we use “analog” film, there are physical particles of a finite size that cause noise in a digital scan.
4
![Page 5: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/5.jpg)
Graphical Models and Their Applications - December 9, 2015
Shot Noise
5
Simulated shot noise (Poisson process) From Wikipedia
![Page 6: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/6.jpg)
Graphical Models and Their Applications - December 9, 2015
Thermal Noise or “Dark Noise”
6
62 minute exposure with no incident light
From Jeff Medkeff (photo.net)
![Page 7: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/7.jpg)
Graphical Models and Their Applications - December 9, 2015
Bias Noise from Amplifiers
7
High ISO exposure (3200)
From Jeff Medkeff (photo.net)
![Page 8: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/8.jpg)
Graphical Models and Their Applications - December 9, 2015
Noise in Real Digital Photographs
• Combination of many different noise sources:
8
From dcresource.com
![Page 9: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/9.jpg)
Graphical Models and Their Applications - December 9, 2015
Film Grain
• If we make a digital scan of a film, we get noise from the small silver particles on the film strip:
9
Das Testament des Dr. Mabuse, 1933
![Page 10: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/10.jpg)
Graphical Models and Their Applications - December 9, 2015
How do we remove image noise?
• Classical techniques: ‣ Linear filtering, e.g. Gauss filtering.
‣ Median filtering
‣ Wiener filtering
‣ Etc.
• Modern techniques: ‣ PDE-based techniques
‣ Wavelet techniques
‣ MRF-based techniques (application of graphical models :)
‣ Etc.
10
![Page 11: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/11.jpg)
Graphical Models and Their Applications - December 9, 2015
• The simplest idea is to use a Gaussian filter:
Original� = 0.5
Linear Filtering
11
� = 1.0� = 1.5
� = 2.0� = 3.0
This removes the noise, but it blurs across edges!
![Page 12: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/12.jpg)
Graphical Models and Their Applications - December 9, 2015
Median Filter
• Replace each pixel by the median of the pixel values in a window around it:
12
Originalw = [3, 3]
w = [5, 5]w = [7, 7]
Sharper edges, but still blurred...
![Page 13: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/13.jpg)
Graphical Models and Their Applications - December 9, 2015
How do we improve on this?
• We need denoising techniques that better preserve boundaries in images and do not blur across them.
• There is a whole host of modern denoising techniques: ‣ We would need a whole semester to go through the important ones in detail.
‣ So we will do the area of denoising some major injustice and restrict ourselves to what we can easily understand with what we have learned so far.
13
![Page 14: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/14.jpg)
Graphical Models and Their Applications - December 9, 2015
• We formulate the problem of image denoising in a Bayesian fashion as a problem of probabilistic inference.
• For that we model denoising using a suitable posterior distribution:
• Idea: ‣ derive a graphical model that models this posterior appropriately
‣ use standard inference techniques (such as sum-product rule or max-product rule) to estimate the true image that we want to recover
Denoising as Probabilistic Inference
14
p(true image|noisy image) = p(T|N)
![Page 15: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/15.jpg)
Graphical Models and Their Applications - December 9, 2015
• For this, we can apply Bayes’ rule and obtain:
normalization term (constant)posterior
image prior for all true imageslikelihood of noisy given true image (observation model)
p(T|N) =p(N|T) · p(T)
p(N)
Modeling the Posterior
15
![Page 16: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/16.jpg)
Graphical Models and Their Applications - December 9, 2015
Modeling the Likelihood
• The likelihood expresses a model of the observation: ‣ Given the true, noise free image T, we assess how likely it is to observe a
particular noisy image N.
‣ If we wanted to model particular real noise phenomena, we could model the likelihood based on real physical properties of the world.
‣ Here, we will simplify things and only use a simple, generic noise model.
‣ Nevertheless, our formulation allows us to easily adapt the noise model without having to change everything...
16
p(N|T)
![Page 17: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/17.jpg)
Graphical Models and Their Applications - December 9, 2015
Modeling the Likelihood
• Simplification: assume that the noise at one pixel is independent of the others.
‣ often reasonable assumption, for example since sites of a CCD sensor are relatively independent.
• Then we will assume that the noise at each pixel is additive and Gaussian distributed:
‣ The variance controls the amount of noise.
17
�2
p(N|T)
p(N|T) =Y
i,j
p(Ni,j |Ti,j)
p(Ni,j |Ti,j) = N (Ni,j � Ti,j |0,�2)
![Page 18: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/18.jpg)
Graphical Models and Their Applications - December 9, 2015
Gaussian Image Likelihood
• We can thus write the Gaussian image likelihood as:
• While this may seem a bit hacky, it works well in many applications. ‣ It is suboptimal when the noise is not really independent,
such as in some high definition (HD) images.
‣ It also is suboptimal when the noise is non-additive, or not really Gaussian, for example as with film grain noise.
18
p(N|T)
p(N|T) =Y
i,j
N (Ni,j � Ti,j |0,�2)
![Page 19: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/19.jpg)
Graphical Models and Their Applications - December 9, 2015
Modeling the Prior
• How do we model the prior distribution of true images? • What does that even mean? ‣ We want the prior to describe how probable it is (a-priori) to have a particular true
image among the set of all possible images.
19
probableimprobable
p(T)
![Page 20: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/20.jpg)
Graphical Models and Their Applications - December 9, 2015
Natural Images
• What distinguishes “natural” images from “fake” ones? ‣ We can take a large database of natural images and study them.
20
![Page 21: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/21.jpg)
Graphical Models and Their Applications - December 9, 2015
Simple Observation
• Nearby pixels often have similar intensity:
• But sometimes there are large intensity changes.
21
![Page 22: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/22.jpg)
Graphical Models and Their Applications - December 9, 2015
Statistics of Natural Images
• Compute the image derivative of all images in an image database and plot a histogram:
22
−200 −100 0 100 200
10−6
10−4
10−2
100
−200 −100 0 100 200
10−6
10−4
10−2
100
x-derivative y-derivative
empirical histogram fit with a Gaussian
![Page 23: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/23.jpg)
Graphical Models and Their Applications - December 9, 2015
Statistics of Natural Images
• Compute the image derivative of all images in an image database and plot a histogram:
23
−200 −100 0 100 200
10−6
10−4
10−2
100
−200 −100 0 100 200
10−6
10−4
10−2
100
x-derivative y-derivative
empirical histogram fit with a Gaussian
• Sharp peak at zero: Neighboring pixels most often have identical intensities.
• Heavy tails: Sometimes, there are strong intensity differences due to discontinuities in the image.
![Page 24: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/24.jpg)
Graphical Models and Their Applications - December 9, 2015
Modeling the prior
• The prior models our a-prior assumption about images ‣ here we want to model the statistics of natural images
‣ more specifically the local neighborhood statistics of each pixel: - nearby pixels have often similar intensity
- but in the presence of boundaries the intensity difference can be large
‣ let’s formulate this as a graphical model...
24
p(T)
![Page 25: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/25.jpg)
Graphical Models and Their Applications - December 9, 2015
Modeling Compatibilities
• Pixel grid:
25
Let’s assume that we want to model how compatible
or consistent a pixel is with its 4 nearest neighbors.
![Page 26: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/26.jpg)
Graphical Models and Their Applications - December 9, 2015
Modeling Compatibilities
• Pixel grid (as nodes of a graph):
26
Denote this by drawing a line (edge) between
two pixels (nodes).
![Page 27: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/27.jpg)
Graphical Models and Their Applications - December 9, 2015
Modeling Compatibilities
• Pixel grid (as nodes of a graph):
27
We do this for all pixels.
![Page 28: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/28.jpg)
Graphical Models and Their Applications - December 9, 2015
product over all the pixels
compatibility of vertical neighbors
compatibility of horizontal neighbors
Markov Random Fields
• This is an undirected graphical model, or more specifically a Markov random field. ‣ Each edge (in this particular graph) corresponds
to a term in the (image) prior that models how compatible two neighboring pixels are in terms of their intensities:
28
p(T) =�
i,j
fH(Ti,j , Ti+1,j) · fV (Ti,j , Ti,j+1)
![Page 29: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/29.jpg)
Graphical Models and Their Applications - December 9, 2015
Modeling the Potentials
• What remains is to model the potentials (or compatibility functions), e.g.:
• Gaussian distributions are inappropriate: ‣ They do not match the statistics of natural images well.
‣ They would lead to blurred discontinuities.
• We need discontinuity-preserving potentials: ‣ One possibility: Student-t distribution.
29
fH(Ti,j , Ti+1,j)
−300 −200 −100 0 100 200 300−10
−9
−8
−7
−6
−5
−4
−3
−2
−1
log-density
fH(Ti,j , Ti+1,j) =�
1 +1
2�2(Ti,j � Ti+1,j)2
⇥��
![Page 30: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/30.jpg)
Graphical Models and Their Applications - December 9, 2015
MRF Model of the (complete) Posterior• We can now put the likelihood and the prior together in a single
MRF model:
30
pixels of the true image (hidden)
pixels of the noisy image (observed)
Ti,j
Ni,j
Edges representing the likelihood
Edges representing the prior
p(T|N) � p(N|T)p(T)
=
0
@Y
i,j
p(Ni,j |Ti,j)
1
A
0
@Y
i,j
fH(Ti,j , Ti+1,j) · fV (Ti,j , Ti,j+1)
1
A
![Page 31: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/31.jpg)
Graphical Models and Their Applications - December 9, 2015
Denoising as Probabilistic Inference
• Probabilistic inference generally means one of three things ‣ computing the maximum of the posterior distribution - here
that is computing the state that is the most probable given our observations (maximum a-posteriori (MAP) estimation)
‣ computing expectations over the posterior distribution, such as the mean of the posterior
‣ computing marginal distributions
• Visualization of the difference between those cases: ‣ assume we have the following posterior distribution:
in particular - the posterior may be multi-modal
31
p(T|N)
−4 −2 0 2 40
0.05
0.1
0.15
Maximum (MAP estimate)
Mean
![Page 32: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/32.jpg)
Graphical Models and Their Applications - December 9, 2015
Probabilistic Inference
• Methods that can be used for MAP estimation ‣ continuous optimization methods
‣ graph-based methods (graph cuts)
‣ belief propagation: in particular max-product algorithm - however: we have a graph with cycles (=loopy) !
- no convergence / correctness guarantees !
- in practice “loopy belief propagation” obtains good results
• Method that can be used for expectations and marginal distributions ‣ belief propagation: in particular sum-product algorithm
- same notes as above - we have a cyclic graph - loopy belief propagation !
32
![Page 33: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/33.jpg)
Graphical Models and Their Applications - December 9, 2015
Denoising as Inference - Continuous Optimization
• The most straightforward idea for maximizing the posterior is to apply well-known continuous optimization techniques.
• Especially gradient techniques have found widespread use, e.g.: ‣ Simple gradient ascent, also called hill-climbing.
‣ Conjugate gradient methods.
‣ And many more.
• Since the posterior may be multi-modal, this will give us a local optimum and not necessarily the global optimum.
33
![Page 34: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/34.jpg)
Graphical Models and Their Applications - December 9, 2015
• Iteratively maximize a function : ‣ Initialize somewhere:
‣ Compute the derivative:
‣ Take a step in the directionof the derivative:
‣ Repeat...
step size
Gradient Ascent
34
−4 −2 0 2 4−0.05
0
0.05
0.1
0.15
f(x)x(0)
ddxf(x) = f �(x)
x(1) ⇥ x(0) + � · f �(x(0))
x(n+1) ⇥ x(n) + � · f �(x(n))
![Page 35: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/35.jpg)
Graphical Models and Their Applications - December 9, 2015
• We can do the same in multiple dimensions:
• Issues: ‣ How to initialize?
- bad initialization with result in “wrong” local optimum
‣ How to choose thestep size ? - the wrong one can lead to
instabilities or slow convergence.
gradient
Gradient Ascent
35
−4 −2 0 2 4−0.05
0
0.05
0.1
0.15
x(n+1) ⇥ x(n) + � ·⇤f(x(n))
�
![Page 36: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/36.jpg)
Graphical Models and Their Applications - December 9, 2015
• We want to maximize the posterior:
• equivalently we can maximize the log-posterior: ‣ numerically much more stable and often more convenient
• for gradient ascent we need the partial derivatives w.r.t. to a particular pixel
• in the following we look at the two derivatives separately ‣ first - the image prior and
‣ second - the likelihood.
Image Denoising with Continuous Optimization
36
p(T|N) � p(N|T)p(T)
�
�Tk,l{log p(T|N)} =
�
�Tk,l{log p(N|T)}+ �
�Tk,l{log p(T)}
log p(T|N) = log p(N|T) + log p(T) + const.
Tk,l
![Page 37: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/37.jpg)
Graphical Models and Their Applications - December 9, 2015
Image Denoising with Continuous Optimization
• Let us first look at the log-prior:
37
log p(T) = log
2
4 1
Z
Y
i,j
fH(Ti,j , Ti+1,j) · fV (Ti,j , Ti,j+1)
3
5
=
X
i,j
log fH(Ti,j , Ti+1,j) + log fV (Ti,j , Ti,j+1) + const
![Page 38: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/38.jpg)
Graphical Models and Their Applications - December 9, 2015
Gradient of the Log-Prior
• Calculate the partial derivative w.r.t. a particular pixel :
• Only the 4 terms from the 4 neighbors remain:
38
�
�Tk,llog p(T) =
�
�Tk,l
X
i,j
log fH(Ti,j , Ti+1,j) + log fV (Ti,j , Ti,j+1) + const
=
X
i,j
�
�Tk,llog fH(Ti,j , Ti+1,j) +
�
�Tk,llog fV (Ti,j , Ti,j+1)
�
�Tk,llog p(T) =
�
�Tk,llog fH(Tk,l, Tk+1,l) +
�
�Tk,llog fH(Tk�1,l, Tk,l)+
+
�
�Tk,llog fV (Tk,l, Tk,l+1) +
�
�Tk,llog fV (Tk,l�1, Tk,l)
Tk,l
![Page 39: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/39.jpg)
Graphical Models and Their Applications - December 9, 2015
Gradient of the Log-Prior
• Almost there... simply apply the chain rule:
• last thing: calculate derivative of the compatibility function (or potential function)
39
⇥
⇥Tk,lfH(Tk,l, Tk+1,l) =
⇥
⇥Tk,l
✓1 +
1
2�2(Tk,l � Tk+1,j)
2
◆��
�
�Tk,llog fH(Tk,l, Tk+1,l) =
��Tk,l
fH(Tk,l, Tk+1,l)
fH(Tk,l, Tk+1,l)
![Page 40: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/40.jpg)
Graphical Models and Their Applications - December 9, 2015
Gradient of the Log-Likelihood
• Let us now look at the log-likelihood
• the partial derivative of the log-likelihood is thus simply:
40
⇥
⇥Tk,llog p(N|T) =
⇥
⇥Tk,llog
�N (Nk,l � Tk,l|0,�2
)
log p(N|T) = log
2
4Y
i,j
p(Ni,j |Ti,j)
3
5
=
X
i,j
log p(Ni,j |Ti,j)
=
X
i,j
logN (Ni,j � Ti,j |0,�2)
![Page 41: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/41.jpg)
Graphical Models and Their Applications - December 9, 2015
Probabilistic Inference
• Methods that can be used for MAP estimation ‣ continuous optimization methods
‣ graph-based methods (graph cuts)
‣ belief propagation: in particular max-product algorithm - however: we have a graph with cycles (=loopy) !
- no convergence / correctness guarantee !
- in practice “loopy belief propagation” obtains good results
• Method that can be used for expectations and marginal distributions ‣ belief propagation: in particular sum-product algorithm
- same notes as above - we have a cyclic graph - loopy belief propagation !
41
![Page 42: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/42.jpg)
Graphical Models and Their Applications - December 9, 2015
Loopy Belief Propagation
• Empirical Observation: [Murphy,Weiss,Jordan@uai’99] ‣ even for graphs with cycles: simply apply belief propagation (BP)...
‣ observation: BP often gives good results even for graphs with cycles (if it converges)
‣ issues - may not converge !
- cycling error - old information is mistaken as new - convergence error - unlike in a tree, neighbors need not be independent.
Loopy BP treats them as if they were
‣ not really well understood under which conditions BP works well for cyclic graphs...
42
![Page 43: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/43.jpg)
Graphical Models and Their Applications - December 9, 2015
Loopy Belief Propagation
• Loopy BP for Image Denoising: [Lan,Roth,Huttenlocher,Black@eccv’06] ‣ different update schemes: synchronous, random, ...
‣ synchronous message updates: all messages are updated simultaneously
‣ checkerboard-like update: alternate updates between neighbors
‣ best results (image denoising) with random updates: at each step, messages are updated with fixed probability
• Some Results from the above paper:
43
originalimage
noisyimage
result from image denoising with loopy BP with different potentials (left: Student t-distribution)
![Page 44: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/44.jpg)
Graphical Models and Their Applications - December 9, 2015
Denoising Results
44
original image noisy image,σ=20
PSNR 22.49dB SSIM 0.528
denoised usinggradient ascent
PSNR 27.60dB SSIM 0.810
![Page 45: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/45.jpg)
Graphical Models and Their Applications - December 9, 2015
Denoising Results
45
original image noisy image,σ=20
PSNR 22.49dB SSIM 0.528
denoised usinggradient ascent
PSNR 27.60dB SSIM 0.810
• Very sharp discontinuities. No blurring across boundaries.
• Noise is removed quite well nonetheless.
![Page 46: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/46.jpg)
Graphical Models and Their Applications - December 9, 2015
Denoising Results
46
original image noisy image,σ=20
PSNR 22.49dB SSIM 0.528
denoised usinggradient ascent
PSNR 27.60dB SSIM 0.810
• Because the noisy image is based on synthetic noise, we can measure the performance:
• PSNR: Peak signal-to-noise ratio • SSIM [Wang et al., 04]: Perceptual similarity
• Gives an estimate of how humans would assess the quality of the image.
PSNR = 20 log10
✓MAXIpMSE
◆
![Page 47: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/47.jpg)
Graphical Models and Their Applications - December 9, 2015
Is this the end of the story?
• No, natural images have many complex properties that we have not modeled. ‣ For example, they have complex structural properties that are not modeled with a
simple MRF based on a 4-connected grid.
‣ Natural images have scale-invariant statistics, our model does not.
‣ Responses to random linear filters are heavy-tailed.
‣ Etc.
47
−200 −100 0 100 200
10−4
10−2
100
−200 −100 0 100 200
10−6
10−4
10−2
100
Derivative histogram on 4 spatial scales
Histograms of random (zero-mean) linear filters
![Page 48: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/48.jpg)
Graphical Models and Their Applications - December 9, 2015
Image Processing & Stereo
• Today we shift gears and look at another problem domain: Image processing
• 4 applications of interest ‣ Image denoising.
‣ Image inpainting.
‣ Super-resolution.
‣ Stereo
• Acknowledgement ‣ Majority of Slides (adapted) from Stefan Roth @ TU Darmstadt
48
![Page 49: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/49.jpg)
Graphical Models and Their Applications - December 9, 2015
Image Inpainting
• In image inpainting the goal in to fill in a “missing” part of an image: ‣ Restoration of old photographs, e.g. scratch removal...
49
user-supplied maskold photograph
[Bertalmio et al., 2000]
![Page 50: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/50.jpg)
Graphical Models and Their Applications - December 9, 2015
Image Inpainting
• There are many different ways to do inpainting: ‣ PDEs: [Bertalmio et al, 2000], ...
‣ Exemplar-based: [Criminisi et al., 2003], ...
‣ And many more.
• But, we can also apply what we already know: ‣ We model the problem in a Bayesian fashion, where we regard the inpainted
image as the true image. We are given the corrupted image with missing pixels.
‣ Then we apply probabilistic inference...
50
p(true image|corrupted image) = p(T|C)
![Page 51: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/51.jpg)
Graphical Models and Their Applications - December 9, 2015
Image Inpainting
• We apply Bayes’ rule:
‣ I know this may be boring, but this general approach really is this versatile...
• Modeling the prior: ‣ Important observation: This is the very same prior that we use for denoising!
- We can re-use the prior model from denoising here.
‣ Once we have a good probabilistic model of images, we can use it in many applications !
51
p(T|C) =p(C|T) · p(T)
p(C)
![Page 52: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/52.jpg)
Graphical Models and Their Applications - December 9, 2015
Inpainting Likelihood
• Again, we assume independence of the pixels:
• Desiderata: ‣ We want to keep all known pixels fixed.
‣ For all unknown pixels all intensities should be equally likely.
• Simple likelihood model:
52
p(C|T) =�
i,j
p(Cij |Tij)
p(Cij |Tij) =�
const, Cij is corrupted�(Tij � Cij), Cij is not corrupted
![Page 53: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/53.jpg)
Graphical Models and Their Applications - December 9, 2015
MRF Model of the Posterior
• The posterior for inpainting has the same graphical model structure as the one for denoising.
• Nonetheless, the potentials representing the likelihood are different.
53
![Page 54: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/54.jpg)
Graphical Models and Their Applications - December 9, 2015
Inpainting Results
54
“Corrupted” image(artificially corrupted for
benchmarking)
Inpainted image obtainedusing gradient ascent
![Page 55: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/55.jpg)
Graphical Models and Their Applications - December 9, 2015
Other Inpainting Results
55
From [Bertalmio et al., 2000]
![Page 56: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/56.jpg)
Graphical Models and Their Applications - December 9, 2015
Image Super-Resolution
• In super-resolution, the goal is to increase the resolution of an image, while making the high-resolution image seem sharp and natural.
• Many applications: ‣ Resizing of old, low-res digital imagery.
‣ Improving resolution of images from cheap cameras.
‣ Up-conversion of standard definition TV and movie footage to high-definition (720p or 1080p).
56
64x64 original image 128x128 super-resolved image
(bicubic interpolation)
From [Tappen et al., 2003]
![Page 57: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/57.jpg)
Graphical Models and Their Applications - December 9, 2015
Image Super-Resolution
• The story parallels what we have seen before. • There are many special purpose super-resolution techniques: ‣ The simplest ones are bilinear and bicubic interpolation.
‣ Example-based methods.
‣ Etc.
• But once again, we can formulate the problem in a Bayesian way and super-resolve images using probabilistic inference:
57
p(high res. image|low res. image) = p(H|L)
![Page 58: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/58.jpg)
Graphical Models and Their Applications - December 9, 2015
Image Super-Resolution
• Yet again apply Bayes rule:
‣ this should be getting really boring by now...
• Modeling the prior of high-resolution images: ‣ again we can use the prior model that we already have!
• Modeling the likelihood: ‣ The likelihood has to model how the low-resolution pixels correspond to the high-
resolution pixels.
‣ For simplicity, assume a fixed zooming factor of 2x.
58
p(H|L) =p(L|H) · p(H)
p(L)
![Page 59: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/59.jpg)
Graphical Models and Their Applications - December 9, 2015
Super-Resolution Likelihood
• If we have a zoom factor of 2x, there is a 2x2 patch of high-resolution pixels that corresponds to a single low-resolution pixel.
• Again assume conditional independence of the low-resolution pixels given the high-resolution image.
• We can thus write the likelihood as:
• We may for example assume that a low-resolution pixel is the average of the 4 high-resolution pixels plus a small amount of Gaussian noise:
59
p(L|H) =�
i,j
p(Li,j |H2i,2j ,H2i+1,2j ,H2i,2j+1,H2i+1,2j+1)
p(Li,j |H2i,2j ,H2i+1,2j ,H2i,2j+1,H2i+1,2j+1) = N�
Li,j �14(H2i,2j + H2i+1,2j + H2i,2j+1 + H2i+1,2j+1); 0,�2
⇥
![Page 60: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/60.jpg)
Graphical Models and Their Applications - December 9, 2015
Super-resolution Results
• Gradient ascent does not work well here, because the likelihood is more complex than for denoising or inpainting.
• But belief propagation can be made to work (with some tricks...):
60
From [Tappen et al., 2003]
original
bicubic interpolationMRF + BP
![Page 61: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/61.jpg)
Graphical Models and Their Applications - December 9, 2015
Super-resolution Results
• Gradient ascent does not work very well here, because the likelihood is more complex than in denoising or inpainting.
• But belief propagation can be made to work (with some tricks...):
61
From [Tappen et al., 2003]
original
bicubic interpolationMRF + BP
![Page 62: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/62.jpg)
Graphical Models and Their Applications - December 9, 2015
Super-resolution Results
• Gradient ascent does not work very well here, because the likelihood is more complex than in denoising or inpainting.
• But belief propagation can be made to work (with some tricks...):
62
From [Tappen et al., 2003]
original
bicubic interpolationMRF + BP
![Page 63: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/63.jpg)
Graphical Models and Their Applications - December 9, 2015
Summary
• Many image processing problems can be formulated as problems of probabilistic inference. ‣ This is only one of many different ways of approaching these problems!
• Advantages: ‣ Unified approach to many different problems, in which important components
(prior) may be re-used.
‣ Is is relatively easy to understand what the various parts do.
‣ Good application performance, despite generality.
• Disadvantages: ‣ Computationally often expensive.
‣ Special purpose techniques often have somewhat better application performance.
63
![Page 64: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/64.jpg)
Graphical Models and Their Applications - December 9, 2015
Image Processing & Stereo
• Last week we started to shift gears and look at another problem domain: Image processing
• 4 applications of interest ‣ Image denoising.
‣ Image inpainting.
‣ Super-resolution.
‣ Stereo
• Acknowledgement ‣ Majority of Slides (adapted) from Stefan Roth @ TU Darmstadt
64
![Page 65: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/65.jpg)
Graphical Models and Their Applications - December 9, 2015
What is Stereo (Vision)?
• Stereo vision, stereopsis, or short stereo is the perception or measurement of depth from two projections.
‣ The human visual system heavily relies on stereo vision: - Our eyes give two slightly shifted projections of the scene.
- But only relative depth can be judged accurately by humans.
‣ Was first described in technical detail by Charles Wheatstone in the 1830s with the invention of the stereoscope.
65
![Page 66: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/66.jpg)
Graphical Models and Their Applications - December 10, 2014 66
[Black]
![Page 67: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/67.jpg)
Graphical Models and Their Applications - December 10, 2014 67
[Black]
![Page 68: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/68.jpg)
Graphical Models and Their Applications - December 10, 2014 68
[Black]
![Page 69: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/69.jpg)
Graphical Models and Their Applications - December 10, 2014
Left image
69
[Black]
![Page 70: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/70.jpg)
Graphical Models and Their Applications - December 10, 2014
Right image
70
[Black]
![Page 71: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/71.jpg)
Graphical Models and Their Applications - December 9, 2015
Binocular Stereo
Left
71
image plane
camera
![Page 72: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/72.jpg)
Graphical Models and Their Applications - December 9, 2015
Right
Binocular Stereo
Left
72
![Page 73: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/73.jpg)
Graphical Models and Their Applications - December 9, 2015
Right
Binocular Stereo
Left
73
![Page 74: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/74.jpg)
Graphical Models and Their Applications - December 9, 2015
Right
Binocular Stereo
Left
74
![Page 75: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/75.jpg)
Graphical Models and Their Applications - December 9, 2015
Right
Binocular Stereo
Left
75
binocular disparity
From known geometry of the cameras and estimated disparity, recover depth in
the scene
![Page 76: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/76.jpg)
Graphical Models and Their Applications - December 9, 2015
left camera
right camera
Stereo Geometry
76
Note: We do not assume a virtual image here, but use the standard pinhole camera setup.
![Page 77: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/77.jpg)
Graphical Models and Their Applications - December 9, 2015
left camera
right camera
Disparity = difference in image position
Stereo Geometry
77
d
d
![Page 78: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/78.jpg)
Graphical Models and Their Applications - December 9, 2015
left camera
right camera
Stereo Geometry
78
d
Disparity = difference in image position
d
![Page 79: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/79.jpg)
Graphical Models and Their Applications - December 9, 2015
left camera
right camera
Stereo Geometry
79
d
f
Z
B
B: Baseline between cameras
f: focal length of cameras
![Page 80: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/80.jpg)
Graphical Models and Their Applications - December 9, 2015
left camera
right camera
Stereo Geometry
80
d
f
Z
B
d
f=
B
Z
Disparity d = fB1Z
B: Baseline between cameras
f: focal length of cameras
![Page 81: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/81.jpg)
Graphical Models and Their Applications - December 9, 2015
Binocular Disparity
Left Right
is depth at pixel is disparity
Estimate:
Search for best match
81
(x, y)d(x, y)
Z(x, y)
Z(x, y) =fB
d(x, y)
[Black]
![Page 82: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/82.jpg)
Graphical Models and Their Applications - December 9, 2015
Binocular Disparity
Left Right
is depth at pixel is disparity
Estimate:
82
(x, y)d(x, y)
Do I need to consider this region?
Z(x, y)
Z(x, y) =fB
d(x, y)
[Black]
![Page 83: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/83.jpg)
Graphical Models and Their Applications - December 9, 2015
Epipolar Geometry
83
O1
point in the world
Camera 1
P
![Page 84: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/84.jpg)
Graphical Models and Their Applications - December 9, 2015
Epipolar Geometry
84
O1
p1
P
pi : projection of point P in camera i
![Page 85: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/85.jpg)
Graphical Models and Their Applications - December 9, 2015
Epipolar Geometry
85
O2O1
p1 p2
P
pi : projection of point P in camera i
![Page 86: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/86.jpg)
Graphical Models and Their Applications - December 9, 2015
p2
Epipolar Geometry
O2O1
p1
baseline
epipoles
e2e1
• Epipole: Image location of the optical center of the other camera. ‣ Can be outside of the visible area.
86
P
![Page 87: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/87.jpg)
Graphical Models and Their Applications - December 9, 2015
p2
Epipolar Geometry
• Epipolar plane: Plane through both camera centers and world point.
O2O1
p1
e2e1
87
epipolar lines
epipolar planeP
![Page 88: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/88.jpg)
Graphical Models and Their Applications - December 9, 2015
p2
Epipolar Geometry
• Epipolar line: Constrains the location where a particular feature from one view can be found in the other. ‣ Here: Possible matches for
O2O1
p1
e2e1
88
epipolar lines
epipolar planeP2
P1
P3P4
p1
P
![Page 89: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/89.jpg)
Graphical Models and Their Applications - December 9, 2015
p2
Epipolar Geometry
• Epipolar lines: ‣ Intersect at the epipoles.
‣ In general not parallel.
O2O1
p1
e2e1
89
q1q2
QP
![Page 90: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/90.jpg)
Graphical Models and Their Applications - December 9, 2015
Special case: Binocular setup
• Special case for stereo cameras with a standard binocular setup: ‣ Epipoles are at infinity.
‣ Epipolar lines are parallel.
‣ Points correspond along the scanlines of the image.
90
p1 p2
e1e2at infinity at infinity
camera 1 camera 2
![Page 91: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/91.jpg)
Graphical Models and Their Applications - December 9, 2015
Special case: Binocular setup
• We can answer this now: ‣ No, we do not have to consider this or similar regions.
91
Left Right
Do I need to consider this region?
![Page 92: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/92.jpg)
Graphical Models and Their Applications - December 9, 2015
Rectification
[Szeliski and Fleet]
• What if our cameras do not have a standard binocular setup?
‣ Rectification aligns epipolar lines with scanlines.
‣ Can be obtained by warping the images.
92
![Page 93: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/93.jpg)
Graphical Models and Their Applications - December 9, 2015
Rectification
93
[Szeliski and Fleet]
![Page 94: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/94.jpg)
Graphical Models and Their Applications - December 9, 2015
Matching
• Matching only has to occur along epipolar lines. • Now in the simpler binocular case where the cameras are pointing
forward.
94
[Szeliski and Fleet]
![Page 95: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/95.jpg)
Graphical Models and Their Applications - December 9, 2015
Assumption for the Rest of the Lecture: Binocular Stereo
95
RightRightLeft Right
binocular disparity
From known geometry of the cameras and estimated disparity, recover depth in
the scene
![Page 96: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/96.jpg)
Graphical Models and Their Applications - December 9, 2015
Correspondence Using Correlation
correlation
disparity
Left Right
scanline
96
[Black]
![Page 97: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/97.jpg)
Graphical Models and Their Applications - December 9, 2015
(Normalized) Correlation
• Image patch and cross correlation ‣ Invariant – only when computed on normalized patches
‣ SSD: Sum of Squared Distances:
97
If the patches are normalized then
DSSD(f1, f2) = ||f1 � f2|| =X
n
(f1(n)� f2(n))2
=X
n
�f21 (n)� 2f1(n)f2(n) + f2
2 (n)�=
X
n
f21 (n) +
X
n
f22 (n)� 2
X
n
f1(n)f2(n)
X
n
f21 (n) =
X
n
f22 (n) = 1
CrossCorrelation (f1, f2) =X
n
f1(n)f2(n)
![Page 98: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/98.jpg)
Graphical Models and Their Applications - December 9, 2015
Template or filter as a vector
Image patch as a vector
Angle between the
vectors
Correlation
98
⌅a ·⌅b = |⌅a||⌅b| cos �
Correlation (sum of product
of “signals”)
Normalized correlation:
⌅a ·⌅b|⌅a||⌅b|
= cos �
![Page 99: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/99.jpg)
Graphical Models and Their Applications - December 9, 2015
Window-based Correlation
• We can identify the following problems: ‣ If the window is small, window-based matching often matches to an incorrect
location, because the patch is not descriptive enough.
‣ Large windows improve matching, but lead to other artifacts, because the disparity is not constant within the window, etc.
[Seitz]
99
m = 3 m = 20
![Page 100: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/100.jpg)
Graphical Models and Their Applications - December 9, 2015
Ordering Constraint
• Assume that the order of features on the epipolar line is the inverse of that in 3D:
100
da
d’c’
O’O’O
A
B
C
ab
c a’
c’
A
B
C
D
O
b’ b b’
From [Ponce & Forsyth]
Ordering constraint works fine Failure!
![Page 101: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/101.jpg)
Graphical Models and Their Applications - December 9, 2015
Cooperative Stereo
• First proposed by Marr & Poggio in 1976: ‣ Assume uniqueness of matches:
- No pixel or feature can be matched twice.
‣ Assume ordering constraint.
• Why do we want to still use theseconstraints? ‣ They help resolve ambiguities in texture-
less areas.
‣ Encourage spatial continuity.
‣ Improve the results by removing spurious depth discontinuities.
101
correlation-based
matching
ground-truth
![Page 102: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/102.jpg)
Graphical Models and Their Applications - December 9, 2015
Search Over Correspondences
Left scanline
Right scanline
Occluded Pixels
Disoccluded Pixels
• Three cases: ‣ Sequential – cost of match
‣ Occluded – cost of no match
‣ Disoccluded – cost of no match
102
![Page 103: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/103.jpg)
Graphical Models and Their Applications - December 9, 2015
Stereo matching with Dynamic Programming
• Results:
• Much better results already! • But no consistency between scanlines!
103
![Page 104: Probabilistic Graphical Models and Their Applications Image … · 2015. 12. 10. · Graphical Models and Their Applications - December 9, 2015 • We formulate the problem of image](https://reader035.fdocuments.us/reader035/viewer/2022071404/60f8bbd6f4785f08030111a9/html5/thumbnails/104.jpg)
Graphical Models and Their Applications - December 9, 2015
Today
• How can we impose regularity constraints that impose global consistency? ‣ This is also called regularization.
• Goals: ‣ We want consistency within and between scanlines.
‣ We want a model of consistency that is well supported by the properties of the real world, i.e. by real scene depth.
‣ We want a model that is computationally manageable.
‣ We would like to find a model of consistency that does not only work for stereo, but also for other applications.
• Approach here: ‣ Markov random fields
104