C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor...
Transcript of C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor...
![Page 1: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/1.jpg)
C280 Computer VisionC280, Computer Vision
Prof. Trevor [email protected]
Lecture 20: Markov Random Fields
![Page 2: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/2.jpg)
Slide CreditsSlide Credits
• Bill FreemanBill Freeman
• Yuri Boykov
h d• others, as noted…
2
![Page 3: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/3.jpg)
Making probability distributions modular, and h f bltherefore tractable:
Probabilistic graphical models
Vision is a problem involving the interactions of many variables: things can seem hopelessly complex. Everything is made tractable, or at least, simpler, if we modularize the problem. That’s what probabilistic graphical models do, and let’s examineThat s what probabilistic graphical models do, and let s examine that.
Readings: Jordan and Weiss intro article fantastic!Readings: Jordan and Weiss intro article—fantastic!Kevin Murphy web page—comprehensive and with
pointers to many advanced
3
topics
![Page 4: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/4.jpg)
A toy exampleA toy example
Suppose we have a system of 5 interacting variables perhaps some areSuppose we have a system of 5 interacting variables, perhaps some are observed and some are not. There’s some probabilistic relationship between the 5 variables, described by their joint probability,P(x1 x2 x3 x4 x5)P(x1, x2, x3, x4, x5).
If we want to find out what the likely state of variable x1 is (say, the position of the hand of some person we are observing), what can we do?p g),
Two reasonable choices are: (a) find the value of x1 (and of all the other variables) that gives the maximum of P(x1 x2 x3 x4 x5); that’s the MAPvariables) that gives the maximum of P(x1, x2, x3, x4, x5); that s the MAP solution.Or (b) marginalize over all the other variables and then take the mean or the maximum of the other variables. Marginalizing, then taking the mean, is
4
g g, g ,equivalent to finding the MMSE solution. Marginalizing, then taking the max, is called the max marginal solution and sometimes a useful thing to do.
![Page 5: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/5.jpg)
To find the marginal probability at x1 we have to take this sum:To find the marginal probability at x1, we have to take this sum:),,,,(
5432 ,,,54321
xxxxxxxxxP
If the system really is high dimensional, that will quickly become intractable. But if there is some modularity in ),,,,( 54321 xxxxxPthen things become tractable again.
Suppose the variables form a Markov chain: x1 causes x2 which causes x3, etc We might draw out this relationship as follows:etc. We might draw out this relationship as follows:
1x 2x 3x 4x 5x
5
1 2 3 4 5
![Page 6: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/6.jpg)
B th h i l f b bilit di t ib ti h
P(a,b) = P(b|a) P(a)
)|,,,()(),,,,( 15432154321 xxxxxPxPxxxxxP
By the chain rule, for any probability distribution, we have:
),|,,()|()( 21543121 xxxxxPxxPxP
),,|,(),|()|()( 32154213121 xxxxxPxxxPxxPxP
),,,|(),,|(),|()|()( 432153214213121 xxxxxPxxxxPxxxPxxPxP
But if we exploit the assumed modularity of the probability distribution over
)|()|()|()|()( 453423121 xxPxxPxxPxxPxP
But if we exploit the assumed modularity of the probability distribution over the 5 variables (in this case, the assumed Markov chain structure), then that expression simplifies:
Now our marginalization summations distribute through those terms:1x 2x 3x 4x 5x
6
1 2 3 4 55432
)|()|()|()|()(),,,,( 453423121,,,
54321x x x x xxxxx
xxPxxPxxPxxPxPxxxxxP
![Page 7: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/7.jpg)
Directed graphical modelsg p
• A directed, acyclic graph.• Nodes are random variables. Can be scalars or vectors, continuous or
discrete.• The direction of the edge tells the parent-child-relation:
• With every node i is associated a conditional pdf defined by all the
parent child
With every node i is associated a conditional pdf defined by all the parent nodes of node i. That conditional probability is
• The joint distribution depicted by the graph is the product of all those
i Px i | x i
• The joint distribution depicted by the graph is the product of all those conditional probabilities:
P Pn
Px1 ... x n Px i | x i
i1
![Page 8: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/8.jpg)
Another modular probabilistic structure, more common in vision problems, is an undirected graph:
1x 2x 3x 4x 5x
Th j i b bili f hi h i i bThe joint probability for this graph is given by:
),(),(),(),(),,,,( 5443322154321 xxxxxxxxxxxxxP
Where is called a “compatibility function”. We can define compatibility functions we result in the same joint probability as
),( 21 xx
8
define compatibility functions we result in the same joint probability as for the directed graph described in the previous slides; for that example, we could use either form.
![Page 9: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/9.jpg)
Undirected graphical modelsg p
• A set of nodes joined by undirected edges.• The graph makes conditional independencies explicit: If two nodes are
not linked, and we condition on every other node in the graph, then those two nodes are conditionally independent.
Conditionally independent, because are not connected by a line in the undirected graphical model
9
![Page 10: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/10.jpg)
Undirected graphical models: cliquesg p q
• Clique: a fully connected set of nodesnot a clique
clique
• A maximal clique is a clique that can’t include more nodes of the graph w/o losing the clique property.
Maximal cliqueNon-maximal clique
10
![Page 11: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/11.jpg)
Undirected graphical models: probability factorizationprobability factorization
• Hammersley-Clifford theorem addresses the pdf factorization implied by a graph: A distribution has the Markov structure implied by an undirected graph iff it can be represented in the factored form
Px 1Z
xcc
Potential functions of c
set of maximal cliques
states of variables in maximal cliqueNormalizing constant
• So for this graph, the factorization is ambiguous:. (something called “factor graphs” makes it explicit).
11
![Page 12: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/12.jpg)
Markov Random Fields
• Allows rich probabilistic models for imagesimages.
• But built in a local, modular way. Learn local relationships, get global effects out.local relationships, get global effects out.
12
![Page 13: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/13.jpg)
MRF nodes as pixels
13Winkler, 1995, p. 32
![Page 14: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/14.jpg)
MRF nodes as patchesMRF nodes as patches
image patches
scene patches
(xi, yi)image
scene patches
( )14
(xi, xj)scene
![Page 15: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/15.jpg)
Network joint probability
1 i
iiji
ji yxxxZ
yxP ),(),(1),(
sceneimage
Scene-scenecompatibility
Image-scenecompatibility
iji ,
g p yfunction
neighboringscene nodes
local observations
p yfunction
15
scene nodes observations
![Page 16: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/16.jpg)
In order to use MRFs:
• Given observations y, and the parameters of the MRF, how infer the hidden variables, x?, ,
• How learn the parameters of the MRF?
16
![Page 17: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/17.jpg)
Outline of MRF section
• Inference in MRF’s.– Gibbs sampling, simulated annealing– Iterated condtional modes (ICM)– Variational methods– Belief propagation– Graph cuts
• Vision applications of inference in MRF’s.• Learning MRF parameters.
17
g p– Iterative proportional fitting (IPF)
![Page 18: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/18.jpg)
Outline of MRF section
• Inference in MRF’s.– Gibbs sampling, simulated annealing– Iterated condtional modes (ICM)– Variational methods– Belief propagation– Graph cuts
• Vision applications of inference in MRF’s.• Learning MRF parameters.
18
g p– Iterative proportional fitting (IPF)
![Page 19: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/19.jpg)
Derivation of belief propagationy1
)( yx )( yx )( yx
y2 y3
),( 11 yx
),( 21 xx
),( 22 yx
),( 32 xx
),( 33 yx
x1 x2 x3
)(P ),,,,,(sumsummean 3213211321
yyyxxxPxxxxMMSE
19
![Page 20: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/20.jpg)
The posterior factorizes
),,,,,(sumsummean 3213211 yyyxxxPx MMSE
),(sumsummean 111321
321
yxxxxxMMSE
xxx
),(),(),(),(
3233
2122
xxyxxxyx
),(mean),(),(
111
3233
1
yxxxxyx
xMMSE
y1 y2 y3
)()(sum
),(),(sum 21222
xxyx
xxyxx
y1
),( 11 yx ),( 22 yx ),( 33 yx
x1
y2
x2
y3
x3
20
),(),(sum 32333
xxyxx
),( 21 xx ),( 32 xx1 2 3
![Page 21: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/21.jpg)
Propagation rules
),,,,,(sumsummean 3213211 yyyxxxPx MMSE
),(sumsummean 111321
321
yxxxxxMMSE
xxx
),(),(),(),(
3233
2122
xxyxxxyx
y1 y2 y3
),(mean),(),(
111
3233
1
yxxxxyx
xMMSE
y1
),( 11 yx ),( 22 yx ),( 33 yx
x1
y2
x2
y3
x3)()(sum
),(),(sum 21222
xxyx
xxyxx
21
),( 21 xx ),( 32 xx1 2 3),(),(sum 3233
3
xxyxx
![Page 22: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/22.jpg)
Propagation rules),(mean 111
1
yxxxMMSE
)()(
),(),(sum 21222
xxyxx
),(),(sum 32333
xxyxx
y1 y2 y3
)( ),( ),(sum)( 23222211
21
2
xMyxxxxMx
y1
),( 11 yx ),( 22 yx ),( 33 yx
x1
y2
x2
y3
x3
22
),( 21 xx ),( 32 xx1 2 3
![Page 23: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/23.jpg)
Propagation rules),(mean 111
1
yxxxMMSE
)()(
),(),(sum 21222
xxyxx
),(),(sum 32333
xxyxx
y1 y2 y3
)( ),( ),(sum)( 23222211
21
2
xMyxxxxMx
y1
),( 11 yx ),( 22 yx ),( 33 yx
x1
y2
x2
y3
x3
23
),( 21 xx ),( 32 xx1 2 3
![Page 24: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/24.jpg)
Belief propagation: the noseyBelief propagation: the nosey neighbor rule
“Given everything that I know, here’s what I think you should think”y
(Given the probabilities of my being in(Given the probabilities of my being in different states, and how my states relate to your states here’s what I think theyour states, here s what I think the probabilities of your states should be)
24
![Page 25: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/25.jpg)
Belief propagation messagesA message: can be thought of as a set of weights on each of your possible states
To send a message: Multiply together all the incoming messages, except from the node you’re sending to,then multiply by the compatibility matrix and marginalize
k
then multiply by the compatibility matrix and marginalize over the sender’s states.
ijNk
jkjji
xi
ji xMxxxM
j \)(ij )(),()(
jii =j
25
=
![Page 26: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/26.jpg)
BeliefsBeliefsTo find a node’s beliefs: Multiply together all the messages coming in to that node.
j )(
)( )(jNk
jkjjj xMxb
)( jNk
26
![Page 27: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/27.jpg)
Belief, and message updatesBelief, and message updates
kj
)(
)()(jNk
jkjjj xMxb
jkjjii
ji xMxxxM ij )(),()(
jii
ijNk
jjjix
iij \)(
ij )()()(
27
ji =
![Page 28: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/28.jpg)
Optimal solution in a chain or tree:pBelief Propagation
• “Do the right thing” Bayesian algorithm.• For Gaussian random variables over time:For Gaussian random variables over time:
Kalman filter.• For hidden Markov models:• For hidden Markov models:
forward/backward algorithm (and MAP variant is Viterbi)variant is Viterbi).
28
![Page 29: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/29.jpg)
No factorization with loops!),(mean 111
1
yxxxMMSE
)()(
),(),(sum 21222
xxyxx
),(),(sum 32333
xxyxx
31 ),( xx
y1
y2
y3y1
x1
x2
y3
x3
29
![Page 30: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/30.jpg)
Justification for running belief propagation in networks with loopsin networks with loops
• Experimental results:– Error-correcting codes Kschischang and Frey, 1998;g
Vision applications Freeman and Pasztor, 1999;
Kschischang and Frey, 1998;McEliece et al., 1998
– Vision applications
• Theoretical results:
, ;Frey, 2000
– For Gaussian processes, means are correct.
– Large neighborhood local maximum for MAP.i d
Weiss and Freeman, 1999
– Equivalent to Bethe approx. in statistical physics.
– Tree-weighted reparameterization
Weiss and Freeman, 2000
Yedidia, Freeman, and Weiss, 2000
30
Tree weighted reparameterizationWainwright, Willsky, Jaakkola, 2001
![Page 31: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/31.jpg)
Results from Bethe free energy analysis
• Fixed point of belief propagation equations iff. Bethe i i i iapproximation stationary point.
• Belief propagation always has a fixed point.• Connection with variational methods for inference: both
minimize approximations to Free Energy,i ti l ll i l i bl– variational: usually use primal variables.
– belief propagation: fixed pt. equs. for dual variables.
• Kikuchi approximations lead to more accurate belief• Kikuchi approximations lead to more accurate belief propagation algorithms.
• Other Bethe free energy minimization algorithms31
• Other Bethe free energy minimization algorithms—Yuille, Welling, etc.
![Page 32: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/32.jpg)
References on BP and GBP• J. Pearl, 1985
– classic• Y. Weiss, NIPS 1998,
– Inspires application of BP to vision• W. Freeman et al learning low-level vision, IJCV 1999
– Applications in super-resolution, motion, shading/paint pp p , , g pdiscrimination
• H. Shum et al, ECCV 2002– Application to stereo
• M. Wainwright, T. Jaakkola, A. Willsky– Reparameterization version
• J. Yedidia, AAAI 2000
32
,– The clearest place to read about BP and GBP.
![Page 33: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/33.jpg)
Outline of MRF section• Inference in MRF’s.
Gibbs sampling simulated annealing– Gibbs sampling, simulated annealing– Iterated conditional modes (ICM)
Belief propagation– Belief propagation• Application example—super-resolution
– Graph cutsp• Application example--stereo
– Variational methods• Application example—blind deconvolution
• Learning MRF parameters.
33– Iterative proportional fitting (IPF)
![Page 34: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/34.jpg)
Stereo problem
L RL R
Squared difference,
d
(L[x] – R[x-d])^2,for some x.
x
Showing local disparity evidence vectors for a set of neighboring positions xd
34
positions, x.
![Page 35: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/35.jpg)
MRF for stereo
Disparity estimate at Local evidence for each disparity b d i i heach pixel based on intensity match to
neighboring frame
35Show Middlebury stereo evaluation web pages.
![Page 36: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/36.jpg)
Log probability or energy function to be optimizedLog probability, or energy function, to be optimized for MAP estimation in MRF’s
E Edata Esmooth
Dp ( f p ) p Vp,q ( f p, fq )
{ p q}N
p { p,q}N
36
![Page 37: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/37.jpg)
Graph cuts
• Algorithm: uses node label swaps or expansions as moves in the algorithm to reduce the energy. Swaps many labels at once not just one at a timeSwaps many labels at once, not just one at a time, as with ICM.
• Find which pixel labels to swap using min cut/maxFind which pixel labels to swap using min cut/max flow algorithms from network theory.
• Can offer bounds on optimality.• See Boykov, Veksler, Zabih, IEEE PAMI 23 (11)
Nov. 2001 (available on web).
37
![Page 38: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/38.jpg)
38
![Page 39: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/39.jpg)
Two moves: swap and expansion
39
![Page 40: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/40.jpg)
The cost of an alpha-beta assignment
D () V ( f )for each node in P labeled
b f
not a member of P
Dp () Vp,q (, fq )qN p ,qP
a member of P
( )for each connected P neighbor:
Vp,q (,)p
Dp () Vp,q (, fq )40
qN p ,qP
for each node in P labeled
![Page 41: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/41.jpg)
Flash of inspiration here…
41
![Page 42: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/42.jpg)
The cost of any possible alpha-beta assignment described as a cut in a graph
Definitions of graph edge weights:
tp Dp () Vp,q (, fq )
N P
Definitions of graph edge weights:
qN p ,qP
ep q Vp q (,)
tp
P p,q p,q ( ,)p
t
qep,qP
tp Dp () Vp,q (, fq )
qN p ,qP
tp
42
![Page 43: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/43.jpg)
The cost of any possible alpha-beta assignment described as a cut in this graph, with edge-weights defined depending on the MRF energies.
tq Dq () Vq ,r(, f r )
rNq ,rP
V ( )
tp
D () V ( f )
ep,q Vp,q (,)p
qep,q
tp Dp () Vp,q (, fq )
qN p ,qP
tp
tp Dp () Vp,q (, fq )
qN p ,qP
43
p
![Page 44: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/44.jpg)
44
![Page 45: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/45.jpg)
45
![Page 46: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/46.jpg)
Slide by Yuri Boykov
Minimum s-t cuts algorithms
Augmenting paths [Ford & Fulkerson 1962] Augmenting paths [Ford & Fulkerson, 1962]
P h l b l [G ldb T j 1986] Push-relabel [Goldberg-Tarjan, 1986]
![Page 47: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/47.jpg)
Slide by Yuri Boykov
“Augmenting Paths”• Find a path from S to• Find a path from S to
T along non-saturated edgesg
“source”
S T“sink”
Increase flow along this path until somethis path until some edge saturates
A graph with two terminals
![Page 48: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/48.jpg)
Slide by Yuri Boykov
“Augmenting Paths”• Find a path from S to• Find a path from S to
T along non-saturated edgesg
“source”
S T“sink”
Increase flow along this path until somethis path until some edge saturates
A graph with two terminals Find next path… Increase flow…Increase flow…
![Page 49: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/49.jpg)
Slide by Yuri Boykov
“Augmenting Paths”• Find a path from S to• Find a path from S to
T along non-saturated edgesg
“source”
S T“sink”
Increase flow along this path until somethis path until some edge saturates
A graph with two terminalsIterate until … all
paths from S to T have at l t t t d dleast one saturated edge
MAX FLOWMIN CUTBreak between the two sets always cutting connections that are at capacity
![Page 50: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/50.jpg)
![Page 51: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/51.jpg)
ICCV 2001
http://citeseer.ist.psu.edu/rd/0%2C251059%2C1%2C0.25%2CDownload/http:
51
//coblitz.codeen.org:3125/citeseer.ist.psu.edu/cache/papers/cs/11281/http:zSzzSzwww.cs.cornell.eduzSzrdzzSzPaperszSzBVZ-iccv99.pdf/boykov99fast.pdf
![Page 52: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/52.jpg)
52
![Page 53: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/53.jpg)
53
![Page 54: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/54.jpg)
54
![Page 55: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/55.jpg)
To Boykov slides
http://palantir.swarthmore.edu/cvpr05/CVPR05_tutorial_yyb.ppt
And see:http://www-static.cc.gatech.edu/gvu/perception/projects/graphcuttextures/#videohttp://www static.cc.gatech.edu/gvu/perception/projects/graphcuttextures/#videoAndhttp://en.wikipedia.org/wiki/Max-flow_min-cut_theoremAnd
55
Andhttp://www.cse.yorku.ca/~aaw/Wang/MaxFlowStart.htm
![Page 56: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/56.jpg)
CVPR’05 Tutorial Discrete Optimization Methods in Vision
Slide by Yuri Boykov
Discrete Optimization Methods in Vision
Part III: Graph Algorithms in VisionPart III: Graph Algorithms in Vision
Yuri Boykov, University of Western OntarioPedro Felzenszwalb, University of ChicagoR i Z bih C ll U i iRamin Zabih, Cornell University
![Page 57: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/57.jpg)
Slide by Yuri Boykov
Outline• Shortest path algorithmShortest path algorithm• s-t Graph cuts
– Hypersurface optimizationHypersurface optimization – Binary energy minimization
• N-way cuts / Multi-label energy minimization• N-way cuts / Multi-label energy minimization – a-expansion algorithm
Applications in Computer Visionpp p
![Page 58: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/58.jpg)
Shortest paths on graphsSlide by Yuri Boykov
Shortest paths on graphs (examples)
T t S th i• Texture Synthesis– Image quilting [Efros & Freeman, 2001]
• Object extraction– live-wire [Falcao, Udupa, Samarasekera, Sharma 1998]
– intelligent scissors [Mortensen, Barrett 1998]
• Scan line stereo Ohta & Kanade 1985– Ohta & Kanade, 1985
– Cox, Hingorani, Rao, 1996
![Page 59: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/59.jpg)
Shortest paths:Slide by Yuri Boykov
Shortest paths: Texture synthesis“Image quilting”Image quilting
![Page 60: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/60.jpg)
2D Graph cut shortest path on a graph
Shortest pathsExample:
find the shortest Graph Cutspapproach
(live wire, intelligent scissors)
find the shortest closed contour in a given
domain of a graph
p
approach
p
Compute the shortest path Compute theCompute the shortest pathp ->p for a point p.
Compute the minimum cut that
separates red region from blue region
Repeat for all points on the gray line Then choose the from blue regiongray line. Then choose the
optimal contour.
live-wire: [Falcao, Udupa, Samarasekera, Sharma 1998] intelligent scissors: [Mortensen, Barrett 1998]
![Page 61: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/61.jpg)
Slide by Yuri Boykov
DP / Shortest-paths • Efficient global optimization tools Efficient global optimization tools • Successes are primarily in 1D problems
C ’t d l ith d d t li i t– Can’t deal with dependent scan-lines in stereo– Can’t handle bubbles (3D snakes) or
bj t b d i i l t i d tobject boundaries in volumetric data
• Graph Cuts can be seen as a “generalization” to N-D
![Page 62: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/62.jpg)
Slide by Yuri Boykov
Stereo example
Independent Multi-scan linepscan-lines(via DP)
Multi-scan line(via Graph Cuts) Ground truth
![Page 63: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/63.jpg)
Graph cuts for segmentation
Slide by Yuri Boykov
Graph cuts for segmentation(simple example à la Boykov&Jolly, ICCV’01)
n-linkst a cuthard constraint
hards
hard constraint
IMinimum cost cut can be
computed in polynomial time
22
exp
pqpq
Iw
(max-flow/min-cut algorithms)pqI
![Page 64: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/64.jpg)
Examples of Graph Cuts in vision
Slide by Yuri Boykov
Examples of Graph Cuts in vision• Image Restoration (e.g. Greig at.al. 1989)• SegmentationSegmentation
– Wu & Leahy 1993– Nested Cuts, Veksler 2000
• Multi-scan-line Stereo, Multi-camera stereoMulti scan line Stereo, Multi camera stereo– Roy & Cox 1998, 1999 – Ishikawa & Geiger 1998, 2003 – Boykov, Veksler, Zabih 1998, 2001 – Kolmogorov & Zabih 2002, 2004
• Object Matching/Recognition (Boykov & Huttenlocher 1999)• N-D Object extraction (photo-video editing, medical imaging)
– Boykov, Jolly, Funka-Lea 2000, 2001, 2004– Boykov & Kolmogorov 2003– Rother, Blake, Kolmogorov 2004
• Texture synthesis (Kwatra, Schodl, Essa, Bobick 2003)• Shape reconstruction (Snow,Viola,Zabih 2000)• Motion (e.g. Xiao, Shah 2004)
![Page 65: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/65.jpg)
s-t graph cuts for video texturesSlide by Yuri Boykov
Graph cuts video textureshttp://www-static.cc.gatech.edu/gvu/perception/projects/graphcuttextures/#video
Graph cuts video textures
a cut
21 21
Short video clip Long video clip3D generalization of “image-quilting”
![Page 66: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/66.jpg)
Pixel interactions:“convex” vs. “discontinuity-preserving”
Slide by Yuri Boykov
Robust“discontinuity preserving”
interactions“Convex”
interactions
V(ΔL)Potts model
V(ΔL)
“linear”
ΔL=Lp-Lq
( )
ΔL=Lp-Lq
model
V(ΔL)V(ΔL)
ΔL=Lp-LqΔL=Lp-Lq
![Page 67: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/67.jpg)
Pixel interactions:“convex” vs. “discontinuity-preserving”
Slide by Yuri Boykov
y p g
“linear” V
truncated “linear” V
![Page 68: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/68.jpg)
a-expansion moveSlide by Yuri Boykov
Basic idea: break multi-way cut computation into a sequence of binary s-t cuts
other labelsa
Iteratively, each label competes with the other labels for space in the image
![Page 69: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/69.jpg)
a-expansion movesSlide by Yuri Boykov
In each a-expansion a given label “a” grabs space from other labels
initial solution-expansionexpansion-expansion-expansion-expansion-expansion-expansion-expansion
For each move we choose expansion that gives the largest decrease in the energy: binary optimization problem
![Page 70: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/70.jpg)
a-expansion algorithm vs. standard discrete energy minimization techniques
Slide by Yuri Boykov
discrete energy minimization techniques
single “one pixel” move single a-expansion movesingle one-pixel move(simulated annealing, ICM,…)
• Large number of pixels can change • Only one pixel can change its label at g p gtheir labels simultaneously
• Finding an optimal move is
y p ga time
• Finding an optimal move is computationally intensive O(2^n)
(s-t cuts)computationally trivial
![Page 71: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/71.jpg)
a-expansion move vs.“standard” moves
Slide by Yuri Boykov
original imagenoisy image
Potts energy minimization
a local minimumw.r.t. expansion moves
a local minimumw.r.t. “one-pixel” moves
![Page 72: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/72.jpg)
a expansions vs simulated annealing
Slide by Yuri Boykov
a-expansions vs. simulated annealing
li d l tii l t d li i ( 89 01)normalized correlation,start for annealing, 24.7% err
simulated annealing, 19 hours, 20.3% err
a-expansions (BVZ 89,01)90 seconds, 5.8% err
Annealing Our method
20000
40000
60000
80000
100000
moo
thne
ss E
nerg
y
0
20000
1 10 100 1000 10000 100000
Time in seconds
Sm
![Page 73: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/73.jpg)
a-expansion algorithm vs.local-update algorithms (SA, ICM, ...)
Slide by Yuri Boykov
a-expansions simulated annealing
• Finds local minimum of ith t t
• Finds local minimum of energy with respect to small “one pixel”
a expansions simulated annealing, …
energy with respect to very strong moves
with respect to small one-pixel moves
• In practice, results do not • Initialization is important
• solution is within the factor of 2 from the global minima
• solution could be arbitrarily far from the global minima
depend on initialization
g• In practice, one cycle through all
labels gives sufficiently good results
• May not know when to stop. Practical complexity may be worse than exhaustive searchworse than exhaustive search
• Applies to a restricted class of energies
• Can be applied to anything
![Page 74: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/74.jpg)
end of Boykov slides
![Page 75: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/75.jpg)
Middlebury stereo web page
75
![Page 76: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/76.jpg)
76
![Page 77: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/77.jpg)
Algorithm comparisons
77
![Page 78: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/78.jpg)
78
![Page 79: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/79.jpg)
Comparison of graph cuts and beliefComparison of graph cuts and belief propagation
Comparison of Graph Cuts with Belief Propagation for Stereo, using IdenticalMRF Parameters ICCV 2003MRF Parameters, ICCV 2003.Marshall F. Tappen William T. Freeman
79
![Page 80: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/80.jpg)
Ground truth graph cuts and beliefGround truth, graph cuts, and belief propagation disparity solution energies
80
![Page 81: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/81.jpg)
Graph cuts versus belief propagation
• Graph cuts consistently gave slightly lower energy solutions for that stereo-problem MRF, although BP ran faster, although there is now a faster graphBP ran faster, although there is now a faster graph cuts implementation than what we used…
• Advantages of Belief Propagation:W k f ibili f i i d– Works for any compatibility functions, not a restricted set like graph cuts.
– I find it very intuitive.– Extensions: sum-product algorithm computes MMSE,
and Generalized Belief Propagation gives you very accurate solutions, at a cost of time.
81
![Page 82: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/82.jpg)
MAP versus MMSE
82
![Page 83: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/83.jpg)
Outline of MRF section
• Inference in MRF’s.– Gibbs sampling, simulated annealing– Iterated condtional modes (ICM)– Variational methods– Belief propagation– Graph cuts
• Vision applications of inference in MRF’s.• Learning MRF parameters.
83
g p– Iterative proportional fitting (IPF)
![Page 84: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/84.jpg)
Vision applications of MRF’s
• Super-resolution• Motion estimationMotion estimation• Labelling shading and reflectance
M h• Many others…
84
![Page 85: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/85.jpg)
Super-resolution
• Image: low resolution image• Scene: high resolution imageScene: high resolution image
ultimate goal...
imag
e
ultimate goal...
ien
e85
sce
![Page 86: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/86.jpg)
Pixel-based images are not resolution
i d d tindependentPixel replication
Cubic splineCubic spline, Cubic splinesharpened
Training-based Polygon-based graphics images are
gsuper-resolution
86
images are resolution independent
![Page 87: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/87.jpg)
3 approaches to perceptual sharpening
(1) Sharpening; boost existing high itude
(1) Sharpening; boost existing high frequencies.
(2) U l i l f b ispatial frequency
ampl
i
(2) Use multiple frames to obtain higher sampling rate in a still frame.
(3) Estimate high frequencies not present in image, although implicitly defined.
In this talk we focus on (3) whicham
plitu
de
87
In this talk, we focus on (3), which we’ll call “super-resolution”. spatial frequency
![Page 88: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/88.jpg)
Super-resolution: other approaches
• Schultz and Stevenson, 1994• Pentland and Horowitz, 1993• fractal image compression (Polvere, 1998; Iterated Systems)• astronomical image processing (eg. Gull and Daniell, 1978;
“pixons” http://casswww.ucsd.edu/puetter.html)• Follow on: Jianchao Yang John Wright Thomas S Huang• Follow-on: Jianchao Yang, John Wright, Thomas S. Huang,
Yi Ma: Image super-resolution as sparse representation of raw image patches. CVPR 2008
88
![Page 89: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/89.jpg)
Training images, ~100,000 image/scene patch pairs
Images from two Corel database categories: “giraffes” and “urban skyline”giraffes and urban skyline .
89
![Page 90: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/90.jpg)
Do a first interpolation
Zoomed low-resolution
Low-resolution
90
![Page 91: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/91.jpg)
Zoomed low-resolution Full frequency original
Low-resolution
91
![Page 92: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/92.jpg)
Full freq. originalRepresentationZoomed low-freq.
92
![Page 93: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/93.jpg)
Full freq. originalRepresentationZoomed low-freq.
True high freqsL b d i t
93
g qLow-band input
(contrast normalized, PCA fitted)
(to minimize the complexity of the relationships we have to learn,we remove the lowest frequencies from the input image,
and normalize the local contrast level).
![Page 94: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/94.jpg)
Gather ~100,000 patches
high freqs
Training data samples (magnified)
......low freqs.
high freqs.
94
![Page 95: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/95.jpg)
True high freqs.Input low freqs.Nearest neighbor estimate
Estimated high freqs.
high freqs.
95Training data samples (magnified)
......low freqs.
![Page 96: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/96.jpg)
Input low freqs.Nearest neighbor estimate
Estimated high freqs.
high freqs.
96Training data samples (magnified)
......low freqs.
![Page 97: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/97.jpg)
Example: input image patch, and closest matches from databasematches from database
Input patchInput patch
Closest imagepatches from database
Correspondinghigh-resolution
patches from database
97
patches from database
![Page 98: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/98.jpg)
98
![Page 99: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/99.jpg)
Scene-scene compatibility function, ( )(xi, xj)
Assume overlapped regions d of hi-resAssume overlapped regions, d, of hi-res. patches differ by Gaussian observation noise:
Uniqueness constraint,99d
Uniqueness constraint,not smoothness.
![Page 100: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/100.jpg)
Image-scene compatibility y
function, (xi, yi)
Assume Gaussian noise takes you from b d i t h t th ti l
x
observed image patch to synthetic sample:
100
![Page 101: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/101.jpg)
Markov network
i t himage patches
(xi, yi)
(x x )scene patches
101
(xi, xj)
![Page 102: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/102.jpg)
Belief PropagationInput
After a few iterations of belief propagation, the algorithm selects spatially consistent high resolution
interpretations for each low-resolution patch of the Input
Iter 0
input image.
Iter. 0
Iter 1Iter. 1
Iter. 3
102
![Page 103: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/103.jpg)
Zooming 2 octavesWe apply the super-resolution
algorithm recursively, zooming up 2 powers of 2, or a factor of 4
i h di i
85 x 51 input
in each dimension.
pu
103Cubic spline zoom to 340x204 Max. likelihood zoom to 340x204
![Page 104: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/104.jpg)
Now we examine the effect of the prior assumptions made about images on the
high resolution reconstruction.
Original50x58
(cubic spline implies thin plate prior)
First, cubic spline interpolation.
50x58 plate prior)
TrueTrue200x232
104
![Page 105: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/105.jpg)
Original50x58
(cubic spline implies thin plate prior)50x58 plate prior)
TrueCubic spline
True200x232
105
![Page 106: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/106.jpg)
Next, train the Markov network algorithm on a world of random noise
images
Original50x58
images.
50x58
i i iTraining images
True
106
![Page 107: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/107.jpg)
The algorithm learns that, in such a world, we add random noise when zoom
to a higher resolution
Original50x58
to a higher resolution.
50x58
i i iTraining images
M kMarkovnetwork
True
107
![Page 108: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/108.jpg)
Next, train on a world of vertically oriented rectangles.
Original50x5850x58
i i iTraining images
True
108
![Page 109: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/109.jpg)
The Markov network algorithm hallucinates those vertical rectangles that
it was trained on
Original50x58
it was trained on.
50x58
i i iTraining images
M kMarkovnetwork
True
109
![Page 110: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/110.jpg)
Now train on a generic collection of images.
Original50x5850x58
Training images
True
110
![Page 111: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/111.jpg)
The algorithm makes a reasonable guess at the high resolution image, based on its
training images
Original50x58
training images.
50x58
Training images
M kMarkovnetwork
True
111
![Page 112: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/112.jpg)
Generic training imagesNext, train on a generic
set of training imagesset of training images. Using the same camera
as for the test image, but a random collection of
photographs.
112
![Page 113: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/113.jpg)
Cubic Spline
Original70x70
Markovnetnet, training:generic
True280x280
113
![Page 114: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/114.jpg)
Kodak Imaging Science Technology Lab test.
3 test images, 640x480, to bezoomed up by 4 in each di idimension.
8 judges, making 2-alternative, forced-choice comparisons.
114
![Page 115: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/115.jpg)
Algorithms comparedAlgorithms compared
• Bicubic Interpolation• Mitra's Directional Filter• Fuzzy Logic Filter•Vector Quantization•Vector Quantization• VISTA
115
![Page 116: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/116.jpg)
Bicubic spline Altamira VISTA
116
![Page 117: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/117.jpg)
Bicubic spline Altamira VISTA
117
![Page 118: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/118.jpg)
User preference test results
“The observer data indicates that six of the observers rankedF ’ l i h h f d f h fi dFreeman’s algorithm as the most preferred of the five testedalgorithms. However the other two observers rank Freeman’s algorithmas the least preferred of all the algorithms….p g
Freeman’s algorithm produces prints which are by far the sharpestout of the five algorithms However this sharpness comes at a priceout of the five algorithms. However, this sharpness comes at a priceof artifacts (spurious detail that is not present in the originalscene). Apparently the two observers who did not prefer Freeman’sl i h h d bj i h if Th h balgorithm had strong objections to the artifacts. The other observers
apparently placed high priority on the high level of sharpness in theimages created by Freeman’s algorithm.”
118
g y g
![Page 119: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/119.jpg)
119
![Page 120: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/120.jpg)
120
![Page 121: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/121.jpg)
Training images
121
![Page 122: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/122.jpg)
Training image
122
![Page 123: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/123.jpg)
Processed image
123
![Page 124: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/124.jpg)
Vision applications of MRF’s
• Super-resolution• Motion estimationMotion estimation• Labelling shading and reflectance
M h• Many others…
124
![Page 125: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/125.jpg)
Motion applicationppimage patches
image
scene patches
125scene
![Page 126: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/126.jpg)
What behavior should we see in aWhat behavior should we see in a motion algorithm?
• Aperture problem• Resolution through propagation ofResolution through propagation of
information• Figure/ground discrimination• Figure/ground discrimination
126
![Page 127: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/127.jpg)
The aperture problemp p
127
![Page 128: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/128.jpg)
The aperture problemp p
128
![Page 129: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/129.jpg)
Motion analysis: related work
• Markov network– Luettgen, Karl, Willsky and collaborators.g , , y
• Neural network or learning-based– Nowlan & T. J. Senjowski; Sereno.j ;
• Optical flow analysis– Weiss & Adelson; Darrell & Pentland; JuWeiss & Adelson; Darrell & Pentland; Ju,
Black & Jepson; Simoncelli; Grzywacz & Yuille; Hildreth; Horn & Schunk; etc.
129
![Page 130: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/130.jpg)
Motion estimation results(maxima of scene probability distributions displayed)
Inference:
Image data
Iterations 0 and 1
130Initial guesses only show motion at edges.
![Page 131: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/131.jpg)
Motion estimation results(maxima of scene probability distributions displayed)
Figure/ground still unresolved here
Iterations 2 and 3
131
unresolved here.
![Page 132: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/132.jpg)
Motion estimation results(maxima of scene probability distributions displayed)
Final result compares well with vector
Iterations 4 and 5
132
pquantized true (uniform) velocities.
![Page 133: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/133.jpg)
Vision applications of MRF’s
• Super-resolution• Motion estimationMotion estimation• Labelling shading and reflectance
M h• Many others…
133
![Page 134: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/134.jpg)
Forming an ImageIlluminate the surface to get:
Surface (Height Map) Shading Image
134
The shading image is the interaction of the shapeof the surface and the illumination
![Page 135: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/135.jpg)
Painting the Surface
Scene ImageScene
Add a reflectance pattern to the surface. Points
Image
135inside the squares should reflect less light
![Page 136: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/136.jpg)
Goal
Image Shading Image Reflectance ImageImage
136
![Page 137: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/137.jpg)
Basic Steps1. Compute the x and y image derivatives2. Classify each derivative as being caused by
i h h di fl t heither shading or a reflectance change3. Set derivatives with the wrong label to zero. 4 Recover the intrinsic images by finding the least-4. Recover the intrinsic images by finding the least-
squares solution of the derivatives.
137Original x derivative image Classify each derivative(White is reflectance)
![Page 138: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/138.jpg)
Learning the Classifiers• Combine multiple classifiers into a strong classifier using
Ad B (F d d S h i )AdaBoost (Freund and Schapire)• Choose weak classifiers greedily similar to (Tieu and Viola
2000))• Train on synthetic images• Assume the light direction is from the right
Shading Training Set Reflectance Change Training Set
138
![Page 139: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/139.jpg)
Using Both Color and G S l I f tiGray-Scale Information
Results withoutconsidering gray-scale
139
![Page 140: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/140.jpg)
Some Areas of the Image Are Locally AmbiguousLocally Ambiguous
Is the change here better explained as
Input?or ?
140Shading Reflectance
![Page 141: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/141.jpg)
Propagating InformationC di bi t b ti• Can disambiguate areas by propagating information from reliable areas of the image i t bi f th iinto ambiguous areas of the image
141
![Page 142: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/142.jpg)
Propagating Information• Consider relationship between neighboring derivativesg g
• Use Generalized Belief Propagation to infer labels
142
p g
![Page 143: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/143.jpg)
Setting Compatibilities
• Set compatibilities according to image contourscontours– All derivatives along a
contour should have the same label
• Derivatives along an image contourimage contour strongly influence each other 0.5 1.0
β=
143
1
1),( jxxi
![Page 144: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/144.jpg)
Improvements Using Propagation
Input Image Reflectance ImageWith Propagation
Reflectance ImageWithout Propagation
144
![Page 145: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/145.jpg)
145
![Page 146: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/146.jpg)
(More Results)(More Results)
Input Image Shading Image Reflectance Image
146
![Page 147: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/147.jpg)
147
![Page 148: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/148.jpg)
148
![Page 149: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/149.jpg)
Outline of MRF section
• Inference in MRF’s.– Gibbs sampling, simulated annealing– Iterated conditional modes (ICM)– Variational methods– Belief propagation– Graph cuts
• Vision applications of inference in MRF’s.• Learning MRF parameters.
149
g p– Iterative proportional fitting (IPF)
![Page 150: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/150.jpg)
Learning MRF parameters, labeled data
Iterative proportional fitting lets you make a maximum likelihood estimate a joint distribution from observations of various marginal distributions.
150
![Page 151: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/151.jpg)
True joint probability
Observed marginal
151
gdistributions
![Page 152: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/152.jpg)
Initial guess at joint probability
152
![Page 153: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/153.jpg)
IPF update equationIPF update equation
Scale the previous iteration’s estimate for the joint probability by the ratio of the true to the predicted marginals.
Gives gradient ascent in the likelihood of the jointGives gradient ascent in the likelihood of the joint probability, given the observations of the marginals.
153See: Michael Jordan’s book on graphical models
![Page 154: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/154.jpg)
Convergence of to correct marginals by IPF algorithm
154
![Page 155: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/155.jpg)
Convergence of to correct marginals by IPF algorithm
155
![Page 156: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/156.jpg)
IPF results for this example: comparison of joint probabilities
True joint probability
Initial guess Final maximumentropy estimate
156
![Page 157: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/157.jpg)
Application to MRF parameter estimation
• Can show that for the ML estimate of the cliqueCan show that for the ML estimate of the clique potentials, c(xc), the empirical marginals equal the model marginals,g
• This leads to the IPF update rule for c(xc)p c( c)
• Performs coordinate ascent in the likelihood of the MRF parameters, given the observed data.
157
p , g
Reference: unpublished notes by Michael Jordan
![Page 158: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/158.jpg)
Learning MRF parameters, labeled data
Iterative proportional fitting lets you make a maximum likelihood estimate a joint distribution from observations of various marginal distributions.
Applied to learning MRF clique potentials:Applied to learning MRF clique potentials: (1) measure the pairwise marginal statistics (histogram state co-occurrences in labeled training data). (2) guess the clique potentials (use measured marginals to start). (3) do inference to calc late the model’s marginals for(3) do inference to calculate the model’s marginals for every node pair.(4) scale each clique potential for each state pair by the
158
( ) q p p yempirical over model marginal
![Page 159: C280, Computer Vision - Peopletrevor/CS280Notes/20MRFs.pdf · C280, Computer Vision Prof. Trevor Darrell trevor@eecs.berkeley.edu Lecture 20: Markov Random Fields. ... Jordan and](https://reader036.fdocuments.us/reader036/viewer/2022070814/5f0dae247e708231d43b8d6b/html5/thumbnails/159.jpg)
Slide Credits
• Bill Freeman• Yuri BoykovYuri Boykov• others, as noted…
159