Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf ·...
Transcript of Markov Networks in Computer Vision - University at Buffalosrihari/CSE674/Chap4/4.5-MRF-CV.pdf ·...
Probabilistic Graphical Models Srihari
1
Markov Networks in Computer Vision
Sargur [email protected]
Probabilistic Graphical Models Srihari
Markov Nets (MRFs) in Computer Vision
1. Image Denoising
2. Image segmentation
3. Stereo reconstruction
4. Object recognition2
Probabilistic Graphical Models Srihari
3
1. Binary Image de-noising• Image-to-image mapping• Observed noisy image
– Binary pixel values yi ∈ {-1,+1}, i=1,..,D• Unknown noise-free image
– Binary pixel values xi ∈ {-1,+1}, i=1,..,D• Noisy image assumed to randomly
flip sign of pixels with small probability
Probabilistic Graphical Models Srihari
4
Markov Random Field Model
• Known– Strong correlation between xi and yi– Neighbor pixels xi and xj are strongly
correlated• Prior knowledge captured using MRF
xi= unknown noise-free pixelyi= known noisy pixel
Probabilistic Graphical Models Srihari
5
Energy Functions
• Graph has two types of cliques– {xi,yi} expresses correlation between variables
• Choose simple energy function –η xiyi• Lower energy (higher probability) when xi and yi have
same sign
– {xi,xj} which are neighboring pixels• Choose β xixj
Probabilistic Graphical Models Srihari
6
Potential Function
• Complete energy function of model
– The hxi term biases towards pixel values that have one particular sign
– Which defines a joint distribution over x and y given by
{ , }
(x, y) i i j i ii i j i
E h x x x x yb h= - -å å å
1(x, y) exp{ (x, y)}p E
Z= -
Probabilistic Graphical Models Srihari
Parameters for Image denoising• Node potential ϕ(Xi) for each pixel Xi
• penalize large discrepancies from observed pixel value yi• Edge potential
– Encode continuity between adjacent pixel values• Penalize cases where inferred value of Xi is too far from
inferred value of neighbor Xj• Important not to over-penalize true edge disparities
(edges between objects or regions)– Leads to oversmoothing of image
• Solution: Bound the penalty using a truncated norm– ε(xi,xj) = min(c||xi-xj||p,distmax) for p ε {1,2}
7
Probabilistic Graphical Models Srihari
8
De-noising problem statement
• We fix y to observed pixels in the noisy image• p(x|y) is a conditional distribution over all noise-
free images– Called Ising model in statistical physics
• We wish to find an image x that has a high probability
Probabilistic Graphical Models Srihari
• Gradient ascent– Set xi=yi for all i– Take one node xj at a time
• evaluate total energy for states xi=+1 and xi=-1• keeping all other node variable fixed
– Set xj to value for which energy is lower• This is a local computation
• which can be done efficiently
– Repeat for all pixels until • a stopping criterion is met
– Nodes updated systematically • by raster scan or randomly
• Finds a local maximum (which need not be global)
• Algorithm is called Iterated Conditional Modes (ICM)
De-noising algorithm
Probabilistic Graphical Models Srihari
10
Image Restoration Results• Parametersβ=1.0 (for xixj)η=2.1 (for xiyi)h=0
Result of ICM Global maximum obtained byGraph Cut algorithm
Noise Free imageNoisy image where 10%of pixels are corrupted
Probabilistic Graphical Models Srihari
2. Image Segmentation Task• Partition the image pixels into regions
corresponding to distinct parts of scene• Different variants of segmentation task
– Many formulated as a Markov network• Multiclass segmentation
– Each variable Xi has a domain {1,..,K} pixels– Value of Xi represents region assignment for pixel i
, e.g., grass, water, sky, car– Classifying each pixel is expensive
• Oversegment image into superpixels (coherent regions) and classify each superpixels
– All pixels within region are assigned same value11
Probabilistic Graphical Models Srihari
Importance of Modeling Correlations between superpixels
12
car
road
building
cow
grass
(a) (b) (c) (d)Original image Oversegmentedimage-superpixelsEach superpixel isa random variable
Classification using node potentials alone-each superpixel classified independently
Segmentation usingpairwise Markov Network encoding interactionsbetween adjacentsuperpixels
Probabilistic Graphical Models Srihari
Graph from Superpixels• A node for each superpixel• Edge between nodes if regions are adjacent• This defines a distribution in terms of this graph
13
Probabilistic Graphical Models Srihari
Log-linear model with features
• Xi: Pixels or Super-pixels• Joint probability distribution over an image• Log-linear model:
F={fi}: Features between variables A,B ε Di:
14
P(X
1,..X
n;θ) = 1
Z(θ)exp θ
ifi(D
i)
i=1
k
∑⎧⎨⎩⎪
⎫⎬⎭⎪
fa0b0(a,b) = I a = a0{ }I b = b0{ }
Z θ( ) = exp θ
ifiξ( )
i∑⎧⎨
⎩
⎫⎬⎭ξ
∑
Probabilistic Graphical Models Srihari
Network Structure• In most applications structure is pairwise
– Variables correspond to pixels– Edges (factors) correspond to
• interactions between adjacent pixels in grid on image
– Each interior pixel has exactly four neighbors• Value space of variables and exact form of
factors depend on task
• Usually formulated:• Factors in terms of energies
– Negative log potentials– Values represent penalties:
» lower value = higher probability15
car
road
building
cow
grass
(a) (b) (c) (d)
Probabilistic Graphical Models Srihari
Model
• Edge potential between every pair of superpixels Xi, Xj
– Encodes a contiguity preference– With a penalty λ whenever Xi≠Xj
– Model can be improved by making penalty depend on presence of an image gradient between pixels
– Even better model:• Non default values for class pairs• Tigers adjacent to vegetation, water below vegetation
16
Probabilistic Graphical Models Srihari
Features for Image Segmentation• Features extracted for each superpixel
– Statistics over color, texture, location• Features either clustered or input to local classifiers to reduce
dimensionality• Node potential is a function of these features
– Factors depend upon pixels in image– Each image defines a different probability
distribution over segment labels for pixels or superpixels
• Model in effect is a Conditional Random Field
17
Probabilistic Graphical Models Srihari
Metric MRFs• Class of MRFs used for labeling• Graph of nodes X1,..Xn related by set of edges E• Wish to assign to each Xi a label in space V={v1,..vk}
• Each node, taken in isolation, has its preference among possible labels
• Also need to impose a soft ”smoothness” constraint that neighboring nodes should take similar values
18
Probabilistic Graphical Models Srihari
Encoding preferences
• Node preferences are node potentials in pairwise MRF
• Smoothness preferences are edge potentials• Traditional to encode these models in negative
log-space– using energy functions• With MAP objective we can ignore the partition
function
19
Probabilistic Graphical Models Srihari
Energy Function
• Energy function
• Goal is to minimize the energy
20
E(x
1,..x
n) = ε
i(x
i)
i∑ + ε
ij(x
ix
j)
{i,j}∑
arg
x1,..xn
min E(x1,..x
n)
Probabilistic Graphical Models Srihari
Smoothness definition
• Slight variant of Ising model
• Obtain lowest possible pairwise energy (0) when neighboring nodes Xi,Xj take the same value and a higher energy λi,j when they do not
21
εij(x
1,x
j) =
0λ
i,j
⎧⎨⎪
⎩⎪
xi= x
j
xi≠ x
j
Probabilistic Graphical Models Srihari
Generalizations
• Potts model extends it to more than two labels• Distance function on labels
– Prefer neighboring nodes to have labels smaller distance apart
22
Probabilistic Graphical Models Srihari
Metric definition
• A function µ: V � Và [0,∞) is a metric if it satisfies– Reflexivity: µ(vk,vl)=0 if and only if k=l– Symmetry:µ(vk,vl)=µ(vl,vk);– Triangle Inequality:µ(vk,vl)+µ(vl,vm) ≥ µ(vk,vm)
• µ is a semi-metric if it satisfies first two• Metric MRF is defined by defining
εi,j(vk,vj) = µ(vk,vl)• A common metric: ε(xi,xj) = min(c||xi-xj||,distmax)
Probabilistic Graphical Models Srihari
3. Stereo Reconstruction• Reconstruct depth disparity of each
image pixel – Variables represent discretized version
of depth dimension (more finely for discretized for close to camera and coarse when away)
• Node potential: a computer vision technique to estimate depth disparity
• Edge potential: a truncated metric– Inversely proportional to image gradient
between pixels– Smaller penalty to large gradient suggesting
occlusion24