Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color...
-
Upload
yandex -
Category
Technology
-
view
983 -
download
5
description
Transcript of Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color...
![Page 1: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/1.jpg)
C. Olsson
Higher-order Segmentation Functionals: Entropy, Color Consistency, Curvature, etc.
Yuri Boykov jointly with
O. Veksler
Andrew Delong
L. Gorelick
C. Nieuwenhuis E. Toppe
I. Ben Ayed M. Tang
A. Delong
H. Isack
A. Osokin
![Page 2: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/2.jpg)
Different surface representations
mesh level-sets graph labeling
on complex
on grid
point cloud labeling
ps
continuous optimization
mixed optimization
Zps p{0,1}ps
combinatorial optimization
![Page 3: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/3.jpg)
this talk
graph labeling
{0,1}ps
combinatorial optimization
Implicit surfaces/bondary
on grid
![Page 4: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/4.jpg)
Sf,E(S) B(S)Sf,E(S)
Image segmentation Basics
4 I
Fg)|Pr(I Bg)|Pr(I
p
pp sfE(S)
bg)|Pr(I
fg)|Pr(Ilnf
p
p
p
S
{0,1}ps
![Page 5: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/5.jpg)
Linear (modular) appearance of region
p
pp sfSfSR ,)(
Examples of potential functions
• Log-likelihoods • Chan-Vese • Ballooning
2)( cIf pp
1pf
)( pp If Prln
![Page 6: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/6.jpg)
Basic boundary regularization for
{0,1}ps
pair-wise discontinuities
][)( qp
Npq
sswSB
![Page 7: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/7.jpg)
Basic boundary regularization for
{0,1}ps
second-order terms
qpqpqp ssssss )()(][ 11
quadratic
][)( qp
Npq
sswSB
![Page 8: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/8.jpg)
Basic boundary regularization for
{0,1}ps
Examples of discontinuity penalties
• Boundary length • Image-weighted boundary length
1pqw
2)( qppq IIw exp
second-order terms
][)( qp
Npq
pq sswSB
![Page 9: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/9.jpg)
Basic boundary regularization for
• corresponds to boundary length | |
– grids [B&K, 2003], via integral geometry
– complexes [Sullivan 1994]
• submodular second-order energy
– can be minimized exactly via graph cuts [Greig et al.’91, Sullivan’94, Boykov-Jolly’01]
pqw
n-links
s
t a cut
{0,1}ps
second-order terms
][)( qp
Npq
pq sswSB
![Page 10: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/10.jpg)
2
any (binary) segmentation energy E(S) is a set function E: S
Ω
Submodular set functions
![Page 11: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/11.jpg)
Submodular set functions
Set function is submodular if for any 2:E
)()()()( TESETSETSE TS,
Significance: any submodular set function can be globally optimized in polynomial time
[Grotschel et al.1981,88, Schrijver 2000]
S T Ω
)||( 9O
![Page 12: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/12.jpg)
Submodular set functions
Set function is submodular if for any 2:E
)()}{()()}{( SEvSETEvTE
TS
S T Ω
an alternative equivalent definition providing intuitive interpretation: “diminishing returns”
v v
Easily follows from the previous definition: E(T))}{()()}{( vSESEvTE
STS TS
Significance: any submodular set function can be globally optimized in polynomial time
[Grotschel et al.1981,88, Schrijver 2000] )||( 9O
![Page 13: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/13.jpg)
Graph cuts for minimization of
submodular set functions
Assume set Ω and 2nd-order (quadratic) function
Function E(S) is submodular if for any
)()()()( 10011100 ,,,, pqpqpqpq EEEE
Nqp )( ,
Significance: submodular 2nd-order boolean (set) function can be globally optimized in polynomial time by graph cuts
[Hammer 1968, Pickard&Ratliff 1973]
Npq
qppq ssEsE)(
, )()( }{ 10,, qp ssIndicator variables
[Boros&Hammer 2000, Kolmogorov&Zabih2003] )|||N(| 2O
![Page 14: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/14.jpg)
Combinatorial optimization
Continuous optimization
submodularity convexity
Global Optimization
?
![Page 15: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/15.jpg)
Assume Gibbs distribution over binary random variables
for
Graph cuts for minimization of posterior energy (MRF)
}{ 10,ps
))((),...,( SEexpssPr n1
Theorem [Boykov, Delong, Kolmogorov, Veksler in unpublished book 2014?]
All random variables sp are positively correlated iff set function E(S) is submodular
That is, submodularity implies MRF with “smoothness” prior
}{ 1s|pS p
![Page 16: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/16.jpg)
Basic segmentation energy
][ qp
Npq
pqp
p
p sswsf
boundary smoothness segment region/appearance
![Page 17: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/17.jpg)
this talk
Higher-order binary segmentation
Curvature (3-rd order)
Convexity (3-rd order)
segment region/appearance
Shape priors (N-th order) Connectivity (N-th order)
Cardinality potentials (N-th order)
Appearance Entropy (N-th order)
Color consistency (N-th order)
Distribution consistency (N-th order)
boundary smoothness
![Page 18: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/18.jpg)
submodular approximations [our work: Trust Region 13, Auxiliary Cuts 13]
global minimum [our work: One Cut 2014]
block-coordinate descent [Zhu&Yuille 96, GrabCut 04]
Overview of this talk
• From likelihoods to entropy
• From entropy to color consistency
• Convex cardinality potentials
• Distribution consistency
• From length to curvature
optimization high-order functionals
other extensions [arXiv13]
![Page 19: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/19.jpg)
][)Pr(Npq
qppq
p
sp sswISEp
|ln),|( 10
assuming known
[Boykov&Jolly, ICCV2001]
image segmentation, graph cut
RGBI p
• parametric models – e.g. Gaussian or GMM • non-parametric models - histograms
}{ 10,ps
pair-wise (quadratic) term unary (linear) term
Given likelihood models
guaranteed globally optimal S
![Page 20: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/20.jpg)
Beyond fixed likelihood models
[Rother, et al. SIGGRAPH’2004]
iterative image segmentation, Grabcut (block coordinate descent )
RGBI p
10 ,S
Models 0 , 1 are iteratively re-estimated
(from initial box)
extra variables • parametric models – e.g. Gaussian or GMM • non-parametric models - histograms
}{ 10,ps][)Pr(Npq
qppq
p
sp sswISEp
|ln),,( 10
pair-wise (quadratic) term mixed optimization term
NP hard mixed optimization! [Vesente et al., ICCV’09]
![Page 21: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/21.jpg)
• Minimize over segmentation S for fixed 0 , 1
• Minimize over 0 , 1 for fixed labeling S
Block-coordinate descent for
fixed for S=const
)( 10 ,,SE
Npq
qppq
p
Sp sswISEp
][)Pr()( |ln,, 10
Npq
qppq
sp
p
sp
p sswIISEpp
][)Pr()Pr()(1
1
0
010
::
|ln|ln,,
Sp0ˆ Sp1
ˆ
distribution of intensities in current bkg. segment ={p:Sp=0}
distribution of intensities in current obj. segment S={p:Sp=1} S
optimal S is computed using graph cuts, as in [BJ 2001]
![Page 22: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/22.jpg)
Iterative learning of color models (binary case )
• GrabCut: iterated graph cuts [Rother et al., SIGGRAPH 04]
start from models 0 , 1 inside and outside some given box
iterate graph cuts and model re-estimation until convergence to a local minimum
][)Pr()(Npq
qppq
p
Sp sswISEp
|ln,, 10
}{ 10,ps
solution is sensitive to initial box
![Page 23: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/23.jpg)
BCD minimization of converges to a local minimum )( 10 ,,SE
E=2.37×106 E=2.41×106 E=1.39×106 E=1.410×106
Iterative learning of color models (binary case ) }{ 10,ps
(interactivity a la “snakes”)
![Page 24: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/24.jpg)
Iterative learning of color models (could be used for more than 2 labels )
• Unsupervised segmentation [Zhu&Yuille, 1996]
][)Pr()(Npq
qppq
p
Sp sswISEp
|ln...,,, 210 || labels
}{ ,...,, 210ps
using level sets + merging heuristic
initialize models 0 , 1 , 2 , from many randomly sampled boxes
iterate segmentation and model re-estimation
until convergence
models compete, stable result if sufficiently many
![Page 25: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/25.jpg)
Iterative learning of color models (could be used for more than 2 labels )
• Unsupervised segmentation [Delong et al., 2012]
|| labels
}{ ,...,, 210ps
using a-expansion (graph-cuts)
initialize models 0 , 1 , 2 , from many randomly sampled boxes
models compete, stable result if sufficiently many
iterate segmentation and model re-estimation
until convergence
][)Pr()(Npq
qppq
p
Sp sswISEp
|ln...,,, 210
![Page 26: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/26.jpg)
Iterative learning of other models (could be used for more than 2 labels )
• Geometric multi-model fitting [Isack et al., 2012]
initialize plane models 0 , 1 , 2 , from many randomly sampled SIFT matches
in 2 images of the same scene
|| labels
using a-expansion (graph-cuts)
iterate segmentation and model re-estimation
until convergence
models compete, stable result if sufficiently many
][-)(Npq
qppq
p
S sswppSEp
...,,, 210
}{ ,...,, 210ps
![Page 27: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/27.jpg)
Iterative learning of other models (could be used for more than 2 labels )
• Geometric multi-model fitting [Isack et al., 2012]
initialize Fundamental matrices 0 , 1 , 2 , from many randomly sampled SIFT matches
in 2 consecutive frames in video
|| labels
using a-expansion (graph-cuts)
iterate segmentation and model re-estimation
until convergence
models compete, stable result if sufficiently many
VIDEO
}{ ,...,, 210ps
][-)(Npq
qppq
p
S sswppSEp
...,,, 210
![Page 28: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/28.jpg)
From color model estimation to entropy and color consistency
global optimization in One Cut
[Tang et al. ICCV 2013]
![Page 29: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/29.jpg)
Interpretation of log-likelihoods: entropy of segment intensities
S
S1
Si
S2
S3
S4 S5
pixels of color i in S
}|{ iISpS pi
||
||
S
Sp is
i
probability of intensity i in S
Sp
pI )( |Prln
i
i
S
i ppS ln||
=
where = {p1 , p2 , ... , pn }
given distribution of intensities
},...,,{ S
n
SSS pppp 21
distribution of intensities observed at S
cross entropy
of distribution pS w.r.t.
H(S| )
i
ii pS ln||
=
![Page 30: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/30.jpg)
Interpretation of log-likelihoods: entropy of segment intensities
][)()()(Npq
qppq sswSH|S|SH|S|SE
Npq
qppq
1S:p
1p
0S:p
0p ssw|Iln|Ilnpp
][)Pr()Pr()( 10 ,,SE
)( 0||| SHS )( 1||| SHS
10 ,min
entropy of intensities in S
entropy of intensities in S
minimization of segments entropy
Note: H(P|Q) H(P) for any two distributions (equality when Q=P) cross-entropy entropy
joint estimation of S and color models [Rother et al., SIGGRAPH’04, ICCV’09]
[Tang et al, ICCV 2013]
![Page 31: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/31.jpg)
Interpretation of log-likelihoods: entropy of segment intensities
)( 10 ,,SE
)( 0||| SHS )( 1||| SHS
10 ,min
entropy of intensities in S
entropy of intensities in S
binary optimization
Note: H(P|Q) H(P) for any two distributions (equality when Q=P) cross-entropy entropy
mixed optimization
[Tang et al, ICCV 2013]
[Rother et al., SIGGRAPH’04, ICCV’09]
][)()()(Npq
qppq sswSH|S|SH|S|SE
Npq
qppq
1S:p
1p
0S:p
0p ssw|Iln|Ilnpp
][)Pr()Pr(
![Page 32: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/32.jpg)
][)()()(Npq
qppq sswSH|S|SH|S|SE
Interpretation of log-likelihoods: entropy of segment intensities
)( 10 ,,SE
)( 0||| SHS )( 1||| SHS
10 ,min
entropy of intensities in S
entropy of intensities in S
common energy for categorical clustering, e.g. [Li et al. ICML’04]
Note: H(P|Q) H(P) for any two distributions (equality when Q=P) cross-entropy entropy
Npq
qppq
1S:p
1p
0S:p
0p ssw|Iln|Ilnpp
][)Pr()Pr(
![Page 33: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/33.jpg)
Minimizing entropy of segments intensities (intuitive motivation)
][)()()(Npq
qppq sswSH|S|SH|S|SE
unsupervised image segmentation (like in Chan-Vese)
high entropy segmentation
break image into two coherent segments with low entropy of intensities
S
S
low entropy segmentation
S
S S
S S
S
![Page 34: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/34.jpg)
more general than Chan-Vese (colors can vary within each segment)
S
S
S
S
break image into two coherent segments with low entropy of intensities
Minimizing entropy of segments intensities (intuitive motivation)
][)()()(Npq
qppq sswSH|S|SH|S|SE
![Page 35: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/35.jpg)
From entropy to color consistency
all pixels i
Minimization of entropy encourages pixels i of the same color bin i to be segmented together
(proof: see next page)
i
12
4
3
5
![Page 36: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/36.jpg)
From entropy to color consistency
][)()()(Npq
qppq sswSH|S|SH|S|SE
S
i
i
S
i
S
i
i
S
i ppSppS ln||ln||
||
||
||
||ln||ln||
S
S
i
iS
S
i
iii SS
||ln||||ln|| i
i
i
i
i SSSS ||ln||||ln|| i
i
i
i
i SSSS
||ln||||ln|| SSSSi
iiii |S|ln|S||S|ln|S| )(
volume balancing
color consistency
|S| |S|
S
S
Si Si
Si = S i
|S| |Si|
i i
pixels in each color bin i prefer to be together (either inside object
or background)
![Page 37: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/37.jpg)
From entropy to color consistency
||ln||||ln|| SSSS
volume balancing
color consistency
S
Si = S i
|S| |Si|
i i
segmentation S with better
color consistency
pixels in each color bin i prefer to be together (either inside object
or background)
S
i
iiii |S|ln|S||S|ln|S| )(
![Page 38: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/38.jpg)
From entropy to color consistency
||ln||||ln|| SSSS
volume balancing
color consistency
S
Si = S i
|S| |Si|
i i
convex function of cardinality |S|
(non-submodular)
pixels in each color bin i prefer to be together (either inside object
or background)
S concave function of
cardinality |Si| (submodular)
Graph-cut constructions for similar cardinality terms (for superpixel consistency)
[Kohli et al. IJCV’09]
In many applications, this term can be either dropped or replaced
with simple unary ballooning [Tang et al. ICCV 2013]
i
iiii |S|ln|S||S|ln|S| )(
![Page 39: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/39.jpg)
|Si|
From entropy to color consistency
||ln||||ln|| SSSS
volume balancing
color consistency
(also, simpler construction)
connect pixels in each color bin to corresponding auxiliary nodes
][Npq
qppq ssw
boundary smoothness
|S| |Si|
i i
In many applications, this term can be either dropped or replaced
with simple unary ballooning [Tang et al. ICCV 2013]
convex function of cardinality |S|
(non-submodular)
L1 color separation works better in practice [Tang et al. ICCV 2013]
i
iiii |S|ln|S||S|ln|S| )(
![Page 40: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/40.jpg)
smoothness + color consistency
One Cut [Tang, et al., ICCV’13]
connect pixels in each color bin to corresponding auxiliary nodes
Grabcut is sensitive to bin size
guaranteed global minimum
bo
x se
gmen
tati
on
linear ballooning
inside the box
![Page 41: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/41.jpg)
smoothness + color consistency
One Cut [Tang, et al., ICCV’13]
bo
x se
gmen
tati
on
ballooning from hard constraints
linear ballooning from
saliency measure
connect pixels in each color bin to corresponding auxiliary nodes
guaranteed global minimum linear ballooning
inside the box
fro
m s
eed
s sa
lien
cy-b
ased
se
gmen
tati
on
![Page 42: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/42.jpg)
photo-consistency + smoothness + color consistency
Color consistency can be integrated into
binary stereo
connect pixels in each color bin to corresponding auxiliary nodes
+ color consistency
photo-consistency+smoothness
![Page 43: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/43.jpg)
Approximating: - Convex cardinality potentials - Distribution consistency - Other high-order region terms
![Page 44: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/44.jpg)
d||SS| | 0
min
General Trust Region Approach (overview)
Trust region
(S)(S)E(S) BH
(S)(S)(S)E BU0
~
1st-order approximation for H(S)
0S
dsubmodular
(easy) hard
![Page 45: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/45.jpg)
||SS||λB(S)(S)U(S)L 00λ
• Constrained optimization
minimize
• Unconstrained Lagrangian Formulation
minimize
d||SS||s.t.
B(S)(S)U(S)E
0
0
~
can be approximated with unary terms [Boykov,Kolmogorov,Cremers,Delong, ECCV’06]
45
General Trust Region Approach (overview)
![Page 46: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/46.jpg)
Approximating L2 distance
0C
2
s dsdCdC,dC
|||| 0SS
p
o
ppp ssd2 )(
unary potentials [Boykov et al. ECCV 2006]
C
p dpd2
dp - signed distance map from C0
![Page 47: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/47.jpg)
Trust Region Approximation
|S|
][)Pr(Npq
qppq
p
Sp ssw|Ilnp
submodular terms
appearance log-likelihoods boundary length
non-submodular term
volume constraint
Linear approx. at S0
S0
S0 submodular approx.
trust region
p
o
ppp ssd )(
L2 distance to S0
![Page 48: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/48.jpg)
Volume Constraint for Vertebrae segmentation
Log-Lik. + length
48
![Page 49: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/49.jpg)
Back to entropy-based segmentation
Interactive segmentation
with box
volume balancing
color consistency
][Npq
qppq ssw
boundary smoothness
|S| |Si|
i i
+ +
submodular terms non-submodular term
global minimum
Approximations (local minima near the box)
![Page 50: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/50.jpg)
Trust Region Approximation
Surprisingly, TR outperforms QPBO, DD, TRWS, BP, etc.
on many high-order [CVPR’13] and/or
non-submodular problems [arXiv13]
![Page 51: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/51.jpg)
Curvature
![Page 52: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/52.jpg)
Pair-wise smoothness: limitations
52
• discrete metrication errors
4-neighborhood
- continuous convex formulations
8-neighborhood
- resolved by higher connectivity
![Page 53: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/53.jpg)
Pair-wise smoothness: limitations
53
• boundary over-smoothing (a.k.a. shrinking bias)
![Page 54: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/54.jpg)
Pair-wise smoothness: limitations
54
- curvature
- needs higher-order smoothness
• boundary over-smoothing (a.k.a. shrinking bias)
multi-view reconstruction
[Vogiatzis et al. 2005]
![Page 55: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/55.jpg)
Higher-order smoothness & curvature for discrete regularization
• Geman and Geman 1983 (line process, simulated annealing)
• Second-order stereo and surface reconstruction – Li & Zuker 2010 (loopy belief propagation) – Woodford et al. 2009 (fusion of proposals, QPBO) – Olsson et al. 2012-13 (fusion of planes, nearly submodular)
• Curvature in segmentation: – Schoenemann et al. 2009 (complex, LP relaxation, many extra variables) – Strandmark & Kahl 2011 (complex, LP relaxation,…) – El-Zehiry & Grady 2010 (grid, 3-clique, only 90 degree accurate, QPBO) – Shekhovtsov et al. 2012 (grid patches, approximately learned, QPBO) – Olsson et al. 2013 (grid patches, integral geometry, partial enumeration) – Nieuwenhuis et al 2014? (grid, 3-cliques, integral geometry, trust region)
this talk good approximation of curvature, better and faster optimization practical !
![Page 56: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/56.jpg)
the rest of the talk:
• Absolute curvature regularization on a grid [Olsson, Ulen, Boykov, Kolmogorov - ICCV 2013]
• Squared curvature regularization on a grid [Nieuwenhuis, Toppe, Gorelick, Veksler, Boykov - arXiv 2013]
![Page 57: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/57.jpg)
Absolute Curvature
dsS
||
Motivating example: for any convex shape 2dsS
||
• no shrinking bias • thin structures
![Page 58: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/58.jpg)
Absolute Curvature
n
n
easy to estimate via approximating
polygons
dsS
||
polygons also work for p [Bruckstein et al. 2001]
![Page 59: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/59.jpg)
curvature on a cell complex (standard geometry)
/2 /4
/4
/2
• Schoenemann et al. 2009 • Strandmark & Kahl 2011
4- or 3-cliques on a cell complex
solved via LP relaxations
![Page 60: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/60.jpg)
curvature on a cell complex (standard geometry)
/2 /4
/4
/2
cell-patch cliques on a complex
• Olsson et al., ICCV 2013
partial enumeration + TRWS
zero gap
reduction to pair-wise Constrain Satisfaction Problem
- new graph: patches are nodes - curvature is a unary potential
- patches overlap, need consistency
- tighter LP relaxation P4 P5 P6
P1 P2 P3
![Page 61: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/61.jpg)
A B C D E F G H
curvature on a cell complex (standard geometry)
0 0 0 0 0 0 0 0 0 0 0
/2 /4
/4
/2
A
A
A
A
B
C
F
F
G
H
E
D 0
0
0
0
2A+B= /2 A+F+G+H = /4
D+E+F = /2
A+C= /4
/4 /2 /2 /2 /4
curvature on a pixel grid (integral geometry)
representative cell-patches representative pixel-patches
![Page 62: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/62.jpg)
2x2 patches 3x3 patches 5x5 patches
zero gap
integral approach to absolute curvature
on a grid
![Page 63: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/63.jpg)
integral approach to absolute curvature
on a grid
2x2 patches 3x3 patches 5x5 patches
zero gap
![Page 64: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/64.jpg)
Squared Curvature with 3-cliques
dsS
2
![Page 65: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/65.jpg)
S
S 3-cliques
with configurations (0,1,0) and (1,0,1)
p p+
p-
general intuition example
Nieuwenhuis et al., arXiv 2013
more responses where curvature is higher
![Page 66: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/66.jpg)
N
n
nn
C
sdss1
22 )(
Δ i
Δ i
1
2
3
…
n
N
N-1
n-1
n+1
n+2
…
…
C
rn
Δ i
5x5 neighborhood
ci i
N
n
nin
1
)(||
)(|| ninn s
![Page 67: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/67.jpg)
rn
n
4
3ddR ||, )(
d
r =1/
zoom-in
Thus, appropriately weighted 3-cliques estimate squared curvature integral
![Page 68: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/68.jpg)
r
rds
rCircle
12
)(
Experimental evaluation
![Page 69: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/69.jpg)
Experimental evaluation
r
rds
rCircle
12
)(
![Page 70: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/70.jpg)
Model is OK on given segments. But, how do we optimize non-submodular
3-cliques (010) and (101)?
1. Standard trick: convert to non-submodular pair-wise binary optimization
2. Our observation: QPBO does not work (unless non-submodular regularization is very weak)
Fast Trust Region [CVPR13, arXiv]
uses local submodular approximations
![Page 71: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/71.jpg)
Segmentation Examples
length-based regularization
![Page 72: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/72.jpg)
elastica [Heber,Ranftl,Pock, 2012]
Segmentation Examples
![Page 73: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/73.jpg)
90-degree curvature [El-Zehiry&Grady, 2010]
Segmentation Examples
![Page 74: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/74.jpg)
our squared curvature
7x7 neighborhood Segmentation Examples
![Page 75: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/75.jpg)
our squared curvature (stronger)
7x7 neighborhood Segmentation Examples
![Page 76: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/76.jpg)
our squared curvature (stronger)
2x2 neighborhood Segmentation Examples
![Page 77: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/77.jpg)
Binary inpainting length squared curvature
![Page 78: Yuri Boykov — Combinatorial optimization for higher-order segmentation functionals: Entropy, Color Consistency, Curvature, etc.](https://reader034.fdocuments.us/reader034/viewer/2022052601/558fca991a28ab0e398b45d3/html5/thumbnails/78.jpg)
Conclusions • Optimization of Entropy is a useful information-
theoretic interpretation of color model estimation
• L1 color separation is an easy-to-optimize objective useful in its own right [ICCV 2013]
• Global optimization matters: one cut [ICCV13]
• Trust region, auxiliary cuts, partial enumeration
General approximation techniques
- for high-order energies [CVPR13]
- for non-submodular energies [arXiv’13]
outperforming state-of-the-art combinatorial optimization methods