Graph Cuts based segmentation
methods:
Daniel Heilper, CS Department, Haifa University
Topics:
Segmentation as Graph cuts problems. MIN-CUT and Normalized CUT approaches Examples of Cost/Energy functions Performance issues – feasibility and
approximations Relations to Clustering and other methods
Motivation:
Human perception takes into account several perceptual aspects and “blends” them together: Proximity:
Human perception Proximity: Closure:
Human perception Similarity Continuation
Early segmentation methods Boundary based methods - based on local information and cannot
guarantee continuous closed edge contours (Derivatives kernels, Harris, Canny)
Region based methods + statistics inside the region. - often generates irregular boundaries and small
Holes.
Boundary based methods Original text Canny:
Boundary based methods Derivatives:
Region based methods
Global optimization approaches based on energy functions
Construct a heuristic energy function which take into account both the local information and the global information.
Graph Representation: The set of points in an arbitrary feature
space are represented as a weighted undirected
graph G = (V,E,w) where the nodes of the graph are the points in the feature space, and an edge is formed between every pair of nodes. The weight on each edge, Wij is a function of the similarity between nodes i and j.
Graph Representation:
Energy cost functions: Energy Function is heuristic for quantization
of a combination of Data Features on an N-D Image.
Simple Examples: Distance metric – Image is sent as a binary image, graph is represented as points in the plane
Energy functions for Graph Cuts In some cases, energy minimization is
NP-Hard, or has high polynomial order which make the method impractical
Choosing a good energy function is not “Pure” science
Energy functions for Graph Cuts The most Natural Energy form is:
V - Boundary Penalty caused by choosing labels for 2 pixels
D – Regional Penalty for assigning a pixel to some label.
First approach: Max-Flow Min-Cut
Refresher: Flow Networks:
Network flow graphs Given A flow Networks where: is a directed graph is a source (not necessary belongs to G) is a sink (not necessary belongs to G) a non-negative capacity function on the
edges (if we can assume )
),( EVG
ctsGN ,,,
Vs
Vt
REc :
( , ) 0c u v ( , )u v E
Network flow graphs Function is a valid flow function if holds: Capacity constraints: , counter symmetry: for each ,
flow conservation: , where , are the edge set that leave V. Similarly are the sedge set that enters V
, ,f u v c u v Ev,u
,v u V u,vfv,uf
)()(
)()(vNevNe
efef )(vN
)(vN
tsVv ,\
Network flow graphs The value of the flow function is
Vv
v,sf
)()(
)()(sNesNe
efef
Network flow graphs
Network flow graphs
Network flow graphs Max-Flow – Min-Cut Theorem: The maximum amount of flow is equal to
the capacity of a minimum cut
Interactive Segmentation application using Graph Cuts:
Motivation: for some applications: Fully automatic segmentation does not give sufficient results.
for example, in medical diagnostics, Doctors handlings a medical scan images where soft tissues, blood vessels, and bones are “blended” together , are very difficult to segment.
Above: CAT scan of Pancreas.
Interactive Segmentation application using Graph Cuts In a nutshell:
User marks some pixels as seeds of “Background” and “Object”, and runs the application several times until a satisfactory result achieved.
Interactive inputs: “seeds” pixels the ratio between regional importance VS boundary
importance Energy function (Intensities, distances, textures, etc..)
Seeds in the flow graph
Interactive Segmentation application using Graph Cuts First run, application fully solves max-flow task Each time the application outputs an insufficient
result, the user adds more “seeds”, (changes the ratio ) and rerun the application.
The segmentation is improved until satisfactory by user.
Only the first run is computationally heavy, while the consecutive runs are fast, and can be seen as an addition to capacities, and hence finding additional “augmenting paths” in the flow function
Past Interactive tools for segmentation: intelligent scissors live wire “shortest path” between certain pixels using
some energy function Difficulties:
the user inputs have to be very accurately positioned at the desired boundary
can not be easily generalized to 3D data.
Energy function: General form of the energy function:
Where
Energy function:
R(A) stands for the sum of the regional penalties of a given labeling
Similarly B(A) is the sum of the boundry penalties of the labeling
3D - Data Video sequence is treated as a 3D volume
image:
Second Method: Normalized Cut: The normalized cut criterion measures both
the total dissimilarity between the different groups as well as the total similarity within the groups.
The key point is that image partitioning is to be done from the big picture downward, rather like a painter first marking out the major areas and then filling in the details.
Normalized Cut Unfortunately, minimizing normalized cut
exactly is NP-complete, even for the special case of graphs on grids(Papadimitriou). However, an approximate discrete solution can be found efficiently
MINCut problem Why minimum cut isn’t always favorable? the minimum cut criteria favors cutting
small sets of isolated nodes in the graph. (for example when cost function is inverse to the distance between nodes)
A case where minimum cut gives a bad partition
.
Definition: Normalized cut
where assoc(A,V) is the total connection from nodes in A to all nodes in the graph
Normalized cut It is easy to see that single nodded cuts
produce MINCut value of at least 1, so minimum normal cut is not likely to be single nodded.
Definition: Normalized Association:
In the same spirit, we can define a measure for total normalized association within groups for a given partition:
NAssoc vs NCut A direct conclusion:
NCut(A,B) = 2 – Nassoc(A,B)
Ncut is normalized measure of disassociation
minimizing Ncut is equivalent to maximizing association
Computing the Optimal Partition An approximate discrete solution can be
found efficiently by converting the problem into a generalized eigenvector problem
NCut as Eigenvector system Matrix representation W is the cost Matrix D is the sum of the costs from node i:
The Partition is a binary matrix size |P|XK where |P| is the number of pixels and K is the number of labels
NCut as Eigenvector system
The Segmentation matrix y:
Each column is a label vector
NCut can be written as:
Recursive two-way Ncut grouping algorithm
1. Given an image or image sequence, set up a weighted graph G=(V, E) and set the weight on the edge connection two nodes to be a measure of the similarity between the two nodes.
2. Solve for eigenvectors with the smallest eigenvalues
Recursive two-way Ncut grouping algorithm
3. Use the eigenvector with the second smallest eigenvalue to bipartition the graph
4. Decide if the current partition should be subdivided and recursively repartition the segmented parts if necessary.
Experiments on energy functions:
where X(i) is the spatial location of node i, and F(i) is a feature vector based on intensity, color, or texture information
If can separate the regional weight from the boundary weight using log operation
(a) A synthetic image showing a noisy ªstepº image. Intensity varies from 0 to 1, and Gaussian noise with sigma = 0.2 is added.
(b) eigenvector with the second smallest eigenvalue (c) resulting partition.
Performance and optimality Solving an eigenvector system takes at
least (|P|^3) time Using iterative power method we can find
the second lowest eigenvector in a O(|P|*|n|) where |P| is #pixels, and |n| is #iterative steps
The NCut at the end of the algorithm has proven constant proximity to the MIN NCut
Finding the Minimum Normalized Cut VS interactive MINCut finding:
Min NCut Pros: Avoids segmenting isolated small sections (VS
“blind” calculation of MINCut) This NP-Hard can be well approximated
Min NCut cons: Difficult to control and analyse
Finding the Minimum Normalized Cut VS interactive MINCut finding
interactive MINCut Pros: Allows a fast correction of the result in an
interactively manner, and converges Easy to control
Energy Minimization Techniques for Graph Cuts Goal: giving a faster algorithm to the MIN-
Cut problem or a general K-Labeling partition of an Image
Some energy functions makes the MinCut problem to be NP-Hard
Other functions reach optimal solution in impractical polynomial time
Known algorithms for energy minimization: Simmulated Anealing – converges well but for
small temperature delta’s is exponential in time, and for large deltas it converges to local minimum which may be far from the minimum
Combinatorial Max flow – Min cut algorithms. Combinatorial algorithms solves the problem in at least O(|P|^3) time (Tarjan & Goldberg).
dynamic programming – restricted to 1-D energy function
linear time algorithm for finding the Max-Flow-Min-Cut/
Updating the labeling function: two types of large moves: α-expansion and
α-β-swap
α-Exapnsion
Take a subset of pixels which are not labeled to have that label
Works for a metric energy function:
α-β swap Changing a label α of some pixels to label β Works for semimetric Energy functions:
α-Exapnsion
α-Exapnsion and {α, β}–Swap
α-Exapnsion and {α, β}–Swap these moves allow large number of pixels
to change their labels simultaneously Note that a move which assigns a given
label α to a single pixel is both an α-β swap and an α-expansion. (α-β swap can be seen as a private case of α-Exapnsion)
{ α} Expansion 1. Start with an arbitrary labeling f 2. Set success := 0 3. For each label α belongs L
3.1. Find ˆ f = argminE(f0) among f0 within one α-expansion of f
3.2. If E(f*) < E(f), set f := f* and success := 1
4. If success = 1 goto 2 5. Return f
{α, β} – Swap algorithm 1. Start with an arbitrary labeling f 2. Set success := 0 3. For each pair of labels {α, β} belongs to L
3.1. Find f* = argminE(f’) among f’ within one α-β swap of f
3.2. If E(f*) < E(f), set f := f* and success := 1
4. If success = 1 goto 2 5. Return f
Algorithms summary A cycle is successful if a strictly better
labeling is found at any iteration. The algorithms stop after the first
unsuccessful cycle since no further improvement is possible.
α-Exapnsion much faster than α-β swap : |L| VS o(|L|^2) iterations (L is the labeling)
Benchmark results
(Results of running the algorithms on the previous image)
References:
[1] J. Shi and J.Malik, “Normalized Cuts and Image Segmentation,” Proc. CVPR 1997.
[2] Boykov, Y., Jolly, M., " Interactive graph cuts for optimal boundary and regionsegmentation of objects in N-D images." In: International Conference on Computer Vision, Vancouver , BC. (2001) 105–112
[3] Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23 (2001) 1222–1239
[4] Goro Obinata, Ashish Dutta: Vision Systems: Segmentation and Pattern Recognition, I-Tech Education and Publishing, Vienna, Austria
Top Related