Graph Cuts based segmentation

methods:

Daniel Heilper, CS Department, Haifa University

Topics:

Segmentation as Graph cuts problems. MIN-CUT and Normalized CUT approaches Examples of Cost/Energy functions Performance issues – feasibility and

approximations Relations to Clustering and other methods

Motivation:

Human perception takes into account several perceptual aspects and “blends” them together: Proximity:

Human perception Proximity: Closure:

Human perception Similarity Continuation

Early segmentation methods Boundary based methods - based on local information and cannot

guarantee continuous closed edge contours (Derivatives kernels, Harris, Canny)

Region based methods + statistics inside the region. - often generates irregular boundaries and small

Holes.

Boundary based methods Original text Canny:

Boundary based methods Derivatives:

Region based methods

Global optimization approaches based on energy functions

Construct a heuristic energy function which take into account both the local information and the global information.

Graph Representation: The set of points in an arbitrary feature

space are represented as a weighted undirected

graph G = (V,E,w) where the nodes of the graph are the points in the feature space, and an edge is formed between every pair of nodes. The weight on each edge, Wij is a function of the similarity between nodes i and j.

Graph Representation:

Energy cost functions: Energy Function is heuristic for quantization

of a combination of Data Features on an N-D Image.

Simple Examples: Distance metric – Image is sent as a binary image, graph is represented as points in the plane

Energy functions for Graph Cuts In some cases, energy minimization is

NP-Hard, or has high polynomial order which make the method impractical

Choosing a good energy function is not “Pure” science

Energy functions for Graph Cuts The most Natural Energy form is:

V - Boundary Penalty caused by choosing labels for 2 pixels

D – Regional Penalty for assigning a pixel to some label.

First approach: Max-Flow Min-Cut

Refresher: Flow Networks:

Network flow graphs Given A flow Networks where: is a directed graph is a source (not necessary belongs to G) is a sink (not necessary belongs to G) a non-negative capacity function on the

edges (if we can assume )

),( EVG

ctsGN ,,,

Vs

Vt

REc :

( , ) 0c u v ( , )u v E

Network flow graphs Function is a valid flow function if holds: Capacity constraints: , counter symmetry: for each ,

flow conservation: , where , are the edge set that leave V. Similarly are the sedge set that enters V

, ,f u v c u v Ev,u

,v u V u,vfv,uf

)()(

)()(vNevNe

efef )(vN

)(vN

tsVv ,\

Network flow graphs The value of the flow function is

Vv

v,sf

)()(

)()(sNesNe

efef

Network flow graphs

Network flow graphs

Network flow graphs Max-Flow – Min-Cut Theorem: The maximum amount of flow is equal to

the capacity of a minimum cut

Interactive Segmentation application using Graph Cuts:

Motivation: for some applications: Fully automatic segmentation does not give sufficient results.

for example, in medical diagnostics, Doctors handlings a medical scan images where soft tissues, blood vessels, and bones are “blended” together , are very difficult to segment.

Above: CAT scan of Pancreas.

Interactive Segmentation application using Graph Cuts In a nutshell:

User marks some pixels as seeds of “Background” and “Object”, and runs the application several times until a satisfactory result achieved.

Interactive inputs: “seeds” pixels the ratio between regional importance VS boundary

importance Energy function (Intensities, distances, textures, etc..)

Seeds in the flow graph

Interactive Segmentation application using Graph Cuts First run, application fully solves max-flow task Each time the application outputs an insufficient

result, the user adds more “seeds”, (changes the ratio ) and rerun the application.

The segmentation is improved until satisfactory by user.

Only the first run is computationally heavy, while the consecutive runs are fast, and can be seen as an addition to capacities, and hence finding additional “augmenting paths” in the flow function

Past Interactive tools for segmentation: intelligent scissors live wire “shortest path” between certain pixels using

some energy function Difficulties:

the user inputs have to be very accurately positioned at the desired boundary

can not be easily generalized to 3D data.

Energy function: General form of the energy function:

Where

Energy function:

R(A) stands for the sum of the regional penalties of a given labeling

Similarly B(A) is the sum of the boundry penalties of the labeling

3D - Data Video sequence is treated as a 3D volume

image:

Second Method: Normalized Cut: The normalized cut criterion measures both

the total dissimilarity between the different groups as well as the total similarity within the groups.

The key point is that image partitioning is to be done from the big picture downward, rather like a painter first marking out the major areas and then filling in the details.

Normalized Cut Unfortunately, minimizing normalized cut

exactly is NP-complete, even for the special case of graphs on grids(Papadimitriou). However, an approximate discrete solution can be found efficiently

MINCut problem Why minimum cut isn’t always favorable? the minimum cut criteria favors cutting

small sets of isolated nodes in the graph. (for example when cost function is inverse to the distance between nodes)

A case where minimum cut gives a bad partition

.

Definition: Normalized cut

where assoc(A,V) is the total connection from nodes in A to all nodes in the graph

Normalized cut It is easy to see that single nodded cuts

produce MINCut value of at least 1, so minimum normal cut is not likely to be single nodded.

Definition: Normalized Association:

In the same spirit, we can define a measure for total normalized association within groups for a given partition:

NAssoc vs NCut A direct conclusion:

NCut(A,B) = 2 – Nassoc(A,B)

Ncut is normalized measure of disassociation

minimizing Ncut is equivalent to maximizing association

Computing the Optimal Partition An approximate discrete solution can be

found efficiently by converting the problem into a generalized eigenvector problem

NCut as Eigenvector system Matrix representation W is the cost Matrix D is the sum of the costs from node i:

The Partition is a binary matrix size |P|XK where |P| is the number of pixels and K is the number of labels

NCut as Eigenvector system

The Segmentation matrix y:

Each column is a label vector

NCut can be written as:

Recursive two-way Ncut grouping algorithm

1. Given an image or image sequence, set up a weighted graph G=(V, E) and set the weight on the edge connection two nodes to be a measure of the similarity between the two nodes.

2. Solve for eigenvectors with the smallest eigenvalues

Recursive two-way Ncut grouping algorithm

3. Use the eigenvector with the second smallest eigenvalue to bipartition the graph

4. Decide if the current partition should be subdivided and recursively repartition the segmented parts if necessary.

Experiments on energy functions:

where X(i) is the spatial location of node i, and F(i) is a feature vector based on intensity, color, or texture information

If can separate the regional weight from the boundary weight using log operation

(a) A synthetic image showing a noisy ªstepº image. Intensity varies from 0 to 1, and Gaussian noise with sigma = 0.2 is added.

(b) eigenvector with the second smallest eigenvalue (c) resulting partition.

Performance and optimality Solving an eigenvector system takes at

least (|P|^3) time Using iterative power method we can find

the second lowest eigenvector in a O(|P|*|n|) where |P| is #pixels, and |n| is #iterative steps

The NCut at the end of the algorithm has proven constant proximity to the MIN NCut

Finding the Minimum Normalized Cut VS interactive MINCut finding:

Min NCut Pros: Avoids segmenting isolated small sections (VS

“blind” calculation of MINCut) This NP-Hard can be well approximated

Min NCut cons: Difficult to control and analyse

Finding the Minimum Normalized Cut VS interactive MINCut finding

interactive MINCut Pros: Allows a fast correction of the result in an

interactively manner, and converges Easy to control

Energy Minimization Techniques for Graph Cuts Goal: giving a faster algorithm to the MIN-

Cut problem or a general K-Labeling partition of an Image

Some energy functions makes the MinCut problem to be NP-Hard

Other functions reach optimal solution in impractical polynomial time

Known algorithms for energy minimization: Simmulated Anealing – converges well but for

small temperature delta’s is exponential in time, and for large deltas it converges to local minimum which may be far from the minimum

Combinatorial Max flow – Min cut algorithms. Combinatorial algorithms solves the problem in at least O(|P|^3) time (Tarjan & Goldberg).

dynamic programming – restricted to 1-D energy function

linear time algorithm for finding the Max-Flow-Min-Cut/

Updating the labeling function: two types of large moves: α-expansion and

α-β-swap

α-Exapnsion

Take a subset of pixels which are not labeled to have that label

Works for a metric energy function:

α-β swap Changing a label α of some pixels to label β Works for semimetric Energy functions:

α-Exapnsion

α-Exapnsion and {α, β}–Swap

α-Exapnsion and {α, β}–Swap these moves allow large number of pixels

to change their labels simultaneously Note that a move which assigns a given

label α to a single pixel is both an α-β swap and an α-expansion. (α-β swap can be seen as a private case of α-Exapnsion)

{ α} Expansion 1. Start with an arbitrary labeling f 2. Set success := 0 3. For each label α belongs L

3.1. Find ˆ f = argminE(f0) among f0 within one α-expansion of f

3.2. If E(f*) < E(f), set f := f* and success := 1

4. If success = 1 goto 2 5. Return f

{α, β} – Swap algorithm 1. Start with an arbitrary labeling f 2. Set success := 0 3. For each pair of labels {α, β} belongs to L

3.1. Find f* = argminE(f’) among f’ within one α-β swap of f

3.2. If E(f*) < E(f), set f := f* and success := 1

4. If success = 1 goto 2 5. Return f

Algorithms summary A cycle is successful if a strictly better

labeling is found at any iteration. The algorithms stop after the first

unsuccessful cycle since no further improvement is possible.

α-Exapnsion much faster than α-β swap : |L| VS o(|L|^2) iterations (L is the labeling)

Benchmark results

(Results of running the algorithms on the previous image)

References:

[1] J. Shi and J.Malik, “Normalized Cuts and Image Segmentation,” Proc. CVPR 1997.

[2] Boykov, Y., Jolly, M., " Interactive graph cuts for optimal boundary and regionsegmentation of objects in N-D images." In: International Conference on Computer Vision, Vancouver , BC. (2001) 105–112

[3] Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23 (2001) 1222–1239

[4] Goro Obinata, Ashish Dutta: Vision Systems: Segmentation and Pattern Recognition, I-Tech Education and Publishing, Vienna, Austria