An efficient image segmentation algorithm using bidirectional

Mahalanobis distance

MASTERS THESISBy: Rahul Suresh

COMMITTEE MEMBERSDr.Stan BirchfieldDr.Adam Hoover

Dr.Brian Dean

Introduction Related work Background theory:

◦ Image as a graph◦ Kruskals’ Minimum Spanning Tree ◦ MST based segmentation

Our algorithm Results Conclusion and future work

Thesis overview

Dividing an image to disjoint regions such that similar pixels are grouped together

What is segmentation

Image Courtesy: [3]

Image Segmentation involves division of image I into K regions: R1, R2, R3, … RK such that:

What is segmentation

Every pixel must be assigned to a region

Regions must be disjoint

Size: 1 pixel to the entire image itself

Pixels within a region share certain characteristics that is not found with pixels in another region.

What is segmentation

f is a function that returns TRUE if the region under consideration is homogenous

Biomedical applications◦ Used as a preprocessing step to identify

anatomical regions for medical diagnosis/analysis.

Applications of segmentation

Brain Tissue MRI Segmentation [1]

CT Jaw segmentation [2]

Object recognition systems:◦ Lower level features such as color and texture are

used to segment the image◦ Only relevant segments (subset of pixels) are fed

to the object recognition system. Saves computational cost, especially for

large scale recognition systems

Applications of segmentation

As a preprocessing step in face and iris recognition

Applications of segmentation

Face segmentation Iris Segmentation

Astronomy: Preprocessing step before further analysis

Applications of segmentation

Segmentation of Nebula [4]

What is good segmentation?

Which segmentation is “correct”?

(Manual segmentations from BSDS)

“Correctness”- Are similar pixels grouped together and dissimilar pixels grouped seperately?

Granularity- Extent of resolution of segmentation◦ Consider example in the previous image

What is good segmentation?

There is ambiguity in defining “good”/ “optimal” segmentation.

An image can have multiple segmentations. “correct”◦ Make evaluation /benchmarking of segmentation

algorithm hard

What is good segmentation?

Related work Background theory:

◦ Image as a graph◦ Kruskals’ Minimum Spanning Tree ◦ MST based segmentation

Our algorithm Results Conclusion and future work

Thesis overview

Related work Background theory:

◦ Image as a graph◦ Kruskals’ Minimum Spanning Tree ◦ MST based segmentation

Our algorithm Results Conclusion and future work

Thesis overview

Some of the popular image segmentation approaches are:

Split and Merge approaches Mean Shift and k-means Spectral theory and normalized cuts Minimum spanning tree

Related work

Split and Merge approaches◦ Iteratively split

If evidence of a boundary exists

◦ Iteratively merge Based on similarity

Quad-tree used

Related work

Image Courtesy: [5]

Mean shift and k-means are related. Mean-shift:

◦ Represent each pixel as a vector [color, texture, space]

◦ Define a window around every point. 1. Update the point to the mean of all the points

within the window.2. Repeat until convergence.

Related work: Mean shift and k means

K-means:◦ Represent each pixel as vector [color, texture, space]◦ Choose K initial cluster centers

1. Assign every pixel to its closet cluster center.2. Recompute the means of all the clusters3. Repeat 1-2 until convergence.

Difference between K means and mean-shift:◦ In K-means, K has to be known beforehand◦ K-means sensitive to initial choice of cluster centers

Related work: Mean shift and k means

Represent image as a graph. Using graph cuts, partitions the image into

regions. In Spectral theory and normalized cuts,

Eigenvalues/vectors of the Laplacian matrix is used to determine the cut

Related work: Spectral theory and Normalized cuts

Use Minimum Spanning Tree to segment image.◦ Proposed by Felzenszwalb & Huttenlocher in 2004.◦ Uses a variant of Kruskals MST to segment images◦ Very efficient- O(NlogN) time

Discussed in detail in the next section

Related work: MST based approach

Related work Background theory:

◦ Image as a graph◦ Kruskals’ Minimum Spanning Tree ◦ MST based segmentation

Our algorithm Results Conclusion and future work

Thesis overview

Graph G=(V,E) is an abstract data type containing a set of vertices V and edges E.

Useful operations using a graph:◦ See if path exists between any 2 vertices◦ Find connected components ◦ Check for cycles◦ Find the shortest point between any 2 vertices◦ Compute minimum spanning ◦ Graph partition based on cuts

Graph algorithms are useful in image processing

Background: Image as a graph

Image graph:◦ Pixels/group of pixels form vertices.◦ Vertices connected to form edges◦ Edge weight represents dissimilarity between vertices

Types of image graph:◦ Image grid◦ Complete graph◦ Nearest neighbor graph

Background: Image as a graph

Image grid:◦ Edges: every vertex (pixel) is connected with its

4 (or 8) x-y neighbors.◦ No of edges m= O(N) [Graph operations are

quick]◦ Fails to capture global properties

Background: Image as a graph

Complete graph:◦ Edges: Connect every vertex (pixel) with every

other vertex◦ No of edges m= O(N2)◦ Captures global properties◦ Graph operations are very expensive

Background: Image as a graph

Nearest neighbor graph:◦ Compromise between grid (fails to capture global

properties) and complete graph (too many edges).◦ Represent every vertex as a combination of color

and x-y features. [e.g. (R, G, B, x, y)]◦ Find the K=O(1) neighbors for each pixel using

Approximate nearest neighbor (ANN)◦ Edges: Connect every pixel to K nearest neighbors

Background: Image as a graph

Related work Background theory:

◦ Image as a graph◦ Kruskal’s Minimum Spanning Tree ◦ MST based segmentation

Our algorithm Results Conclusion and future work

Thesis overview

Tree is a graph which is:◦ Connected◦ Has no cycles

Spanning tree: contains all the vertices of graph G◦ A graph can have multiple spanning trees

Minimum spanning tree is a spanning tree which has the least sum of weights of edges

Background: Kruskal’s MST

Background: Kruskal’s MST

• Sorting: O(mlog(m)) time• FindSet and Merge: O(mα(N)) time [very slow growing]

OVERALL TIME: O(m log(m))

Related work Background theory:

◦ Image as a graph◦ Kruskal’s Minimum Spanning Tree ◦ MST based segmentation

Our algorithm Results Conclusion and future work

Thesis overview

Use Minimum Spanning Tree to segment image. In Kruskal’s MST algorithm,

◦ Edges are sorted in ascending order of weights◦ Edges are added in order to the spanning tree as long as a

cycle is not formed.◦ All vertices added to ONE spanning tree

If Kruskal’s is applied directly to image segmentation:◦ We will end up with ONE segment (entire image)

Background: MST Segmentation

Variant of Kruskal’s used in image segmentation.1. Create an image grid graph.2. Sort edges in the increasing order of weights3. For every edge ei in E,

1. If FindSet(ui) ≠ FindSet(vi) AND IsSimilar(ui ,vi)=TRUE

Merge(FindSet(ui) ,FindSet(vi) )

Instead of one MST, we end up with a forest of K trees

◦ Each tree represents a region

Background: MST Segmentation

We add an edge ei connecting regions Ru and Rv to a tree only if :

Background: MST Segmentation

• D(Ru Rv): edge weight connecting vertices u and v• Int(Ri): maximum edge weight in region Ri

WE MERGE IF THE EDGE WEIGHT IS LOWER THAN THE MAXIMUM EDGE WEIGHT IN

EITHER REGIONS!!

Drawback 1: LEAK

Background: MST Segmentation

Felzenszwalb and Huttenlocher 2004

Drawback 2: SENSITIVITY TO PARAMETER k◦ Notice how granularity changes by varying k

Background: MST Segmentation

• k is arbitrary• k is affected by the size of the image

Related work Background theory:

◦ Image as a graph◦ Kruskal’s Minimum Spanning Tree ◦ MST based segmentation

Our algorithm Results Conclusion and future work

Thesis overview

Objective Constructing image grid Sort edges in ascending order For every edge

◦ If Merge criterion is satisfied Merge

Our algorithm: overview

Improve upon the drawbacks of MST ALGORITHM:◦ Addressing Leak:

Represent regions as a Gaussian distribution. Use Bidirectional Mahalanobis distance to compare Gaussians.

◦ Overcome sensitivity to parameter k: Propose parameter τ that is

independent of image size Works well for 2-2.5

Provide a mathematical intuition for it. Propose an approximation that enables real-time

implementation.

Our algorithm: Objective

How to compare two regions in a graph?

MST APPROACH

D(u,v)u v

Int(Ru) Int(Rv)

Check if D(u,v) < Int(Ru) && D(u,v) < Int(Rv)• Leak can happen

How to compare two regions in a graph?

OUR APPROACH

D(u,v)u v

Represent each region as a Gaussian Check if the Gaussians are similar:

Mahalanobis distance is less than 2.5

Objective Constructing image grid Sort edges in ascending order For every edge

◦ If Merge criterion is satisfied Merge

Our algorithm: overview

Initialize Vertices: ◦ Every pixel is mapped to a vertex◦ Information about vertex vi is stored at the ‘i’th entry of

the disjoint set data structure D. The ‘i’th entry in D contains following information:

◦ Root node◦ Zeroth, first and second order moments◦ List of all the edges connected to vertex v i

Our algorithm: Building image grid

Initialize Edges:◦ Between neighboring pixels in x-y space◦ Number of edges m= O(N)◦ Use List to maintain edges

Edge weight: ◦ Euclidean distance between pixels to begin with◦ Mahalanobis distance between Gaussians as region grows

Note that Euclidean distance is a special instance of Mahalanobis distance

Our algorithm: Building image grid

Objective Constructing image grid Sort edges in ascending order For every edge

◦ If Merge criterion is satisfied Merge

Our algorithm: overview

Our algorithm- Naïve approach

Objective Constructing image grid Sort edges in ascending order For every edge

◦ If Merge criterion is satisfied Merge

Our algorithm: overview

While adding edge ei to the MST, regions Ru and Rv are merged if the following criterion is satisfied:

Our algorithm: Merge Criterion

Forces small regions to merge

Bidirectional Mahalanobis

Around 2.5 is a good threshold

Objective Constructing image grid Sort edges in ascending order For every edge

◦ If Merge criterion is satisfied Merge

Our algorithm: overview

Merging regions Ru and Rv

◦ Update information at the root node of the disjoint set data-structure (Similar to MST)

◦ Updating information about root node, zeroth, first and second order moment is easy

However, after merging Ru and Rv

◦ All edges connected to either Ru or Rv have to be updated w.r.t. (Ru ∪

Rv )

◦ The edges have to be re-sorted.

◦ The above operations will slow down the overall running time to O(N2).

Our algorithm: Merge

To speed up weight update that needs to be performed after every iteration,◦ For every region in the DSDS, we store the pointers to

all the edges connected to it.◦ When 2 regions are merged, we merge their neighbor

lists also◦ Assuming that the number of neighbors for every

region is constant, every iteration of merging neighbors can also be accomplished in O(1) time

Our algorithm: Speed-up

Re-sorting the edges after every merge is an expensive operation. ◦ We use skip lists to maintain edges. ◦ Skip list is a data structure that helps maintain sorted

items. Every insert, delete and search operation takes O(logN)

amortized time.

Although the asymptotic running time is O(NplogN), it is still slower than MST

Our algorithm: Speed-up

Do not update weights or re-sort edges after every iteration.

This runs in O(NlogN) time◦ Speed comparable to MST

Our experiments show that the approximated algorithm still improves upon the drawback of MST:◦ Leak◦ Sensitivity to parameter k

Our algorithm: Approximation

Related work Background theory:

◦ Image as a graph◦ Kruskals’ Minimum Spanning Tree ◦ MST based segmentation

Our algorithm Results Conclusion and future work

Thesis overview

Tested the algorithm on:◦ Synthetic images◦ Berkeley Segmentation Dataset

Compared its performance with MST based segmentation algorithm

Results

Our algorithm overcomes leak!

Results

Gradient ramp MST merges the entire image into 1 region

Our algorithm has 2 stable

regions

Effect of parameter tau on granularity

Results

MST based Segmentatio

n

Our algorithm

k is arbitrary! Varies for different image

sizes

• τ represents the distance between Gaussians!• Best results when 2< τ<2.5

Results

We ran the segmentation exhaustively for multiple values of τ

Studied the effect of τ on the number of regions formed

Notice that curve flattens for τ in the range 2-2.5.◦ Represents “stable” regions◦ Segmentation unaffected by

parameter change

ResultsMona Lisa

Man Notice the flat regions

cf. Yu 2007

We ran the algorithm exhaustively on Berkeley Segmentation dataset.

Our algorithm produced more “correct” segmentations than MST segmentations.

Segmentation was sharper in our algorithm. Some specific example illustrated in the

subsequent slides

Results: BSDS dataset

Results

Notice the Man on the Hill

Results

Notice the face of the man kneeling down!

Results

Notice how a small leak has merged grass with part of

bison’s body

Results: Our results are much sharper

Related work Background theory:

◦ Image as a graph◦ Kruskal’s Minimum Spanning Tree ◦ MST based segmentation

Our algorithm Results Conclusion and future work

Thesis overview

Proposed a new segmentation algorithm that improves upon the drawbacks of MST:◦ Leak

Represent regions as Gaussians Use bidirectional Mahalanobis distance to compare

regions◦ Sensitivity to parameter k

τ = 2.5 works well for all images, represents normalized distance between Gaussian distributions

Shown experimentally to be “stable”

Conclusion

In the worst case scenerio, ◦ naïve version of our algorithm runs in O(N2) time. ◦ Using skip list improves to O(NlogN) but still not as

fast as MST An approximated algorithm is proposed:

◦ Runs in O(NlogN) time and speed comparable to MST◦ Still overcomes the two drawbacks of MST based

segmentation

Conclusion

Pre-processing: Homographic filtering Efficiency: Speed up the original version of

the algorithm using more sophisticated priority queues.

Benchmarking: Mathematical study of the accuracy of our segmentation algorithm on BSDS dataset

Future work

[1] http://picsl.upenn.edu/Project/BrainTissueSegmentation[2] http://etidweb.tamu.edu/faculty/beasley/publications/Beasley_Mediviz_VisualizeEdgeWeights_2011.pdf[3] http://www.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/[4] http://atc.udg.edu/~llado/CVpapers/icip11b.pdf[5] Stan Birchfield. Image Segmentation. Lecture Notes

References