Image Segmentation
description
Transcript of Image Segmentation
![Page 1: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/1.jpg)
Image SegmentationRob AtlasNick Bridle
Evan Radkoff
![Page 2: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/2.jpg)
Image Segmentation
Separate an image into sets of pixels which represent some structure in the image
Main application is object and boundary detection
![Page 3: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/3.jpg)
High Level Overview of Papers
Image Segmentation
Mean Shift
Interactive Cuts
Normalized Cuts
Local approaches Graph cuts
Comaniciu et al., 1999 Shi and Malik, 2000
Rother et al., 2004Li et al., 2004
![Page 4: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/4.jpg)
Mean Shift Analysis
Dorin Comaniciu, Peter Meer
Rutgers University, ICCV 1999
![Page 5: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/5.jpg)
Spatial-Range Domain
Spatial domain: pixel locationsRange domain: intensity valuesSpatial-Range domain: concatenation of the two
{ x, y, R, G, B }Parameters:∂s: Spatial domain normalization∂r: Range domain normalization
{ x / ∂s , y / ∂s , R / ∂r , G / ∂r , B / ∂r }
5 dimensions for RGB, 3 for grayscale
![Page 6: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/6.jpg)
Kernel Density Estimate
For n points {xi}i=1..n in a d-dimensional Euclidean space with kernel function K(x) and window radius h:
![Page 7: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/7.jpg)
Epanechnikov Kernel
cd : volume of the unit d-dimensional sphere
![Page 8: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/8.jpg)
Estimating the Density Gradient
Goal:
where Sh is a hypersphere containing nx data points
![Page 9: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/9.jpg)
Mean Shift Vector
Vector with a direction towards the largest increase in density, thus leading to a local density maximum.
![Page 10: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/10.jpg)
Mean Shift Procedure
1) Compute the mean shift vector Mh(x)2) Translate the window Sh(x) by Mh(x)3) If not converged, go to step 1.
Mathematically guaranteed to converge
![Page 11: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/11.jpg)
Applying Mean Shift to Images
Spatial-Range Domain: { x / ∂s , y / ∂s , R / ∂r , G / ∂r , B / ∂r }
The authors apply the mean shift procedure to all points in the spatial-range domain.
Two applications: Filtering, SegmentationTwo parameters: spatial resolution and range resolution
![Page 12: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/12.jpg)
Mean Shift Filteringxi: source pixel zi: filtered pixelfor each xiapply the mean shift procedure on xi to find the
convergence point yispacial domain of zi = spacial domain of xirange domain of zi = range domain of yi
![Page 13: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/13.jpg)
Results
![Page 14: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/14.jpg)
Results
![Page 15: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/15.jpg)
Results
![Page 16: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/16.jpg)
Segmentation
Instead of changing pixels ranges (colors), group them by which point their mean shift procedure converges to.
![Page 17: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/17.jpg)
Advantages:
No oversight requiredPreserves detail when appropriate
Disadvantages:
Cannot choose how many segments are made
![Page 18: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/18.jpg)
Normalized Cuts and Image Segmentation...
Shi and Malik, 2000
![Page 19: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/19.jpg)
Basic Idea
Treat image segmentation as a graph-partitioning problem
Big Questions:
1. How do we define a good partitioning?2. How do we do partition efficiently?
![Page 20: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/20.jpg)
Images as a graph
G=(V,E)
Vertices: pixels
Edges: similarity
![Page 21: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/21.jpg)
Graph cuts
Assumes:Graph is fully connected
Given: G=(V,E)Do:
Create disjoint vertex sets A, B s.t.
and
![Page 22: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/22.jpg)
Similarity metric
Weight between edges in graph:
X(i): Location of pixel iF(i): Feature vector describing pixel i
Simple case - F(i)=1 for segmenting pointsIntensity, color, texture
, : Parameters
![Page 23: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/23.jpg)
2-way graph cut
Minimize:
Existing efficient methods for optimal 2-way cut
![Page 24: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/24.jpg)
Recursive minimum cuts
Iteratively bipartition graph according to minimum cut (Wu and Leahy, 1993)
The problem: favors small clusters
![Page 25: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/25.jpg)
Want to minimize cut score
But, we also want to partition out larger areas with more connections
So, what we're really interested in is the proportion of total weights that we are cutting for each segment
Intuition for normalized cuts
![Page 26: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/26.jpg)
Normalized cuts
Goal: partition into disassociated groups
![Page 27: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/27.jpg)
Minimizing association between groups ~ maximizing association within groups
![Page 28: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/28.jpg)
Computational complexity
Minimizing normalized cut: NP-complete
Can compute approximate solution by solving a generalized eigenvalue problem
![Page 29: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/29.jpg)
Eigenvalue system
D - diagonal matrix of the total weight from each node to every other
W - matrix of weights between nodes
y - eigenvectors
lambda - eigenvalues
However, this is not in standard form for solving
![Page 30: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/30.jpg)
A solvable representation
First eigenvector (eigenvalue=0):
![Page 31: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/31.jpg)
Second smallest eigenvalue describes solution
Remember, the smallest eigenvalue is 0
![Page 32: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/32.jpg)
Recursive two-way Ncut
Third, fourth, etc eigenvalues correspond to eigenvectors that subdivide the existing graphs
However, error accumulates with each
Better to recompute partitioning on each subgraph iteratively
![Page 33: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/33.jpg)
Segmentation Algorithm
1. Create weighted graph G=(V,E)2. Solve and take the eigenvector with the second smallest eigenvalue3. Bipartition the graph with this eigenvector 4. If Ncut is below threshold value, recursively partition each segment
![Page 34: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/34.jpg)
Partitioning by an eigenvector
The partitioning eigenvector contains one continuous value per pixel
Since these are continuous, not discrete, they don't directly tell us which points are in which segment
Instead, we try different cutoffs, and select the one that produces the minimal NCut value
![Page 35: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/35.jpg)
Partitioning by an eigenvector
![Page 36: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/36.jpg)
Examples - segmenting point sets
![Page 37: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/37.jpg)
Examples - segmenting noisy data
![Page 38: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/38.jpg)
Examples - images
![Page 39: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/39.jpg)
Example - image with texture metric
![Page 40: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/40.jpg)
Interactive Graph Cuts
User can provide hints about where the objects are in the scene
Ideally, the less work the user has to do, the better
![Page 41: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/41.jpg)
Foreground Extraction
Goal is to partition the pixels into two sets: foreground and background
![Page 42: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/42.jpg)
Foreground Extraction
Given pixel intensity values
Output set of alphas
which determine how much each pixel belongs to the foreground
"Hard segmentation":
"Soft segmentation":
![Page 43: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/43.jpg)
GrabCut
Rother et al.,SIGGRAPH 2004
![Page 44: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/44.jpg)
GrabCut: Handling Color Images
Previous work: histogram of gray values for foreground and background
GrabCut:- histograms are intractable for color- instead use Gaussian Mixture Model (GMM) to represent the distribution of color
![Page 45: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/45.jpg)
GrabCut: Algorithm
![Page 46: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/46.jpg)
GrabCut: Border Matting
Perform hard segmentation on image, and then relax alpha values on the border of the object
![Page 47: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/47.jpg)
![Page 48: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/48.jpg)
Lazy Snapping
Li et al., SIGGRAPH 2004
![Page 49: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/49.jpg)
Lazy Snapping: Energy Function
Minimize the Gibbs energy function
Run k-means clustering on foreground and background points
E1 defined in terms of these mean colors:
![Page 50: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/50.jpg)
Lazy Snapping: Speeding it up
Speed is crucial for a responsive user experience
Use the watershed algorithm to divide the image into small approximately constant patches, then do graph cut using these as the nodes
![Page 51: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/51.jpg)
Lazy Snapping: Boundary Editing
Want boundary to "snap" to the user-defined polygon
Energy term E2 becomes
![Page 52: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/52.jpg)
Lazy Snapping: Results
![Page 53: Image Segmentation](https://reader036.fdocuments.us/reader036/viewer/2022081512/56814589550346895db27181/html5/thumbnails/53.jpg)
Thanks for listening.
Questions?