Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The...
Transcript of Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The...
![Page 1: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/1.jpg)
Image segmentation
![Page 2: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/2.jpg)
The goals of segmentation• Group together similar-looking pixels for
efficiency of further processing“B tt ”• “Bottom-up” process
• Unsupervised
“superpixels”
X. Ren and J. Malik. Learning a classification model for segmentation.ICCV 2003.
![Page 3: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/3.jpg)
The goals of segmentation• Separate image into coherent “objects”
• “Bottom-up” or “top-down” process?• Supervised or unsupervised?• Supervised or unsupervised?
image human segmentation
Berkeley segmentation database:http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/
![Page 4: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/4.jpg)
Inspiration from psychology• The Gestalt school: Grouping is key to visual
perception
The Muller-Lyer illusion
http://en.wikipedia.org/wiki/Gestalt_psychology
![Page 5: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/5.jpg)
The Gestalt school• Elements in a collection can have properties
that result from relationships “Th h l i t th th f it t ”• “The whole is greater than the sum of its parts”
subjective contours occlusion
familiar configuration
http://en.wikipedia.org/wiki/Gestalt_psychology
![Page 6: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/6.jpg)
Emergence
http://en.wikipedia.org/wiki/Gestalt_psychology
![Page 7: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/7.jpg)
Gestalt factors
![Page 8: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/8.jpg)
Grouping phenomena in real life
Forsyth & Ponce, Figure 14.7
![Page 9: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/9.jpg)
Grouping phenomena in real life
Forsyth & Ponce, Figure 14.7
![Page 10: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/10.jpg)
Gestalt factors
• These factors make intuitive sense but are• These factors make intuitive sense, but are very difficult to translate into algorithms
![Page 11: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/11.jpg)
Segmentation as clustering
Source: K. Grauman
![Page 12: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/12.jpg)
Segmentation as clustering• K-means clustering based on intensity or
color is essentially vector quantization of the i tt ib timage attributes• Clusters don’t have to be spatially coherent
Image Intensity-based clusters Color-based clusters
![Page 13: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/13.jpg)
Segmentation as clustering
Source: K. Grauman
![Page 14: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/14.jpg)
Segmentation as clustering• Clustering based on (r,g,b,x,y) values
enforces more spatial coherence
![Page 15: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/15.jpg)
K-Means for segmentation• Pros
• Very simple method• Converges to a local minimum of the error function• Converges to a local minimum of the error function
• Cons• Memory-intensiveMemory intensive• Need to pick K• Sensitive to initialization• Sensitive to outliers• Sensitive to outliers• Only finds “spherical”
clusters
![Page 16: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/16.jpg)
Mean shift clustering and segmentation• An advanced and versatile technique for
clustering-based segmentation
http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html
D. Comaniciu and P. Meer, Mean Shift: A Robust Approach toward Feature Space Analysis, PAMI 2002.
![Page 17: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/17.jpg)
Mean shift algorithm• The mean shift algorithm seeks modes or local
maxima of density in the feature space
imageFeature space
(L*u*v* color values)
![Page 18: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/18.jpg)
Mean shiftSearchwindow
Center ofmass
Mean Shiftvectorvector
Slide by Y. Ukrainitz & B. Sarel
![Page 19: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/19.jpg)
Mean shiftSearchwindow
Center ofmass
Mean Shiftvectorvector
Slide by Y. Ukrainitz & B. Sarel
![Page 20: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/20.jpg)
Mean shiftSearchwindow
Center ofmass
Mean Shiftvectorvector
Slide by Y. Ukrainitz & B. Sarel
![Page 21: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/21.jpg)
Mean shiftSearchwindow
Center ofmass
Mean Shiftvectorvector
Slide by Y. Ukrainitz & B. Sarel
![Page 22: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/22.jpg)
Mean shiftSearchwindow
Center ofmass
Mean Shiftvectorvector
Slide by Y. Ukrainitz & B. Sarel
![Page 23: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/23.jpg)
Mean shiftSearchwindow
Center ofmass
Mean Shiftvectorvector
Slide by Y. Ukrainitz & B. Sarel
![Page 24: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/24.jpg)
Mean shiftSearchwindow
Center ofmass
Slide by Y. Ukrainitz & B. Sarel
![Page 25: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/25.jpg)
Mean shift clustering• Cluster: all data points in the attraction basin
of a mode• Attraction basin: the region for which all
trajectories lead to the same mode
Slide by Y. Ukrainitz & B. Sarel
![Page 26: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/26.jpg)
Mean shift clustering/segmentation• Find features (color, gradients, texture, etc)• Initialize windows at individual feature points
P f hift f h i d til• Perform mean shift for each window until convergence• Merge windows that end up near the same “peak” or mode
![Page 27: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/27.jpg)
Mean shift segmentation results
http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html
![Page 28: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/28.jpg)
More results
![Page 29: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/29.jpg)
More results
![Page 30: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/30.jpg)
Mean shift pros and cons• Pros
• Does not assume spherical clusters• Just a single parameter (window size)• Just a single parameter (window size) • Finds variable number of modes• Robust to outliers
• Cons• Output depends on window size• Computationally expensive• Computationally expensive• Does not scale well with dimension of feature space
![Page 31: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/31.jpg)
Images as graphs
jwij
i
j
i
• Node for every pixel• Edge between every pair of pixels (or every pair
of “sufficiently close” pixels)E h d i i ht d b th ffi it• Each edge is weighted by the affinity or similarity of the two nodes
Source: S. Seitz
![Page 32: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/32.jpg)
Segmentation by graph partitioning
jwij
i
j
A B C
i
• Break Graph into Segmentsp g• Delete links that cross between segments• Easiest to break links that have low affinity
similar pixels should be in the same segments– similar pixels should be in the same segments– dissimilar pixels should be in different segments
Source: S. Seitz
![Page 33: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/33.jpg)
Measuring affinity• Suppose we represent each pixel by a
feature vector x, and define a distance f ti i t f thi f tfunction appropriate for this feature representation
• Then we can convert the distance between• Then we can convert the distance between two feature vectors into an affinity with the help of a generalized Gaussian kernel:help of a generalized Gaussian kernel:
⎟⎞
⎜⎛ 2)(di t1
⎟⎠⎞
⎜⎝⎛− 2
2 ),(dist2
exp ji xxσ
![Page 34: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/34.jpg)
Scale affects affinity• Small σ: group only nearby points• Large σ: group far-away points
![Page 35: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/35.jpg)
Graph cut
A B
• Set of edges whose removal makes a graph disconnectedC t f t f i ht f t d• Cost of a cut: sum of weights of cut edges
• A graph cut gives us a segmentationWh t i “ d” h t d h d fi d ?• What is a “good” graph cut and how do we find one?
Source: S. Seitz
![Page 36: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/36.jpg)
Minimum cut• We can do segmentation by finding the
minimum cut in a graphEffi i t l ith i t f d i thi• Efficient algorithms exist for doing this
Minimum cut example
![Page 37: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/37.jpg)
Minimum cut• We can do segmentation by finding the
minimum cut in a graphEffi i t l ith i t f d i thi• Efficient algorithms exist for doing this
Minimum cut example
![Page 38: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/38.jpg)
Normalized cut• Drawback: minimum cut tends to cut off very
small, isolated components
i hCuts with lesser weightthan the
Ideal Cutideal cut
* Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 39: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/39.jpg)
Normalized cut• Drawback: minimum cut tends to cut off very
small, isolated components• This can be fixed by normalizing the cut by
the weight of all the edges incident to the segmentsegment
• The normalized cut cost is:
),(),(
),(),(
VBwBAw
VAwBAw
+
w(A, B) = sum of weights of all edges between A and B
J. Shi and J. Malik. Normalized cuts and image segmentation. PAMI 2000
![Page 40: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/40.jpg)
Normalized cut• Let W be the adjacency matrix of the graph• Let D be the diagonal matrix with diagonal
entries D(i, i) = Σj W(i, j) • Then the normalized cut cost can be written as
DyyyWDy
T
T )( −
where y is an indicator vector whose value should be 1 in the ith position if the ith feature
Dyy
ppoint belongs to A and a negative constant otherwise
J. Shi and J. Malik. Normalized cuts and image segmentation. PAMI 2000
![Page 41: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/41.jpg)
Normalized cut• Finding the exact minimum of the normalized cut
cost is NP-complete, but if we relax y to take on bit l th i i i th l darbitrary values, then we can minimize the relaxed
cost by solving the generalized eigenvalue problem (D − W)y = λDyproblem (D W)y = λDy
• The solution y is given by the generalized eigenvector corresponding to the second smallesteigenvector corresponding to the second smallest eigenvalue
• Intutitively, the ith entry of y can be viewed as a y, y y“soft” indication of the component membership of the ith feature• Can use 0 or median value of the entries as the splitting point
(threshold), or find threshold that minimizes the Ncut cost
![Page 42: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/42.jpg)
Normalized cut algorithm1. Represent the image as a weighted graph
G = (V,E), compute the weight of each edge, d i th i f ti i D d Wand summarize the information in D and W
2. Solve (D − W)y = λDy for the eigenvector with the second smallest eigenvaluewith the second smallest eigenvalue
3. Use the entries of the eigenvector to bipartition the graphbipartition the graph
To find more than two clusters:To find more than two clusters:• Recursively bipartition the graph
R k l t i l f• Run k-means clustering on values of several eigenvectors
![Page 43: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/43.jpg)
Example result
![Page 44: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/44.jpg)
Challenge• How to segment images that are a “mosaic of
textures”?
![Page 45: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/45.jpg)
Using texture features for segmentation• Convolve image with a bank of filters
J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001.
![Page 46: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/46.jpg)
Using texture features for segmentation• Convolve image with a bank of filters• Find textons by clustering vectors of filter bank
outputsTexton mapImage
J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001.
![Page 47: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/47.jpg)
Using texture features for segmentation• Convolve image with a bank of filters• Find textons by clustering vectors of filter bank
outputs• The final texture feature is a texton histogram
t d i i d t “l lcomputed over image windows at some “local scale”
J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001.
![Page 48: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/48.jpg)
Pitfall of texture features
• Possible solution: check for “interveningPossible solution: check for intervening contours” when computing connection weights
J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001.
![Page 49: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/49.jpg)
Example results
![Page 50: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/50.jpg)
Results: Berkeley Segmentation Engine
http://www.cs.berkeley.edu/~fowlkes/BSE/
![Page 51: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/51.jpg)
Normalized cuts: Pro and con• Pros
• Generic framework, can be used with many different features and affinity formulationsfeatures and affinity formulations
• Cons• High storage requirement and time complexity• Bias towards partitioning into equal segments
![Page 52: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/52.jpg)
Segments as primitives for recognition
J. Tighe and S. Lazebnik, ECCV 2010
![Page 53: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/53.jpg)
Top-down segmentation
E B t i d S Ull “Cl ifi t dE. Borenstein and S. Ullman, “Class-specific, top-down segmentation,” ECCV 2002
A Levin and Y Weiss “Learning to Combine Bottom UpA. Levin and Y. Weiss, “Learning to Combine Bottom-Up and Top-Down Segmentation,” ECCV 2006.
![Page 54: Image segmentation - Computer Sciencecs.unc.edu/~lazebnik/spring11/lec25_segmentation.pdf · The Gestalt school • Elements in a collection can have properties that result from relationships](https://reader030.fdocuments.us/reader030/viewer/2022032611/5b573bb27f8b9a022e8d7f86/html5/thumbnails/54.jpg)
Top-down segmentation
Normalized cuts
Top down
E B t i d S Ull “Cl ifi t d
Top-down segmentation
E. Borenstein and S. Ullman, “Class-specific, top-down segmentation,” ECCV 2002
A Levin and Y Weiss “Learning to Combine Bottom UpA. Levin and Y. Weiss, “Learning to Combine Bottom-Up and Top-Down Segmentation,” ECCV 2006.