Transcript of Perceptual Organization: Segmentation and Optical Flow.
- Slide 1
- Perceptual Organization: Segmentation and Optical Flow
- Slide 2
- Inspiration from psychology The Gestalt school: Grouping is key
to visual perception The whole is greater than the sum of its parts
http://en.wikipedia.org/wiki/Gestalt_psychology subjective contours
occlusion familiar configuration
- Slide 3
- Gestalt grouping factors
- Slide 4
- Emergence http://en.wikipedia.org/wiki/Gestalt_psychology
- Slide 5
- Motion and perceptual organization Even impoverished motion
data can evoke a strong percept G. Johansson, Visual Perception of
Biological Motion and a Model For Its Analysis", Perception and
Psychophysics 14, 201-211, 1973. YouTube video
- Slide 6
- Image segmentation
- Slide 7
- The goals of segmentation Obtain primitives for other tasks
Perceptual organization, recognition Graphics, image
manipulation
- Slide 8
- Goal 1: Primitives for other tasks Group together
similar-looking pixels for efficiency of further processing
Bottom-up process Unsupervised X. Ren and J. Malik. Learning a
classification model for segmentation. ICCV 2003.Learning a
classification model for segmentation. superpixels
- Slide 9
- Image parsing or semantic segmentation: Segments as primitives
for recognition J. Tighe and S. Lazebnik, ECCV 2010, IJCV 2013
- Slide 10
- Goal 2: Recognition Separate image into coherent objects
Bottom-up or top-down process? Supervised or unsupervised? Berkeley
segmentation database:
http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/
http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/
image human segmentation
- Slide 11
- Goal 3: Image manipulation Interactive segmentation for
graphics
- Slide 12
- Approaches to segmentation Segmentation as clustering
Segmentation as graph partitioning Segmentation as labeling
- Slide 13
- Segmentation as clustering Source: K. Grauman
- Slide 14
- Image Intensity-based clustersColor-based clusters Segmentation
as clustering K-means clustering based on intensity or color is
essentially vector quantization of the image attributes Clusters
dont have to be spatially coherent
- Slide 15
- Segmentation as clustering Source: K. Grauman
- Slide 16
- Segmentation as clustering Clustering based on (r,g,b,x,y)
values enforces more spatial coherence
- Slide 17
- Segmentation as graph partitioning Node for every pixel Edge
between every pair of pixels (or every pair of sufficiently close
pixels) Each edge is weighted by the affinity or similarity of the
two nodes w ij i j Source: S. Seitz
- Slide 18
- Measuring affinity Represent each pixel by a feature vector x
and define an appropriate distance function small large Role
of
- Slide 19
- Segmentation as graph partitioning Break Graph into Segments
Delete links that cross between segments Easiest to break links
that have low affinity similar pixels should be in the same
segments dissimilar pixels should be in different segments ABC
Source: S. Seitz w ij i j
- Slide 20
- Graph cut Set of edges whose removal makes a graph disconnected
Cost of a cut: sum of weights of cut edges A graph cut gives us a
segmentation What is a good graph cut and how do we find one? A B
Source: S. Seitz
- Slide 21
- Minimum cut We can do segmentation by finding the minimum cut
in a graph Efficient algorithms exist for doing this Minimum cut
example
- Slide 22
- Minimum cut We can do segmentation by finding the minimum cut
in a graph Efficient algorithms exist for doing this Minimum cut
example
- Slide 23
- Normalized cut Drawback: minimum cut tends to cut off very
small, isolated components Ideal Cut Cuts with lesser weight than
the ideal cut * Slide from Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
- Slide 24
- Normalized cut To encourage larger segments, normalize the cut
by the total weight of edges incident to the segment The normalized
cut cost is: Intuition: big segments will have a large w(A,V), thus
decreasing ncut(A, B) Finding the globally optimal cut is
NP-complete, but a relaxed version can be solved using a
generalized eigenvalue problem w(A, B) = sum of weights of all
edges between A and B J. Shi and J. Malik. Normalized cuts and
image segmentation. PAMI 2000Normalized cuts and image
segmentation.
- Slide 25
- Normalized cut: Algorithm Let W be the affinity matrix of the
graph (n x n for n pixels) Let D be the diagonal matrix with
entries D(i, i) = j W(i, j) Solve generalized eigenvalue problem (D
W)y = Dy for the eigenvector with the second smallest eigenvalue
The ith entry of y can be viewed as a soft indicator of the
component membership of the ith pixel Use 0 or median value of the
entries of y to split the graph into two components To find more
than two components: Recursively bipartition the graph Run k-means
clustering on values of several eigenvectors
- Slide 26
- Example result
- Slide 27
- Challenge How to define affinities for segmenting highly
textured images?
- Slide 28
- Segmenting textured images Convolve image with a bank of
filters Find textons by clustering vectors of filter bank outputs
J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture
Analysis for Image Segmentation". IJCV 43(1),7-27,2001."Contour and
Texture Analysis for Image Segmentation" Texton mapImage Filter
bank
- Slide 29
- Segmenting textured images Convolve image with a bank of
filters Find textons by clustering vectors of filter bank outputs
Represent pixels by texton histograms computed over neighborhoods
at some local scale Define affinities as similarities between local
texton histograms J. Malik, S. Belongie, T. Leung and J. Shi.
"Contour and Texture Analysis for Image Segmentation". IJCV
43(1),7-27,2001."Contour and Texture Analysis for Image
Segmentation"
- Slide 30
- Pitfall of texture features Possible solution: check for
intervening contours when computing affinities J. Malik, S.
Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for
Image Segmentation". IJCV 43(1),7-27,2001."Contour and Texture
Analysis for Image Segmentation"
- Slide 31
- Results: Berkeley Segmentation Engine
http://www.cs.berkeley.edu/~fowlkes/BSE/
- Slide 32
- Berkeley Segmentation Engine
http://www.cs.berkeley.edu/~fowlkes/BSE/
- Slide 33
- Pro Generic framework, can be used with many different features
and affinity formulations Con High storage requirement and time
complexity: involves solving a generalized eigenvalue problem of
size n x n, where n is the number of pixels Normalized cuts: Pro
and con
- Slide 34
- Efficient graph-based segmentation Runs in time nearly linear
in the number of edges Easy to control coarseness of segmentations
Results can be unstable P. Felzenszwalb and D. Huttenlocher,
Efficient Graph-Based Image Segmentation, IJCV 2004Efficient
Graph-Based Image Segmentation
- Slide 35
- Segmentation as labeling Suppose we want to segment an image
into foreground and background Binary labeling problem Credit: N.
Snavely
- Slide 36
- Segmentation as labeling Suppose we want to segment an image
into foreground and background Binary labeling problem User
sketches out a few strokes on foreground and background How do we
label the rest of the pixels? Source: N. Snavely
- Slide 37
- Binary segmentation as energy minimization Define a labeling L
as an assignment of each pixel with a 0-1 label (background or
foreground) Find the labeling L that minimizes data term smoothness
term How similar is each labeled pixel to the foreground or
background? Encourage spatially coherent segments Source: N.
Snavely
- Slide 38
- : distance from pixel to background : distance from pixel to
foreground { computed by creating a color model from user- labeled
pixels Source: N. Snavely
- Slide 39
- Slide 40
- Neighboring pixels should generally have the same labels Unless
the pixels have very different intensities : similarity in
intensity of p and q = 10.0 = 0.1 Source: N. Snavely
- Slide 41
- Binary segmentation as energy minimization For this problem, we
can efficiently find the global minimum using the max flow / min
cut algorithm Source: N. Snavely Y. Boykov and M.-P. Jolly,
Interactive Graph Cuts for Optimal Boundary and Region Segmentation
of Objects in N-D Images, ICCV 2001Interactive Graph Cuts for
Optimal Boundary and Region Segmentation of Objects in N-D
Images
- Slide 42
- Recall: Stereo as energy minimization I1I1 I2I2 D Energy
functions of this form can be minimized using graph cuts Y. Boykov,
O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via
Graph Cuts, PAMI 2001Fast Approximate Energy Minimization via Graph
Cuts W1(i )W1(i )W 2 (i+D(i )) D(i )D(i ) data term smoothness
term
- Slide 43
- GrabCut C. Rother, V. Kolmogorov, and A. Blake, GrabCut
Interactive Foreground Extraction using Iterated Graph Cuts,
SIGGRAPH 2004GrabCut Interactive Foreground Extraction using
Iterated Graph Cuts