Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth,...
-
date post
22-Dec-2015 -
Category
Documents
-
view
217 -
download
1
Transcript of Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth,...
![Page 1: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/1.jpg)
ComputerVision
Segmentation
Marc PollefeysCOMP 256
Some slides and illustrations from D. Forsyth, T. Darrel,...
![Page 2: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/2.jpg)
ComputerVision
Aug 26/28 - Introduction
Sep 2/4 Cameras Radiometry
Sep 9/11 Sources & Shadows Color
Sep 16/18 Linear filters & edges
(hurricane Isabel)
Sep 23/25 Pyramids & Texture Multi-View Geometry
Sep30/Oct2 Stereo Project proposals
Oct 7/9 Tracking (Welch) Optical flow
Oct 14/16 - -
Oct 21/23 Silhouettes/carving (Fall break)
Oct 28/30 - Structure from motion
Nov 4/6 Project update Proj. SfM
Nov 11/13 Camera calibration Segmentation
Nov 18/20 Fitting Prob. segm.&fit.
Nov 25/27 Matching templates (Thanksgiving)
Dec 2/4 Matching relations Range data
Dec ? Final project
Tentative class schedule
![Page 3: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/3.jpg)
ComputerVision Segmentation and Grouping
• Motivation: not information is evidence
• Obtain a compact representation from an image/motion sequence/set of tokens
• Should support application
• Broad theory is absent at present
• Grouping (or clustering)– collect together
tokens that “belong together”
• Fitting– associate a model
with tokens– issues
• which model?• which token goes
to which element?• how many
elements in the model?
![Page 4: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/4.jpg)
ComputerVision General ideas
• tokens– whatever we need to
group (pixels, points, surface elements, etc., etc.)
• top down segmentation– tokens belong
together because they lie on the same object
• bottom up segmentation– tokens belong
together because they are locally coherent
• These two are not mutually exclusive
![Page 5: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/5.jpg)
ComputerVision
Why do these tokens belong together?
![Page 6: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/6.jpg)
ComputerVision
![Page 7: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/7.jpg)
ComputerVision
Basic ideas of grouping in humans
• Figure-ground discrimination– grouping can be
seen in terms of allocating some elements to a figure, some to ground
– impoverished theory
• Gestalt properties– elements in a
collection of elements can have properties that result from relationships (Muller-Lyer effect)
• gestaltqualitat
– A series of factors affect whether elements should be grouped together
• Gestalt factors
![Page 8: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/8.jpg)
ComputerVision
![Page 9: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/9.jpg)
ComputerVision
![Page 10: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/10.jpg)
ComputerVision
![Page 11: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/11.jpg)
ComputerVision
![Page 12: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/12.jpg)
ComputerVision
![Page 13: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/13.jpg)
ComputerVision
![Page 14: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/14.jpg)
ComputerVision
![Page 15: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/15.jpg)
ComputerVision
Technique: Shot Boundary Detection
• Find the shots in a sequence of video– shot boundaries
usually result in big differences between succeeding frames
• Strategy:– compute interframe
distances– declare a boundary
where these are big
• Possible distances– frame differences– histogram differences– block comparisons– edge differences
• Applications:– representation for
movies, or video sequences
• find shot boundaries• obtain “most
representative” frame
– supports search
![Page 16: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/16.jpg)
ComputerVision
Technique: Background Subtraction
• If we know what the background looks like, it is easy to identify “interesting bits”
• Applications– Person in an office– Tracking cars on a
road– surveillance
• Approach:– use a moving
average to estimate background image
– subtract from current frame
– large absolute values are interesting pixels
• trick: use morphological operations to clean up pixels
![Page 17: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/17.jpg)
ComputerVision
![Page 18: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/18.jpg)
ComputerVision
![Page 19: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/19.jpg)
ComputerVision
![Page 20: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/20.jpg)
ComputerVision
![Page 21: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/21.jpg)
ComputerVision
![Page 22: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/22.jpg)
ComputerVision
![Page 23: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/23.jpg)
ComputerVision
![Page 24: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/24.jpg)
ComputerVision
![Page 25: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/25.jpg)
ComputerVision
![Page 26: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/26.jpg)
ComputerVision
![Page 27: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/27.jpg)
ComputerVision Segmentation as clustering
• Cluster together (pixels, tokens, etc.) that belong together
• Agglomerative clustering– attach closest to
cluster it is closest to– repeat
• Divisive clustering– split cluster along best
boundary– repeat
• Point-Cluster distance– single-link clustering– complete-link
clustering– group-average
clustering
• Dendrograms– yield a picture of
output as clustering process continues
![Page 28: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/28.jpg)
ComputerVision Simple clustering algorithms
![Page 29: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/29.jpg)
ComputerVision
![Page 30: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/30.jpg)
ComputerVision K-Means
• Choose a fixed number of clusters• Choose cluster centers and point-cluster allocations
to minimize error
• can’t do this by search, because there are too many possible allocations.
x j i2
jelements of i'th cluster
iclusters
![Page 31: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/31.jpg)
ComputerVision
K-means clustering using intensity alone and color alone
Image Clusters on intensity Clusters on color
![Page 32: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/32.jpg)
ComputerVision
K-means using color alone, 11 segments
Image Clusters on color
![Page 33: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/33.jpg)
ComputerVision
K-means usingcolor alone,11 segments.
![Page 34: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/34.jpg)
ComputerVision
K-means using colour andposition, 20 segments
![Page 35: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/35.jpg)
ComputerVision Graph theoretic clustering
• Represent tokens using a weighted graph.– affinity matrix
• Cut up this graph to get subgraphs with strong interior links
![Page 36: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/36.jpg)
ComputerVision
![Page 37: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/37.jpg)
ComputerVision
![Page 38: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/38.jpg)
ComputerVision
![Page 39: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/39.jpg)
ComputerVision
![Page 40: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/40.jpg)
ComputerVision
![Page 41: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/41.jpg)
ComputerVision Measuring Affinity
Intensity
Texture
Distance
aff x, y exp 12 i
2
I x I y 2
aff x, y exp 12 d
2
x y
2
aff x, y exp 12 t
2
c x c y 2
![Page 42: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/42.jpg)
ComputerVision Scale affects affinity
![Page 43: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/43.jpg)
ComputerVision Eigenvectors and cuts
• Simplest idea: we want a vector a giving the association between each element and a cluster
• We want elements within this cluster to, on the whole, have strong affinity with one another
• We could maximize
• But need the constraint
• This is an eigenvalue problem - choose the eigenvector of A with largest eigenvalue
aTAa
aTa1
![Page 44: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/44.jpg)
ComputerVision Example eigenvector
points
matrix
eigenvector
![Page 45: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/45.jpg)
ComputerVision
![Page 46: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/46.jpg)
ComputerVision More than two segments
• Two options– Recursively split each side to get a tree,
continuing till the eigenvalues are too small
– Use the other eigenvectors
![Page 47: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/47.jpg)
ComputerVision More than two segments
![Page 48: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/48.jpg)
ComputerVision Normalized cuts
• Current criterion evaluates within cluster similarity, but not across cluster difference
• Instead, we’d like to maximize the within cluster similarity compared to the across cluster difference
• Write graph as V, one cluster as A and the other as B
• Maximize
• i.e. construct A, B such that their within cluster similarity is high compared to their association with the rest of the graph
assoc(A,A)assoc(A,V )
assoc(B,B)assoc(B,V )
![Page 49: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/49.jpg)
ComputerVision
![Page 50: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/50.jpg)
ComputerVision
![Page 51: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/51.jpg)
ComputerVision Normalized cuts
• Write a vector y whose elements are 1 if item is in A, -b if it’s in B
• Write the matrix of the graph as W, and the matrix which has the row sums of W on its diagonal as D, 1 is the vector with all ones.
• Criterion becomes
• and we have a constraint
• This is hard to do, because y’s values are quantized
miny
yT D W yyTDy
yTD1 0
![Page 52: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/52.jpg)
ComputerVision
![Page 53: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/53.jpg)
ComputerVision Normalized cuts
• Instead, solve the generalized eigenvalue problem
• which gives
• Now look for a quantization threshold that maximises the criterion --- i.e all components of y above that threshold go to one, all below go to -b
maxy yT D W y subject to yTDy 1
D W yDy
![Page 54: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/54.jpg)
ComputerVision
Figure from “Image and video segmentation: the normalised cut framework”, by Shi and Malik, copyright IEEE, 1998
![Page 55: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/55.jpg)
ComputerVision
Figure from “Normalized cuts and image segmentation,” Shi and Malik, copyright IEEE, 2000
![Page 56: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/56.jpg)
ComputerVision Image parsing:
Unifying Segmentation, Detection and RecognitionTu, Chen, Yuille & Zhu ICCV’03 (Marr prize!)
![Page 57: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/57.jpg)
ComputerVision Image parsing
• Generative models for each classsome random samples for 5 and H
some random face samples (PCA model)
![Page 58: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/58.jpg)
ComputerVision DDMCMC
Data Driven Markov Chain Monte Carlo
![Page 59: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/59.jpg)
ComputerVision Image parsing
![Page 60: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/60.jpg)
ComputerVision Image parsing
![Page 61: Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...](https://reader035.fdocuments.us/reader035/viewer/2022081519/56649d7a5503460f94a5dc1b/html5/thumbnails/61.jpg)
ComputerVision Next class:
Fitting
Reading: Chapter 15