Photo by Svetlana Lazebnik Which parts are hard to model? From Alexei Efros.
The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D)...
-
Upload
arline-robbins -
Category
Documents
-
view
220 -
download
7
Transcript of The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D)...
![Page 1: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/1.jpg)
The University of
Ontario
CS 433/653
Algorithms for Image Analysis
Segmentation (2D)
Acknowledgements: Alexei Efros, Steven Seitz
![Page 2: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/2.jpg)
The University of
Ontario
CS 433/653 Algorithms for Image Analysis Segmentation (2D)
Blobs• Need for blobs• Extracting blobs
– Thresholding, region growing, mean-shift– Minimum spanning trees
Object extraction• Intelligent scissors (also known as live-wire)• Snakes• Region+boundary methods
Local, greedy, and global processing
Extra Reading: Sonka at.al. Ch. 5 Gonzalez and Woods, Ch. 10
![Page 3: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/3.jpg)
The University of
OntarioHW assignment 1
2D image segmentation• Live-wire (based on Dijkstra algorithm for shortest
paths on a graph)• Implement and experiment on two images• Bonus for comparing with local/greedy methods• Some useful code libraries and sample images will
be posted within a couple of days
![Page 4: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/4.jpg)
The University of
OntarioGoal: Extract “Blobs”
What are “blobs”?• Regions of an image that are somehow coherent
Why?• Object extraction, object removal, compositing, etc.• …but are “blobs” objects?• No, not in general
![Page 5: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/5.jpg)
The University of
OntarioBlob’s coherence
Simplest way to define blob coherence is as similarity in brightness or color:
The tools become blobsThe house, grass, and sky make different blobs
![Page 9: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/9.jpg)
The University of
OntarioThresholding
Basic segmentation operation:mask(x,y) = 1 if im(x,y) > T mask(x,y) = 0 if im(x,y) < T
T is threshold• User-defined• Or automatic
Same as histogram partitioning:
![Page 12: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/12.jpg)
The University of
OntarioRegion growing
Is this same as global threshold?
• Start with initial set of pixels K (initial seed(s))• Add to pixels p in K their neighbors q if |Ip-Iq| < T • Repeat until nothing changes
![Page 14: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/14.jpg)
The University of
Ontario
What can go wrongwith region growing ?
Region growth may “leak” through a single week spot
in the boundary
![Page 15: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/15.jpg)
The University of
OntarioRegion growing
See region leaks into sky due to a weak boundary
between them
![Page 16: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/16.jpg)
The University of
Ontario
Due to Pedro Felzenszwalb and Daniel Huttenlocher
Motivating example
This image has three perceptually distinct regions
Difference along border between A and B is less then differences
within C
A CB Q: Where would image thresholding fail?
Q: Where would region growing fail?
A: Region A would be divided in two sets and region C will be split into a large number of arbitrary small subsets
A: Either A and B are merged or region C is split into many small subsets Also, B and C are merged
![Page 17: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/17.jpg)
The University of
OntarioColor-Based Blob Segmentation
Automatic Histogram Partitioning• Given image with N colors, choose K• Each of the K colors defines a region
– not necessarily contiguous
• Performed by computing color histogram, looking for modes
• This is what happens when you downsample image color range, for instance in Photoshop
![Page 18: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/18.jpg)
The University of
OntarioFinding Modes in a Histogram
How Many Modes Are There?• Easy to see, hard to compute
![Page 19: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/19.jpg)
The University of
OntarioMean Shift [Comaniciu & Meer]
1. Initialize random seed, and fixed window
2. Calculate center of gravity ‘x’ of the window (the“mean”)
3. Translate the search window to the mean
4. Repeat Step 2 until convergence
Iter
ativ
e M
ode
Sea
rch
x
o
x x
mode
![Page 21: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/21.jpg)
The University of
OntarioMean-shift results
More Examples: http://www.caip.rutgers.edu/~comanici/segm_images.html
![Page 22: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/22.jpg)
The University of
OntarioIssues:
Although often useful, all these approaches work only some of the time, and are considered rather “hacky”.
Can’t even handle our tiger:
Problem is that blobs != objects!
![Page 24: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/24.jpg)
The University of
OntarioExtracting objects
Many approaches proposed• color cues• region cues• contour cues
We will consider a few of these Today:
• Intelligent Scissors (contour-based)– E. N. Mortensen and W. A. Barrett, Intelligent Scissors for Image Composition, in
ACM Computer Graphics (SIGGRAPH `95), pp. 191-198, 1995
![Page 26: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/26.jpg)
The University of
OntarioIntelligent Scissors
Approach answers a basic question• Q: how to find a path from seed to mouse that follows object boundary
as closely as possible?• A: define a path that stays as close as possible to edges
![Page 27: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/27.jpg)
The University of
OntarioIntelligent Scissors Basic Idea
• Define edge score for each pixel– edge pixels have low cost
• Find lowest cost path from seed to mouse
seed
mouse
Questions• How to define costs?• How to find the path?
![Page 28: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/28.jpg)
The University of
OntarioPath Search (basic idea) Graph Search Algorithm
• Computes minimum cost path from seed to all other pixels
Note: diagonal “paths” are scaled by a factor .
2 … Why?
![Page 29: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/29.jpg)
The University of
Ontario
Good segmentation method shouldhave Invariance to Image Rotation
After object rotation
L
L
Path’s cost along the top boundary Path’s cost along the top-left boundary
147*2 105*2 After diagonal links are adjusted by2
145*2*2
2 2 2 2 2 2 2
2
2
2
2
2
image gradient scores
![Page 30: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/30.jpg)
The University of
Ontario
Instead of nodes, image gradient scores can be assigned directly to graph edges
Graph• node for every pixel p• link between every adjacent pair of pixels, p,q• cost c for each link
Note: each link has a cost• this is a little different than the figure before
where each pixel (graph node) had a cost
p
qc
Treat the image as a graph
![Page 31: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/31.jpg)
The University of
OntarioDefining the costs
Treat the image as a graph
Want to hug image edges: how to define cost of a link?
p
qc
• the link should follow the intensity edge– want intensity to change rapidly to the link
• c - |difference of intensity to the link|
T
T
![Page 32: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/32.jpg)
The University of
OntarioDefining the costs
p
qc
c can be computed using a cross-correlation filter• assume it is centered at p
Also typically scale c by its length• set c = (max-|filter response|)
– where max = maximum |filter response| over all pixels in the image
![Page 33: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/33.jpg)
The University of
OntarioDefining the costs
p
qc 1
-1w-1-1
11
c can be computed using a cross-correlation filter• assume it is centered at p
Also typically scale c by its length• set c = (max-|filter response|)
– where max = maximum |filter response| over all pixels in the image
Why no scalar?2
![Page 34: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/34.jpg)
The University of
Ontario
(see Cormen et.al. “Introduction to Algorithms”, p.595) Dijkstra’s shortest path algorithm
0531
33
4 9
2
link cost
1. init node costs to , set p = seed point, cost(p) = 02. expand p as follows:
for each of p’s neighbors q that are not expanded• set cost(q) = min( cost(p) + cpq, cost(q) )
ALG
OR
ITH
M
![Page 35: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/35.jpg)
The University of
OntarioDijkstra’s shortest path algorithm
4
1 0
5
3
3 2 3
9
531
33
4 9
2
11
1. init node costs to , set p = seed point, cost(p) = 02. expand p as follows:
for each of p’s neighbors q that are not expanded• set cost(q) = min( cost(p) + cpq, cost(q) )
– if q’s cost changed, make q point back to p• put q on the ACTIVE list (if not already there)
ALG
OR
ITH
M
![Page 36: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/36.jpg)
The University of
OntarioDijkstra’s shortest path algorithm
4
1 0
5
3
3 2 3
9
531
33
4 9
2
15
233
3 2
4
1. init node costs to , set p = seed point, cost(p) = 02. expand p as follows:
for each of p’s neighbors q that are not expanded• set cost(q) = min( cost(p) + cpq, cost(q) )
– if q’s cost changed, make q point back to p• put q on the ACTIVE list (if not already there)
3. set r = node with minimum cost on the ACTIVE list4. repeat Step 2 for p = r
ALG
OR
ITH
M
![Page 37: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/37.jpg)
The University of
OntarioDijkstra’s shortest path algorithm
3
1 0
5
3
3 2 3
6
531
33
4 9
2
4
3 1
4
52
33
3 2
4
1. init node costs to , set p = seed point, cost(p) = 02. expand p as follows:
for each of p’s neighbors q that are not expanded• set cost(q) = min( cost(p) + cpq, cost(q) )
– if q’s cost changed, make q point back to p• put q on the ACTIVE list (if not already there)
3. set r = node with minimum cost on the ACTIVE list4. repeat Step 2 for p = r
ALG
OR
ITH
M
![Page 38: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/38.jpg)
The University of
OntarioDijkstra’s shortest path algorithm
3
1 0
5
3
3 2 3
6
531
33
4 9
2
4
3 1
4
52
33
3 2
4
2
1. init node costs to , set p = seed point, cost(p) = 02. expand p as follows:
for each of p’s neighbors q that are not expanded• set cost(q) = min( cost(p) + cpq, cost(q) )
– if q’s cost changed, make q point back to p• put q on the ACTIVE list (if not already there)
3. set r = node with minimum cost on the ACTIVE list4. repeat Step 2 for p = r
ALG
OR
ITH
M
![Page 39: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/39.jpg)
The University of
OntarioPath Search (basic idea)
A
B
Dijkstra algorithm
- processed nodes (distance to A is known)- active nodes (front)
- active node with the smallest distance value
![Page 40: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/40.jpg)
The University of
OntarioDijkstra’s shortest path algorithm
Properties• It computes the minimum cost path from the seed to every
node in the graph. This set of minimum paths is represented as a tree
• Running time, with N pixels:– O(N2) time if you use an active list– O(N log N) if you use an active priority queue (heap)– takes < second for a typical (640x480) image
• Once this tree is computed once, we can extract the optimal path from any point to the seed in O(N) time.
– it runs in real time as the mouse moves
• What happens when the user specifies a new seed?
![Page 41: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/41.jpg)
The University of
OntarioLivewire extensions
Directed graphs Restricted search space
• Restricted domain (e.g. near a priori model)• Restricted backward search
Different edge weight functions• Image-Edge strength• Image-Edge Curvature • Proximity to known approximate model/boundary
Multi-resolution processing
![Page 42: The University of Ontario CS 433/653 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz.](https://reader030.fdocuments.us/reader030/viewer/2022012922/56649e935503460f94b98a8d/html5/thumbnails/42.jpg)
The University of
OntarioResults
http://www.cs.washington.edu/education/courses/455/03wi/projects/project1/artifacts/index.html