Lecture 11: Texture - cs.princeton.edu
Transcript of Lecture 11: Texture - cs.princeton.edu
![Page 1: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/1.jpg)
COS 429: Computer Vision
Lecture 11: Texture
Acknowledgment: slides from Antonio Torralba, Kristen Grauman, Jitendra Malik, Alyosha Efros, Tom Funkhouser, Szymon Rusinkiewicz
![Page 2: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/2.jpg)
Texture
What is a texture?
Torralba
![Page 3: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/3.jpg)
Texture
What is a texture?
Torralba
![Page 4: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/4.jpg)
Texture
What is a texture?
Torralba
![Page 5: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/5.jpg)
Texture
• Texture: stochastic pattern that is stationary(“looks the same” at all locations)
• May be structured or random
Wei & Levoy
![Page 6: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/6.jpg)
Texture
Stochastic Stationary
![Page 7: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/7.jpg)
Texture
Stochastic Stationary
![Page 8: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/8.jpg)
Goal
• Computational representation of texture– Textures generated by same stationary stochastic
process have same representation– Perceptually similar textures have similar
representations
5, 7, 34, 2, 199, 12Hypothetical texture representation
![Page 9: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/9.jpg)
Applications
• Segmentation• 3D Reconstruction• Classification• Synthesis
http://animals.nationalgeographic.com/
![Page 10: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/10.jpg)
Applications
• Segmentation• 3D Reconstruction• Classification• Synthesis
![Page 11: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/11.jpg)
Applications
• Segmentation• 3D Reconstruction• Classification• Synthesis
Grauman
![Page 12: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/12.jpg)
Applications
• Segmentation• 3D Reconstruction• Classification• Synthesis
Grauman
![Page 13: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/13.jpg)
Applications
• Segmentation• 3D Reconstruction• Classification• Synthesis
Input OutputEfros
![Page 14: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/14.jpg)
Texture Representation?
• What makes a good texture representation?– Textures generated by same stationary stochastic
process have same representation– Perceptually similar textures have similar
representations
![Page 15: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/15.jpg)
Statistics of filter banks
![Page 16: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/16.jpg)
Filter-Based Texture Representation
• Research suggests that the human visual system performs local spatial frequency analysis(Gabor filters)
J. J. Kulikowski, S. Marcelja, and P. Bishop. Theory of spatial position and spatial frequency relations in the receptive fields of simple cells in the visual cortex. Biol. Cybern, 43:187-198, 1982.
![Page 17: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/17.jpg)
Texture Representation
• Analyze textures based on the responses of linear filters – Use filters that look like patterns
(spots, edges, bars, …)– Compute magnitudes of filter responses
• Represent textures with statistics of filter responses within local windows– Histogram of feature responses for all pixels in window
Grauman
![Page 18: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/18.jpg)
Texture Representation Example
original image
derivative filter responses, squared
statistics to summarize patterns in small
windows
mean d/dx
value
mean d/dy
value Win. #1 4 10
…
Grauman
![Page 19: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/19.jpg)
Texture Representation Example
original image
derivative filter responses, squared
statistics to summarize patterns in small
windows
mean d/dx
value
mean d/dy
value Win. #1 4 10
Win.#2 18 7
…
Grauman
![Page 20: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/20.jpg)
Texture Representation Example
original image
derivative filter responses, squared
statistics to summarize patterns in small
windows
mean d/dx
value
mean d/dy
value Win. #1 4 10
Win.#2 18 7
Win.#9 20 20
…
…
Grauman
![Page 21: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/21.jpg)
Texture Representation Example
statistics to summarize patterns in small
windows
mean d/dx
value
mean d/dy
value Win. #1 4 10
Win.#2 18 7
Win.#9 20 20
…
…
Dimension 1 (mean d/dx value)Dim
ensi
on 2
(mea
n d/
dy v
alue
)
Grauman
![Page 22: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/22.jpg)
Texture Representation Example
statistics to summarize patterns in small
windows
mean d/dx
value
mean d/dy
value Win. #1 4 10
Win.#2 18 7
Win.#9 20 20
…
…
Dimension 1 (mean d/dx value)Dim
ensi
on 2
(mea
n d/
dy v
alue
)
Far: dissimilartextures
Close: similartextures
Grauman
![Page 23: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/23.jpg)
Filter Banks
• Previous example used two filters, resulting in 2-dimensional feature vector– x and y derivatives revealed local structure
• Filter bank: many filters– Higher-dimensional feature space– Distance still related to similarity of local structure
Grauman
![Page 24: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/24.jpg)
Filter banks
• What filters to put in the bank?– Combination of different scales, orientations, patterns
scales
orientations
“Edges” “Bars”
“Spots”
Grauman
![Page 25: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/25.jpg)
Imag
e fro
m h
ttp://
ww
w.te
xase
xplo
rer.c
om/a
ustin
cap2
.jpg
Grauman
![Page 26: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/26.jpg)
Showing magnitude of responses
Grauman
![Page 27: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/27.jpg)
Grauman
![Page 28: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/28.jpg)
Grauman
![Page 29: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/29.jpg)
Grauman
![Page 30: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/30.jpg)
Grauman
![Page 31: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/31.jpg)
Grauman
![Page 32: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/32.jpg)
Grauman
![Page 33: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/33.jpg)
Grauman
![Page 34: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/34.jpg)
Grauman
![Page 35: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/35.jpg)
Grauman
![Page 36: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/36.jpg)
Grauman
![Page 37: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/37.jpg)
Grauman
![Page 38: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/38.jpg)
Grauman
![Page 39: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/39.jpg)
Grauman
![Page 40: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/40.jpg)
Grauman
![Page 41: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/41.jpg)
Grauman
![Page 42: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/42.jpg)
Grauman
![Page 43: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/43.jpg)
Grauman
![Page 44: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/44.jpg)
Grauman
![Page 45: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/45.jpg)
You Try: Can you match the texture to the response?
Mean abs responses
Filters A
B
C
1
2
3
Derek Hoiem
![Page 46: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/46.jpg)
Application: Retrieval
• Retrieve similar imagesbased on texture
Y. Rubner, C. Tomasi, and L. J. Guibas. The earth mover's distance as a metric for image retrieval. International Journal of Computer Vision, 40(2):99-121, November 2000,
![Page 47: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/47.jpg)
Textons
• Elements (“textons”) either identical or come from some statistical distribution
• Can analyze in natural images
Olhausen & Field
![Page 48: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/48.jpg)
Clustering Textons
• Output of bank of n filters can be thought of as vector in n-dimensional space
• Can cluster these vectors using k-means [Malik et al.]
• Result: dictionary of most common textures
![Page 49: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/49.jpg)
K-means clustering
![Page 50: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/50.jpg)
Revisiting k-means
Most well-known and popular clustering algorithm:
Start with some initial cluster centers
Iterate:– Assign/cluster each example to closest center– Recalculate centers as the mean of the points in a cluster
Credit: David Kauchak
![Page 51: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/51.jpg)
K-means: an example
Credit: David Kauchak
![Page 52: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/52.jpg)
K-means: Initialize centers randomly
Credit: David Kauchak
![Page 53: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/53.jpg)
K-means: assign points to nearest center
Credit: David Kauchak
![Page 54: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/54.jpg)
K-means: readjust centers
Credit: David Kauchak
![Page 55: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/55.jpg)
K-means: assign points to nearest center
Credit: David Kauchak
![Page 56: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/56.jpg)
K-means: readjust centers
Credit: David Kauchak
![Page 57: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/57.jpg)
K-means: assign points to nearest center
Credit: David Kauchak
![Page 58: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/58.jpg)
K-means: readjust centers
Credit: David Kauchak
![Page 59: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/59.jpg)
K-means: assign points to nearest center
No changes: DoneCredit: David Kauchak
![Page 60: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/60.jpg)
K-means
Iterate:– Assign/cluster each example to closest center– Recalculate centers as the mean of the points in a
cluster
How do we do this?Credit: David Kauchak
![Page 61: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/61.jpg)
K-means
Iterate:– Assign/cluster each example to closest center– Recalculate centers as the mean of the points in a
cluster
Where are the cluster centers?Credit: David Kauchak
![Page 62: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/62.jpg)
K-means
Iterate:– Assign/cluster each example to closest center– Recalculate centers as the mean of the points in a
cluster
How do we calculate these?Credit: David Kauchak
![Page 63: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/63.jpg)
Iterate:– Assign/cluster each example to closest center– Recalculate centers as the mean of the points in a
cluster
K-means
Mean of the points in the cluster:
µ(C) = 1|C |
xx∈C∑
Credit: David Kauchak
![Page 64: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/64.jpg)
K-means loss function
K-means tries to minimize what is called the “k-means” loss function:
loss= d(xi,µk )2 where µk is cluster center for xi
i=1
n
∑
that is, the sum of the squared distances from each point to the associated cluster center
Credit: David Kauchak
![Page 65: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/65.jpg)
Minimizing k-means loss
Iterate: 1. Assign/cluster each example to closest center 2. Recalculate centers as the mean of the points in a cluster
loss= d(xi,µk )2 where µk is cluster center for xi
i=1
n
∑
Does each step of k-means move towards reducing this loss function (or at least not increasing)?
Credit: David Kauchak
![Page 66: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/66.jpg)
Minimizing k-means loss
Iterate: 1. Assign/cluster each example to closest center 2. Recalculate centers as the mean of the points in a cluster
loss= d(xi,µk )2 where µk is cluster center for xi
i=1
n
∑
This isn’t quite a complete proof/argument, but:
1. Any other assignment would end up in a larger loss
1. The mean of a set of values minimizes the squared error
Credit: David Kauchak
![Page 67: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/67.jpg)
Minimizing k-means loss
Iterate: 1. Assign/cluster each example to closest center 2. Recalculate centers as the mean of the points in a cluster
loss= d(xi,µk )2 where µk is cluster center for xi
i=1
n
∑
Does this mean that k-means will always find the minimum loss/clustering?
Credit: David Kauchak
![Page 68: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/68.jpg)
Minimizing k-means loss
Iterate: 1. Assign/cluster each example to closest center 2. Recalculate centers as the mean of the points in a cluster
loss= d(xi,µk )2 where µk is cluster center for xi
i=1
n
∑
NO! It will find a minimum.
Unfortunately, the k-means loss function is generally not convex and for most problems has many, many minima
We’re only guaranteed to find one of them
Credit: David Kauchak
![Page 69: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/69.jpg)
K-means: Initialize centers randomly
What would happen here?
Seed selection ideas?Credit: David Kauchak
![Page 70: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/70.jpg)
K-means: Initialize furthest from centers
Pick a random point for the first center
Credit: David Kauchak
![Page 71: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/71.jpg)
K-means: Initialize furthest from centers
What point will be chosen next?
Credit: David Kauchak
![Page 72: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/72.jpg)
K-means: Initialize furthest from centers
Furthest point from center
What point will be chosen next?Credit: David Kauchak
![Page 73: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/73.jpg)
K-means: Initialize furthest from centers
Furthest point from center
What point will be chosen next?Credit: David Kauchak
![Page 74: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/74.jpg)
K-means: Initialize furthest from centers
Furthest point from center
Any issues/concerns with this approach?Credit: David Kauchak
![Page 75: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/75.jpg)
Furthest points concerns
If k = 4, which points will get chosen?
Credit: David Kauchak
![Page 76: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/76.jpg)
Furthest points concerns
If we do a number of trials, will we get different centers?
Credit: David Kauchak
![Page 77: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/77.jpg)
Furthest points concerns
Doesn’t deal well with outliers
Credit: David Kauchak
![Page 78: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/78.jpg)
K-means
• But usually k-means works pretty well• Especially with large number of points and large
number of centers k• Variations: kmeans++, etc• Alternatives: spectral clustering, hierarchical
(bottom-up, agglomerative or top-down, divisive)
![Page 79: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/79.jpg)
Coming back to textons
![Page 80: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/80.jpg)
Clustering Textons
• Output of bank of n filters can be thought of as vector in n-dimensional space
• Can cluster these vectors using k-means [Malik et al.]
• Result: dictionary of most common textures
![Page 81: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/81.jpg)
Clustering Textons
Image
Clustered TextonsTexton to Pixel Mapping
Malik
![Page 82: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/82.jpg)
Using Texture in Segmentation
• Compute histogram of how many times each of the k clusters occurs in a neighborhood
• Define similarity of histograms hi and hj using χ2
• Different histograms → separate regions
�2 =1
2
X
k
(hi(k)� hj(k))2
hi(k) + hj(k)
![Page 83: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/83.jpg)
Application: Segmentation
Malik
![Page 84: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/84.jpg)
Texture synthesis
![Page 85: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/85.jpg)
Markov Random Fields
• Different way of thinking about textures• Premise: probability distribution of a pixel
depends on values of neighbors• Probability the same throughout image– Extension of Markov chains
![Page 86: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/86.jpg)
Motivation from Language
• Shannon (1948) proposed a way to synthesize new text using N-grams– Use a large text to compute probability distributions of
each letter given N–1 previous letters – Starting from a seed repeatedly sample the conditional
probabilities to generate new letters– Can do this with image patches!
Efros
![Page 87: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/87.jpg)
Texture Synthesis Based on MRF
• For each pixel in destination:– Take already-synthesized neighbors– Find closest match in original texture– Copy pixel to destination
• Efros & Leung 1999– Speedup by Wei & Levoy 2000– Extension to copying whole blocks
by Efros & Freeman 2001
Wei & Levoy
![Page 88: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/88.jpg)
• Compute output pixels in scanline order (top-to-bottom,left-to-right)
Efros & Leung Algorithm
Efros
![Page 89: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/89.jpg)
• Find candidate pixels based on similarities of pixel features in neighborhoods
Efros & Leung Algorithm
Efros
![Page 90: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/90.jpg)
Efros & Leung Algorithm
• Similarities of pixel neighborhoods can be computed with squared differences (SSD)of pixel colors and/or filter bank responses
||( – )||2
Efros
![Page 91: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/91.jpg)
Efros & Leung Algorithm
• For each pixel p:– Find the best matching K windows from the input image– Pick one matching window at random– Assign p to be the center pixel of that window
input image
p
Efros
![Page 92: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/92.jpg)
Synthesis Results
Efros
![Page 93: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/93.jpg)
Synthesis Results
white bread brick wall
Efros
![Page 94: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/94.jpg)
Hole Filling
• Fill pixels in “onion skin” order– Within each “layer”, pixels with most neighbors
are synthesized first– Normalize error by the number of known pixels– If no close match can be found, the pixel is not
synthesized until the end
![Page 95: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/95.jpg)
Hole Filling
Efros
![Page 96: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/96.jpg)
Extrapolation
![Page 97: Lecture 11: Texture - cs.princeton.edu](https://reader031.fdocuments.us/reader031/viewer/2022012507/6182ed95ab12314c641b3c5e/html5/thumbnails/97.jpg)
Special video
https://ghc.anitab.org/ghc-17-livestream/
(Wednesday keynote, 16:20 min - 44:00 min)