Image alignment
description
Transcript of Image alignment
![Page 1: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/1.jpg)
Image alignment
Image from http://graphics.cs.cmu.edu/courses/15-463/2010_fall/
![Page 2: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/2.jpg)
A look into the past
http://blog.flickr.net/en/2010/01/27/a-look-into-the-past/
![Page 3: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/3.jpg)
A look into the pastLeningrad during the blockade
http://komen-dant.livejournal.com/345684.html
![Page 4: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/4.jpg)
Bing streetside images
http://www.bing.com/community/blogs/maps/archive/2010/01/12/new-bing-maps-application-streetside-photos.aspx
![Page 5: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/5.jpg)
Image alignment: Applications
Panorama stitching
![Page 6: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/6.jpg)
Image alignment: Applications
Recognitionof objectinstances
![Page 7: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/7.jpg)
Image alignment: Challenges
Small degree of overlap
Occlusion,clutter
Intensity changes
![Page 8: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/8.jpg)
Feature-based alignment• Search sets of feature matches that agree in terms of:
a) Local appearanceb) Geometric configuration
?
![Page 9: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/9.jpg)
Feature-based alignment: Overview• Alignment as fitting
• Affine transformations• Homographies
• Robust alignment • Descriptor-based feature matching• RANSAC
• Large-scale alignment• Inverted indexing• Vocabulary trees
• Application: searching the night sky
![Page 10: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/10.jpg)
Alignment as fitting• Previous lectures: fitting a model to features in one image
i
i Mx ),(residual
Find model M that minimizes
M
xi
![Page 11: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/11.jpg)
Alignment as fitting• Previous lectures: fitting a model to features in one image
• Alignment: fitting a model to a transformation between pairs of features (matches) in two images
i
i Mx ),(residual
i
ii xxT )),((residual
Find model M that minimizes
Find transformation T that minimizes
M
xi
T
xixi'
![Page 12: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/12.jpg)
2D transformation models
• Similarity(translation, scale, rotation)
• Affine
• Projective(homography)
![Page 13: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/13.jpg)
Let’s start with affine transformations• Simple fitting procedure (linear least squares)• Approximates viewpoint changes for roughly planar
objects and roughly orthographic cameras• Can be used to initialize fitting for more complex
models
![Page 14: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/14.jpg)
Fitting an affine transformation• Assume we know the correspondences, how do we
get the transformation?
),( ii yx ),( ii yx
2
1
43
21
tt
yx
mmmm
yx
i
i
i
i
tMxx ii
Want to find M, t to minimize
n
iii
1
2|||| tMxx
![Page 15: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/15.jpg)
Fitting an affine transformation• Assume we know the correspondences, how do we
get the transformation?
),( ii yx ),( ii yx
2
1
43
21
tt
yx
mmmm
yx
i
i
i
i
i
i
ii
ii
yx
ttmmmm
yxyx
2
1
4
3
2
1
10000100
![Page 16: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/16.jpg)
Fitting an affine transformation
• Linear system with six unknowns• Each match gives us two linearly independent
equations: need at least three to solve for the transformation parameters
i
i
ii
ii
yx
ttmmmm
yxyx
2
1
4
3
2
1
10000100
![Page 17: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/17.jpg)
Fitting a plane projective transformation• Homography: plane projective transformation
(transformation taking a quad to another arbitrary quad)
![Page 18: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/18.jpg)
Homography• The transformation between two views of a planar
surface
• The transformation between images from two cameras that share the same center
![Page 19: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/19.jpg)
Application: Panorama stitching
Source: Hartley & Zisserman
![Page 20: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/20.jpg)
Fitting a homography• Recall: homogeneous coordinates
Converting to homogeneousimage coordinates
Converting from homogeneousimage coordinates
![Page 21: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/21.jpg)
Fitting a homography• Recall: homogeneous coordinates
• Equation for homography:
Converting to homogeneousimage coordinates
Converting from homogeneousimage coordinates
11 333231
232221
131211
yx
hhhhhhhhh
yx
![Page 22: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/22.jpg)
Fitting a homography• Equation for homography:
ii xHx
11 333231
232221
131211
i
i
i
i
yx
hhhhhhhhh
yx
0 ii xHx
iT
iiT
i
iT
iiT
iT
iT
i
iT
iT
iT
i
i
yxx
yyx
xhxhxhxhxhxh
xhxhxh
3
2
1
12
31
23
1
00
00
3
2
1
hhh
xxxx
xx
TTii
Tii
Tii
TTi
Tii
Ti
T
xyxy
3 equations, only 2 linearly
independent
![Page 23: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/23.jpg)
Direct linear transform
• H has 8 degrees of freedom (9 parameters, but scale is arbitrary)
• One match gives us two linearly independent equations• Homogeneous least squares: find h minimizing ||Ah||2
• Eigenvector of ATA corresponding to smallest eigenvalue• Four matches needed for a minimal solution
0
00
00
3
2
1111
111
hhh
xxxx
xxxx
Tnn
TTn
Tnn
Tn
T
TTT
TTT
xy
xy
0hA
![Page 24: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/24.jpg)
Robust feature-based alignment• So far, we’ve assumed that we are given a set of
“ground-truth” correspondences between the two images we want to align
• What if we don’t know the correspondences?
),( ii yx ),( ii yx
![Page 25: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/25.jpg)
Robust feature-based alignment• So far, we’ve assumed that we are given a set of
“ground-truth” correspondences between the two images we want to align
• What if we don’t know the correspondences?
?
![Page 26: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/26.jpg)
Robust feature-based alignment
![Page 27: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/27.jpg)
Robust feature-based alignment
• Extract features
![Page 28: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/28.jpg)
Robust feature-based alignment
• Extract features• Compute putative matches
![Page 29: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/29.jpg)
Robust feature-based alignment
• Extract features• Compute putative matches• Loop:
• Hypothesize transformation T
![Page 30: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/30.jpg)
Robust feature-based alignment
• Extract features• Compute putative matches• Loop:
• Hypothesize transformation T• Verify transformation (search for other matches consistent
with T)
![Page 31: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/31.jpg)
Robust feature-based alignment
• Extract features• Compute putative matches• Loop:
• Hypothesize transformation T• Verify transformation (search for other matches consistent
with T)
![Page 32: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/32.jpg)
Generating putative correspondences
?
![Page 33: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/33.jpg)
Generating putative correspondences
• Need to compare feature descriptors of local patches surrounding interest points
( ) ( )=?
featuredescriptor
featuredescriptor
?
![Page 34: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/34.jpg)
Feature descriptors• Recall: feature detection and description
![Page 35: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/35.jpg)
• Simplest descriptor: vector of raw intensity values• How to compare two such vectors?
• Sum of squared differences (SSD)
– Not invariant to intensity change
• Normalized correlation
– Invariant to affine intensity change
Feature descriptors
i
ii vu 2)SSD( vu,
jj
jj
i ii
vu
vu
22 )()(
))((||||)(
||||)()(
vu
vuvvvv
uuuuvu,
![Page 36: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/36.jpg)
Disadvantage of intensity vectors as descriptors• Small deformations can affect the matching
score a lot
![Page 37: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/37.jpg)
Review: Alignment• 2D alignment models• Feature-based alignment outline• Descriptor matching
![Page 38: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/38.jpg)
• Descriptor computation:• Divide patch into 4x4 sub-patches• Compute histogram of gradient orientations (8 reference
angles) inside each sub-patch• Resulting descriptor: 4x4x8 = 128 dimensions
Feature descriptors: SIFT
David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60 (2), pp. 91-110, 2004.
![Page 39: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/39.jpg)
• Descriptor computation:• Divide patch into 4x4 sub-patches• Compute histogram of gradient orientations (8 reference
angles) inside each sub-patch• Resulting descriptor: 4x4x8 = 128 dimensions
• Advantage over raw vectors of pixel values• Gradients less sensitive to illumination change• Pooling of gradients over the sub-patches achieves
robustness to small shifts, but still preserves some spatial information
Feature descriptors: SIFT
David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60 (2), pp. 91-110, 2004.
![Page 40: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/40.jpg)
Feature matching
?
• Generating putative matches: for each patch in one image, find a short list of patches in the other image that could match it based solely on appearance
![Page 41: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/41.jpg)
Problem: Ambiguous putative matches
Source: Y. Furukawa
![Page 42: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/42.jpg)
Rejection of unreliable matches• How can we tell which putative matches are more reliable?• Heuristic: compare distance of nearest neighbor to that of
second nearest neighbor• Ratio of closest distance to second-closest distance will be high
for features that are not distinctive
David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60 (2), pp. 91-110, 2004.
Threshold of 0.8 provides good separation
![Page 43: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/43.jpg)
RANSAC• The set of putative matches contains a very high
percentage of outliers
RANSAC loop:1. Randomly select a seed group of matches2. Compute transformation from seed group3. Find inliers to this transformation 4. If the number of inliers is sufficiently large, re-compute
least-squares estimate of transformation on all of the inliers
Keep the transformation with the largest number of inliers
![Page 44: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/44.jpg)
RANSAC example: Translation
Putative matches
![Page 45: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/45.jpg)
RANSAC example: Translation
Select one match, count inliers
![Page 46: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/46.jpg)
RANSAC example: Translation
Select one match, count inliers
![Page 47: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/47.jpg)
RANSAC example: Translation
Select translation with the most inliers
![Page 48: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/48.jpg)
Scalability: Alignment to large databases• What if we need to align a test image with thousands
or millions of images in a model database?• Efficient putative match generation
• Approximate descriptor similarity search, inverted indices
Model database
?Test image
![Page 49: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/49.jpg)
Large-scale visual search
Figure from: Kristen Grauman and Bastian Leibe, Visual Object Recognition, Synthesis Lectures on Artificial Intelligence and Machine Learning, April 2011, Vol. 5, No. 2, Pages 1-181
Inverted indexing
Reranking/Geometric verification
![Page 50: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/50.jpg)
Example indexing technique: Vocabulary trees
D. Nistér and H. Stewénius, Scalable Recognition with a Vocabulary Tree, CVPR 2006
Test image
Database
Vocabulary tree with inverted
index
![Page 51: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/51.jpg)
Descriptor space
Goal: find a set of representative prototypes or cluster centers to which descriptors can be quantized
![Page 52: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/52.jpg)
K-means clustering• Want to minimize sum of squared Euclidean
distances between points xi and their nearest cluster centers mk
Algorithm:• Randomly initialize K cluster centers• Iterate until convergence:
• Assign each data point to the nearest center• Recompute each cluster center as the mean of all points
assigned to it
k
ki
kiMXDcluster
clusterinpoint
2)(),( mx
![Page 53: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/53.jpg)
K-means demo
Source: http://shabal.in/visuals/kmeans/1.htmlAnother demo: http://www.kovan.ceng.metu.edu.tr/~maya/kmeans/
![Page 54: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/54.jpg)
Recall: Visual codebook for generalized Hough transform
…
Source: B. Leibe
Appearance codebook
![Page 55: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/55.jpg)
Hierarchical k-means clustering of
descriptor space (vocabulary tree) Slide credit: D. Nister
![Page 56: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/56.jpg)
Slide credit: D. NisterVocabulary tree/inverted index
![Page 57: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/57.jpg)
Populating the vocabulary tree/inverted indexSlide credit: D. Nister
Model images
![Page 58: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/58.jpg)
Populating the vocabulary tree/inverted indexSlide credit: D. Nister
Model images
![Page 59: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/59.jpg)
Populating the vocabulary tree/inverted indexSlide credit: D. Nister
Model images
![Page 60: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/60.jpg)
Populating the vocabulary tree/inverted indexSlide credit: D. Nister
Model images
![Page 61: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/61.jpg)
Looking up a test image Slide credit: D. Nister
Test imageModel images
![Page 62: Image alignment](https://reader035.fdocuments.us/reader035/viewer/2022062218/5681662f550346895dd995ee/html5/thumbnails/62.jpg)
Cool application of large-scale alignment: searching the night sky
http://www.astrometry.net/
A
B
CD