Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of...
-
date post
21-Dec-2015 -
Category
Documents
-
view
213 -
download
0
Transcript of Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of...
![Page 1: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/1.jpg)
Automatic Image Stitching using Invariant Features
Matthew Brown and David Lowe, University of British Columbia
![Page 2: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/2.jpg)
Introduction
• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°
![Page 3: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/3.jpg)
Introduction
• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°
![Page 4: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/4.jpg)
Introduction
• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°– Panoramic Mosaic = 360 x 180°
![Page 5: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/5.jpg)
Recognising Panoramas
![Page 6: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/6.jpg)
Recognising Panoramas
• 1D Rotations ()– Ordering matching images
![Page 7: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/7.jpg)
Recognising Panoramas
• 1D Rotations ()– Ordering matching images
![Page 8: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/8.jpg)
Recognising Panoramas
• 1D Rotations ()– Ordering matching images
![Page 9: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/9.jpg)
Recognising Panoramas
• 2D Rotations (, )– Ordering matching images
• 1D Rotations ()– Ordering matching images
![Page 10: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/10.jpg)
Recognising Panoramas
• 1D Rotations ()– Ordering matching images
• 2D Rotations (, )– Ordering matching images
![Page 11: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/11.jpg)
Recognising Panoramas
• 1D Rotations ()– Ordering matching images
• 2D Rotations (, )– Ordering matching images
![Page 12: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/12.jpg)
Recognising Panoramas
![Page 13: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/13.jpg)
Overview
• Feature Matching• Image Matching• Bundle Adjustment• Multi-band Blending• Results• Conclusions
![Page 14: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/14.jpg)
Overview
• Feature Matching• Image Matching• Bundle Adjustment• Multi-band Blending• Results• Conclusions
![Page 15: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/15.jpg)
Overview
• Feature Matching– SIFT Features– Nearest Neighbour Matching
• Image Matching• Bundle Adjustment• Multi-band Blending• Results• Conclusions
![Page 16: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/16.jpg)
Overview
• Feature Matching– SIFT Features– Nearest Neighbour Matching
• Image Matching• Bundle Adjustment• Multi-band Blending• Results• Conclusions
![Page 17: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/17.jpg)
Invariant Features
• Schmid & Mohr 1997, Lowe 1999, Baumberg 2000, Tuytelaars & Van Gool 2000, Mikolajczyk & Schmid 2001, Brown & Lowe 2002, Matas et. al. 2002, Schaffalitzky & Zisserman 2002
![Page 18: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/18.jpg)
SIFT Features
• Invariant Features– Establish invariant frame
• Maxima/minima of scale-space DOG x, y, s• Maximum of distribution of local gradients
– Form descriptor vector• Histogram of smoothed local gradients• 128 dimensions
• SIFT features are…– Geometrically invariant to similarity transforms,
• some robustness to affine change
– Photometrically invariant to affine changes in intensity
![Page 19: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/19.jpg)
Overview
• Feature Matching– SIFT Features– Nearest Neighbour Matching
• Image Matching• Bundle Adjustment• Multi-band Blending• Results• Conclusions
![Page 20: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/20.jpg)
Nearest Neighbour Matching
• Nearest neighbour matching
• Use k-d tree– k-d tree recursively bi-partitions data at mean in the
dimension of maximum variance– Approximate nearest neighbours found in O(n log n)
• Find k-NN for each feature– k number of overlapping images (we use k = 4)
[ Beis Lowe 1997, Nene Nayar 1997, Gray Moore 2000, Shakhnarovich 2003 ]
![Page 21: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/21.jpg)
K-d tree
![Page 22: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/22.jpg)
K-d tree
![Page 23: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/23.jpg)
Overview
• Feature Matching– SIFT Features– Nearest Neighbour Matching
• Image Matching• Bundle Adjustment• Multi-band Blending• Results• Conclusions
![Page 24: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/24.jpg)
Overview
• Feature Matching• Image Matching• Bundle Adjustment• Multi-band Blending• Results• Conclusions
![Page 25: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/25.jpg)
Overview
• Feature Matching• Image Matching• Bundle Adjustment• Multi-band Blending• Results• Conclusions
![Page 26: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/26.jpg)
Overview
• Feature Matching• Image Matching
– RANSAC for Homography
• Bundle Adjustment• Multi-band Blending• Results• Conclusions
![Page 27: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/27.jpg)
Overview
• Feature Matching• Image Matching
– RANSAC for Homography
• Bundle Adjustment• Multi-band Blending• Results• Conclusions
![Page 28: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/28.jpg)
RANSAC for Homography
![Page 29: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/29.jpg)
RANSAC for Homography
![Page 30: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/30.jpg)
RANSAC for Homography
![Page 31: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/31.jpg)
RANSAC: 1D Line Fitting
least squares line
![Page 32: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/32.jpg)
RANSAC: 1D Line Fitting
![Page 33: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/33.jpg)
RANSAC: 1D Line FittingRANSACline
![Page 34: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/34.jpg)
The RANSAC Algorithm
function H = RANSAC(points, nIterations){
bestInliers = 0; bestH = zeros(3, 3);
for (i = 0; i < nIterations; i++){
samplePoints = RandomSample(points);H = ComputeTransform(samplePoints);nInliers = Consistent(H);
if (nInliers > bestInliers){
bestInliers = nInliers;bestH = H;
} // end if
} // end for
} // end RANSAC
![Page 35: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/35.jpg)
2D Transforms
• Linear (affine)
• Homography
![Page 36: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/36.jpg)
Finding the panoramas
![Page 37: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/37.jpg)
Finding the panoramas
![Page 38: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/38.jpg)
Finding the panoramas
![Page 39: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/39.jpg)
Finding the panoramas
![Page 40: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/40.jpg)
Connected Components
![Page 41: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/41.jpg)
Overview
• Feature Matching• Image Matching• Bundle Adjustment• Multi-band Blending• Results• Conclusions
![Page 42: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/42.jpg)
Bundle Adjustment
• Adjust rotation, focal length of each image to minimise error in matched features
![Page 43: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/43.jpg)
Bundle Adjustment
• Adjust rotation, focal length of each image to minimise error in matched features
![Page 44: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/44.jpg)
Overview
• Feature Matching• Image Matching• Bundle Adjustment• Multi-band Blending• Results• Conclusions
![Page 45: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/45.jpg)
Overview
• Feature Matching• Image Matching• Bundle Adjustment• Multi-band Blending• Results• Conclusions
![Page 46: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/46.jpg)
Multi-band Blending
• Burt & Adelson 1983– Blend frequency bands over range
![Page 47: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/47.jpg)
Low frequency ( > 2 pixels)
High frequency ( < 2 pixels)
2-band Blending
![Page 48: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/48.jpg)
Linear Blending
![Page 49: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/49.jpg)
2-band Blending
![Page 50: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/50.jpg)
![Page 51: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/51.jpg)
![Page 52: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/52.jpg)
Multi-band Blending
• No blending
![Page 53: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/53.jpg)
Multi-band Blending
• Linear blending
– Each pixel is a weighted sum
![Page 54: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/54.jpg)
Multi-band Blending
• Multi-band blending
– Each pixel is a weighted sum (for each band)
![Page 55: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/55.jpg)
Multi-band Blending
• Linear blending • Multi-band blending
![Page 56: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/56.jpg)
Overview
• Feature Matching• Image Matching• Bundle Adjustment• Multi-band Blending• Results• Conclusions
![Page 57: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/57.jpg)
Overview
• Feature Matching• Image Matching• Bundle Adjustment• Multi-band Blending• Results• Conclusions
![Page 58: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/58.jpg)
Results
![Page 59: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/59.jpg)
Results
![Page 60: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/60.jpg)
Overview
• Feature Matching• Image Matching• Bundle Adjustment• Multi-band Blending• Results• Conclusions
![Page 61: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/61.jpg)
Overview
• Feature Matching• Image Matching• Bundle Adjustment• Multi-band Blending• Results• Conclusions
![Page 62: Automatic Image Stitching using Invariant Features Matthew Brown and David Lowe, University of British Columbia.](https://reader031.fdocuments.us/reader031/viewer/2022013101/56649d595503460f94a391e3/html5/thumbnails/62.jpg)
Conclusions
• Fully automatic panoramas– A recognition problem…
• Invariant feature based method– SIFT features, RANSAC, Bundle Adjustment, Multi-
band Blending– O(nlogn)
• Future Work– Advanced camera modelling
• radial distortion, camera motion, scene motion, vignetting, exposure, high dynamic range, flash …
– Full 3D case – recognising 3D objects/scenes in unordered datasets. “PhotoTourism”.
http://www.autostitch.net