BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 ·...
Transcript of BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 ·...
![Page 1: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/1.jpg)
Image Resampling & InterpolationHarris corner detection
BIM496: Computer VisionMehmet Kilicarslan
![Page 2: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/2.jpg)
Reading• Szeliski: 4.1
![Page 3: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/3.jpg)
Image Scaling
This image is too big to fit on the screen. How can we generate a half-sized version?
Source: S. Seitz
![Page 4: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/4.jpg)
Image sub-sampling
Throw away every other row and column to create a 1/2 size image
- called image sub-sampling
1/4
1/8
Source: S. Seitz
![Page 5: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/5.jpg)
Image sub-sampling
1/4 (2x zoom) 1/8 (4x zoom)
Why does this look so crufty?
1/2
Source: S. Seitz
![Page 6: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/6.jpg)
Image sub-sampling – another example
Source: F. Durand
![Page 7: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/7.jpg)
Even worse for synthetic images
Source: L. Zhang
![Page 8: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/8.jpg)
Aliasing
• Occurs when your sampling rate is not high enough to capture the amount of detail in your image
• Can give you the wrong signal/image—an alias
• To do sampling right, need to understand the structure of your signal/image
• Enter Monsieur Fourier…– “But what is the Fourier Transform? A visual introduction.”
https://www.youtube.com/watch?v=spUNpyF58BY&t=444s• To avoid aliasing:
– sampling rate ≥ 2 * max frequency in the image• said another way: ≥ two samples per cycle
– This minimum sampling rate is called the Nyquist rateSource: L. Zhang
![Page 9: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/9.jpg)
Wagon-wheel effect
(See http://www.michaelbach.de/ot/mot-wagonWheel/index.html) Source: L. Zhang
![Page 10: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/10.jpg)
Nyquist limit – 2D example
Good sampling
Bad sampling
![Page 11: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/11.jpg)
Aliasing• When downsampling by a factor of two– Original image has frequencies that are too high
• How can we fix this?
![Page 12: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/12.jpg)
Gaussian pre-filtering
G 1/4
G 1/8
Gaussian 1/2
• Solution: filter the image, then subsampleSource: S. Seitz
![Page 13: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/13.jpg)
Subsampling with Gaussian pre-filtering
G 1/4 G 1/8Gaussian 1/2
• Solution: filter the image, then subsampleSource: S. Seitz
![Page 14: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/14.jpg)
Compare with...
1/4 (2x zoom) 1/8 (4x zoom)1/2
Source: S. Seitz
![Page 15: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/15.jpg)
Gaussian pre-filtering• Solution: filter
the image, thensubsample
blur
F0 H*
subsample blur subsample …F1
F1 H*
F2F0
![Page 16: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/16.jpg)
blur
F0 H*subsample blur subsample …
F1
F1 H*
F2F0{Gaussian
pyramid
![Page 17: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/17.jpg)
![Page 18: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/18.jpg)
Gaussian pyramids [Burt and Adelson, 1983]
• In computer graphics, a mip map [Williams, 1983]• A precursor to wavelet transform
Gaussian Pyramids have all sorts of applications in computer visionSource: S. Seitz
![Page 19: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/19.jpg)
Gaussian pyramids [Burt and Adelson, 1983]
• How much space does a Gaussian pyramid take compared to the original image?
Source: S. Seitz
![Page 20: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/20.jpg)
Gaussian Pyramid
![Page 21: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/21.jpg)
Upsampling• This image is too small for this screen:• How can we make it 10 times as big?• Simplest approach:
repeat each rowand column 10 times
• (“Nearest neighborinterpolation”)
![Page 22: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/22.jpg)
Image interpolation
Recall that a digital images is formed as follows:
• It is a discrete point-sampling of a continuous function• If we could somehow reconstruct the original function, any new
image could be generated, at any resolution and scale
1 2 3 4 5
Adapted from: S. Seitz
d = 1 in this example
![Page 23: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/23.jpg)
Image interpolation
1 2 3 4 5
d = 1 in this example
Recall that a digital images is formed as follows:
• It is a discrete point-sampling of a continuous function• If we could somehow reconstruct the original function, any new
image could be generated, at any resolution and scale
Adapted from: S. Seitz
![Page 24: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/24.jpg)
Image interpolation
1 2 3 4 52.5
1
• Convert to a continuous function:
• Reconstruct by convolution with a reconstruction filter, h
• What if we don’t know ?• Guess an approximation:• Can be done in a principled way: filtering
d = 1 in this example
Adapted from: S. Seitz
![Page 25: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/25.jpg)
Image interpolation“Ideal” reconstruction
Nearest-neighbor interpolation
Linear interpolation
Gaussian reconstruction
Source: B. Curless
![Page 26: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/26.jpg)
Reconstruction filters• What does the 2D version of this hat function look like?
Often implemented without cross-correlation• E.g., http://en.wikipedia.org/wiki/Bilinear_interpolation
Better filters give better resampled images• Bicubic is common choice
performs linear interpolation
(tent function) performs bilinear interpolation
Cubic reconstruction filter
![Page 27: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/27.jpg)
Image interpolation
Nearest-neighbor interpolation Bilinear interpolation Bicubic interpolation
Original image: x 10
![Page 28: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/28.jpg)
Image interpolationAlso used for resampling
![Page 29: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/29.jpg)
Modern methods
From Romano, et al: RAISR: Rapid and Accurate Image Super Resolution, https://arxiv.org/abs/1606.01299
![Page 30: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/30.jpg)
Questions?
![Page 31: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/31.jpg)
Feature extraction: Corners and blobs
![Page 32: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/32.jpg)
Motivation: Automatic panoramas
Credit: Matt Brown
![Page 33: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/33.jpg)
http://gigapan.com /GigaPan
Also see Google Zoom Views: https://www.google.com/culturalinstitute/beta/project/gigapixels
Motivation: Automatic panoramas
![Page 34: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/34.jpg)
Why extract features?• Motivation: panorama stitching–We have two images – how do we combine them?
![Page 35: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/35.jpg)
Why extract features?• Motivation: panorama stitching–We have two images – how do we combine them?
Step 1: extract featuresStep 2: match features
![Page 36: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/36.jpg)
Why extract features?• Motivation: panorama stitching–We have two images – how do we combine them?
Step 1: extract featuresStep 2: match featuresStep 3: align images
![Page 37: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/37.jpg)
Application: Visual SLAM• (aka Simultaneous Localization and Mapping)
![Page 38: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/38.jpg)
Image matching
by Diva Sian
by swashford
TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAA
![Page 39: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/39.jpg)
Harder case
by Diva Sian by scgbt
![Page 40: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/40.jpg)
Harder still?
![Page 41: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/41.jpg)
NASA Mars Rover imageswith SIFT feature matches
Answer below (look for tiny colored squares…)
![Page 42: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/42.jpg)
Feature Matching
![Page 43: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/43.jpg)
Feature Matching
![Page 44: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/44.jpg)
Invariant local featuresFind features that are invariant to transformations
– geometric invariance: translation, rotation, scale– photometric invariance: brightness, exposure, …
Feature Descriptors
![Page 45: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/45.jpg)
Advantages of local featuresLocality – features are local, so robust to occlusion and clutter
Quantity– hundreds or thousands in a single image
Distinctiveness: – can differentiate a large database of objects
Efficiency– real-time performance achievable
![Page 46: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/46.jpg)
More motivation… Feature points are used for:– Image alignment
• (e.g., mosaics)– 3D reconstruction– Motion tracking
• (e.g. for AR)– Object recognition– Image retrieval– Robot navigation– … other
![Page 47: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/47.jpg)
Approach1. Feature detection: find it2. Feature descriptor: represent it3. Feature matching: match it
Feature tracking: track it, when motion
![Page 48: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/48.jpg)
Local features: main components1) Detection: Identify the interest
points
2) Description: Extract vector feature descriptor surrounding each interest point.
3) Matching: Determine correspondence between descriptors in two views
],,[ )1()1(11 dxx !=x
],,[ )2()2(12 dxx !=x
Kristen Grauman
![Page 49: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/49.jpg)
Snoop demo
What makes a good feature?
![Page 50: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/50.jpg)
Want uniquenessLook for image regions that are unusual– Lead to unambiguous matches in other images
How to define “unusual”?
![Page 51: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/51.jpg)
Local measures of uniquenessSuppose we only consider a small window of pixels– What defines whether a feature is a good or bad
candidate?
Credit: S. Seitz, D. Frolova, D. Simakov
![Page 52: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/52.jpg)
Local measures of uniqueness
“flat” region:no change in all directions
“edge”: no change along the edge direction
“corner”:significant change in all directions
• How does the window change when you shift it?• Shifting the window in any direction causes a big
change
Credit: S. Seitz, D. Frolova, D. Simakov
![Page 53: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/53.jpg)
Consider shifting the window W by (u,v)• how do the pixels in W change?• compare each pixel before and after by
summing up the squared differences (SSD)• this defines an SSD “error” E(u,v):
• We are happy if this error is high• Slow to compute exactly for each pixel
and each offset (u,v)
Harris corner detection: the math
W(u,v)
![Page 54: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/54.jpg)
Taylor Series expansion of I:
If the motion (u,v) is small, then first order approximation is good
Plugging this into the formula on the previous slide…
Small motion assumption
![Page 55: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/55.jpg)
Corner detection: the math
Consider shifting the window W by (u,v)• define an SSD “error” E(u,v):
W(u,v)
![Page 56: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/56.jpg)
Corner detection: the math
Consider shifting the window W by (u,v)• define an SSD “error” E(u,v):
• Thus, E(u,v) is locally approximated as a quadratic error function
W(u,v)
![Page 57: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/57.jpg)
The surface E(u,v) is locally approximated by a quadratic form.
The second moment matrix
Let’s try to understand its shape.
![Page 58: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/58.jpg)
Horizontal edge:
uv
E(u,v)
![Page 59: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/59.jpg)
Vertical edge:
uv
E(u,v)
![Page 60: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/60.jpg)
General caseWe can visualize H as an ellipse with axis lengths determined by the eigenvalues of H and orientation determined by the eigenvectors of H
direction of the slowest change
direction of the fastest change
(lmax)-1/2(lmin)-1/2
const][ =úû
ùêë
évu
Hvu
Ellipse equation:lmax, lmin : eigenvalues of H
![Page 61: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/61.jpg)
Quick eigenvalue/eigenvector reviewThe eigenvectors of a matrix A are the vectors x that satisfy:
The scalar l is the eigenvalue corresponding to x– The eigenvalues are found by solving:
– In our case, A = H is a 2x2 matrix, so we have
– The solution:
Once you know l, you find x by solving
![Page 62: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/62.jpg)
Corner detection: the math
Eigenvalues and eigenvectors of H• Define shift directions with the smallest and largest change in error• xmax = direction of largest increase in E• lmax = amount of increase in direction xmax
• xmin = direction of smallest increase in E• lmin = amount of increase in direction xmin
xmin
xmax
![Page 63: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/63.jpg)
Corner detection: the mathHow are lmax, xmax, lmin, and xmin relevant for feature detection?
• What’s our feature scoring function?
![Page 64: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/64.jpg)
Corner detection: the mathHow are lmax, xmax, lmin, and xmin relevant for feature detection?
• What’s our feature scoring function?Want E(u,v) to be large for small shifts in all directions
• the minimum of E(u,v) should be large, over all unit vectors [u v]• this minimum is given by the smaller eigenvalue (lmin) of H
![Page 65: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/65.jpg)
Interpreting the eigenvalues
l1
l2
“Corner”l1 and l2 are large,l1 ~ l2;E increases in all directions
l1 and l2 are small;E is almost constant in all directions
“Edge” l1 >> l2
“Edge” l2 >> l1
“Flat” region
Classification of image points using eigenvalues of M:
![Page 66: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/66.jpg)
Corner detection summaryHere’s what you do
• Compute the gradient at each point in the image• Create the H matrix from the entries in the gradient• Compute the eigenvalues.
• Find points with large response (lmin > threshold)• Choose those points where lmin is a local maximum as features
![Page 67: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/67.jpg)
Corner detection summaryHere’s what you do
• Compute the gradient at each point in the image• Create the H matrix from the entries in the gradient• Compute the eigenvalues.
• Find points with large response (lmin > threshold)• Choose those points where lmin is a local maximum as features
![Page 68: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/68.jpg)
The Harris operatorlmin is a variant of the “Harris operator” for feature detection
• The trace is the sum of the diagonals, i.e., trace(H) = h11 + h22
• Very similar to lmin but less expensive (no square root)• Called the “Harris Corner Detector” or “Harris Operator”• Lots of other detectors, this is one of the most popular
![Page 69: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/69.jpg)
The Harris operator
Harris operator
![Page 70: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/70.jpg)
Harris detector example
![Page 71: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/71.jpg)
f value (red high, blue low)
![Page 72: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/72.jpg)
Threshold (f > value)
![Page 73: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/73.jpg)
Find local maxima of f
![Page 74: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/74.jpg)
Harris features (in red)
![Page 75: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/75.jpg)
Weighting the derivatives• In practice, using a simple window W doesn’t
work too well
• Instead, we’ll weight each derivative value based on its distance from the center pixel
![Page 76: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/76.jpg)
Harris Detector [Harris88]
• Second moment matrix
úúû
ù
êêë
é*=
)()()()(
)(),( 2
2
DyDyx
DyxDxIDI III
IIIg
ssss
sssµ
96
1. Image derivatives
2. Square of derivatives
3. Gaussian filter g(sI)
Ix Iy
Ix2 Iy2 IxIy
g(Ix2) g(Iy2) g(IxIy)
4. Cornerness function – both eigenvalues are strong
har5. Non-maxima suppression
1 2
1 2
dettrace
MM
l ll l
== +
(optionally, blur first)
![Page 77: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/77.jpg)
Harris Corners – Why so complicated?• Can’t we just check for regions with lots of
gradients in the x and y directions?– No! A diagonal line would satisfy that criteria
Current Window
![Page 78: BIM496: Computer Visionceng.eskisehir.edu.tr/mkilicarslan/BIM496/duyuru/lec02... · 2020-02-25 · Image sub-sampling Throw away every other row and column to create a 1/2size image-called](https://reader034.fdocuments.us/reader034/viewer/2022050413/5f89890d17a20258092ed0d5/html5/thumbnails/78.jpg)
Questions?