CSCE 643 Computer Vision: Structure from Motion Jinxiang Chai.
CSCE 643 Computer Vision: Template Matching, Image Pyramids and Denoising
description
Transcript of CSCE 643 Computer Vision: Template Matching, Image Pyramids and Denoising
CSCE 643 Computer Vision:Template Matching, Image Pyramids and Denoising
Jinxiang Chai
Today’s class
• Template matching
• Gaussian Pyramids
• Laplacian Pyramids
• Image denoising
Template matching• Goal: find in image
Template matching• Goal: find in image
• Main challenge: What is a good similarity or distance measure between two patches?
Template matching• Goal: find in image
• Main challenge: What is a good similarity or distance measure between two patches?– Correlation– Zero-mean correlation– Sum Square Difference– Normalized Cross
Correlation
Slide: Hoiem
Matching with filters• Goal: find in image• Method 0: filter the image with eye patch
Input Filtered Image
],[],[],[,
lnkmflkgnmhlk
What went wrong?
f = imageg = filter
Slide: Hoiem
Problem with Correlation
Problem with Correlation
Solution
Slide: Hoiem
Matching with filters• Goal: find in image• Method 1: filter the image with zero-mean eye
Input Filtered Image (scaled) Thresholded Image
],[)],[(],[,
lkgflnkmfnmhlk
True detections
False detections
mean of f
Slide: Hoiem
Matching with filters• Goal: find in image• Method 2: SSD
Input 1- sqrt(SSD) Thresholded Image
2
,
)],[],[(],[ lnkmflkgnmhlk
True detections
Relationship SSD and Correlation
Matching with filters• Goal: find in image• Method 2: SSD
Input 1- sqrt(SSD)
2
,
)],[],[(],[ lnkmflkgnmhlk
What’s the potential downside of SSD?
Slide: Hoiem
Matching with filters• Goal: find in image• Method 3: Normalized cross-correlation - subtracting the mean - dividing by the standard deviation
Matching with filters• Goal: find in image• Method 3: Normalized cross-correlation
5.0
,
2,
,
2
,,
)],[()],[(
)],[)(],[(],[
lknm
lk
nmlk
flnkmfglkg
flnkmfglkgnmh
Matlab: normxcorr2(template, im)
mean image patchmean template
Slide: Hoiem
Matching with filters• Goal: find in image• Method 3: Normalized cross-correlation
Input Normalized X-Correlation Thresholded Image
True detections
Matching with filters• Goal: find in image• Method 3: Normalized cross-correlation
Input Normalized X-Correlation Thresholded Image
True detections
Slide: Hoiem
Q: What is the best method to use?
A: Depends• SSD: faster, sensitive to overall intensity• Normalized cross-correlation: slower, invariant
to local average intensity and contrast
Q: What if we want to find larger or smaller eyes?
A: Image Pyramid
Gaussian Pyramid
Low-Pass Filtered ImageImage
GaussianFilter Sample Low-Res
Image
Gaussian Pyramid
filter mask
Repeat– Filter– Subsample
Until minimum resolution reached – can specify desired number of levels (e.g., 3-level pyramid)
Gaussian pyramid
Template Matching with Image Pyramids
Input: Image, Template1. Match template at current scale
2. Downsample image
3. Repeat 1-2 until image is very small
4. Take responses above some threshold, perhaps with non-maxima suppression
Laplacian pyramid
Laplacian FilterThe filtering kernel for Laplacian is
Finite difference allows us to approximate the Laplacian using the following filtering kernel.
Laplacian Pyramid
Laplacian Pyramid (subband images)- Created from Gaussian pyramid by subtraction
The Laplacian of Gaussian Kernel
),()()),(( 22 yxIGyxIG
2
22
24
222222 2
yx
eyxxG
xGG
is the Laplacian operator:
The Laplacian of Gaussian Filter Kernel
0.1
4.1
Laplacian Pyramid
What happens in frequency domain?
Laplacian Pyramid
What happens in frequency domain? Bandpass filters
Can we reconstruct the original from the laplacian pyramid?
Laplacian Pyramid
What happens in frequency domain? Bandpass filters
Can we reconstruct the original from the laplacian pyramid?- Yes, laplacian pyramid is a complete image representation which encodes fine to course structure in a different level
Image representation
• Pixels: great for spatial resolution, poor access to frequency
• Fourier transform: great for frequency, not for spatial info such as locations.
• Pyramids/filter banks: balance between spatial and frequency information
Major uses of image pyramids
• Compression
• Object detection– Scale search– Features
• Detecting stable interest points
• Image blending/mosaicing
• Registration– Course-to-fine
Multi-resolution SIFT Feature Detection
- Object recognition from local scale-invariant features [pdf link], ICCV 09
- David G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, 60, 2 (2004), pp. 91-110
Pyramid Blending
laplacianlevel
4
laplacianlevel
2
laplacianlevel
0
left pyramid right pyramid blended pyramid
Hierarchical Image Registration1. Compute Gaussian pyramid2. Align with coarse pyramid3. Successively align with finer
pyramids– Search smaller range
Why is this faster?
- Hierarchical Model-Based Motion Estimation, ECCV 92
Denoising
Additive Gaussian Noise
Gaussian Filter
Gaussian
input Gaussian filter
Median Filter
• For each neighbor in image, sliding the window• Sort pixel values• Set the center pixel to the median
Median Filter
input Gaussian filter Median filter
Median Filter Examples
input Median 7X7
Median Filter Examples
Median 11X11Median 3X3
Median Filter Examples
Median 11X11Median 3X3
Straight edges kept
Sharp features lost
Median Filter Properties
Can remove outliers (peppers and salts)
Window size controls size of structure
Preserve some details but sharp corners and edges might get lost
Comparison of Mean, Gaussian, and Median
original Mean with 6 pixels
Comparison of Mean, Gaussian, and Median
original Gaussian with 6 pixels
Comparison of Mean, Gaussian, and Median
original Median with 6 pixels
Common Problems
Mean: blurs image, removes simple noise, no details are preserved
Common Problems
Mean: blurs image, removes simple noise, no details are preserved
Gaussian: blurs image, preserves details only for small σ.
Common Problems
Mean: blurs image, removes simple noise, no details are preserved
Gaussian: blurs image, preserves details only for small σ.
Median: preserves some details, good at removing strong noise
Common Problems
Mean: blurs image, removes simple noise, no details are preserved
Gaussian: blurs image, preserves details only for small σ.
Median: preserves some details, good at removing strong noise
Can we find a filter that not only smooths regions but preserves sharp features such as edges?
Common ProblemsMean: blurs image, removes simple noise, no details
are preservedGaussian: blurs image, preserves details only for small
σ.Median: preserves some details, good at removing
strong noise
Can we find a filter that not only smooths regions but preserves sharp features such as edges?
- yes, Tomasi's Bilateral Filter!
Bilateral Filter Example
Gaussian filter Bilateral filter
Bilateral Filter Example
Gaussian filter Bilateral filter
1D Graphical ExampleCenter Sample u
It is clear that in weighting this neighborhood, we would like to preserve the step
Neighborhood
I(p)
p
)(uneighborp
The Weights
)2
)(exp()( 2
2
cc
pupW
p
I(p)
Filtered Values
)2
)(exp()( 2
2
cc
pupW
p
I(p)
Filtered value
Edges Are Smoothed
)2
)(exp()( 2
2
cc
pupW
p
I(p)
Filtered value
What Causes the Problem?
)2
)(exp()( 2
2
cc
pupW
p
I(p)
Filtered value
What Causes the Problem?
)2
)(exp()( 2
2
cc
pupW
p
I(p)
Filtered value
Same weights for these two pixels!!
The Weights
)2
))()((exp()( 2
2
ss
pIuIpW
)2
)(exp()( 2
2
cc
pupW
p
I(p)
Bilateral filter
2
2 2
2
2 2
2 2
2 2
( )'
c s
c s
I u I pu p
p N u
I u I pu p
p N u
e e I pI u
e e
Denoise Feature preserving
Normalization
Bilateral Filtering
Filter Parameters
As proposed by Tomasi and Manduchi, the filter is controlled by 3 parameters:
The filter can be applied for several iterations in order to further strengthen its edge-preserving smoothing
N(u) – The neighbor size of the filter support,c – The variance of the spatial distances,s – The variance of the value distances,
)(
)(
)(*)(
)(*)(*)()(
uNp sc
uNp sc
pWpW
pIpWpWuI
Bilateral Filter Results
Original
Bilateral Filter Results
σc = 3, σs = 3
Bilateral Filter Results
σc = 6, σs = 3
Bilateral Filter Results
σc = 12, σs = 3
Bilateral Filter Results
σc = 12, σs = 6
Bilateral Filter Results
σc = 15, σs = 8
Summary– Template matching (SSD or Normxcorr2)
• SSD can be done with linear filters, is sensitive to overall intensity
– Gaussian pyramid• Coarse-to-fine search, multi-scale detection
– Laplacian pyramid• More compact image representation• Can be used for compositing in graphics
– Image denoising