Post on 20-Dec-2015
CSCE 641 Computer Graphics:Image Filtering & Feature Detection
Jinxiang Chai
Review
2D Fourier Transform
Nyquist sampling theory - Sampling Rate ≥ 2 * max frequency in the signal
Antialiasing - prefiltering
Gaussian pyramid - Mipmap
Review
2D Fourier Transform
Nyquist sampling theory - Sampling Rate ≥ 2 * max frequency in the signal
Antialiasing - prefiltering
Gaussian pyramid - Mipmap
Review: 2D Fourier Transform
Fourier Transform:
Inverse Fourier transform:
dvduevuFyxf vyuxi )(2),(),(
dydxeyxfvuF vyuxi )(2),(),(
Review: Nyquist Sampling Rate
T 2T…-2T -T… 0
x
fs(x)
f(x)
x
Fs(u)
u-fmax fmax
F(u)
u-fmax fmax
)()( uboxuFs
Inverse Fourier transform
Fourier transform
Sampling Rate ≥ 2 * max frequency in the signal!
Review: Antialiasing
Increase the sampling rate to above twice the highest frequency
Introduce an anti-aliasing filter to reduce fmax
u-fmax fmax
Fs(u)
u-fmax fmax
Review: Antialiasing
Increase the sampling rate to above twice the highest frequency
Introduce an anti-aliasing filter to reduce fmax
u-fmax fmax
Fs(u)
u-fmax fmax
- Sinc filter
- Mean filter
- Triangular filter
- Gaussian filter
Known as a Gaussian Pyramid– MipMap (Williams, 1983)
Review: Image Pyramids
Outline
Image Processing
- Median filtering
- Bilateral filtering
- Edge detection
- Corner detection
Filtering
input Gaussian filter
Median Filter
• For each neighbor in image, sliding the window
• Sort pixel values• Set the center pixel to the median
Median Filter
input Gaussian filter Median filter
Median Filter Examples
input Median 7X7
Median Filter Examples
Median 11X11Median 3X3
Median Filter Examples
Median 11X11Median 3X3
Straight edges kept
Sharp features lost
Median Filter Properties
Can remove outliers (peppers and salts)
Window size controls size of structure
Preserve some details but sharp corners and edges might get lost
Comparison of Mean, Gaussian, and Median
original Mean with 6 pixels
Comparison of Mean, Gaussian, and Median
original Gaussian with 6 pixels
Comparison of Mean, Gaussian, and Median
original Median with 6 pixels
Common Problems
Mean: blurs image, removes simple noise, no details are preserved
Common Problems
Mean: blurs image, removes simple noise, no details are preserved
Gaussian: blurs image, preserves details only for small σ.
Common Problems
Mean: blurs image, removes simple noise, no details are preserved
Gaussian: blurs image, preserves details only for small σ.
Median: preserves some details, good at removing strong noise
Common Problems
Mean: blurs image, removes simple noise, no details are preserved
Gaussian: blurs image, preserves details only for small σ.
Median: preserves some details, good at removing strong noise
Can we find a filter that not only smooths regions but preserves sharp features such as edges?
Common Problems
Mean: blurs image, removes simple noise, no details are preserved
Gaussian: blurs image, preserves details only for small σ.
Median: preserves some details, good at removing strong noise
Can we find a filter that not only smooths regions but preserves sharp features such as edges?
- yes, bilateral filter
Outline
Image Processing
- Median filtering
- Bilateral filtering
- Edge detection
- Corner detection
What Is Bilateral Filter?
Bilateral
- Affecting or undertaken by two sides equally
Property:
- Convolution filter
- Smooth image but preserve edges
- Operates in the domain and the range of image
Bilateral Filter Example
Gaussian filter Bilateral filter
Bilateral Filter Example
Gaussian filter Bilateral filter
1D Graphical ExampleCenter Sample u
It is clear that in weighting this neighborhood, we would like to preserve the step
Neighborhood
I(p)
p
)(uneighborp
The Weights
)2
)(exp()( 2
2
c
c
pupW
p
I(p)
Filtered Values
)2
)(exp()( 2
2
c
c
pupW
p
I(p)
Filtered value
Edges Are Smoothed
)2
)(exp()( 2
2
c
c
pupW
p
I(p)
Filtered value
What Causes the Problem?
)2
)(exp()( 2
2
c
c
pupW
p
I(p)
Filtered value
What Causes the Problem?
)2
)(exp()( 2
2
c
c
pupW
p
I(p)
Filtered value
Same weights for these two pixels!!
The Weights
)2
))()((exp()( 2
2
s
s
pIuIpW
)2
)(exp()( 2
2
c
c
pupW
p
I(p)
Bilateral filter
2
2 2
2
2 2
2 2
2 2
( )
'
c s
c s
I u I pu p
p N u
I u I pu p
p N u
e e I p
I u
e e
Denoise Feature preserving
Normalization
Bilateral FilteringBilateral Filtering
Kernel Properties
• Per each sample, we can define a ‘Kernel’ that averages its neighborhood
• This kernel changes from sample to sample!• The sum of the kernel entries is 1 due to the
normalization,• The center entry in the kernel is the largest,• Subject to the above, the kernel can take any
form (as opposed to filters which are monotonically decreasing).
p sc
p sc
pWpW
pIpWpWuI
)(*)(
)(*)(*)()(
Filter Parameters
As proposed by Tomasi and Manduchi, the filter is controlled by 3 parameters:
The filter can be applied for several iterations in order to further strengthen its edge-preserving smoothing
N(u) – The neighbor size of the filter support,c – The variance of the spatial distances,s – The variance of the value distances,
)(
)(
)(*)(
)(*)(*)()(
uNp sc
uNp sc
pWpW
pIpWpWuI
Bilateral Filter
input
Bilateral Filter
input
Bilateral Filter
input
Wc
Bilateral Filter
input
Wc Ws
Bilateral Filter
input
Wc Ws Ws*Wc
Bilateral Filter
input
Wc Ws Ws*Wc
Output
Bilateral Filter Results
Original
Bilateral Filter Results
σc = 3, σs = 3
Bilateral Filter Results
σc = 6, σs = 3
Bilateral Filter Results
σc = 12, σs = 3
Bilateral Filter Results
σc = 12, σs = 6
Bilateral Filter Results
σc = 15, σs = 8
Additional Comments
The bilateral filter is a powerful filter: - Can work with any reasonable distance function
Ws and Wc,
- Easily extended to higher dimension signals, e.g. Images, video, mesh, animation data etc.
- Easily extended to vectored-signals, e.g. RGB images or RGBD images etc.
[Fleishman et al, siggraph 03]Bilateral Mesh Denoising
Readings on Image Filtering
• Section 3.3.1 in Szeliski book
• Tomasi's Bilateral Filter
• more about bilateral filter, click here
Outline
Image Processing
- Median filtering
- Bilateral filtering
- Edge detection
- Corner detection
Edge Detection
What are edges in this image?
Edge Detection
What are edges in this image?
Edge Detection
One of the most important image processing problems
- really easy for human - really difficult for computers - fundamental for computer vision - very important for many graphics applications
How to tell a pixel is on the edge?
Edge Detection
One of the most important image processing problems
- really easy for human - really difficult for computers - fundamental for computer vision - very important for many graphics applications
How to tell a pixel is on the edge?
Edge Detection
One of the most important image processing problems
- really easy for human - really difficult for computers - fundamental for computer vision - very important for many graphics applications
How to tell a pixel is on the edge?
Edge Detection
One of the most important image processing problems
- really easy for human - really difficult for computers - fundamental for computer vision - very important for many graphics applications
How to tell a pixel is on the edge?- Related to the first-derivative.
- How to compute the first-derivatives of an image?
Gradient
A vector variable - Direction of the maximum growth of the function
- Magnitude of the growth
- Perpendicular to the edge direction
2
)/
/tan(
),(
),(
22
xf
yfa
y
f
x
fyxf
yy
fx
x
fyxf
),( yxf
Gradients
How to Calculate Gradient?
How to compute ?x
f
Intensity
Pixel column
The Good ole’ Taylor Series
Subtracting the second from the first we obtain
...)(''2
1)(')()( 2 xfhxhfxfhxf
...)(''2
1)(')()( 2 xfhxhfxfhxf
or…
)(2
)()()(' 2hO
h
hxfhxfxf
Discrete Gradient Estimation
For discrete functions, we can use the first order approximation of the gradient
where h corresponds to the step size
h
hxfhxfxf
2
)()()('
Discrete Gradient Estimation
For discrete functions, we can use the first order approximation of the gradient
where h corresponds to the step size
For our purposes, h corresponds to the width of 1 pixel =>
h
hxfhxfxf
2
)()()('
2
)1,()1,(),(2
),1(),1(),(
yxIyxI
y
yxI
yxIyxI
x
yxI
Discrete Gradient Estimation
So how can we compute the image gradient efficiently?
- Using our good old friend convolution!
1
0
1
*
101*
Iy
I
Ix
I
- Dropped off the “divide by 2” for speed considerations.- This only scales the gradient.
Taking the Discrete Derivative
] 1 0 1[
abs()
Basic Edge Detection Step 1
INPUT IMAGE
1) EdgeEnhancement
Horizontal [-1 0 1]
Vertical [-1 0 1]T
),( yxI
x
yxI
),(
y
yxI
),(
But isn’t edge detection susceptible to noise?
Basic Edge Detection Steps 1-2
INPUT IMAGE
2) EdgeEnhancement
Horizontal [-1 0 1]
Vertical [-1 0 1]T
),( yxI
x
yxI
),(
y
yxI
),(
1) NoiseSmoothing
16/
121
242
121
Discrete Gradient EstimationRemember that the gradient is a vector and we have calculated the coefficients in the x and y directions at each point in the image
After convolving, we get the magnitude of the gradient from at each point (pixel) from
In practice, we often sum the absolute values of the components for computational efficiency
22
),(
y
I
x
IyxG
Basic Edge Detection (cont’d)
INPUT IMAGE
1) NoiseSmoothing
2) EdgeEnhancement
Horizontal [-1 0 1]
Vertical [-1 0 1]T
),( yxI
x
yxI
),(
y
yxI
),(
2
122 ),(),(
|),(|
y
yxI
x
yxIyxI
“GRADIENT” IMAGE
16/
121
242
121
Basic Edge Detection (cont’d)
INPUT IMAGE
1) NoiseSmoothing
2) EdgeEnhancement
Horizontal [-1 0 1]
Vertical [-1 0 1]T
),( yxI
x
yxI
),(
y
yxI
),(
2
122 ),(),(
|),(|
y
yxI
x
yxIyxI
“GRADIENT” IMAGE
16/
121
242
121
What does the gradient image mean?
Basic Edge Detection (cont’d)
INPUT IMAGE
1) NoiseSmoothing
2) EdgeEnhancement
Horizontal [-1 0 1]
Vertical [-1 0 1]T
),( yxI
x
yxI
),(
y
yxI
),(
2
122 ),(),(
|),(|
y
yxI
x
yxIyxI
“GRADIENT” IMAGE
16/
121
242
121
What does the gradient image mean?
- Magnitude of intensity changes around each pixel
Discrete Gradient EstimationSo how do we segment the edges from the rest of the scene?
- Thresholding!
The Effects of ThresholdingWhile edge features are independent of illumination, the edge strength is not!
Results from threshold values of 50 and 100
Basic Edge Detection Summary
INPUT IMAGE
1) NoiseSmoothing
EDGE IMAGE
2) EdgeEnhancement
Horizontal [-1 0 1]
Vertical [-1 0 1]T
),( yxI
x
yxI
),(
y
yxI
),(
2
122 ),(),(
|),(|
y
yxI
x
yxIyxI
“GRADIENT” IMAGE
3)Threshold
16/
121
242
121
The effects of Filtering Noise
Threshold20
Gaussian SmoothingUnsmoothed Edges
Threshold50
Sobel Edge DetectionOne of the dominant edge detection schemes uses the Sobel
operators
Convolving each of these with the original image generates horizontal and vertical gradient images that are combined as before
101
202
101
VSobel
121
000
121
HSobel
Sobel Edge Detection
One of the dominant edge detection schemes uses the Sobel operators
101
202
101
VSobel
121
000
121
HSobel
- Can be approximated as a derivative of Gaussian
- First Gaussian smoothing and then compute derivatives
Ix
G
x
IG
)(
Sobel Edge Detection
One of the dominant edge detection schemes uses the Sobel operators
101
202
101
VSobel
121
000
121
HSobel
- Can be approximated as a derivative of Gaussian
- First Gaussian smoothing and then compute derivatives
Ix
G
x
IG
)( Why?
Sobel Edge Detection
One of the dominant edge detection schemes uses the Sobel operators
101
202
101
VSobel
121
000
121
HSobel
- Can be approximated as a derivative of Gaussian
- First Gaussian smoothing and then compute derivatives
- In practice we may still need to smooth for noise
Ix
G
x
IG
)(
Robert and Prewitt Edge Detectors
The Prewitt is similar to the Sobel, but uses a different kernel
Roberts was an early edge detector kernel
101
101
101
VP
111
000
111
HP
11
111R
11
112R
Sobel Edge Detector
Java Applet
Second Derivative Edge Detector
Sobel Operator can produce thick edges; ideally we are looking for infinitely thin boundaries
Second Derivative Edge Detector
Sobel Operator can produce thick edges; ideally we are looking for infinitely thin boundaries
An alternative approach is to look for local extrema in the first derivative
A peak in the first derivative corresponds what in the second derivative?
Second Derivative Edge Detector
Sobel Operator can produce thick edges; ideally we are looking for infinitely thin boundaries
An alternative approach is to look for local extrema in the first derivative
A peak in the first derivative corresponds what in the second derivative?
Localization with Laplacian
An equivalent measure of the second derivative in 2D is the Laplacian:
Numerically, we approximate the Laplacian using the following filtering kernel:
Zeros crossing of the filter corresponds to positions of maximum gradient.
- can be used for detecting edges - might be sensitive to noise - need to filter the image
The Laplacian of Gaussian Kernel
),()()),(( 22 yxIGyxIG
Review: Properties
Linearity:
Time shift:
Derivative:
Integration:
Convolution:
)()()()( ubGuaFxbgxaf
)()( 020 uFexxf uxi
)()(
uuFx
xdf
u
uFdxxf
)()(
)()()()( uGuFxgxf
)()()()( uGuFxgxf
The Laplacian of Gaussian Kernel
),()()),(( 22 yxIGyxIG
2
22
24
222222 2
yx
eyx
x
G
x
GG
The Laplacian of Gaussian Filter Kernel
0.1
4.1
Laplacian Edge Detector
Java applet: Click here
Readings
• Section 4.2 in Szeliski book
Outline
Image Processing
- Median filtering
- Bilateral filtering
- Edge detection
- Corner detection
Corner Detection
Why do we need to detect corners?
- Corners correspond to points in both the world and image spaces
- Very important for image based rendering and modeling
- Very useful in computer vision
What are Corners?
Point features
What are Corners?
Point features
Where two edges come together
Where the image gradient has significant components in the x and y direction
We will establish corners from the gradient rather than the edge images
Basic Ideas
What are gradients along x and y directions?
Basic Ideas
What are gradients along x and y directions?
Basic Ideas
What are gradients along x and y directions?
How to measure corners based on the gradient images?
Basic Ideas
What are gradients along x and y directions?
How to measure corners based on the gradient images?How to measure corners based on the gradient images?
- two major axes in the local window!
Basic Idea
• Corners: two major axes of gradients in the local window
• How to evaluate this?
Basic Idea
• Corners: two major axes of gradients in the local window
• How to evaluate this?
y
x
I
IGradient vector
Basic Idea
Ix
Iy
Basic Idea
Ix
Iy
Ix
Iy
Ix
Iy
Finding Two Major Axes
• Compute covariance matrix
• Compute two eigen values: λ1 and λ2, which indicate magnitude of two dominant axes
• If both λ1 and λ2 are large, then we have two dominant axes (corners).
1. Compute the image gradients
2. Define a neighborhood size as an area of interest around each pixel
Corner Detection Algorithm
6160531918
5855531513
5555501313
1010101111
1012121110
y
yxII
x
yxII yx
),(,
),(
3x3 neighborhood
3. For each image pixel (i,j), construct the following matrix from it and its neighborhood values
e.g.
Corner Detection Algorithm (cont’d)
6160531918
5855531513
5555501313
1010101111
1012121110
xI
2
2
),(yyx
yxx
T
y
x
y
xji III
III
I
I
I
IC
22222
2222)3,3(
5553155550
13101011]1,1[
C
Similar to covariance matrix (Ix,Iy)T!
Corner Detection Algorithm (cont’d)
4. For each matrix C(i,j), determine the 2 eigenvalues λ(i.j)= [λ1, λ2].
- Eigenvalues indicate magnitude of two dominant axes
- If either λ1 or λ2 is close to zero, then this is not a corner.
Simple case:
Corner Detection Algorithm (cont’d)
4. For each matrix C(i,j), determine the 2 eigenvalues λ(i.j)= [λ1, λ2].
Simple case:
Isolated pixelsInterior Region Edge Corner
λ1, λ2=0 Large λ1 and small λ2 Large λ1 and large λ2 small λ1 and small λ2
Corner Detection Algorithm (cont’d)
4. For each matrix C(i,j), determine the 2 eigenvalues λ(i.j)= [λ1, λ2].
- This is just a rotated version of the one on last slide
- If either λ1 or λ2 is close to zero, then this is not a corner.
General case:
Eigen-values and Corner
- λ1 is large
- λ2 is large
Eigen-values and Corner
- λ1 is large
- λ2 is small
Eigen-values and Corner
- λ1 is small
- λ2 is small
Corner Detection Algorithm (cont’d)
4. For each matrix C(i,j), determine the 2 eigenvalues λ(i.j)= [λ1, λ2].
5. If both λ1 and λ2 are big, we have a corner (Harris also checks the ratio of λs is not too high)
ISSUE: The corners obtained will be a function of the threshold !
Image Gradients
Image Gradients
Closeup of image orientation at each pixel
The Orientation Field
Corners are detected where both λ1 and λ2 are big
The Orientation Field
Corners are detected where both λ1 and λ2 are big
Corner Detection Sample Results
Threshold=25,000 Threshold=10,000
Threshold=5,000
Results
Required Readings
• Section 4.1.1 in Szeliski book
• Harris corner detector
Homework #1
• Due by 9/23, midnight
• Submit it via csnet.
• If you do not have the csnet account, submit it to me via email.
Summary
Image Processing - Median filtering
- Bilateral filtering
- Edge detection
- Corner detection
Next Lecture
SIFT features
Image composition and blending
Image warping
Image morphing