Lecture’4:’ Pixels’and’Filters - Stanford Computer...
56
Lecture 4- Fei-Fei Li Lecture 4: Pixels and Filters Dr. Juan Carlos Niebles Stanford AI Lab Professor FeiFei Li Stanford Vision Lab 6Oct16 1
Transcript of Lecture’4:’ Pixels’and’Filters - Stanford Computer...
Lecture 4-Fei-Fei Li
Lecture 4: Pixels and Filters
Dr. Juan Carlos NieblesStanford AI Lab
Professor Fei-‐Fei LiStanford Vision Lab
6-‐Oct-‐161
Lecture 4-Fei-Fei Li
What we will learn today?
• Images as functions• Linear systems (filters)• Convolution and correlation
6-‐Oct-‐162
Some background reading:Forsyth and Ponce, Computer Vision, Chapter 7
Lecture 4-Fei-Fei Li
• An image contains discrete number of pixels– A simple example– Pixel value:• “grayscale” (or “intensity”): [0,255]
3
Images as functions
231
75
148
6-‐Oct-‐16
Lecture 4-Fei-Fei Li
• An image contains discrete number of pixels– A simple example– Pixel value:• “grayscale” (or “intensity”): [0,255]• “color”
– RGB: [R, G, B]– Lab: [L, a, b]– HSV: [H, S, V]
4
Images as functions
[249, 215, 203]
[90, 0, 53]
[213, 60, 67]
6-‐Oct-‐16
Lecture 4-Fei-Fei Li 6-‐Oct-‐165
Images as functions• An Image as a function f from R2 to RM:
• f( x, y ) gives the intensity at position ( x, y ) • Defined over a rectangle, with a finite range:
f: [a,b] x [c,d ] à [0,255]Domainsupport
range
Lecture 4-Fei-Fei Li 6-‐Oct-‐166
Images as functions• An Image as a function f from R2 to RM:
• f( x, y ) gives the intensity at position ( x, y ) • Defined over a rectangle, with a finite range:
f: [a,b] x [c,d ] à [0,255]
( , )( , ) ( , )
( , )
r x yf x y g x y
b x y=• A color image:
Domainsupport
range
Lecture 4-Fei-Fei Li 6-‐Oct-‐167
• Images are usually digital (discrete):– Sample the 2D space on a regular grid
• Represented as a matrix of integer valuespixel
Images as discrete functions
Lecture 4-Fei-Fei Li 6-‐Oct-‐168
Cartesian coordinates
Images as discrete functions
Notation for discrete functions
Lecture 4-Fei-Fei Li
What we will learn today?
• Images as functions• Linear systems (filters)• Convolution and correlation
6-‐Oct-‐169
Some background reading:Forsyth and Ponce, Computer Vision, Chapter 7
Lecture 4-Fei-Fei Li 6-‐Oct-‐1610
Systems and Filters• Filtering:– Form a new image whose pixels are a combination original pixel values
Goals: -‐Extract useful information from the images• Features (edges, corners, blobs…)
-‐ Modify or enhance image properties: • super-‐resolution; in-‐painting; de-‐noising
Lecture 4-Fei-Fei Li 6-‐Oct-‐1611
Super-‐resolutionDe-‐noising
In-‐painting
Bertamioet al
Lecture 4-Fei-Fei Li
2D discrete-‐space systems (filters)
6-‐Oct-‐1612
Lecture 4-Fei-Fei Li
Filter example #1: Moving Average
• 2D DS moving average over a 3 × 3 window of neighborhood
6-‐Oct-‐1613
111
111
111
],[],[91],)[(
,
lnkmhlkfnmhflk
=
h
Lecture 4-Fei-Fei Li 6-‐Oct-‐1614
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Courtesy of S. Seitz( f ∗h)[m,n]= f [k, l]k,l∑ h[m− k,n− l]
Filter example #1: Moving Average
Lecture 4-Fei-Fei Li 6-‐Oct-‐1615
15
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Filter example #1: Moving Average
( f ∗h)[m,n]= f [k, l]k,l∑ h[m− k,n− l]
Lecture 4-Fei-Fei Li 6-‐Oct-‐1616
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Filter example #1: Moving Average
( f ∗h)[m,n]= f [k, l]k,l∑ h[m− k,n− l]
Lecture 4-Fei-Fei Li 6-‐Oct-‐1617
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Filter example #1: Moving Average
( f ∗h)[m,n]= f [k, l]k,l∑ h[m− k,n− l]
Lecture 4-Fei-Fei Li 6-‐Oct-‐1618
0 10 20 30 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Filter example #1: Moving Average
( f ∗h)[m,n]= f [k, l]k,l∑ h[m− k,n− l]
Lecture 4-Fei-Fei Li 6-‐Oct-‐1619
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30 30 30 20 10
0 20 40 60 60 60 40 20
0 30 60 90 90 90 60 30
0 30 50 80 80 90 60 30
0 30 50 80 80 90 60 30
0 20 30 50 50 60 40 20
10 20 30 30 30 30 20 10
10 10 10 0 0 0 0 0
Source: S. Seitz
Filter example #1: Moving Average
( f ∗h)[m,n]= f [k, l]k,l∑ h[m− k,n− l]
Lecture 4-Fei-Fei Li 6-‐Oct-‐1620
In summary:• Replaces each pixel with an average of its neighborhood.
• Achieve smoothing effect (remove sharp features)
111
111
111
h[⋅ , ⋅ ]
Filter example #1: Moving Average
Lecture 4-Fei-Fei Li 6-‐Oct-‐1621
Filter example #1: Moving Average
Lecture 4-Fei-Fei Li 6-‐Oct-‐1622
• Image segmentation based on a simple threshold:
Filter example #2: Image Segmentation
255,
Lecture 4-Fei-Fei Li
Classification of systems
6-‐Oct-‐1623
• Amplitude properties• Linearity• Stability• Invertibility
• Spatial properties• Causality• Separability• Memory• Shift invariance• Rotation invariance
Lecture 4-Fei-Fei Li
Shift-‐invariance
6-‐Oct-‐1624
If then
for every input image f[n,m] and shifts n0,m0
Lecture 4-Fei-Fei Li
Is the moving average system is shift invariant?
6-‐Oct-‐1625
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30 30 30 20 10
0 20 40 60 60 60 40 20
0 30 60 90 90 90 60 30
0 30 50 80 80 90 60 30
0 30 50 80 80 90 60 30
0 20 30 50 50 60 40 20
10 20 30 30 30 30 20 10
10 10 10 0 0 0 0 0
Lecture 4-Fei-Fei Li
Is the moving average system is shift invariant?
6-‐Oct-‐1626
Yes!
Lecture 4-Fei-Fei Li
Linear Systems (filters)
6-‐Oct-‐1627
• Linear filtering:– Form a new image whose pixels are a weighted sum of original pixel values
– Use the same set of weights at each point
• S is a linear system (function) iff it S satisfies
superposition property
Lecture 4-Fei-Fei Li
• Is the moving average a linear system?
• Is thresholding a linear system?– f1[n,m] + f2[n,m] > T– f1[n,m] < T– f2[n,m]<T
6-‐Oct-‐1628
Linear Systems (filters)
No!
Lecture 4-Fei-Fei Li
LSI (linear shift invariant) systems
6-‐Oct-‐1629
Impulse response
Lecture 4-Fei-Fei Li
LSI (linear shift invariant) systems
6-‐Oct-‐1630
111
111
111
h
Example: impulse response of the 3 by 3 moving average filter:
Lecture 4-Fei-Fei Li
LSI (linear shift invariant) systems
6-‐Oct-‐1631
An LSI system is completely specified by its impulse response.
superposition
sifting property of the delta function
Discrete convolution
f [n,m] ⇤ h[n,m]
Lecture 4-Fei-Fei Li
What we will learn today?
• Images as functions• Linear systems (filters)• Convolution and correlation
6-‐Oct-‐1632
Some background reading:Forsyth and Ponce, Computer Vision, Chapter 7
Lecture 4-Fei-Fei Li
Discrete convolution (symbol: )
6-‐Oct-‐1633
• Fold h[k,l] about origin to form h[−k,−l]• Shift the folded results by n,m to form h[n − k,m − l]• Multiply h[n − k,m − l] by f[k, l]• Sum over all k,l• Repeat for every n,m
h[k,l]
h[-‐k,-‐l] h[n-‐k,m-‐l]
n
Lecture 4-Fei-Fei Li 6-‐Oct-‐1634
• Fold h[k,l] about origin to form h[−k,−l]• Shift the folded results by n,m to form h[n − k,m − l]• Multiply h[n − k,m − l] by f[k, l]• Sum over all k,l• Repeat for every n,m
h[k,l]
h[-‐k,-‐l] h[n-‐k,m-‐l]
n
Discrete convolution (symbol: )
Lecture 4-Fei-Fei Li 6-‐Oct-‐1635
• Fold h[k,l] about origin to form h[−k,−l]• Shift the folded results by n,m to form h[n − k,m − l]• Multiply h[n − k,m − l] by f[k, l]• Sum over all k,l• Repeat for every n,m
h[n-‐k,m-‐l]
f[k,l]
f[k,l] x h[n-‐k,m-‐l]
Sum (f[k,l] x h[n-‐k,m-‐l])
n
Discrete convolution (symbol: )
Lecture 4-Fei-Fei Li
Convolution in 2D -‐ examples
6-‐Oct-‐1636
•0•0•0
•0•1•0
•0•0•0
Original
?=
Courtesy of D
Lowe
*
Lecture 4-Fei-Fei Li
Convolution in 2D -‐ examples
6-‐Oct-‐1637
•0•0•0
•0•1•0
•0•0•0
Original Filtered (no change)
=
Courtesy of D
Lowe
*
Lecture 4-Fei-Fei Li
Convolution in 2D -‐ examples
6-‐Oct-‐1638
•0•0•0
•1•0•0
•0•0•0
Original
?=
Courtesy of D
Lowe
*
Lecture 4-Fei-Fei Li
Convolution in 2D -‐ examples
6-‐Oct-‐1639
•0•0•0
•1•0•0
•0•0•0
Shifted rightBy 1 pixel
=
Courtesy of D
LoweOriginal
*
Lecture 4-Fei-Fei Li
Convolution in 2D -‐ examples
6-‐Oct-‐1640
?•1•1•1
•1•1•1
•1•1•1
=
Courtesy of D
LoweOriginal
*
Lecture 4-Fei-Fei Li
Convolution in 2D -‐ examples
6-‐Oct-‐1641
•1•1•1
•1•1•1
•1•1•1
Blur (with abox filter)
=
Courtesy of D
LoweOriginal
*
Lecture 4-Fei-Fei Li
Convolution in 2D -‐ examples
6-‐Oct-‐1642
•1•1•1•1•1•1•1•1•1
•0•0•0•0•2•0•0•0•0 - = ?
(Note that filter sums to 1)
•1•1•1•1•1•1•1•1•1
•0•0•0•0•1•0•0•0•0 -
•0•0•0•0•1•0•0•0•0
+
“details of the image”
Courtesy of D
Lowe
Original
Lecture 4-Fei-Fei Li 6-‐Oct-‐1643
• What does blurring take away?
original smoothed (5x5)
–
detail
=
sharpened
=
• Let’s add it back:
original detail
+ a
Lecture 4-Fei-Fei Li
Convolution in 2D –Sharpening filter
6-‐Oct-‐1644
•1•1•1•1•1•1•1•1•1
•0•0•0•0•2•0•0•0•0 -
Sharpening filter: Accentuates differences with local average
=
Courtesy of D
Lowe
Original
Lecture 4-Fei-Fei Li
Image support and edge effect
6-‐Oct-‐1645
•A computer will only convolve finite support signals. • That is: images that are zero for n,m outside some rectangular region
•MATLAB’s conv2 performs 2D DS convolution of finite-‐support signals.
N1 ×M1
N2 ×M2 (N1 + N2 − 1) × (M1 +M2 − 1)* =
Lecture 4-Fei-Fei Li
Image support and edge effect
6-‐Oct-‐1646
•A computer will only convolve finite support signals. •What happens at the edge?
fh
• zero “padding”• edge value replication• mirror extension• more (beyond the scope of this class)
-‐> Matlab conv2 uses zero-‐padding
Lecture 4-Fei-Fei Li
What we will learn today?
• Images as functions• Linear systems (filters)• Convolution and correlation
6-‐Oct-‐1647
Some background reading:Forsyth and Ponce, Computer Vision, Chapter 7
Lecture 4-Fei-Fei Li
(Cross) correlation (symbol: )
6-‐Oct-‐1648
Cross correlation of two 2D signals f[n,m] and g[n,m]
• Equivalent to a convolution without the flip
(k, l) is called the lag
(g* is defined as the complex conjugate of g. In this class, g(n,m) are real numbers, hence g*=g.)
rfg[n,m] = f [n,m] ⇤ g⇤[�n,�m]
Lecture 4-Fei-Fei Li
(Cross) correlation – example
6-‐Oct-‐1649
MATLAB’s xcorr2
Courtesy of J. Fessler
Lecture 4-Fei-Fei Li 6-‐Oct-‐1650
(Cross) correlation – exampleLeft Right
scanline
Norm. cross corr. score
Lecture 4-Fei-Fei Li 6-‐Oct-‐1651
Convolution vs. (Cross) Correlation
Lecture 4-Fei-Fei Li
Convolution vs. (Cross) Correlation• A convolution is an integral that expresses the amount of overlap of one function as it is shifted over another function. – convolution is a filtering operation
• Correlation compares the similarity of two sets of data. Correlation computes a measure of similarity of two input signals as they are shifted by one another. The correlation result reaches a maximum at the time when the two signals match best .– correlation is a measure of relatedness of two signals
6-‐Oct-‐1652
Lecture 4-Fei-Fei Li
Figure from “Computer Vision for Interactive Computer Graphics,” W.Freeman et al, IEEE Computer Graphics and Applications, 1998 copyright 1998, IEEE
Cross Correlation Application: Vision system for TV remote control
- uses template matching
6-‐Oct-‐1653
Lecture 4-Fei-Fei Li
properties
6-‐Oct-‐1654
• Commutative property:
• Associative property:
• Distributive property:
The order doesn’t matter!
Lecture 4-Fei-Fei Li
properties
6-‐Oct-‐1655
• Shift property:
• Shift-‐invariance:
Lecture 4-Fei-Fei Li
What we have learned today?
• Images as functions• Linear systems (filters)• Convolution and correlation
6-‐Oct-‐1656
