Lecture 36: Course review CS4670 / 5670: Computer Vision Noah Snavely.
Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid...
-
Upload
marion-marshall -
Category
Documents
-
view
223 -
download
0
Transcript of Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid...
![Page 1: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/1.jpg)
Lecture 1: Images and image filtering
CS4670/5670: Intro to Computer VisionKavita Bala
Hybrid Images, Oliva et al., http://cvcl.mit.edu/hybridimage.htm
![Page 2: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/2.jpg)
Lecture 1: Images and image filtering
CS4670: Computer VisionKavita Bala
Hybrid Images, Oliva et al., http://cvcl.mit.edu/hybridimage.htm
![Page 3: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/3.jpg)
Lecture 1: Images and image filtering
CS4670: Computer VisionKavita Bala
Hybrid Images, Oliva et al., http://cvcl.mit.edu/hybridimage.htm
![Page 4: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/4.jpg)
CS4670: Computer VisionKavita Bala
Hybrid Images, Oliva et al., http://cvcl.mit.edu/hybridimage.htm
Lecture 1: Images and image filtering
![Page 5: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/5.jpg)
Reading and Announcements
• Szeliski, Chapter 3.1-3.2
• For CMS, wait and then contact Randy Hess ([email protected])
![Page 6: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/6.jpg)
What is an image?
![Page 7: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/7.jpg)
What is an image?
Digital Camera
The EyeSource: A. Efros
We’ll focus on these in this class
(More on this process later)
![Page 8: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/8.jpg)
What is an image?• A grid (matrix) of intensity values
(common to use one byte per value: 0 = black, 255 = white)
=
255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 20 0 255 255 255 255 255 255 255
255 255 255 75 75 75 255 255 255 255 255 255
255 255 75 95 95 75 255 255 255 255 255 255
255 255 96 127 145 175 255 255 255 255 255 255
255 255 127 145 175 175 175 255 255 255 255 255
255 255 127 145 200 200 175 175 95 255 255 255
255 255 127 145 200 200 175 175 95 47 255 255
255 255 127 145 145 175 127 127 95 47 255 255
255 255 74 127 127 127 95 95 95 47 255 255
255 255 255 74 74 74 74 74 74 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255
![Page 9: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/9.jpg)
Images as functions
• An image contains discrete numbers of pixels
• Pixel value– grayscale/intensity
• [0,255]
– Color• RGB [R, G, B], where [0,255] per channel• Lab [L, a, b]: Lightness, a and b are color-opponent
dimensions• HSV [H, S, V]: Hue, saturation, value
![Page 10: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/10.jpg)
Images as functions
• Can think of image as a function, f, from R2 to R or RM:– Grayscale: f (x,y) gives intensity at position (x,y)
• f: [a,b] x [c,d] [0,255]
– Color: f (x,y) = [r(x,y), g(x,y), b(x,y)]
![Page 11: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/11.jpg)
A digital image is a discrete (sampled, quantized) version of this function
What is an image?
x
y
f (x, y)
![Page 12: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/12.jpg)
Image transformations• As with any function, we can apply operators
to an image
g (x,y) = f (x,y) + 20 g (x,y) = f (-x,y)
![Page 13: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/13.jpg)
Image transformations• As with any function, we can apply operators
to an image
g (x,y) = f (x,y) + 20 g (x,y) = f (-x,y)
![Page 14: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/14.jpg)
Filters
• Filtering– Form a new image whose pixels are a combination
of the original pixels• Why?
– To get useful information from images• E.g., extract edges or contours (to understand shape)
– To enhance the image• E.g., to blur to remove noise• E.g., to sharpen to “enhance image” a la CSI
![Page 15: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/15.jpg)
Super-resolution
Noise reduction
![Page 16: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/16.jpg)
Noise reduction• Given a camera and a still scene, how can
you reduce noise?
Take lots of images and average them!
How to formulate as filtering?Source: S. Seitz
![Page 17: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/17.jpg)
Image filtering• Modify the pixels in an image based on some
function of a local neighborhood of each pixel
5 14
1 71
5 310
Local image data
7
Modified image data
Some function S
Source: L. Zhang
![Page 18: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/18.jpg)
Filters: examples
000
010
000
Original (f) Identical image (g)
Source: D. Lowe
* =Kernel (k)
![Page 19: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/19.jpg)
Filters: examples
111
111
111
Blur (with a mean filter) (g)
Source: D. Lowe
* =
Original (f)
Kernel (k)
![Page 20: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/20.jpg)
Mean filtering
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
1 1 1
1 1 1
1 1 1 *
![Page 21: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/21.jpg)
Mean filtering/Moving average
![Page 22: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/22.jpg)
Mean filtering/Moving average
![Page 23: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/23.jpg)
Mean filtering/Moving average
![Page 24: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/24.jpg)
Mean filtering/Moving average
![Page 25: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/25.jpg)
Mean filtering/Moving average
![Page 26: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/26.jpg)
Mean filtering
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
1 1 1
1 1 1
1 1 1 * =0 10 20 30 30 30 20 10
0 20 40 60 60 60 40 20
0 30 60 90 90 90 60 30
0 30 50 80 80 90 60 30
0 30 50 80 80 90 60 30
0 20 30 50 50 60 40 20
10 20 30 30 30 30 20 10
10 10 10 0 0 0 0 0
![Page 27: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/27.jpg)
Mean filtering/Moving Average
• Replace each pixel with an average of its neighborhood
• Achieves smoothing effect– Removes sharp features
111
111
111
![Page 28: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/28.jpg)
Filters: Thresholding
![Page 29: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/29.jpg)
Linear filtering• One simple version: linear filtering
– Replace each pixel by a linear combination (a weighted sum) of its neighbors
– Simple, but powerful– Cross-correlation, convolution
• The prescription for the linear combination is called the “kernel” (or “mask”, “filter”)
0.5
0.5 00
10
0 00
kernel
8
Modified image data
Source: L. Zhang
Local image data
6 14
1 81
5 310
![Page 30: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/30.jpg)
Filter Properties
• Linearity– Weighted sum of original pixel values– Use same set of weights at each point– S[f + g] = S[f] + S[g] – S[k f + m g] = k S[f] + m S[g]
![Page 31: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/31.jpg)
Linear Systems
• Is mean filtering/moving average linear?– Yes
• Is thresholding linear? – No
![Page 32: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/32.jpg)
Filter Properties
• Linearity– Weighted sum of original pixel values– Use same set of weights at each point– S[f + g] = S[f] + S[g] – S[p f + q g] = p S[f] + q S[g]
• Shift-invariance– If f[m,n] g[m,n], then f[m-p,n-q] g[m-p, n-q] – The operator behaves the same everywhere
S S
![Page 33: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/33.jpg)
Cross-correlation
This is called a cross-correlation operation:
Let be the image, be the kernel (of size 2k+1 x 2k+1), and be the output image
• Can think of as a “dot product” between local neighborhood and kernel for each pixel
![Page 34: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/34.jpg)
Convolution• Same as cross-correlation, except that the
kernel is “flipped” (horizontally and vertically)
• Convolution is commutative and associative
This is called a convolution operation:
![Page 35: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/35.jpg)
Cross-correlation
This is called a cross-correlation operation:
Let be the image, be the kernel (of size 2k+1 x 2k+1), and be the output image
• Can think of as a “dot product” between local neighborhood and kernel for each pixel
![Page 36: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/36.jpg)
Stereo head
Camera on a mobile vehicle
![Page 37: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/37.jpg)
Example image pair – parallel cameras
![Page 38: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/38.jpg)
Intensity profiles
• Clear correspondence between intensities, but also noise and ambiguity
![Page 39: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/39.jpg)
region A
Normalized Cross Correlation
region B
vector a vector b
write regions as vectors
a
b
![Page 40: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/40.jpg)
left image band
right image band
cross correlation
1
0
0.5
x
![Page 41: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/41.jpg)
Convolution• Same as cross-correlation, except that the
kernel is “flipped” (horizontally and vertically)
• Convolution is commutative and associative
This is called a convolution operation:
![Page 42: Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., //cvcl.mit.edu/hybridimage.htm.](https://reader033.fdocuments.us/reader033/viewer/2022042702/56649dd45503460f94acc44e/html5/thumbnails/42.jpg)
Convolution
Adapted from F. Durand