Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan,...

147
Acknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping and Sampling Connelly Barnes CS 4810 University of Virginia

Transcript of Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan,...

Page 1: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Acknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin

Image Filtering, Warping and Sampling

Connelly BarnesCS 4810

University of Virginia

Page 2: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Outline

2

‣ Image Processing‣ Image Warping‣ Image Sampling

Page 3: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Image Processing

3

‣ What about the case when the modification that we would like to make to a pixel depends on the pixels around it?‣ Blurring‣ Edge Detection‣ Etc.

Page 4: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Multi-Pixel Operations

4

‣ In the simplest case, we define a mask of weights which tells us how the values at adjacent pixels should be combined to generate the new value.

Page 5: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Blurring

5

‣ To blur across pixels, define a mask:‣ Whose value is largest at the center pixel‣ Whose entries sum to one

Original Blur

Filter =

Page 6: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Blurring

6

Pixel(x,y): red = 36 green = 36 blue = 0

Filter =

Page 7: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Blurring

7

Pixel(x,y): red = 36 green = 36 blue = 0

Pixel(x,y).red and its red neighbors

X - 1 X X + 1

Y - 1 36 109 146

Y 32 36 109

Y + 1 32 36 73

Filter =

Page 8: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Blurring

8

X - 1 X X + 1

Y - 1 36 109 146

Y 32 36 109

Y + 1 32 36 73

New value for Pixel(x,y).red = (36 * 1/16) + (109 * 2/16) + (146 * 1/16) (32 * 2/16) + (36 * 4/16) + (109 * 2/16) (32 * 1/16) + (36 * 2/16) + (73 * 1/16)

Pixel(x,y).red and its red neighbors

Filter =

Page 9: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Blurring

9

X - 1 X X + 1

Y - 1 36 109 146

Y 32 36 109

Y + 1 32 36 73

New value for Pixel(x,y).red = 62.69

Pixel(x,y).red and its red neighbors

Filter =

Page 10: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Blurring

10

Original

New value for Pixel(x,y).red = 63

Blur

Page 11: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Blurring

11

‣ Repeat for each pixel and each color channel‣ Note 1: Keep source and destination separate to avoid “drift”‣ Note 2: For boundary pixels, not all neighbors are used, and you

need to normalize the mask so that the sum of the values is correct

Page 12: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Blurring

12

‣ In general, the mask can have arbitrary size:‣ We can express a smaller mask as a bigger one by padding

with zeros.

Original Blur

Page 13: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Blurring

13

‣ More non-zero entries to give rise to a wider blur

Original Narrow Blur Wide Blur

Page 14: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Blurring

14

‣ A general way for defining the entries of an nxn blurring mask is to use the values of a Gaussian:

‣ σ equals the mask radius (“n/2 for an n x n mask”)‣ x is i’s horizontal distance from center pixel‣ y is j’s vertical distance from center pixel‣ Don’t forget to normalize!

Gaussian[i, j] = e�x

2+y

2

2�2

Page 15: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Bivariate Gaussian Function

15

Gaussian[i, j] = e�x

2+y

2

2�2

aka “Normal Distribution”

Page 16: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Edge Detection

16

‣ To find the edges in an image, define a mask:‣ Whose value is largest at the center pixel‣ Whose entries sum to zero.

‣ Edge pixels are those whose value is larger (or smaller) than those of its neighbors.

Original Highlighted Edges

Filter =

Page 17: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Edge Detection

17

Pixel(x,y): red = 36 green = 36 blue = 0

Filter =

Page 18: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Edge Detection

18

Pixel(x,y): red = 36 green = 36 blue = 0

X - 1 X X + 1

Y - 1 36 109 146

Y 32 36 109

Y + 1 32 36 73

Filter =

Pixel(x,y).red and its red neighbors

Page 19: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Edge Detection

19

X - 1 X X + 1

Y - 1 36 109 146

Y 32 36 109

Y + 1 32 36 73

New value for Pixel(x,y).red = (36 * -1) + (109 * -1) + (146 * -1) (32 * -1) + (36 * 8) + (109 * -1) (32 * -1) + (36 * -1) + (73 * -1)

Filter =

Pixel(x,y).red and its red neighbors

Page 20: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Edge Detection

20

X - 1 X X + 1

Y - 1 36 109 146

Y 32 36 109

Y + 1 32 36 73

New value for Pixel(x,y).red = -285

Filter =

Pixel(x,y).red and its red neighbors

Page 21: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Edge Detection

21

X - 1 X X + 1

Y - 1 36 109 146

Y 32 36 109

Y + 1 32 36 73

New value for Pixel(x,y).red = 0

Filter =

Pixel(x,y).red and its red neighbors

Page 22: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Edge Detection

22

New value for Pixel(x,y).red = 0

Page 23: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Edge Mask is a Derivative Filter

23

@G

@x

/ � x

2e�

x

2+y

2

2�2

Page 24: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Outline

24

‣ Image Processing‣ Image Warping‣ Image Sampling

Page 25: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Image Warping

25

‣ Move pixels of image‣ Mapping‣ Resampling

Source image Destination image

Warp

Page 26: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Overview

26

‣ Mapping‣ Forward‣ Reverse

‣ Resampling‣ Point sampling‣ Triangle filter‣ Gaussian filter

Page 27: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Mapping

27

‣ Transformation: describe the destination location (x,y) for every source location (u,v)

v

u

y

x

Page 28: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Example Mappings

28

‣ Scale by factor:‣ x = factor * u‣ y = factor * v

Scale0.8

y

x

v

u

Page 29: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Example Mappings

29

‣ Rotate by θ degrees:‣ x = ucosθ - vsinθ ‣ y = usinθ + vcosθ

Rotate30 deg

v

u

y

x

Page 30: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Example Mappings

30

‣ Shear in X by factor:‣ x = u + factor * v‣ y = v

‣ Shear in Y by factor:‣ x = u‣ y = v + factor * u

Shear X1.3

Shear Y1.3

v

u

v

u

y

x

y

x

Page 31: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Other Mappings

31

‣ Any function of u and v:‣ x = fx(u,v)‣ y = fy(u,v)

Fish-eye

“Swirl”

“Rain”

Page 32: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Image Warping Attempt 1 (Forward Mapping)

32

for (int u = 0; u < umax; u++) for (int v = 0; v < vmax; v++) float x = fx(u,v); float y = fy(u,v); dst(x,y) = src(u,v);

Source image Destination image

(u,v)(x,y)

f

Page 33: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Forward Mapping

33

Rotate-30

v

u

y

x

Page 34: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Forward Mapping

34

Rotate-30

v

u

y

x

Many source pixels can map to same destination pixel

Page 35: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Forward Mapping

35

Rotate-30

v

u

y

x

Some destination pixels may not be covered

Page 36: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Image Warping Attempt 2 (Reverse Mapping)

36

for (int x = 0; x < xmax; x++) for (int y = 0; y < ymax; y++) float u = fx-1(x,y); float v = fy-1(x,y); dst(x,y) = src(u,v);

Source image Destination image

(u,v)(x,y)

f

Page 37: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Reverse Mapping – GOOD!

37

‣ Iterate over destination image‣ Must resample source‣ May oversample, but much simpler!

Rotate-30

v

u

y

x

Page 38: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Resampling

38

Source image Destination image

(u,v)(x,y)

Issue: (u,v) does not usuallyhave integer coordinates

Page 39: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Overview

39

‣ Mapping‣ Forward‣ Reverse

‣ Resampling‣ Nearest Point Sampling‣ Bilinear Sampling‣ Gaussian Sampling

Page 40: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Nearest Point Sampling

40

int iu = floor(u+0.5);

int iv = floor(v+0.5);

dst[x,y] = src[iu,iv];

Rotate-30

Scale0.5

v

u

y

x

Page 41: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Bilinear Sampling

41

‣ Bilinearly interpolate four closest pixelsa = linear interpolation of src(x1,y1) and src(x2,y1)b = linear interpolation of src(x1,y2) and src(x2,y2) dst(x,y) = linear interpolation of “a” and “b”

x1 = floor( x ); x2 = x1 + 1; y1 = floor( y ); y2 = y1 + 1; dx = x – x1; dy = y – y1; a = src(x1,y1)*(1-dx) + src(x2,y1)*dx; b = src(x1,y2)*(1-dx) + src(x2,y2)*dx; dst(x,y) = a*(1-dy) + b*dy;

(x1,y1)

(x2,y2)

(x2,y1)

(x1,y2)

(x,y)

b

a

Page 42: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Bilinear Sampling

42

‣ Bilinearly interpolate four closest pixelsa = linear interpolation of src(x1,y1) and src(x2,y1)b = linear interpolation of src(x1,y2) and src(x2,y2) dst(x,y) = linear interpolation of “a” and “b”

x1 = floor( x ); x2 = x1 + 1; y1 = floor( y ); y2 = y1 + 1; dx = x – x1; dy = y – y1; a = src(x1,y1)*(1-dx) + src(x2,y1)*dx; b = src(x1,y2)*(1-dx) + src(x2,y2)*dx; dst(x,y) = a*(1-dy) + b*dy;

(x1,y1)

(x2,y2)

(x2,y1)

(x1,y2)

(x,y)

b

a

Make sure to test that the pixels (x1,y1), (x2,y2), (x1,y2), and (x2,y1) are within the image.

Page 43: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Gaussian Sampling

43

‣ Compute weighted sum of pixel neighborhood:‣ The blending weights are

the normalized values ofa Gaussian function.

(x,y)

Page 44: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

44

NearestNeighbor

Page 45: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Filtering Methods Comparison

45Bilinear

Page 46: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Filtering Methods Comparison

46Gaussian

Page 47: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Filtering Methods Comparison

47Gaussian

Trade-offs:

1. Jagged edges versus blurring

2. Computational speed

Page 48: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Image Warping Implementation

48

for (int x = 0; x < xmax; x++) for (int y = 0; y < ymax; y++) float u = fx-1(x,y); float v = fy-1(x,y); dst(x,y) = resample_src(u,v,w);

Source image Destination image

(u,v)

(x,y)

f

Page 49: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Image Warping Implementation

49

for (int x = 0; x < xmax; x++) for (int y = 0; y < ymax; y++) float u = fx-1(x,y); float v = fy-1(x,y); dst(x,y) = resample_src(u,v,w);

Source image Destination image

(u,v)

(x,y)

f

w

Page 50: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Example: Scale (src, dst, s)

50

float w = ??; for (int x = 0; x < xmax; x++) for (int y = 0; y < ymax; y++) float u = x / s; float v = y / s; dst(x,y) = resample_src(u,v,w);

Scale0.5

y

x

v

u

(u,v)

(x,y)

Page 51: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Example: Scale (src, dst, s)

51

Scale0.5

y

x

v

u

(u,v)

(x,y)w=1.0/s

float w = ??; for (int x = 0; x < xmax; x++) for (int y = 0; y < ymax; y++) float u = x / s; float v = y / s; dst(x,y) = resample_src(u,v,w);

Page 52: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Example: Rotate (src, dst, theta)

52

float w = ??; for (int x = 0; x < xmax; x++) for (int y = 0; y < ymax; y++) float u = x*cos(-θ) - y*sin(-θ); float v = x*sin(-θ) + y*cos(-θ); dst(x,y) = resample_src(u,v,w);

Rotate30

v

u

(u,v)

x = ucosθ - vsinθ y = usinθ + vcosθ

y

x

(x,y)

Page 53: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Example: Rotate (src, dst, theta)

53

Rotate30

v

u

(u,v)

x = ucosθ - vsinθ y = usinθ + vcosθ

y

x

(x,y)w=1.0

float w = ??; for (int x = 0; x < xmax; x++) for (int y = 0; y < ymax; y++) float u = x*cos(-θ) - y*sin(-θ); float v = x*sin(-θ) + y*cos(-θ); dst(x,y) = resample_src(u,v,w);

Page 54: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Example: Swirl (src, dst, theta) ???

54

float w = ??; for (int x = 0; x < xmax; x++) for (int y = 0; y < ymax; y++) float u = rot(dist(x,xcenter)*theta); float v = rot(dist(y,ycenter)*theta); dst(x,y) = resample_src(u,v,w);

Swirl45

y

x

v

u

(u,v) f(x,y)

Page 55: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Outline

55

‣ Image Processing‣ Image Warping‣ Image Sampling

Page 56: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Sampling Questions

56

‣ How should we sample an image:‣ Nearest Point Sampling?‣ Bilinear Sampling?‣ Gaussian Sampling?‣ Something Else?

Page 57: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Image Representation

57

What is an image?An image is a discrete collection of pixels, each representing a

sample of a continuous function.

Continuous image Digital image

Page 58: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Sampling

58

Let’s look at a 1D example:

Continuous Function Discrete Samples

Page 59: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Sampling

59

At in-between positions, values are undefined. How do we determine the value of a sample at these locations?

Discrete Samples

?

Page 60: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Sampling

60Discrete Samples

We need to reconstruct a continuous function, turning a collection of discrete samples into a 1D function that we can sample at arbitrary locations.

?

At in-between positions, values are undefined. How do we determine the value of a sample at these locations?

Page 61: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Sampling

61Discrete Samples

?

In other words: “How do we define the in-between values?”

At in-between positions, values are undefined. How do we determine the value of a sample at these locations?

We need to reconstruct a continuous function, turning a collection of discrete samples into a 1D function that we can sample at arbitrary locations.

Page 62: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Nearest Point Sampling

62

The value at a point is the value of the closest discrete sample.

Discrete SamplesReconstructed Function

?

Page 63: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Nearest Point Sampling

63

The value at a point is the value of the closest discrete sample.

Discrete SamplesReconstructed Function

?

The reconstruction:

ü Interpolates the samples

û Is not continuous

Page 64: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Bilinear Sampling

64

The value at a point is the (bi)linear interpolation of the twosurrounding samples.

Discrete SamplesReconstructed Function

?

Page 65: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Bilinear Sampling

65

The value at a point is the (bi)linear interpolation of the twosurrounding samples.

Discrete SamplesReconstructed Function

?

The reconstruction:

ü Interpolates the samples

û Is not smooth

Page 66: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Gaussian Sampling

66

The value at a point is the Gaussian average of the surroundingsamples.

Discrete SamplesReconstructed Function

?

Page 67: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Gaussian Sampling

67

The value at a point is the Gaussian average of the surroundingsamples.

Discrete SamplesReconstructed Function

?

The reconstruction:

û Does not interpolate

ü Is smooth

Page 68: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Image Sampling

68

‣ How do we reconstruct a function from a collection of samples?

?Samples Reconstruction

Page 69: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Image Sampling

69

‣ How do we reconstruct a function from a collection of samples?‣ To answer this question, we need to understand what kind of

information the samples contain.

?Samples ReconstructionOriginal Function

Page 70: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Image Sampling

70

‣ How do we reconstruct a function from a collection of samples?‣ To answer this question, we need to understand what kind of

information the samples contain.‣ Signal processing helps us understand this better.

?Samples ReconstructionOriginal Function

Page 71: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

71

‣ Fourier analysis provides a way for expressing (or approximating) any signal as a sum of scaled and shifted cosine functions.

The Building Blocks for the Fourier Decomposition

cos(0θ) cos(1θ) cos(2θ) cos(3θ)

cos(4θ) cos(5θ) cos(6θ)…

π π π π

π π π

-π -π -π -π

-π -π -π

Page 72: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

72

‣ As higher frequency components are added to the approximation, finer details are captured.

Initial Function 0th Order Approximation

0th Order Component

f(θ)

f0(θ)=a0(cos0(θ+φ0)

Page 73: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

73

‣ As higher frequency components are added to the approximation, finer details are captured.

Initial Function 1st Order Approximation

+

1st Order Component

0th Order Approximation

f(θ)

f1(θ)=a1(cos1(θ+φ1)

Page 74: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

74

‣ As higher frequency components are added to the approximation, finer details are captured.

Initial Function 2nd Order Approximation

+

2nd Order Component

1st Order Approximation

f(θ)

f2(θ)=a2(cos2(θ+φ2)

Page 75: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

75

‣ As higher frequency components are added to the approximation, finer details are captured.

Initial Function 3rd Order Approximation

+

3rd Order Component

2nd Order Approximation

f(θ)

f3(θ)=a3(cos3(θ+φ3)

Page 76: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

76

‣ As higher frequency components are added to the approximation, finer details are captured.

Initial Function 4th Order Approximation

+

4th Order Component

3rd Order Approximation

f(θ)

f4(θ)=a4(cos4(θ+φ4)

Page 77: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

77

‣ As higher frequency components are added to the approximation, finer details are captured.

Initial Function 5th Order Approximation

+

5th Order Component

4th Order Approximation

f(θ)

f5(θ)=a5(cos5(θ+φ5)

Page 78: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

78

‣ As higher frequency components are added to the approximation, finer details are captured.

Initial Function 6th Order Approximation

+

6th Order Component

5th Order Approximation

f(θ)

f6(θ)=a6(cos6(θ+φ6)

Page 79: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

79

‣ As higher frequency components are added to the approximation, finer details are captured.

Initial Function 7th Order Approximation

+

7th Order Component

6th Order Approximation

f(θ)

f7(θ)=a7(cos7(θ+φ7)

Page 80: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

80

‣ As higher frequency components are added to the approximation, finer details are captured.

Initial Function 8th Order Approximation

+

8th Order Component

7th Order Approximation

f(θ)

f8(θ)=a8(cos8(θ+φ8)

Page 81: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

81

‣ As higher frequency components are added to the approximation, finer details are captured.

Initial Function 9th Order Approximation

+

9th Order Component

8th Order Approximation

f(θ)

f9(θ)=a9(cos9(θ+φ9)

Page 82: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

82

‣ As higher frequency components are added to the approximation, finer details are captured.

Initial Function 10th Order Approximation

+

10th Order Component

9th Order Approximation

f(θ)

f10(θ)=a10(cos10(θ+φ10)

Page 83: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

83

‣ As higher frequency components are added to the approximation, finer details are captured.

Initial Function 11th Order Approximation

+

11th Order Component

10th Order Approximation

f(θ)

f11(θ)=a11(cos11(θ+φ11)

Page 84: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

84

‣ As higher frequency components are added to the approximation, finer details are captured.

Initial Function 12th Order Approximation

+

12th Order Component

11th Order Approximation

f(θ)

f12(θ)=a12(cos12(θ+φ12)

Page 85: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

85

‣ As higher frequency components are added to the approximation, finer details are captured.

Initial Function 13th Order Approximation

+

13th Order Component

12th Order Approximation

f(θ)

f13(θ)=a13(cos13(θ+φ13)

Page 86: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

86

‣ As higher frequency components are added to the approximation, finer details are captured.

Initial Function 14th Order Approximation

+

14th Order Component

13th Order Approximation

f(θ)

f14(θ)=a14(cos14(θ+φ14)

Page 87: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

87

‣ As higher frequency components are added to the approximation, finer details are captured.

Initial Function 15th Order Approximation

+

15th Order Component

14th Order Approximation

f(θ)

f15(θ)=a15(cos15(θ+φ15)

Page 88: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

88

‣ As higher frequency components are added to the approximation, finer details are captured.

Initial Function 16th Order Approximation

+

16th Order Component

15th Order Approximation

f(θ)

f16(θ)=a16(cos16(θ+φ16)

Page 89: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

89

‣ Combining all of the frequency components together, we get the initial function.

Initial Function

+ + + +

+ + + +

=

f(θ)

f0(θ) f1(θ) f2(θ) f3(θ) f4(θ)

f5(θ) f6(θ) f7(θ) f8(θ)

ak: amplitude of the kth frequency componentφk: shift of the kth frequency component

Page 90: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Question

90

‣ As higher frequency components are added to the approximation, finer details are captured.

‣ If we have n samples, what is the highest frequency that can be represented?

Initial Function

+ + + +

+ + + +

=

f(θ)

f0(θ) f1(θ) f2(θ) f3(θ) f4(θ)

f5(θ) f6(θ) f7(θ) f8(θ)

Page 91: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Question

91

‣ As higher frequency components are added to the approximation, finer details are captured.

‣ If we have n samples, what is the highest frequency that can be represented?

Initial Function

+ + + +

+ + + +

=

f(θ)

f0(θ) f1(θ) f2(θ) f3(θ) f4(θ)

f5(θ) f6(θ) f7(θ) f8(θ)

Each frequency component has two degrees of freedom:• Amplitude• Shift

With n samples we can represent the first n/2 frequencycomponents

Page 92: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Sampling Theorem

92

‣ A signal can be reconstructed from its samples, if the original signal has no frequencies above 1/2 the sampling frequency – Shannon’s Theorem

‣ The minimum sampling rate for band-limited function is called the “Nyquist rate”

A signal is band-limited if its highestnon-zero frequency is bounded.The frequency is called the bandwidth.

Page 93: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Question

93

‣ What if we have only n samples and we try to reconstruct a function with frequencies larger than the Nyquist frequency (n/2)?

Page 94: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Aliasing

94

‣ When a high-frequency signal is sampled with insufficiently many samples, it will be perceived as a lower-frequency signal. This masking of higher frequencies as lower ones is referred to as aliasing.

π-π

Page 95: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Aliasing

95

‣ When a high-frequency signal is sampled with insufficiently many samples, it will be perceived as a lower-frequency signal. This masking of higher frequencies as lower ones is referred to as aliasing.

π-π

Page 96: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Aliasing

96

‣ When a high-frequency signal is sampled with insufficiently many samples, it will be perceived as a lower-frequency signal. This masking of higher frequencies as lower ones is referred to as aliasing.

π-π

Page 97: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Aliasing

97

‣ When a high-frequency signal is sampled with insufficiently many samples, it will be perceived as a lower-frequency signal. This masking of higher frequencies as lower ones is referred to as aliasing.

π-π

Page 98: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Temporal Aliasing

98

‣ Artifacts due to limited temporal resolution

Page 99: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

99

NearestNeighbor

Page 100: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Sampling

100

‣ There are two problems:‣ You don’t have enough samples to correctly reconstruct your

high-frequency information‣ You corrupt the low-frequency information because the high-

frequencies mask themselves as lower ones.

Page 101: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Anti-Aliasing

101

Two possible ways to address aliasing:‣ Sample at higher rate• Pre-filter to form band-limited signal

Page 102: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Anti-Aliasing

102

Two possible ways to address aliasing:‣ Sample at higher rate‣ Not always possible‣ Still rendering to fixed resolution

• Pre-filter to form band-limited signal

Page 103: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Anti-Aliasing

103

Two possible ways to address aliasing:• Sample at higher rate‣ Pre-filter to form a band-limited signal‣ You still don’t get your high frequencies, but at least the low

frequencies are uncorrupted.

Page 104: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

104

‣ If we just look at how much information each frequency contributes, we obtain the power spectrum of the signal:

Initial Function

+ + + +

+ + + +

=

Page 105: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Analysis

105

‣ If we just look at how much information each frequency contributes, we obtain the power spectrum of the signal:

Initial Function

+ + + +

+ + + +

=

Power Spectrum

Page 106: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Pre-Filtering

106

‣ Band-limit by discarding the high-frequency components of the Frequency decomposition.

Initial Power Spectrum Band-Limited Power Spectrum

Page 107: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Pre-Filtering

107

‣ Band-limit by discarding the high-frequency components of the Fourier decomposition.

‣ We can do this by multiplying the frequency components by a 0/1 function:

1

X =

Initial Power Spectrum Band-Limited SpectrumFrequency Filter

Page 108: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Pre-Filtering

108

‣ Band-limit by discarding the high-frequency components of the Fourier decomposition.

‣ We can do this by multiplying the frequency components by a 0/1 function:

1

X =

Initial Power Spectrum Band-Limited SpectrumFrequency Filter

Page 109: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Fourier Theory

109

‣ A fundamental fact from Fourier theory is that multiplication in the frequency domain is equivalent to convolution in the spatial domain.

Page 110: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Convolution

110

‣ To convolve two functions f and g, we resample the function f using the weights given by g.

f(θ)

g(θ)1

Page 111: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Convolution

111

‣ To convolve two functions f and g, we resample the function f using the weights given by g.

f(θ)

g(θ)

(f∗g)(θ)

1

1 0

Page 112: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Convolution

112

‣ To convolve two functions f and g, we resample the function f using the weights given by g.

f(θ)

g(θ).6

(f∗g)(θ)

.4

.6 .4 0

Page 113: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Convolution

113

‣ To convolve two functions f and g, we resample the function f using the weights given by g.

f(θ)

(f∗g)(θ)

g(θ).6.4

.4 .6 0

Page 114: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Convolution

114

‣ To convolve two functions f and g, we resample the function f using the weights given by g.

f(θ)

g(θ)

(f∗g)(θ)

1

1 00

Page 115: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Convolution

115

‣ To convolve two functions f and g, we resample the function f using the weights given by g.

f(θ)

g(θ).6

(f∗g)(θ)

.4

.6 .40

Page 116: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Convolution

116

‣ To convolve two functions f and g, we resample the function f using the weights given by g.

f(θ)

(f∗g)(θ)

g(θ).6.4

.4 .60

Page 117: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Convolution

117

‣ To convolve two functions f and g, we resample the function f using the weights given by g.

f(θ)

g(θ)

(f∗g)(θ)

1

10 0

Page 118: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Convolution

118

‣ To convolve two functions f and g, we resample the function f using the weights given by g.

f(θ)

g(θ)

0.5

(f∗g)(θ).5 .50

Page 119: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Convolution

119

‣ To convolve two functions f and g, we resample the function f using the weights given by g.

f(θ)

g(θ)

(f∗g)(θ)

1

10 0

Page 120: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Convolution

120

‣ To convolve two functions f and g, we resample the function f using the weights given by g.

f(θ)

g(θ)

(f∗g)(θ)

1

10

Page 121: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Convolution

121

‣ To convolve two functions f and g, we resample the function f using the weights given by g.

‣ Nearest point, bilinear, and Gaussian interpolation are just convolutions with different filters.

*

*

*

=

=

=

Page 122: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Convolution

122

‣ Recall that convolution in the spatial domain is the equal to multiplication in the frequency domain.

‣ In order to avoid aliasing, we need to convolve with a filter whose power spectrum has value:‣ 1 at low frequencies‣ 0 at high frequencies

1

X =

Initial Power Spectrum Band-Limited SpectrumFrequency Filter

Page 123: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Nearest Point Convolution

123

* =

Filter Spectrum

Discrete Samples Reconstruction Filter Reconstructed Function

Page 124: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Bilinear Convolution

124

* =Discrete Samples Reconstruction Filter Reconstructed Function

Filter Spectrum

Page 125: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Gaussian Convolution

125

* =Discrete Samples Reconstruction Filter Reconstructed Function

Filter Spectrum

Page 126: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Convolution

126

‣ The ideal filter for avoiding aliasing has a power spectrum with values:‣ 1 at low frequencies‣ 0 at high frequencies

‣ The sinc function has such a power spectrum and is referred to as the ideal reconstruction filter:

Page 127: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

The Sinc Filter

127

‣ The ideal filter for avoiding aliasing has a power spectrum with values:‣ 1 at low frequencies‣ 0 at high frequencies

‣ The sinc function has such a power spectrum and is referred to as the ideal reconstruction filter:

Reconstruction Filter

Filter Spectrum

Page 128: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

The Sinc Filter

128

‣ Limitations:‣ Has negative values, giving rise to negative weights in the

interpolation.‣ The discontinuity in the frequency domain (power spectrum)

results in ringing artifacts known as the Gibbs Phenomenon.

Reconstruction Filter

* =

Initial Function Reconstructed Function

Page 129: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

The Sinc Filter

129

‣ Limitations:‣ Has negative values, giving rise to negative weights in the

interpolation.‣ The discontinuity in the frequency domain (power spectrum)

results in ringing artifacts near spatial discontinuities, known as the Gibbs Phenomenon.

Reconstruction Filter

* =

Initial Function Reconstructed Function

Page 130: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Summary

130

There are different ways to sample an image:‣ Nearest Point Sampling‣ Linear Sampling‣ Gaussian Sampling‣ Sinc Sampling

These methods have advantages and disadvantages.

Page 131: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Summary – Nearest

131

üCan be implemented efficiently because the filter is non-zero in a very small region.

? Interpolates the samples.û Is discontinuous.ûDoes not address the aliasing problem, giving bad results when a

signal is under-sampled.

* =

Discrete Samples Reconstruction Filter Reconstructed Function

Page 132: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Summary – Linear

132

üCan be implemented efficiently because the filter is non-zero in a very small region.

? Interpolates the samples.û Is not smooth.ûPartially addresses the aliasing problem, but can still give bad

results when a signal is under-sampled.

* =

Discrete Samples Reconstruction Filter Reconstructed Function

Page 133: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Summary – Gaussian

133

û Is slow to implement because the filter is non-zero in a large region.

? Does not interpolate the samples.üIs smooth.üAddresses the aliasing problem by killing off the high frequencies.

* =

Discrete Samples Reconstruction Filter Reconstructed Function

Page 134: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Summary – Sinc

134

û Is slow to implement because the filter is non-zero in a large region.

? Does not interpolate the samples.ûAssigns negative weights.ûRinging at discontinuities.üAddresses the aliasing problem by killing off the high frequencies.

* =

Discrete Samples Reconstruction Filter Reconstructed Function

Page 135: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Summary

135

Question:‣ Is it good if a reconstruction method is interpolating? (Consider

the case when you are down-scaling an image?)

Page 136: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Summary

136

It appears that we have been mixing the sampling problem with the reconstruction problem.

However, our motivation for the choice of filter is the same in both cases. We want a filter whose spectrum goes to zero so that:

‣ Sampling: High frequency samples are killed off, the signal becomes band-limited, and we can sample discretely.

‣ Reconstruction: We do not end up reconstructing a function with high frequency components.

Page 137: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Image Sampling

137

Given a signal sampled at m positions, if we would like to re-sample at n positions we need to:

1. Reconstruct a function with maximum non-zero frequency no larger than min(m/2,n/2).

2. Sample the reconstructed function at the n positions.

Page 138: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Image Sampling

138

Example:

Page 139: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Image Sampling

139

Example:

Page 140: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Image Sampling

140

Example:

Page 141: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Gaussian Sampling

141

Recall:To avoid aliasing, we kill off high-frequency components, by

convolving with a function whose power spectrum is zero at high frequencies.

We use a Gaussian for function reconstruction and sampling because it smoothly kills of the high frequency components.

Page 142: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Gaussian Sampling

142

Q: What variance Gaussian should we use?A: The variance of the Gaussian should be between 0.5 and 1.0

times the distance between samples.

Page 143: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Gaussian Sampling

143

Q: What variance Gaussian should we use?A: The variance of the Gaussian should be between 0.5 and 1.0

times the distance between samples.

Power spectra of the Gaussians used for reconstructing and sampling a function with 20 samples

Page 144: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Gaussian Sampling

144

Scaling Example:Q: Suppose we have data represented by 20 samples that we would

like to down-sample to 5 samples. What variance should we use?

Page 145: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Gaussian Sampling

145

Scaling Example:Q: Suppose we have data represented by 20 samples that we would

like to down-sample to 5 samples. What variance should we use?A: The distance between two adjacent samples in the final array

corresponds to a distance of 4 units in the initial array.The variance of the Gaussian should be between 2.0 and 4.0.

Page 146: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Gaussian Sampling

146

Scaling Example:Q: Suppose we have data represented by 20 samples that we would

like to up-sample to 40 samples. What variance should we use?

Page 147: Image Filtering, Warping and SamplingAcknowledgement: slides by Jason Lawrence, Misha Kazhdan, Allison Klein, Tom Funkhouser, Adam Finkelstein and David Dobkin Image Filtering, Warping

Gaussian Sampling

147

Scaling Example:Q: Suppose we have data represented by 20 samples that we would

like to up-sample to 40 samples. What variance should we use?A: Because the initial samples can’t represent frequencies higher

than 10, we shouldn’t use a Gaussian with smaller variance since this would introduce high-frequency components into the reconstruction. The variance of the Gaussian should remain between 0.5 and 1.0.