Multiscale Analysis of Images

Gilad LermanMath 5467

(stealing slides from Gonzalez & Woods, and Efros)

The Multiscale Nature of Images

Recall: Gaussian pre-filtering

G 1/4

G 1/8

Gaussian 1/2

Solution: filter the image, then subsample• Filter size should double for each ½ size reduction.

Subsampling with Gaussian pre-filtering

G 1/4 G 1/8Gaussian 1/2

Solution: filter the image, then subsample• Filter size should double for each ½ size reduction.

Image Pyramids

Known as a Gaussian Pyramid [Burt and Adelson, 1983]• In computer graphics, a mip map [Williams, 1983]• A precursor to wavelet transform

A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose

Figure from David Forsyth

Gaussian pyramid construction

filter mask

Repeat• Filter• Subsample

Until minimum resolution reached • can specify desired number of levels (e.g., 3-level pyramid)

The whole pyramid is only 4/3 the size of the original image!

What does blurring take away? (recall)

original

What does blurring take away? (recall)

smoothed (5x5 Gaussian)

High-Pass filter

smoothed – original

Band-pass filtering

Laplacian Pyramid (subband images)Created from Gaussian pyramid by subtraction

Gaussian Pyramid (low-pass images)

Laplacian Pyramid

How can we reconstruct (collapse) this pyramid into the original image?

Need this!

Originalimage

Image resampling (interpolation)

So far, we considered only power-of-two subsampling• What about arbitrary scale reduction?

• How can we increase the size of the image?

Recall how a digital image is formed

• It is a discrete point-sampling of a continuous function• If we could somehow reconstruct the original function, any

new image could be generated, at any resolution and scale

1 2 3 4 5

d = 1 in this example

Image resampling

So far, we considered only power-of-two subsampling• What about arbitrary scale reduction?

• How can we increase the size of the image?

Recall how a digital image is formed

• It is a discrete point-sampling of a continuous function• If we could somehow reconstruct the original function, any

new image could be generated, at any resolution and scale

1 2 3 4 5

d = 1 in this example

Image resampling

So what to do if we don’t know

1 2 3 4 52.5

1 d = 1 in this example

• Answer: guess an approximation• Can be done in a principled way: filtering

Image reconstruction• Convert to a continuous function

• Reconstruct by cross-correlation:

Resampling filtersWhat does the 2D version of this hat function look like?

Better filters give better resampled images• Bicubic is common choice

Why not use a Gaussian?

What if we don’t want whole f, but just one sample?

performs linear interpolation

(tent function) performs bilinear interpolation

Bilinear interpolation

Smapling at f(x,y):

What happens in the frequency domain?

• Laplacian Pyramid gives rise to different frequency bands

• Alternatively can obtain similar pyramids according to frequency bands…

Example of Frequency-bands “pyramid”

Taken from Rajashekar and Simoncelli (http://www.cns.nyu.edu/ftp/lcv/rajashekar08a.pdf)

Applications (more efficient with wavelets)

•Coding

High frequency coefficients need fewer bits, so one can

collapse a compressed Laplacian pyramid•Denoising/Restoration

Setting “most” coefficients in Laplacian pyramid to zero•Image Blending

Denoising demonstration (motivation)

Taken from Rajashekar and Simoncelli (http://www.cns.nyu.edu/ftp/lcv/rajashekar08a.pdf)

Observation: spatial correlation in noise-free

image (and not the noisy image)

Conclusion: Noise is more evident in the

high-frequency components, than the low-

frequency components (smooth components)

Denoising demonstration (idea)

Taken from Rajashekar and Simoncelli (http://www.cns.nyu.edu/ftp/lcv/rajashekar08a.pdf)

Partition into sub-bands of frequencies (top: high freq., bottom: low)

y-given image (vectorized), - estimated denoised image (vectorized)

The middle graph – intensity thresholding.

No thresholding for smoothest component and keeping only higher intensities for

higher components (the filter is learned empirically)

In practice, can have blurring and not competitive with NLM, BM3d, dictionary learning

ˆ( )x y

Application: Pyramid Blending

0

1

0

1

0

1

Left pyramid Right pyramidblend

Image Blending

Feathering

01

01

+

=

Encoding transparency

I(x,y) = (R, G, B, )

Iblend = left Ileft + right Iright

Ileft Iright

leftright

Affect of Window Size

0

1 left

right0

1

Affect of Window Size

0

1

0

1

Good Window Size

0

1

“Optimal” Window: smooth but not ghostedBurt and Adelson (83): Choose by pyramids…

Pyramid Blending

Pyramid Blending (Color)

Laplacian Pyramid: Blending

General Approach:1. Build Laplacian pyramids LA and LB from images A and B

2. Build a Gaussian pyramid GR from selected region R (black white corresponding images)

3. Form a combined pyramid LS from LA and LB using nodes of GR as weights:• LS(i,j) = GR(I,j,)*LA(I,j) + (1-GR(I,j))*LB(I,j)

4. Collapse the LS pyramid to get the final blended image

laplacianlevel

4

laplacianlevel

2

laplacianlevel

0

left pyramid right pyramid blended pyramid

Blending Regions

Simplification: Two-band Blending

Brown & Lowe, 2003• Only use two bands: high freq. and low freq.• Blends low freq. smoothly• Blend high freq. with no smoothing: use binary mask

Can be explored in a project…