Final Review

Image Processing

CSE 166

Lecture 18

Topics covered

• Basis vectors• Matrix-based transforms• Basis images• Wavelet transform• Image compression• Image watermarking• Morphological image processing• Edge detection• Segmentation

CSE 166, Spring 2020 2

Matrix-based transforms

CSE 166, Spring 2020 3

Forward transform

Inverse transform

where

General inverse transform using basis vectors

CSE 166, Spring 2020 4

Matrix-based transforms using orthonormal basis vectors

• In vector form

CSE 166, Spring 2020 5

Matrix-based transforms using orthonormal basis vectors

• In matrix form

CSE 166, Spring 2020 6

Matrix-based transforms using biorthonormal basis vectors

• In matrix form

CSE 166, Spring 2020 7

Matrix-based transform

CSE 166, Spring 2020 8

Example: 8-point DFT of f(x) = sin(2πx)

real part + imaginary part

Matrix-based transforms

• Discrete Fourier transform (DFT)• Discrete Hartley transform (DHT)• Discrete cosine transform (DCT)• Discrete sine transform (DST)• Walsh-Hadamard (WHT)• Slant (SLT)• Haar (HAAR)• Daubechies (DB4)• Biorthogonal B-spline (BIOR3.1)

CSE 166, Spring 2020 9

Basis vectors of matrix-based 1D transforms

CSE 166, Spring 2020 10

N = 16

real part

imaginary part

Standard basis(for reference)

Basis vectors of matrix-based 1D transforms

CSE 166, Spring 2020 11

N = 16

basis dual Standard basis(for reference)

Matrix-based transformsin two dimensions

CSE 166, Spring 2020 12

Forward transform

Inverse transform

where

Matrix-based transformsin two dimensions

• If r and s are separable and symmetric, and M = N, then

– For orthonormal basis vectors

– For biorthonormal basis vectors

CSE 166, Spring 2020 13

Matrix-based transforms in two dimensions using basis images

• Inverse transform

CSE 166, Spring 2020 14

where

Each Su,v is a basis image

Basis images of matrix-based 2D transforms

CSE 166, Spring 2020 15

Standard basis images (for reference)

8-by-8 array of8-by-8

basis images

Basis images of matrix-based 2D transforms

CSE 166, Spring 2020 16

Discrete Fourier transform (DFT) basis images

real part imaginary part

Basis images of matrix-based 2D transforms

CSE 166, Spring 2020 17

Discrete Hartley transform (DHT) basis images

Basis images of matrix-based 2D transforms

CSE 166, Spring 2020 18

Discrete cosine transform (DCT) basis images

Basis images of matrix-based 2D transforms

CSE 166, Spring 2020 19

Discrete sine transform (DST) basis images

Basis images of matrix-based 2D transforms

CSE 166, Spring 2020 20

Walsh-Hadamard transform (WHT) basis images

Basis images of matrix-based 2D transforms

CSE 166, Spring 2020 21

Slant transform (SLT) basis images

Basis images of matrix-based 2D transforms

CSE 166, Spring 2020 22

Haar transform (HAAR) basis images

Wavelet transforms

• A scaling function is used to create a series of approximations of a function or image, each differing by a factor of 2 in resolution from its nearest neighboring approximations.

• Wavelet functions (wavelets) are then used to encode the differences between adjacent approximations.

• The discrete wavelet transform (DWT) uses those wavelets, together with a single scaling function, to represent a function or image as a linear combination of the wavelets and scaling function.

CSE 166, Spring 2020 23

Scaling functions and set of basis vectors

• Father scaling function

• Set of basis functions

– Integer translation k

– Binary scaling j

• Basis of the function space spanned by

CSE 166, Spring 2020 24

Scaling function, multiresolution analysis

1. The scaling function is orthogonal to its integer translates.

2. The function spaces spanned by the scaling function at low scales are nested within those spanned at higher scales.𝑉−∞⊂⋯⊂ 𝑉−1⊂ 𝑉0⊂ 𝑉1⊂⋯⊂ 𝑉∞

3. The only function representable at every scale (all 𝑉𝑗) is f(x) = 0.

4. All measureable, square-integrable functions can be represented as 𝑗 → ∞

CSE 166, Spring 2020 25

• Given father scaling function , there exists a mother wavelet function whose integer translations and binary scalings

span the difference between any two adjacent scaling spaces

• The orthogonal compliment of

Wavelet functions

CSE 166, Spring 2020 26

Relationship between scaling and wavelet function spaces

CSE 166, Spring 2020 27

Union

V is basis of the function space spanned by scaling function

Wj is orthogonal complement of Vj in Vj+1

Scaling function coefficients and wavelet function coefficients

• Refinement (or dilation) equation

where are scaling function coefficients

• And

where are wavelet function coefficients

• Relationship

CSE 166, Spring 2020 28

1D discrete wavelet transform

• Forward

• Inverse

CSE 166, Spring 2020 29

Approximation

Details

for real signals

2D discrete wavelet transform

• Forward

CSE 166, Spring 2020 30

Approximation

Details

where

Directionalwavelets

for real signals

2D discrete wavelet transform

• Inverse

CSE 166, Spring 2020 31

for real signals

2D discrete wavelet transform

CSE 166, Spring 2020 32

Decomposition

Horizontal detailsApproximation

Vertical details

Diagonal details

2D discrete wavelet transform

CSE 166, Spring 2020 33

3-levelwavelet

decomposition

Wavelets in image processing

1. Wavelet transform

2. Alter transform

3. Inverse wavelet transform

CSE 166, Spring 2020 34

Wavelet-based edge detection

CSE 166, Spring 2020 35

Zero horizontal

details

Zero lowest scale

approximation

Vertical edges

Edges

Wavelet-based noise removal

CSE 166, Spring 2020 36

Noisy imageThreshold

details

Zero highest

resolution details

Zero details for all levels

Data compression

• Data redundancy

where compression ratio

CSE 166, Spring 2020 37

where

Data redundancy in images

CSE 166, Spring 2020 38

Coding redundancy

Spatial redundancy

Irrelevant information

Does not need all 8 bits

Information is unnecessarily

replicated

Information is not useful

Image information

• Entropy

– It is not possible to encode input image with fewer than bits/pixel

CSE 166, Spring 2020 39

where

Fidelity criteria,objective (quantitative)

• Total error

• Root-mean-square error

• Mean-square signal to noise ratio (SNR)

CSE 166, Spring 2020 40

Fidelity criteriasubjective (qualitative)

CSE 166, Spring 2020 41

Approximations

CSE 166, Spring 2020 42

5.17 15.67 14.17

(a) (b) (c)

Objective (quantitative) qualityrms error (in intensity levels)

Subjective (qualitative) quality, relative

Lower is better

(a) is better than (b).

(b) is better than (c)

Compression system

CSE 166, Spring 2020 43

Compression methods

• Huffman coding• Golomb coding• Arithmetic coding• Lempel-Ziv-Welch (LZW) coding• Run-length coding• Symbol-based coding• Bit-plane coding• Block transform coding• Predictive coding• Wavelet coding

CSE 166, Spring 2020 44

Block-transform coding

CSE 166, Spring 2020 45

Encoder

Decoder

Block-transform coding

• Example: discrete cosine transform

• Inverse

CSE 166, Spring 2020 46

where

Block-transform coding

CSE 166, Spring 2020 47

Discrete cosine transformWalsh-Hadamard transform

4x4 subimages (4x4 basis images)

Block-transform coding

CSE 166, Spring 2020 48

Fourier transform

cosine transform

Walsh-Hadamardtransform

2.32 1.78 1.13rms error

Retain 32 largest

coefficients

Error image

8x8 subimages

Lower is better

Block-transform coding

CSE 166, Spring 2020 49

2x2 4x4 8x8

Reconstruction error versus subimage size

DCT subimage size:

JPEG uses block DCT-based coding

CSE 166, Spring 2020 50

Compression reconstruction

Scaled error image

Zoomed compression

reconstruction

25:1

52:1

Compression ratio

Predictive coding model

CSE 166, Spring 2020 51

Encoder

Decoder

Predictive coding

CSE 166, Spring 2020 52

Input image

Prediction error image

Example: previous pixel coding

Histograms

Wavelet coding

CSE 166, Spring 2020 53

Encoder

Decoder

Wavelet coding

CSE 166, Spring 2020 54

Detail coefficients below 25 are truncated to zero

JPEG-2000 uses wavelet-based coding

CSE 166, Spring 2020 55

Compression reconstruction

Scaled error image

Zoomed compression

reconstruction

25:1

52:1

Compression ratio

JPEG-2000 uses wavelet-based coding

CSE 166, Spring 2020 56

Compression reconstruction

Scaled error image

Zoomed compression

reconstruction

75:1

105:1

Compression ratio

Image watermarking

• Visible watermarks

• Invisible watermarks

CSE 166, Spring 2020 57

Visible watermark

CSE 166, Spring 2020 58

Original image minus watermark

Watermarked image

Watermark

Invisible image watermarking system

CSE 166, Spring 2020 59

Encoder

Decoder

Invisible watermark

CSE 166, Spring 2020 60

Two least significant

bits

JPEGcompressed

Fragile invisible

watermark

Originalimage Extracted

watermark

Example: watermarking using two least significant bits

Invisible watermark

CSE 166, Spring 2020 61

Watermarked images

(different watermarks)

Example: DCT-based watermarking

Extracted robust

invisible watermark

Reflection and translation

CSE 166, Spring 2020 62

Translation

Reflection

Sets of pixels: objects and structuring elements (SEs)

CSE 166, Spring 2020 63

Border of background

pixels around objects

Tight border around SE

Reflection about the origin

CSE 166, Spring 2020 64

Origin

Don’t care elements

Erosion

CSE 166, Spring 2020 65

Example: square SE

Complement of A (i.e., set of elements not in A)

Erosion

CSE 166, Spring 2020 66

Example: elongated SE

Erosion

CSE 166, Spring 2020 67

11x11

45x4515x15

Shrinks

Dilation

CSE 166, Spring 2020 68

Square SEExamples

Elongated SE

Dilation

CSE 166, Spring 2020 69

Expands

Opening

CSE 166, Spring 2020 70

Structuring element rolls along inner boundary

Closing

CSE 166, Spring 2020 71

Structuring element rolls along outer boundary

Opening and closing

CSE 166, Spring 2020 72

Opening

Closing

Dilation

Erosion

Morphological image processing

CSE 166, Spring 2020 73

Noisy input

Dilation

Erosion

Opening

Dilation

Erosion

Closing

Boundary extraction

CSE 166, Spring 2020 74

Erosion Set difference

Boundary extraction

CSE 166, Spring 2020 75

Hole filling

CSE 166, Spring 2020 76

Given point in hole

Hole filling

CSE 166, Spring 2020 77

Given points in holes All holes filled

Connected components

CSE 166, Spring 2020 78

Given point in A

Connected components

CSE 166, Spring 2020 79

15 connected

components

X-ray image

Threshold (negative)

Image segmentation

• General approach

1. Spatial filtering

2. Additional processing

3. Thresholding

• Global thresholding (simplest)

CSE 166, Spring 2020 80

where

Image segmentation

CSE 166, Spring 2020 81

Edge-based

Region-based

Input Edges Segmentation

Derivatives in 1D

• Forward difference

• Backward difference

• Central difference

CSE 166, Spring 2020 82

Image derivatives

CSE 166, Spring 2020 83

Detection of isolated points

CSE 166, Spring 2020 84

Laplacian(second derivative)

Input Segmentation

Threshold absolute

value

Line detection

CSE 166, Spring 2020 85

Threshold absolute

value

Threshold value

Laplacian(second

derivative)Input

Double lines

Line detection, specific directions

CSE 166, Spring 2020 86

Spatial filters

Line detection, specific directions

CSE 166, Spring 2020 87

+45°

Negative values set

to zero

Threshold

Edge models

CSE 166, Spring 2020 88

Step Ramp Roof edge

Edge models

CSE 166, Spring 2020 89

Step

RampRoofedge

Ramp edge

CSE 166, Spring 2020 90

First derivative

Second derivative

One point

Two points

Noise and image derivatives

CSE 166, Spring 2020 91

First derivative

Second derivativeInput

Noise

Edge detection

1. Image smoothing for noise reduction

2. Detection of image points (edge point candidates)

3. Edge localization (select from candidates, set of edge points)

CSE 166, Spring 2020 92

Gradient and edge direction

CSE 166, Spring 2020 93

Gradient direction is orthogonal to edge direction

Gradient operators

CSE 166, Spring 2020 94

Forward difference

Gradients

CSE 166, Spring 2020 95

Magnitude of

vertical gradient

Magnitude of

horizontal gradient

Magnitude of

gradient vector

Input

Gradients

CSE 166, Spring 2020 96

Magnitude of

vertical gradient

Magnitude of

horizontal gradient

Magnitude of

gradient vector

Input

Smooth image prior to computing gradients.Results in more selective edges

Edge detection

1. Smooth the input image

2. Compute the gradient magnitude image

3. Apply nonmaximal suppression to the gradient magnitude image

4. Threshold the resulting image

CSE 166, Spring 2020 97

Edge detection

CSE 166, Spring 2020 98

Without smoothing With smoothing

Threshold magnitude of gradient vector

Advanced edge detection

CSE 166, Spring 2020 99

Magnitude of gradient vector

(with smoothing)Input

Marr-Hildreth CannyFigure 10.25 in textbook looks better

Canny edge detector

1. Smooth the input image with a Gaussian filter

2. Compute the gradient magnitude and angle images

3. Apply nonmaximal suppression to the gradient magnitude image

4. Use double thresholding and connectivity analysis to detect and link edges

CSE 166, Spring 2020 100

Double thresholding

• Use a high threshold to start edge curves and a low threshold to continue them– Define two thresholds TH and TL– Starting with output of nonmaximal suppression, find a point q0,

which is a local maximum greater than TH– Start tracking an edge chain at pixel location q0 in one of the two

directions– Stop when gradient magnitude is less than TL

CSE 166, Spring 2020 101

TH

TL

q0

Double thresholding

CSE 166, Spring 2020 102

TH = 15 and TL = 5T = 5T = 15

Single threshold Double threshold

Thresholding

CSE 166, Spring 2020 103

Single threshold Dual thresholds

Histograms

Thresholding

• Single threshold

• Dual thresholds

CSE 166, Spring 2020 104

where

where

Noise and thresholding

CSE 166, Spring 2020 105

Noise

Varying background and thresholding

CSE 166, Spring 2020 106

Intensity ramp Product of input and intensity ramp

Input

Basic global thresholding

CSE 166, Spring 2020 107

Intensity ramp ThresholdInput

Basic global thresholding

CSE 166, Spring 2020 108

Value used to stop iterating

Optimum global thresholding

CSE 166, Spring 2020 109

Histogram

Basic global thresholding

Optimum global

thresholding using Otsu’s

method

Input

Otsu’s method

CSE 166, Spring 2020 110

Image smoothing to improve global thresholding

CSE 166, Spring 2020 111

Without smoothing

With smoothing

Otsu’s method

Image smoothing does not always improve global thresholding

CSE 166, Spring 2020 112

Without smoothing

With smoothing

Otsu’s method

Edges to improve global thresholding

CSE 166, Spring 2020 113

Mask image (thresholded

gradient magnitude)

Input

Masked input

Optimum global

thresholding using Otsu’s

method

Edges to improve global thresholding

CSE 166, Spring 2020 114

Input

Masked input

Optimum global

thresholding using Otsu’s

method

Mask image (thresholded absolute Laplacian)

Variable thresholding

CSE 166, Spring 2020 115

Global thresholding

Local standard

deviations

Local thresholding

using standard

deviations

Input

Variable thresholding

CSE 166, Spring 2020 116

Global thresholding

Local thresholding using moving

averages

Input(spot shading)

Variable thresholding

CSE 166, Spring 2020 117

Global thresholding

Local thresholding using moving

averages

Input(sinusoidal shading)

Segmentation by region growing

CSE 166, Spring 2020 118

Initial seed

image

InputX-ray image

Final seed

image

Output image

Segmentation by region growing

CSE 166, Spring 2020 119

Segmentation by region growing

CSE 166, Spring 2020 120

Difference image thresholded with the smallest of the dual

thresholds

Segmentation by region growing

Difference image thresholded using

dual thresholds

Difference image

Advanced segmentation methods

• k-means clustering

• Superpixels

• Graph cuts

CSE 166, Spring 2020 121

Segmentation using k-means clustering

CSE 166, Spring 2020 122

Input Segmentation using k-means, k = 3

Segmentation using k-means clustering

CSE 166, Spring 2020 123

Superpixels

CSE 166, Spring 2020 124

Image of 4,000 superpixels with

boundaries

Image of 4,000 superpixels

Input image of 480,000 pixels

Superpixels

CSE 166, Spring 2020 125

500 superpixels 250 superpixels1,000 superpixels

Superpixels for image segmentation

CSE 166, Spring 2020 126

Segmentation using k-means, k = 3

Superpixel image (100 superpixels)

Input image of 301,678 pixels

Segmentation using k-means, k = 3

Images as graphs

CSE 166, Spring 2020 127

Simple graph with edges only between

4-connected neighbors

Stronger (greater weight) edges are darker

Graph cuts for image segmentation

CSE 166, Spring 2020 128

Cut the weak edges

Graph cuts for image segmentation

CSE 166, Spring 2020 129

Smoothed input Graph cut segmentation

Input

CSE 166

• Image acquisition

• Geometric transformations and image interpolation

• Intensity transformations

• Spatial filtering

• Fourier transform and filtering in the frequency domain

• Image restoration

• Color image processing

• Basis vectors

• Matrix-based transforms

• Basis images

• Wavelet transform

• Image compression

• Image watermarking

• Morphological image processing

• Edge detection

• Segmentation

CSE 166, Spring 2020 130