ENEE631 Digital Image Processing (Spring'04) Sampling Issues in Image and Video Spring ’04...

ENEE631 Digital Image Processing (Spring'04)

Sampling Issues in Image and VideoSampling Issues in Image and Video

Spring ’04 Instructor: Min Wu

ECE Department, Univ. of Maryland, College Park

www.ajconline.umd.edu (select ENEE631 S’04) [email protected]

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

04

)

Based on ENEE631 Based on ENEE631 Spring’04Spring’04Section 15Section 15

ENEE631 Digital Image Processing (Spring'04) Lec24 – 2-D and 3-D Sampling [3]

Sampling: From 1-D to 2-D and 3-DSampling: From 1-D to 2-D and 3-D

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

04

)


Review: 1-D SamplingReview: 1-D Sampling Time domain

– Multiply continuous-time signal with periodic impulse train

Frequency domain– Duality: sampling in one domain tiling in another domain

FT of an impulse train is an impulse train (proper scaling & stretching)

Review Oppenheim “Sig. & Sys” Chapt.7 (Sampling) Chapt.3,4,5 (FS,FT,DFT)

x(t)

p(t) = k ( t - kT)T

xs(t)

P() = k ( - 2k/T) *2/T

2/TX()

Xs()

2/T

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Review: 1-D Sampling TheoremReview: 1-D Sampling Theorem

1-D Sampling Theorem

– A 1-D signal x(t) bandlimited within [-B,B] can be uniquely determined by its samples x(nT) if s > 2B (sample fast enough).

– Using the samples x(nT), we can reconstruct x(t) by filtering the impulse version of x(nT) by an ideal low pass filter

Sampling below Nyquist rate (2B) cause Aliasing

Xs() with s < 2B Aliasing

s=2/T

B

Xs() with s > 2B

Perfect Reconstructable

s=2/T

B-s

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Extend to 2-D Sampling with Rectangular GridExtend to 2-D Sampling with Rectangular Grid

Bandlimited 2-D signal

– Its FT is zero outside a bounded region ( |x|> x0, |y|> y0 ) in spatial freq. domain

– Real-word multi-dimensional signals often exhibit diamond or football shape of support

With spectrum normalization, we will get spherical shape of support

Jain’s Fig.4.6

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

/20

04

)


2-D Sampling (cont’d)2-D Sampling (cont’d) 2-D Comb function

comb(x,y; x, y) = m,n ( x - mx, y - ny ) ~ separable function

FT: COMB(x, y) = comb(x, y; 1/x, 1/y) / xy

Sampling vs. Replication (tiling)

– Nyquist rates (2x0 and 2y0) Aliasing

Jain’s Fig.4.7

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


2-D Sampling: Beyond Rectangular Grid2-D Sampling: Beyond Rectangular Grid Sampling at nonrectangular grid

– May give more efficient sampling density when spectrum region of support is not rectangular

Sampling density measured by #samples needed per unit area

– E.g. interlaced grid for diamond-shaped region of support

equiv. to rotate 45-deg. of rectangular grid

spectrum rotate by thesame degree

From Wang’s book preprint Fig.4.2

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

/20

04

)


General Sampling LatticeGeneral Sampling Lattice Lattice in K-dimension space R K

– A set of all possible vectors represented as integer weighted combinations of K linearly independent basis vectors

Generating matrix V (sampling matrix)

V = [v1, v2, …, vk] => lattice points x = V ne.g., identity matrix V ~ square lattice

Voronoi cell of a lattice– A “unit cell” of a lattice, whose translations cover the whole space– Consists of vectors that are closer to the origin than to other lattice points

cell boundaries are equidistant lines between surrounding lattice points

Sampling density d() = 1 / |det(V)|– |det(V)| measures volume of a cell; d() is # lattice points in unit volume

K

jkjj

K nn1

,| ZR vxx

From Wang’s book preprint Fig.3.1U

MC

P E

NE

E6

31

Slid

es

(cre

ate

d b

y M

.Wu

© 2

00

1/2

00

4)


Sampling Density:d1 = 1d2 = 2 / 3

)(hexagonal 12/1

02/3

ar)(rectangul 10

01

2

1

V

V


Example of LatticesExample of Lattices

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

04

)


Frequency Domain View & Reciprocal LatticeFrequency Domain View & Reciprocal Lattice Reciprocal lattice # for a lattice (with generating matrix V)

– Generating matrix of # is U = (VT)-1 – Basis vectors for and # are orthonormal to each other: VT U = I– Denser lattice has sparser reciprocal lattice # : det(U) = 1 / det(V)

Frequency domain view of sampling over lattice

– Sampling in spatial domain Repetition in freq. Domain– Repetition grid in freq. domain can be described by reciprocal lattice– Intuition for “reciprocal”

[e.g.] rectangular grid that sample faster horizontally than vertically=>the repetition in frequency domain is slower horizontally than vertically

Aliasing and prefiltering to avoid aliasing

– Aliasing happens when Voronoi cell of reciprocal lattice can’t completely cover signal spectrum

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

/20

04

)


Sampling EfficiencySampling Efficiency Consider spherical signal spectrum support

– Most real-world signals have symmetric freq. contents in many directions– The multi-dim spectrum can be approximated well by a sphere (with proper

scaling spectrum support)

Voronoi cell of reciprocal lattice need to cover the sphere to avoid aliasing– Tighter fit of the Voronoi cell to the sphere requires less sampling density

What lattice gives the best sphere-covering capability? Sampling Efficiency = volume(unit sphere) / d() prefer close to 1

From Wang’s book preprint Fig.4.2 & 3.5U

MC

P E

NE

E6

31

Slid

es

(cre

ate

d b

y M

.Wu

© 2

00

1/2

00

4)


Sampling Lattice ConversionSampling Lattice Conversion


Intermediate

Original

Targeted

UMCP ENEE631 Slides (created by M.Wu © 2001)


Recall: 1-D Upsample and DownsampleRecall: 1-D Upsample and Downsample

From Crochiere-Rabiner “Multirate DSP” book Fig.2.15-16

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


General Procedures for Sampling Rate ConversionGeneral Procedures for Sampling Rate Conversion


UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Example: Frame Rate ConversionExample: Frame Rate Conversion Video sampling: formulate as a 3-D sampling problem

Note: different signal characteristics and visual sensitivities along spatial and temporal dimensions (see Wang’s Sec.3.3 on video sampling)

General Approach to frame rate conversion– Upsample => LPF => Downsample

Interlaced 50 fields/sec 60 fields/sec– Analyze in terms of 2-D sampling lattice (y, t)

– Convert odd field rate and even field rate separately do 25 30 rate conversion twice not fully utilize info. in the other fields

– Deinterlace first then convert frame rate do 50 60 frame rate conversion: 50 300 60

– Simplify 50 60 by converting 5 frames 6 frames each of output 6 frames is from two nearest frames of the 5

originals weights are inversely proportional to the distance between I/O

– May do motion-interpolation for hybrid-coded video

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

/20

04

)



UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Case Studies on Sampling and Resampling Case Studies on Sampling and Resampling

in Video Processingin Video Processing

Reading Assignment: Wang’s book Chapter 4Reading Assignment: Wang’s book Chapter 4

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

04

)


Video Format Conversion for NTSC Video Format Conversion for NTSC PAL PAL

Require both temporal and spatial rate conversion– NTSC 525 lines per picture, 60 fields per second– PAL 625 lines per picture, 50 fields per second

Ideal approach (direct conversion)– 525 lines 60 field/sec 13125 line 300 field/sec

625 lines 50 field/sec

4-step sequential conversion– Deinterlace => line rate conversion

=> frame rate conversion => interlace

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Simplified Video Format ConversionSimplified Video Format Conversion

50 field/sec 60 field/sec

– Simplified after deinterlacing to 5 frames 6 frames– Conversion involves two adjacent frames only

625 lines 525 lines

– Simplified to 25 lines 21 lines– Conversion involves two adjacent lines only

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Interlaced Video and DeinterlacingInterlaced Video and Deinterlacing Interlaced video

Odd field at 0 Even field at t Odd field at 2t Even field at 3t …

Deinterlacing

– Merge to get a complete frame with odd and even field

Examples from http://www.geocities.com/lukesvideo/interlacing.html

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


De-interlacing: Practical ApproachesDe-interlacing: Practical Approaches

Spatial interpolation

– Vertical interpolation within the same field (1-D upsample by 2)

– Line averaging ~ average the line above and below D=(C+E)/2

Temporal interpolation

– 2-frame field merging => artifacts– 3-frame field averaging D=(K+R)/2

fill in the missing odd field by averaging odd fields before and after

Spatial-temporal interpolation

– Line-and-field averaging D=(C+E+K+R)/4

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Motion-Compensated De-interlacing Motion-Compensated De-interlacing

Stationary video scenes

– Temporary deinterlacing approach yield good result

Scenes with rapid temporal changes

– Artifacts incurred from temporal interpolation– Spatial interpolation alone is better than involving temporal

interpolation

Switching between spatial & temporal interpolation modes

– Based on motion detection result– Hard switching or weighted average– Motion-compensated interpolation

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)

ENEE631 Digital Image Processing (Spring'04) Sampling Issues in Image and Video Spring ’04...

Documents

Transcript of ENEE631 Digital Image Processing (Spring'04) Sampling Issues in Image and Video Spring ’04...