Post on 21-Feb-2018
7/24/2019 3 MD-Sampling S12 v5 2p
1/18
13 -
D. Malah Video Signal Processing - Spring 2012
Multidimensional Sampling.3
Basics of Lattice Theory
Sampling Over Lattices
Sampling of Video Signals
Filtering Operations in Cameras and Display Devices
References
Wang et. al., Ch. 3.
Bovik, Ch. 2. (by E. Dubois)
E. Dubois, The Sampling and Reconstruction of Time-Varying Imagery with
Application in Video Systems, Proc. IEEE, Vol 73, No. 4, April 1985, pp. 502-522.
Tekalp, Ch.3.
23 -
D. Malah Video Signal Processing - Spring 2012
Basics of Lattice Theory
As video sampling is done at points on a regular grid (not necessarily rectangular)
in the 3-D space, Lattice Theory can provide the mathematical foundation for this
operation.
Lattice - definition:
A lattice, ,in the real K-D space, , is the set of all possible vectors thatcan be represented as integer-weighted combinations of a set of K
linearly independent basis vectors,
K
1 2v , , , ,Kk k K
That is:
1
x x v ,K
Kk k k
k
Lattice n n
1 2V v ,v , ..., vKGenerating matrix:
1 2, K
Kn n n x Vn n [ , , ..., ]
7/24/2019 3 MD-Sampling S12 v5 2p
2/18
33 -
D. Malah Video Signal Processing - Spring 2012
Examples of Lattices
1
1 0
0 1V =
Orthogonal Lattice1
3
22 1
2
0
1V =
Hexagonal Lattice
2
[Wang]
Unit Cells
1x
2x
43 -
D. Malah Video Signal Processing - Spring 2012
Lattice Properties - 1
The basis vectors uniquely define a Lattice.
The basis vectors associated with a Lattice are non-unique.
Examples
3
2
3 12
3
0V =
3 2;
Any point on the grid of a Lattice is determined by a set of indices:
1 2n [ , , , ] K
Kn n n
Theorem: Unit Cell
For a given lattice , there exists a unit cell so that its translation to
all lattice point form a covering of the entire space :
K
( ifx
x) = x ( y x yK
; ,
where, is the translation of by x. ( |x p x p This is a tiling of by unit cells.
K
4
1 1
0 1
V =
4 1
7/24/2019 3 MD-Sampling S12 v5 2p
3/18
7/24/2019 3 MD-Sampling S12 v5 2p
4/18
73 -
D. Malah Video Signal Processing - Spring 2012
Lattice Properties - 4
Volume of a unit Cell
All unit cells of a given lattice have the same volume: det| V |
Sampling Density
Number of unit cells in a unit volume of :K 1det
() =| V |
d
Reciprocal Lattice
Hence: andT T T T I V I V U UV U V U
Let , and *x V m y Un then,
x,y x y m V Un m nT T T T
For a given lattice with a generating matrix , its reciprocal lattice
Is defined as a lattice with a generating matrix U, given by:
V *
1U (V )T .
Thus, the basis functions of V and U are biorthonormal:
1
;0,
,
v u ,,
T
k k
k
kk
Useful property
83 -
D. Malah Video Signal Processing - Spring 2012
1
1 0
0 1V =
3
22 1
2
0
1V =
1
1 0
0 1U =
2 1
3 3
20 1
U =
1U (V )
T V
11() =d
11*( ) =d
Lattice Properties - 5
Reciprocal Lattice Examples
2
2
3( ) =d 2
3
2
*( ) =d
1
2
1
*
2
*
1det*( ) = | V |
()d
d
1det
det| U |
| V |
2*( )
2( )
[Wang]
1
x
1x
2x 2f
2f
1
f
1f
7/24/2019 3 MD-Sampling S12 v5 2p
5/18
93 -
D. Malah Video Signal Processing - Spring 2012
D Periodic Signal-Lattice of a KPeriodicity
1-D periodic signal:
; - non-singularx x Vn , n VK
,x x nT n
K-D periodic signal:
Hence, considering as the generating matrix of a lattice,
the particular lattice is called: Periodicity Lattice.
The Voronoi cell of the lattice is then called the Fundamental Period.
V
103 -
D. Malah Video Signal Processing - Spring 2012
Sampling Over Lattices
The lattice structure provides a tool for uniform sampling of
multidimensional signals, but not necessarily on a rectangular (hypercube)
grid.
Sampling on a lattice
n Vn , n Ks c where,
- Continuous space signal,(x) x Kc
- Sampled space signal,(n) n Ks
Alternatively, Impulse Sampling:
n(x) n x Vn , xKK
s s
Sampled multi-
dimensional Signal
7/24/2019 3 MD-Sampling S12 v5 2p
6/18
113 -
D. Malah Video Signal Processing - Spring 2012
Sampled Space Fourier Transform - SSFT
exp 2n
f n f VnK
Ts s j
SSFT:
Recall,
DSFT: exp 2n
(f ) (n) ( f n)K
T
sd j
CSFT: ( ) ( ) exp( 2 )f x f x xK
T
cc j d
If V = I, SSFT reduces to the DSFTRemark:
Discrete Space
F. T.
Continuous Space
F. T.
123 -
D. Malah Video Signal Processing - Spring 2012
Since and , we get:
Periodicity of the SSFT
The SSFT is periodic: 1
;f Um f U VTs s
Thus, U, the periodicity lattice, is the generating matrix of the
reciprocal lattice and the fundamental period is the
Voronoi cell
*
*( )
Proof
f Vn : (f Um) Vn f Vn m U VnT T T T T
2n
f n exp f VnK
Ts s j
exp 2 1 ;( )j
exp 2 exp 2( (f Um) Vn) ( f Vn)T Tj j
TU V I
m nT
.f Vn m nT T
[Wang]
2
1f
7/24/2019 3 MD-Sampling S12 v5 2p
7/18
133 -
D. Malah Video Signal Processing - Spring 2012
Inverse SSFT
1
exp 2*( )
n f f Vn f , n
T Ks s j d
d
*1
| det( ) |d Vol
V
Proof by:
exp 20
T d
j dotherwise
*( )
, n mf V(n m) f
,
where,
Linear convolution of signals sampled on the same lattice:
SSFT
K
s c s c
s s s s
s s s s
h h
h h
h H
m
n Vn ; n Vn
n * n n m m
n * n f f
Then,
Let:
143 -
D. Malah Video Signal Processing - Spring 2012
Generalized Nyquist Theorem
If a continuous signal is sampled over a lattice , with a
Generating matrix V, then the SSFT of the sampled signal
, ,(x) Kc
x
n Vn , n Ks c
Is given in terms of the CSFT of the original continuous signal by:
SSFTm
f n f UmK
s s cd
And, it is possible to reconstruct the original signal from ,
Iff the support region of the CSFT of the original signal is limited within
the Voronoi cell of the reciprocal lattice, i.e.,
(x)c ns
0c for *f f The reconstruction can then be done by filtering the sampled signal
(impulse sampling) by a filter whose CSFT is:
1
0
*, f
,
r
fdH
otherwise
7/24/2019 3 MD-Sampling S12 v5 2p
8/18
153 -
D. Malah Video Signal Processing - Spring 2012
[Wang]
fs
1f 1f
1f
1f
11
f
2f
2
2
2f
2
2f
1 3/r
1 2/
1 2/
1 2/
1 2/
1 2/
1 2/
1 2/
1 2/
163 -
D. Malah Video Signal Processing - Spring 2012
Proof of the Generalized Nyquist Theorem
Since we can apply the Inv-CSFT to determine : n Vn ,s c ns
2(n) (V n) (f )exp( f Vn) f K
T
s c c j d
exp 2*m ( )
f Um ( (f Um) Vn) f K
Tc j d
Using: exp 2 exp 2 (since )T T T T T j j ( (f Um) Vn) ( f Vn) m U Vn m n
we get:
exp 2
exp 2
K
K
Ts c
Tc
j d
d
*
*
m ( )
m( )
(n) f Um ( f Vn) f
f Um ( f Vn) f
Comparing with the expression for the Inv-SSFT:
1
exp 2*( )
n f f Vn f , n
T Ks s j d
d
we find,
m
f f UmK
s cd
7/24/2019 3 MD-Sampling S12 v5 2p
9/18
173 -
D. Malah Video Signal Processing - Spring 2012
Reconstruction by Interpolation
y
x x y y yK
r r sh d
n y
x n x y y Vn yK K
r s rh d
n
(y) n y Vn , yK
Ks s
Using impulse sampling:
Hence,
and we get the Interpolation Formula:
n
x n x VnK
r s rh
Reconstructed
Signal
183 -
D. Malah Video Signal Processing - Spring 2012
Summary of Relations between Fourier Transforms
( ) ( ) exp( 2 )f x f x xK
T
cc j d
CSFT:
DSFT: 2n
(f ) (n)exp( f n)K
T
sd j
exp 2n
f n f VnK
Ts s j
SSFT:
Sampling: n Vn , n K
s c
1
Unit Hypercube
K
K
Ts d
Kd s
T Ts d d s
s c
Kd c
I
d
d I
*
*
m
m
f V f , f
f Uf , f
f Uf ; f V f U V
f f Um , f
f Uf Um , f
1 1f Uf V f ; f V f U f T Tc d d d c c
If is band-limited to within then:( )xc * f Uf , f Kd cd I Note
7/24/2019 3 MD-Sampling S12 v5 2p
10/18
193 -
D. Malah Video Signal Processing - Spring 2012
Sampling Efficiency
1 (no aliasing)
*() =
( )
Vol Unit Sphere
Vol
Sampling Efficiency:
Assume that the support of signal spectrum is a unit sphere.
To avoid aliasing, should enclose the sphere.*( )
Definition
Since *( ) ()Vol d () =()
Vol Unit Sphere
d
The closer is to 1 (from below), the higher the effinciency()
Examples (2D)
Vol Unit Sphere
2D:
3D:43
[Wang]
4 4d ( ) ; ( ) / 4 4d ( ) ; ( ) / 2 3 2 3d ( ) ; ( ) /
203 -
D. Malah Video Signal Processing - Spring 2012
Sampling of Video Signals
From the properties of the HVS
To avoid flicker: 60 to 70 frames/sec (orfps) (high illumination)
At a viewing distance of 3 times screen height:
3180 180
( )s s sf df f f cpdh
Thus, for 25 cpd (low visibility of lines), the spatial resolution required is:
180 18025 480
3 3( / )
sf f lines cycles picture height
and, for an image aspect ratio (IAR) of 4:3, we get 640 pixels/line
NTSC: 60 fields/sec (Interlaced), 240 active lines/ field
HDTV: IAR=16:9, 60 fps, 720 1080 lines/fr, 1280 1920 pixels/line
Computer Display: 72 fps (progressive), 1024 lines, 1280 pixels/line
7/24/2019 3 MD-Sampling S12 v5 2p
11/18
213 -
D. Malah Video Signal Processing - Spring 2012
Video Raster Scan
Corresponds to Sampling of the 3D Video signal in the temporal and
vertical directions.
Thus, we consider sampling a continuous 2D video signal along the
temporal and vertical axes.
The Sampling Lattices for Progressive and Interlaced scans are:
Progressive:
Interlaced:
12
1 1 1
02 0= =
00V U
t
y
t
y
12
2 2 1 12
02
= =0V U
t
t y
t t
y
Where: - time between fields; - distance between consecutive linest y
223 -
D. Malah Video Signal Processing - Spring 2012
11
12
1 1 1
02 0= =
00
T t
y
t
y
(V )V U
1
1
*
Progressive ScanNearest
aliasing
centers
[Wang]
(Fig. 3.6)
t
yf
7/24/2019 3 MD-Sampling S12 v5 2p
12/18
233 -
D. Malah Video Signal Processing - Spring 2012
2
2
*
Interlaced ScanNearest
aliasing
centers
[Wang]
(Fig. 3.6)
1
2
1
22 2 1 1
2
02= =0
T
t
y y
t ty
(V )V U
t
243 -
D. Malah Video Signal Processing - Spring 2012
Comparison of Interlaced and Progressive Scans
Same sampling density: 1 21 2
1 1 1
det det 2() = ( ) =
| V | | V |d d
t y
Same nearest aliases along the vertical frequency axis, at .1
y No motion: Same vertical resolution.
With motion, Interlaced has a closer alias, hence a lower resolution.
1
2 t In temporal direction, nearest alias for Progressive is closer, at ,
hence Interlaced has less flicker.
Different mixed (off-axis) aliases: for Progressive;
for Interlaced, resulting in Interline flicker for Interlaced.
1 1
2( , )
t y 1 1
2 2( , )
t y
For a signal with isotropic spectral support, the interlaced scan ismore efficient (assuming equal temporal and spatial frequency
scales).
7/24/2019 3 MD-Sampling S12 v5 2p
13/18
253 -
D. Malah Video Signal Processing - Spring 2012
Sampling Video in 3D - 1
Assume frequency axes were calibrated., ,x y tf f f
Assume signal spectrum support in the normalized frequency
domain to be a Unit Sphere.
Progressive sampling intervals: , ,x y t
Aligned samples 3D Cube - Orthorhombic Lattice (ORT)
To avoid aliasing: 12
x y t 4388 6
( ) ; ( )d ORT ORT
Progressive
1
1
1
0 0
00
0 0
x
y
t
U=
[Wang]
(Fig. 3.7)
x
y
tfx
fy
ft
t1 / t
1 / x
1 /
263 -
D. Malah Video Signal Processing - Spring 2012
Sampling Video in 3D 2
Interlaced sampling intervals: 2 2, , /x y t
Vertically aligned samples ALI Lattice
To avoid aliasing: 1 12 3;x y t 434 3
4 3 3 3( ) ; ( )d ALI ALI
Interlaced - 1
[Wang]
(Fig. 3.7)
1
12
1 2
0 0
00
0
U=
x
y
t t
x
y
t
fx
fy
ft
2/t
7/24/2019 3 MD-Sampling S12 v5 2p
14/18
273 -
D. Malah Video Signal Processing - Spring 2012
Sampling Video in 3D 3
Interlaced sampling intervals: 2 2, , /x y t
Horizontally shifted samples by BCO Lattice
To avoid aliasing: 1 12 2 2
;x t y 4 23 2
( ) ; ( )d BCO BCO
Interlaced - 2
2/x
[Wang]
(Fig. 3.7)
x
y
t
fx
fy
ft
283 -
D. Malah Video Signal Processing - Spring 2012
Sampling Video in 3D 4
Interlaced sampling intervals: 2 2, , /x y t
each field contains all the lines, but the samples in the same
field are interleaved in the even and odd lines
FCO Lattice To avoid aliasing: 1 1
2 2;x y t 4 2
3 2( ) ; ( )d FCO FCO
Interlaced - 3
Same as BCO, but better visually.
[Wang]
(Fig. 3.7)
xy
tfx
fy
ft
7/24/2019 3 MD-Sampling S12 v5 2p
15/18
293 -
D. Malah Video Signal Processing - Spring 2012
[Wang]
Sampling Video in 3D 5
Summary
303 -
D. Malah Video Signal Processing - Spring 2012
1-On Spatial and Temporal Aliasing
Moving Sinusoidal Pattern (vertical lines)
1 /h
f cycle cm
Q. The pattern is moving horizontally at the speed of 3 cm/sec.
If we sample the pattern at:
What is the apparent motion and sinusoidal frequency?
3 3 sec, , ,
/ ; /s x s y s tf f samples cm f frames
Qualititative answer: On the camera image plane well get a stationary image.
1 0 0( , , ) ( , , )h v t
f f f
[Wang](Fig. 3.8)
7/24/2019 3 MD-Sampling S12 v5 2p
16/18
313 -
D. Malah Video Signal Processing - Spring 2012
Analytic Answer
Applying the CSFT:
1 0 1 0
3 3
( , ) ( , ), ( , )
( ) ,
x y
t x x y y
f f
f f v f v
Thus, the 3D CSFT has a pair of impulses at: 1 0 3 1 0 3( , , ) ( , , ),( , , )x y tf f f
1-1
3
3-
xf
tftf
x
f 3, , ,s x s y s t
f f f
,: 2 6s t tAliasing f < f
1 0 0 1 0 0
:
( , , ) ( , , ), ( , , )x y t
Aliased components at
f f f
2-On Spatial and Temporal Aliasing
[Wang](Fig. 3.8)
tf
323 -
D. Malah Video Signal Processing - Spring 2012
Filtering Operations in Cameras and Display Devices
Camera Aperture
, ,x y t Consider a cubic sampling lattice at intervals:
sampling frequencies, , ,
1 1 1, ,s x s x s tf f f
x y t
Ideal pre-filter would be a lowpass filter with cutoff frequenciesat half the sampling frequencies.
Practically we have the following image acquisition characteristics:
Temporal Aperture
- Integration over exposure time , corresponding to a temporal filter:e
,
1, [0, ]
( )
0,p t
t eh t e
otherwise
CSFT ,sin( )
( ) exp( ) tp t t tt
f eH f j f e
f
Thus, the cutoff freq. is . Since usually , we get aliasing.1
e e t
This is preferred on the blurring due to selection of a larger .e
7/24/2019 3 MD-Sampling S12 v5 2p
17/18
333 -
D. Malah Video Signal Processing - Spring 2012
Spatial Aperture
Spatial sensor integration is modeled by:
2 2 2
, ,
1( , ) exp( ( ) / 2 )2
p x yh x y x y
2 2 2, , ,( ) exp( ( ) / 2 )
CSFT
p x y x y x yH f f f f
1
2
Is selected so that the normalized filters response is 0.5 at half
the sampling frequencies (horizontal & vertical). Hence for:
, ,
, , , ,2 ln 2
s x y
s x s y s x y
ff f f
Combined Aperture
, , ,( , , ) ( ) ( , ) CSFT
p p t p x yh x y t h t h x y , , , ,( , ) ( ) ( , )p x y t p t t p x y x yH f f f H f H f f
343 -
D. Malah Video Signal Processing - Spring 2012
,1 ; 480 / .60
s ye t f lines pic height
( , )ph y t
( , )p y tH f f
tf
f
t
y
Combined Aperture
( 0)x
At f
[Wang]
7/24/2019 3 MD-Sampling S12 v5 2p
18/18
353 -
D. Malah Video Signal Processing - Spring 2012
Comments
Human Vision Considerations
Human viewer tolerates more aliasing than resolution
loss (blurring).
The preservation of the signal in the pass-band is more important
to the viewer than attenuation outside the pass-band.
Digital cameras may capture at higher sampling rates and then
implement explicit filtering before converting to lower resolution.
Digital Filtering
In a CRT, a beam scans a phosphor screen. Width of beam
determines vertical filtering. A Wide beam causes blurring, hence a
thin beam is preferred (viewer may observe scan lines).
Temporal filtering is determined by phosphor response decay time.
HVS performs temporal interpolation
Note: Combination of camera and display apertures results in a maximum
vertical resolution that is only K=0.7 of the theoretical limit
(K - Kell factor).
Display Aperture