Introduction to Signal Spaces - College of Engineering, Michigan
Transcript of Introduction to Signal Spaces - College of Engineering, Michigan
Introduction to Signal Spaces
Introduction to Signal Spaces
Selin AviyenteDepartment of Electrical and Computer Engineering
Michigan State University
January 12, 2010
Introduction to Signal Spaces
Motivation
Outline
1 Motivation
2 Vector Space
3 Inner Product Space
4 Normed Vector Space
5 L2 and l2 Spaces
6 Basis
7 Fourier Series
Introduction to Signal Spaces
Motivation
Signal Expansion
Our goal in this course is to expand any signal x fromsome space S in terms of elementary signals {φi} such asx =
∑i αiφi .
The set {φi} should be complete for S.The expansion coefficients, αi should be sparse, explainthe signal with as few coefficients as possible. Attractive forsignal processing applications like compression.There should be a fast algorithm to compute αi from thesignal.{φi} should be a meaningful set of functions, e.g.sinusoids.
Introduction to Signal Spaces
Vector Space
Outline
1 Motivation
2 Vector Space
3 Inner Product Space
4 Normed Vector Space
5 L2 and l2 Spaces
6 Basis
7 Fourier Series
Introduction to Signal Spaces
Vector Space
Vector Space
A linear vector space X is a collection of vectors (over thecomplex or real field) together with two operations; vectoraddition and scalar multiplication, which for all x , y ∈ X andα ∈ C/R satisfy the following:
1 Commutativity: x + y = y + x for all x, y ∈ X .2 Associativity: x + (y + z) = (x + y) + z.3 Distributivity: α(x + y) = αx + αy.4 Additive Identity: There is a zero vector such that x + 0 = x.5 Additive Inverse: For each x ∈ X there is a unique vector−x such that x +−x = 0.
6 Multiplicative Identity: 1x = x.
Introduction to Signal Spaces
Vector Space
Examples
Set of real numbers (R): It is a real vector space withaddition defined in the usual way and multiplication byscalars defined as ordinary multiplication. The null vectoris the real number zero.n-dimensional real or complex vector spaces: Rn,Cn.Lp[0, T ] = {f (t)| ∫ |f (t)|pdt < ∞}, 1 ≤ p < ∞. Used forrepresenting continuous time signals.lp = {x [n]|∑ |x [n]|p < ∞}, 1 ≤ p < ∞. Used for discretetime signals.C[0, T ]: The collection of all real-valued continuousfunctions on the interval [0, T ].The collection of functions that are piecewise constantbetween integers.Is R+ a vector space?
Introduction to Signal Spaces
Vector Space
Subspace
A subspace, S, of a vector space V is a subset of V that isclosed under vector addition and scalar multiplication.
Example: Is X1 = {x ∈ C[0, T ] :∫ T
0 x(t)dt = 1} asubspace of C[0, T ]?Answer: No, if x, y ∈ X1, then check whether x + y ∈ X1.
∫ T
0x(t) + y(t)dt =
∫ T
0x(t)dt +
∫ T
0y(t)dt = 2 6= 1 (1)
Introduction to Signal Spaces
Vector Space
Subspace
A subspace, S, of a vector space V is a subset of V that isclosed under vector addition and scalar multiplication.
Example: Is X1 = {x ∈ C[0, T ] :∫ T
0 x(t)dt = 1} asubspace of C[0, T ]?Answer: No, if x, y ∈ X1, then check whether x + y ∈ X1.
∫ T
0x(t) + y(t)dt =
∫ T
0x(t)dt +
∫ T
0y(t)dt = 2 6= 1 (1)
Introduction to Signal Spaces
Inner Product Space
Outline
1 Motivation
2 Vector Space
3 Inner Product Space
4 Normed Vector Space
5 L2 and l2 Spaces
6 Basis
7 Fourier Series
Introduction to Signal Spaces
Inner Product Space
Inner Product
An inner product of two vectors x, y ∈ V on a vector spaceis a complex-valued scalar assigned to the two vectors andsatisfies the following properties:
1 < x, y >= (< y, x >)∗.2 < αx, y >= α < x, y >.3 < x + y, z >=< x, z > + < y, z >.4 < x, x >≥ 0, < x, x >= 0 iff x = 0.
An inner product space (pre-Hilbert space) is a linearvector space in which an inner product can be defined forall elements of the space and a norm is given by‖x‖ = (< x, x >)1/2.Cauchy-Schwarz Inequality: For all x, y in an inner productspace, | < x, y > | ≤ ‖x‖‖y‖. Equality holds if and only ifx = λy or y = 0.
Introduction to Signal Spaces
Inner Product Space
Inner Product Space
A complete inner product space is called a Hilbert space.Completeness means that every sequence of vectors inthe vector space converge to a vector in the vector space,e.g. the space of rational numbers is not complete.A sequence of vectors {xn} is called a Cauchy sequence if‖xn − xm‖ → 0 as n, m →∞. If every Cauchy sequence inV converges to a vector in V , then V is called complete.Example: V = CN , < x, y >= yHx =
∑Ni=1 xiy∗i .
Example: V = L2[a, b], < x, y >=∫ b
a x(t)y∗(t)dt .In this course, the space of signals we will consider will bea Hilbert space.
Introduction to Signal Spaces
Normed Vector Space
Outline
1 Motivation
2 Vector Space
3 Inner Product Space
4 Normed Vector Space
5 L2 and l2 Spaces
6 Basis
7 Fourier Series
Introduction to Signal Spaces
Normed Vector Space
Normed Vector Space
A norm is a function (‖ · ‖ : V → R) such that the followingproperties hold:
1 ‖x‖ ≥ 0 with equality iff x = 0.2 ‖αx‖ = |α|‖x‖.3 Triangle Inequality: ‖x + y‖ ≤ ‖x‖+ ‖y‖ with equality iff
x = αy.
The norm measures the size of a vector. A vector spacetogether with a norm function is called the normed vectorspace.The notion of length (norm) gives meaning to the distancebetween two vectors in V , d(x, y) = ‖x− y‖.
Introduction to Signal Spaces
Normed Vector Space
Normed Vector Space: Examples
For example, V = RN , ‖x‖ =√∑N
i=1 x2i .
V = lp,‖x‖ = (∑
n xp[n])1/p.The normed space C[a, b] consists of continuous functionson the interval [a, b] together with the norm‖x‖ = maxa≤t≤b|x(t)|.You can define different norms for the same vector spaceyielding different normed spaces.A norm can be defined directly from an inner product,though not every norm can be defined from an innerproduct.
Introduction to Signal Spaces
L2 and l2 Spaces
Outline
1 Motivation
2 Vector Space
3 Inner Product Space
4 Normed Vector Space
5 L2 and l2 Spaces
6 Basis
7 Fourier Series
Introduction to Signal Spaces
L2 and l2 Spaces
L2 Space
For an interval a ≤ t ≤ b, the space L2([a, b]) is the set ofall square integrable functions defined on a ≤ t ≤ b, i.e.L2([a, b]) = {f : [a, b] → C;
∫ ba |f (t)|2dt < ∞}.
This corresponds to the space of all signals whose energyis finite.Functions that are discontinuous are allowed as membersof this space.The space L2[a, b] is infinite dimensional.If a = 0 and b = 1, then f (t) = 1/t is an example of afunction that does not belong to L2[0, 1].The inner product on L2[a, b] is defined as< f , g >=
∫ ba f (t)g∗(t)dt .
Introduction to Signal Spaces
L2 and l2 Spaces
l2 Space
For many applications, the signal is already discrete. Insuch cases, we represent the signal as a sequence.The space l2 is the set of all sequences with
∑n |xn|2 < ∞.
The inner product on this space is defined as< x , y >=
∑n xny∗n .
Introduction to Signal Spaces
Basis
Outline
1 Motivation
2 Vector Space
3 Inner Product Space
4 Normed Vector Space
5 L2 and l2 Spaces
6 Basis
7 Fourier Series
Introduction to Signal Spaces
Basis
Span and Basis
A necessary and sufficient condition for the set of vectorsx1, x2, . . . , xn to be linearly independent is that theexpression
∑nk=1 αkxk = 0 implies αk = 0 for all
k = 1, 2, 3, . . . , n.The span of set S ⊂ V is the subspace of V containing alllinear combinations of vectors in S. WhenS = {x1, x2, . . . , xN}, span(S) = {∑N
i=1 αixi}.A subset of linearly independent vectors {x1, x2, . . . , xN} iscalled a basis for V when its span equals to V . In thiscase, we say that the dimension of V is equal to N. V isinfinite-dimensional if it contains an infinite number oflinearly independent vectors.Any two bases for a finite-dimensional vector spacecontain the same number of elements.
Introduction to Signal Spaces
Basis
Basis Examples
Once you have a basis for a vector space you can expressany vector as a linear combination of the elements of thisset.Example: For R3, the basis is (1, 0, 0), (0, 1, 0), (0, 0, 1).Example: The space of infinite sequences, such as l2, isspanned by the infinite set {δ[n − k ]}k ∈ Z. Since they arelinearly independent, the space is infinite-dimensional.Example: The space of second order polynomials, P2 isspanned by {1, x , x2}.Note: In infinite dimensional spaces, the concept of basisis a bit different since we have not defined infinite sums.
Introduction to Signal Spaces
Basis
Orthogonality
The vectors x and y are said to be orthogonal if< x, y >= 0.The collection of vectors, ei ,i = 1, . . . , N, is said to beorthogonal if each ei has unit length and < ei , ej >= δij .Two subspaces V1 and V2 of V are said to be orthogonal ifeach vector in V1 is orthogonal to every vector in V2.
Introduction to Signal Spaces
Basis
Orthogonal and Orthonormal Basis
A set of vectors {x1, x2, . . . , } form an orthonormal basis forV if they span V and < xi , xj >= δij .In this course, we will work with infinite dimensionalfunction spaces such as L2 and l2.In Hilbert space, orthonormal bases are more useful. Anyorthonormal set in a Hilbert space can be extended to forman orthonormal basis.Fourier Series Theorem: Let {xn} be an orthonormal set ina Hilbert space, H. Then the following statements areequivalent:
1 {xn} is an orhonormal basis.2 For any x ∈ H, x =
∑n < x , xn > xn.
3 For any two vectors, < x , y >=∑
n < x , xn >< y , yn >.4 For any x , ‖x‖2 =
∑n | < x , xn > |2.
Introduction to Signal Spaces
Basis
Orthogonal Projection Theorem
Suppose V0 is a subspace of an inner product space V .Suppose {e1, e2, . . . , eN} is an orthonormal basis for V0.Let v ∈ V and define v0 as the orthogonal projection of vonto V0, then v0 =
∑Ni=1 < v , ei > ei and it is the vector
closest to v in the space V0. Moreover, the error v − v0 isorthogonal to v0.Example: V = R2 and V0 = R, then v0 =< v , e1 > e1. InR2, the inner product is the dot product. Therefore,v0 = vcos(θ).In this course, we will consider orthogonal projections ininfinite dimensional vector spaces. Wavelet expansion isequivalent to projecting a signal onto different subspaces.
Introduction to Signal Spaces
Basis
Gram-Schmidt Orthogonalization
Given any countable linearly independent set {yn} in aninner product space, it is always possible to construct anorthonormal set from it.We can construct an orthonormal set {xn} as follows:
1 x1 = y1‖y1‖ .
2 xk+1 =yk+1−
∑ki=1<yk+1,xi>xi
‖yk+1−∑k
i=1<yk+1,xi>xi‖ .
Introduction to Signal Spaces
Fourier Series
Outline
1 Motivation
2 Vector Space
3 Inner Product Space
4 Normed Vector Space
5 L2 and l2 Spaces
6 Basis
7 Fourier Series
Introduction to Signal Spaces
Fourier Series
Fourier Analysis
There are many practical reasons for expanding a functionas a trigonometric sum. If f (t) is a signal, then adecomposition of f into a trigonometric sum gives adescription of its component frequencies.The space of functions considered are periodic functions,L2[−T/2, T/2] with the inner product< f , g >= 1
T
∫f (t)g∗(t)dt .
The functions {ejω0kt} form an orthonormal basis.By Fourier Theory, they span the space. Orthonormalitycan be proven as follows:
< ejω0kt , ejω0lt > =1T
∫ T/2
−T/2ejω0(k−l)tdt
=1T
[ej(k−l)ω0T/2
j(k − l)ω0− e−j(k−l)ω0T/2
j(k − l)ω0]
=
{1, k = l
sinc(π(k − l)) = 0, k 6= l(2)
Therefore, any f (t) ∈ L2[−T/2, T/2] can be written asf (t) =
∑k F [k ]ejkω0t where F [k ] = 1
T
∫f (t)e−jkω0tdt .
Introduction to Signal Spaces
Fourier Series
If we need to find the best approximation of x(t) using Nterms in the summation, this is equivalent to finding theorthogonal projection of x(t) onto a subspace spanned by{1, ejω0t , . . . , ejω0Nt} and the expansion coefficients that willminimize the error are given by < x(t), ejkω0t >.Fourier transform is an extension for aperiodic signals andcan be defined as F (ω) =< f (t), ejωt >.
Introduction to Signal Spaces
Fourier Series
Shannon’s Sampling Theorem
If f (t) is continuous and bandlimited to ωm, then f (t) isuniquely defined by its samples taken at the samplingfrequency of 2ωm (the minimum sampling frequency,Nyquist rate), with T = π/ωm.
f (t) =∞∑
n=−∞f (nT )sincT (t − nT )
sincT (t) =sin(πt/T )
πt/T(3)
This implies that the space of continuous and bandlimitedfunctions form a vector space (UT functions whose Fouriertransform is limited to [−π/T , π/T ]) andh(t − nT ) = sincT (t − nT ) form an orthogonal basis for thisspace.In this case, f (nT ) = 1
T < f (t), h(t − nT ) >.Proof: Fourier transform of h(t) is H(ω) = Trect(t/2π/T ).You first need to prove orthogonality (HW 1),< h(t − nT ), h(t − pT ) >= T δ(n − p).
< f (t), h(t − pT ) > =∑
n
f (nT ) < h(t − nT ), h(t − pT ) >
= Tf (pT )
f (pT ) =1T
< f (t), h(t − pT ) > (4)