Linear Algebra

Definitions Dot product of vectors A=<a1,a2,…an> and B=<b1,b2, …, bn>

A B = a1*b1 + a2*b2 + … + an * bn

The length of a vector <a1, a2, … , an> is (a12 + a2

2 + … + an2)½

A set of vectors are linearly independent if none of them can be expressed as a linear combination of the others

A set of n vectors is a basis for an n-dimensional space if they are all linearly independent

Vectors are orthogonal if their dot product = 0 (ex: <1/5½,2/5½ >, <-2/5½,1/5½ >) Two orthogonal vectors are orthonormal if their lengths are all unity

<1,0,0>

<0,1,0>

<0,0,1>

Interpretation1.We can represent any vector by a linear combination of an orthogonal set<1/5½,2/5½ > • <4,7> = 18/5½ and <-2/5½,1/5½>• <4,7> = -1/5½

(18/5½)<1/5½,2/5½ > + (-1/5½)<-2/5½,1/5½> = <20/5, 35/5> = <4,7>2.Orthogonal vectors extends the concept of: “perpendicular”

Extends Euclidean space algebra to higher dimensions

Extend Linear Algebra to CalculusTwo functions, f(x) and g(x) and are orthogonal over the interval with weighting function w(x) if

If, in addition, f(x) ≠ g(x) ≠ 0 and w(x) ≠ 0

= 0 , and = 0

the functions f(x) and g(x) are said to be orthornormal

Interpretation: Sets of periodic functions are orthornormal

Fourier Decomposition

• Fourier showed us that we can decompose any time domain signal into a (possibly infinite) series of waves described by a set of orthonormal functions

• Each wave has a well-defined frequency and can vary in amplitude

Fourier Series• An infinite sum of sinusoids modeling a continuous signal is an

example of a Fourier Series• Each point on the Fourier number line represents a particular

sinusoid • Fourier transform: Decompose a signal into Fourier components,

perform processing, and recombine the results to solve an original problem

We use transforms to alter a problem that cannot be solved in one domain into another, where the solution is possible. Many DSP algorithms are able to compute useful results by analyzing a signals component frequencies

Fourier Square Wave Synthesis

Fourier DecompositionExample: decompose a signal into 9 cosine and sine waves

The decomposition models the signal’s phase and amplitude

The Fourier Transform Family• Fourier Series

A possibly infinite decomposed weighted sum of sinusoidal functions that models a periodic signal

• Fourier TransformA linear operation that maps an arbitrary function with into a spectrum of its frequency components

• Discrete Fourier Transform (DFT)A Fourier Transform applied to a discrete periodic series of measurements

• Discrete Time Fourier Transform (DTFT)A Fourier Transform applied to a an aperiodic discrete series of measurements

• Fast Fourier Transform: Fast way to calculate DFT

Orthonormal Sinusoids

OrthonormalAttenuating/Amplifying Sinusoids

Parameters•Frequency of attenuation or amplification•Speed of attenuation or amplification

Alternative Basis Functions

Choosing a Basis Function Set1. Any orthonormal function set works, but all choices are not equal

2. A good choice:a.Has desirable mathematical characteristics. Those with sharp edges fail

b.Represents frequencies, amplitudes, phases. Pure sines and cosines fail

Integral: Cos (πx/3) * cos (2 πx/3)cos (πx/3) and cos (2 πx/3)

Orthonormal Basis Sets (cont.)

• The winners: – Continuous version: e-j2πft = cos(2πft) – j sin(2πft) where t

is point in time, f is a frequency, polar notation models both phase angle and magnitude

– Discrete version: e-j2πfn/N = cos(2πfn/N) – j sin(2π fn/N) where n is a signal measurement, f is one of the N orthonormal functions

• Note: The angular frequency, ωm, measured in radians per second is defined as follows: ωm = 2πfF0 = 2πf/T0 measures how fast a particular basis function oscillates with respect to the fundamental frequency, F0, having period, T0. For every one time F0 rotates about the unit circle, 2πfF0

rotates f times faster.

The Fourier Transform

The Fourier Algorithm

• This Signal has– Two component frequencies: 1Khz, and 2Khz– The second signal has a 1350 phase difference from the first– We measure the signal at N (8) measurements per cycle (fs=8kHz)

• Fourier analysis– Multiply each measurement by the corresponding sin and cosine

values of e-2πfn/N and sum the results

– The resulting frequency array will have non zero elements at indices X[1] and X[2] as we would expect

– The phase angles are relative to the cosine of f0

• X[1] phase is -900 (-π/2) because sin(π/2) = cos(0) and cos(f0) leads by 900

• X[2] phase is +45 (π/4) because cos(2*f0) trails by that amount

• Negative frequencies X[6] and X[7] equal -X[2] and -X[1] respectively

Signal: x(t) = sin(2π*1000t) + 0.5 sin(2*2000πt+3π/4)

Understanding Digital Signal Processing, Third Edition, Richard Lyons(0-13-261480-4) © Pearson Education, 2011.

Understanding Digital Signal Processing, Third Edition, Richard Lyons(0-13-261480-4) © Pearson Education, 2011.

Understanding Digital Signal Processing, Third Edition, Richard Lyons(0-13-261480-4) © Pearson Education, 2011.

Second Example

• Comparisons with the first example– Same signal: x(t) = sin(2π*1000t) + 0.5 sin(2*2000πt+3π/4)

– Sampling starts three time units later (see above figure)

• Fourier results (We apply the same Fourier algorithm)

– The same frequency components have non zero magnitudes

– The cosine and sine sums are different, but the magnitudes are the same

– The phase of X[1] from -90 and +45; the third sample moves 1350 forward.

– The phase of X[2] goes from +45 to -45 because the third sample, because each sample is π /2 apart, and starting with the 3π/4, the third sample starts with 3 * (π /2) + 3π/4 = 9 π /4, π /2 ahead of the basis function

Understanding Digital Signal Processing, Third Edition, Richard Lyons(0-13-261480-4) © Pearson Education, 2011.

Example Resultsf First Example Second Example

M ϕ Real Imag M ϕ Real Imag

0 0 0 0 0 0 0 0 0

1 4 -90 0.000 -4.000

4 +45 2.8284

2.8284

2 2 +45 1.414 1.414 2 -45 1.4142

-1.4142

3 0 0 0 0 0 0 0 0

4 0 0 0 0 0 0 0 0

5 0 0 0 0 0 0 0 0

6 2 -45 1.414 -1.414

2 +45 1.4142

1.4142

7 4 +90 0.000 4.000 4 -45 2.8284

-2.8284

NotesThe results are scaled by N/2 (4 + 2 + 2 +4 = 12

The frequency domain is symmetrical about point N/2X[0] = X[4] in this example, but not generally

DFT Properties• The DFT is a linear transformation:

c1DFT(x1[n])+c2DFT(x2[n]) = DFT(c1x[n] + c2x2[n])

• The resulting magnitudes are amplified by N/2 (8/2 in this case). This means the 4 should be 1 and the 2 should be ½.

• A shift in the measurements correspond to a shift in the DFT phase calculations (as seen in the examples)

• The frequency domain is symmetric.

DFT Symmetry• The resulting frequency domain is symmetric

– X[n-k] = X[n+k] when k ≠N/2 and k ≠0,

– The upper frequencies represent negative frequencies (the extension of the signal if moving back in time)

– X[0] represents zero DC bias, a non oscillating constant

– X[N/2] represents the component fs/2

Negative Frequencies Positive Frequencies

DFT and its Inverse

• Definition: Continuous Fourier Transform and Inverse– Transform: X(w) = ∫-∞, ∞ x(t)e-iwtdt

– Inverse: x(t) = (1/2π)∫ -∞, ∞ X(w)eiwtdw

• Discrete Fourier Transform and Inverse– Transform: X[k] = ∑n=0,N-1 x[n] e-i2πkn/N

– Inverse: x[k] = (1/N)∑n=0,N-1 X[k] ei2 πkn/N

• Note: We can change the scale factor if we want the transform and its inverse to be completely symmetric

• Conclusion: We can use DFT to go back and forth between the time and frequency domains

DFT Correlation Algorithmpublic double[] DFT( double[] time){ int N = time.length;

double fourier[] = new double[2*N], real, imag;for (int k=0; k<N; k++){ for (int i=0; i<N; i++) { real = Math.cos(2*Math.PI*k*i/N); imag = -Math.sin(2*Math.PI*k*i/N); fourier[2*k] += time[i]*real; fourier[2*k+1]+= time[i]*imag;} }return fourier;

}

Note: even indices = real part, odd indices = imaginary part

Complexity: O(N2) because of the double loop of N eachExample: For 512 samples, loops 262144 timesEvaluation: Too slow, but FFT is O(N lg N)

Relation to Speech• For analysis we breakup

signal into overlapping windows

• Why? – Speech is quasi-periodic,

not periodic– Vocal musculature is

always changing– Within a small window of

time, we assume constancy, and process as if the signal were periodic

Typical Characteristics10-30 ms length1/3 to 1/2 overlap

DFT Issues• Implementation difficulties

– Impact: Algorithm runs too slow, especially for real time processing– Solution: FFT algorithm

• Spectral Leakage: Occurs when a signal has frequencies that fall between those of the DFT basis functions– Result: Magnitude leaks over the entire spectrum– Solutions: Windowing and DFT padding

• Inadequate frequency resolution– Impact: DFT magnitudes represent wide ranges of frequencies– Solution: DFT zero padding

• Signal rough edges: – Impact: Inaccuracies because windows don’t have smooth transistions– Solution: Windowing

DFT Zero Padding

• Each frequency domain bin represents a range of frequencies • Assume sampling rate fs = 16000, N = 512• Frequency domain 3, contains magnitudes for frequencies

between 16000/512*3 (93.75) to 16000/512*4 (125)• Each frequency bin models a range of 31.25 frequencies

• Padding the time domain frame with zeroes adds high frequency components, but does not negatively impact the result, or impact the lower frequencies of the spectrum

DFT Zero Padding

Understanding Digital Signal Processing, Third Edition, Richard Lyons(0-13-261480-4) © Pearson Education, 2011.

FFT zero padding can increase SNR

Definition: SNR = 10log( Psignal/Pnoise

Understanding Digital Signal Processing, Third Edition, Richard Lyons(0-13-261480-4) © Pearson Education, 2011.