Introduction to adaptive signal processing

EECS0712 Adaptive Signal Processing1

Introduction to Adaptive SignalProcessing

EECS0712 Adaptive Signal Processing1

Introduction to Adaptive SignalProcessing

Assoc. Prof. Dr. Peerapol YuvapoositanonDept. of Electronic Engineering

CESdSP ASP1-1EECS0712 Adaptive Signal Processing

http://embedsigproc.wordpress.com/eecs0712Assoc. Prof. Dr. P.Yuvapoositanon

Course Outline

• Introduction to Adaptive Signal Processing• Adaptive Algorithms Families:• Newton’s Method and Steepest Descent• Least Mean Squared (LMS)• Recursive Least Squares (RLS)• Kalman Filtering• Applications of Adaptive Signal Processing in

Communications and Blind Equalization

• Introduction to Adaptive Signal Processing• Adaptive Algorithms Families:• Newton’s Method and Steepest Descent• Least Mean Squared (LMS)• Recursive Least Squares (RLS)• Kalman Filtering• Applications of Adaptive Signal Processing in

Communications and Blind EqualizationCESdSP ASP1-2

EECS0712 Adaptive Signal Processinghttp://embedsigproc.wordpress.com/eecs0712

Assoc. Prof. Dr. P.Yuvapoositanon

Evaluation

• Assignment= 20 %• Midterm = 30 %• Final = 50 %

CESdSPEECS0712 Adaptive Signal Processing


ASP1-3

Textbooks



ASP1-4

http://embedsigproc.wordpress.com/eecs0712-adaptive-signal-processing/



ASP1-5

QR code



ASP1-6

Adaptive Signal Processing

• Definition: Adaptive signal processing is thedesign of adaptive systems for signal-processing applications.

[http://encyclopedia2.thefreedictionary.com/adaptive+signal+processing]

• Definition: Adaptive signal processing is thedesign of adaptive systems for signal-processing applications.

[http://encyclopedia2.thefreedictionary.com/adaptive+signal+processing]



ASP1-7

System Identification

• Let’s consider a system called “plant”• We need to know its characteristics, i.e., The

impulse response of the system



Plant Comparison



Error of Plant Outputs



Error of Estimation

• Error of estimation is represented by thesignal energy of error

2 2

2 2

( )

2

e d y

d dy y



2 2

2 2

( )

2

e d y

d dy y

Adaptive System

• We can do it adaptively



• Adjust the weight for minimum error e

One-weight



2 2

2 2

2 20 0 0 0

( )

2

( ) 2( )( ) ( )I I

e d y

d dy y

w x w x w x w x

CESdSP

2 2

2 2

2 20 0 0 0

( )

2

( ) 2( )( ) ( )I I

e d y

d dy y

w x w x w x w x

ASP1-14EECS0712 Adaptive Signal Processing


Error Curve

• Parabola equation



Partial diff. and set to zero

• Partial differentiation

• Set to zero

• Result:

22 2

0 0 0 00 0

2 20 0

( ) 2( )( ) ( )

2 2

I II I

I

ew x w x w x w x

w w

w x w x

• Partial differentiation

• Set to zero

• Result:

CESdSP

22 2

0 0 0 00 0

2 20 0

( ) 2( )( ) ( )

2 2

I II I

I

ew x w x w x w x

w w

w x w x

2 20 00 2 2 Iw x w x

0 0Iw w



Multiple Weight Plants

• We calculate the weight adaptively• Questions:

– What is the type of signal “x” to be used, e.g.Sine, Cosine or Random signals ?

– If there is more than one weight w0 , i.e., w0….wN-

1, how do we calculate the solution?

• We calculate the weight adaptively• Questions:

– What is the type of signal “x” to be used, e.g.Sine, Cosine or Random signals ?

– If there is more than one weight w0 , i.e., w0….wN-

1, how do we calculate the solution?



Plants with Multiple Weight

• If we have multiple weights

CESdSP

10 1w w z w



• In the case of two-weight

Two-weight



Input

• From

• We construct the x as vector with firstelement is the most recent

(3), (2), (1), (0), ( 1), ( 2),...x x x x x x

• From

• We construct the x as vector with firstelement is the most recent

CESdSP

[ (3) (2) (1) (0)...]Tx x x xx



Plants with Multiple Weight(aka “Transversal Filter”)

• If we have multiple weights( )x n ( 1)x n

CESdSP

0 ( )w x n0 ( 1)w x n

0 0( ) ( ) ( 1)y n w x n w x n



Regression input signal vector

• If the current time is n, we have “Regressioninput signal vector”

[ ( ) ( 1) ( 2) ( 3)...]Tx n x n x n x n x

CESdSP

[ ( ) ( 1) ( 2) ( 3)...]Tx n x n x n x n x



00 1

1[ ]T

ww ww

w

CESdSP

00 1

1[ ]T

ww ww

w

00 1

1

ˆ [ ]

II I T

I

ww w

w

w



Convolution

• Output of plant is a convolution

• Ex For N=2

1

1

( ) ( )N

kk

y n w x n k

• Output of plant is a convolution

• Ex For N=2

CESdSP

1

1

( ) ( )N

kk

y n w x n k

0 0( ) ( 0) ( 1)y n w x n w x n



0 1

0 1

0 1

0 1

0 1

(3) (3) (2)

(2) (2) (1)

(1) (1) (0)

(0) (0) ( 1)

( 1) ( 1) ( 2)

y w x w x

y w x w x

y w x w x

y w x w x

y w x w x

CESdSP

0 1

0 1

0 1

0 1

0 1

(3) (3) (2)

(2) (2) (1)

(1) (1) (0)

(0) (0) ( 1)

( 1) ( 1) ( 2)

y w x w x

y w x w x

y w x w x

y w x w x

y w x w x



• We can use a vector-matrix multiplication• For example, for n=3 we construct y(3) as

• For example, for n=1 we construct y(1) as

0 1 0 1

(3)(3) (3) (2) [ ] (3)

(2)T

xy w x w x w w

x

w x

• We can use a vector-matrix multiplication• For example, for n=3 we construct y(3) as

• For example, for n=1 we construct y(1) as

CESdSP

0 1 0 1

(3)(3) (3) (2) [ ] (3)

(2)T

xy w x w x w w

x

w x

0 1 0 1

(1)(1) (1) (0) [ ] (1)

(0)T

xy w x w x w w

x

w x



0 1 0 1

0 1 0 1

0 1 0 1

0 1 0 1

(3)(3) (3) (2) [ ] (3)

(2)

(2)(2) (2) (1) [ ] (2)

(1)

(1)(1) (1) (0) [ ] (1)

(0)

(2)(0) (0) ( 1) [ ] (0

(1)

T

T

T

T

xy w x w x w w

x

xy w x w x w w

x

xy w x w x w w

x

xy w x w x w w

x

w x

w x

w x

w x )

CESdSP

0 1 0 1

0 1 0 1

0 1 0 1

0 1 0 1

(3)(3) (3) (2) [ ] (3)

(2)

(2)(2) (2) (1) [ ] (2)

(1)

(1)(1) (1) (0) [ ] (1)

(0)

(2)(0) (0) ( 1) [ ] (0

(1)

T

T

T

T

xy w x w x w w

x

xy w x w x w w

x

xy w x w x w w

x

xy w x w x w w

x

w x

w x

w x

w x )



• The error squared is

• Let us stop there to consider Random signaltheory first.

2 2

2 2

2 2

( )

2

ˆ ˆ( ) 2( )( ) ( )T T T T

e d y

d dy y

w x w x w x w x

• The error squared is

• Let us stop there to consider Random signaltheory first.

CESdSP

2 2

2 2

2 2

( )

2

ˆ ˆ( ) 2( )( ) ( )T T T T

e d y

d dy y

w x w x w x w x



Review of Random Signals



Wireless Transmissions

• Ideal signal transmission

11 00 11 00 11 0011 11 11 000011



ASP2-30

11 00 11 00 11 0011 11 11 000011

Information

Information is Random

Random variable



Random Variable

• Random variable is a function• For a single time Coin Tossing

1,( )

-1,

x HX x

x T

• Random variable is a function• For a single time Coin Tossing

CESdSP

1,( )

-1,

x HX x

x T



Our signal x(n) is a RandomVariable

• For a series of Coin Tossing

1,( )

-1,

i

ii

x HX x

x T

• For a series of Coin Tossing

CESdSP

1,( )

-1,

i

ii

x HX x

x T

0 1 2 3 4{ , , , , ,....}x x x x x x



Coin tossing and Random Variable

• If random

• We have random variable X0 1 2 3 4

{ , , , , }

{ , , , , }

x H H T H T

x x x x x

CESdSP

• If random

• We have random variable X

0 1 2 3 4( ) { ( ), ( ), ( ), ( ), ( )}

{ ( ), ( ), ( ), ( ), ( )}

{1,1, 1,1, 1}

iX x X x X x X x X x X x

X H X H X T X H X T



Random Digital Signal

• If the random variable is a function of time, itis called a stochastic process



Probability Mass Function

• We need also to define the probability of eachrandom variable

( ) { ( ), ( ), ( ), ( ), ( )}

{1,1, 1,1, 1}

X x X H X H X T X H X T

CESdSP

( ) { ( ), ( ), ( ), ( ), ( )}

{1,1, 1,1, 1}

X x X H X H X T X H X T



Probability Mass Function

• PMF is for Discrete distribution function



Time and Emsemble



Probability of X(2)



Probability Density Function

• PDF is for Continuous Distribution Function



Probability Density Function

• PDF values can be > 1 as long as its area undercurve is 1

2

CESdSP

1/2

2

1

1



Cumulative Distribution Function

CESdSP

( ( )) Pr[ ( )]P x n X x n x



( )

( ( )) ( )x n

P x n p z dz

x x

CESdSP

( )

( ( )) ( )x n

P x n p z dz

x x



Expectation Operator

{}E

CESdSP

{}E



Expected Value

• Expected value is known as the “Mean”

{ } ( )X XE x xp x dx

CESdSP

{ } ( )X XE x xp x dx



Example of Expected Value(Discrete)

• We toss a die N times and get a set ofoutcomes

• Suppose we roll a die with N=6, we might get

{ ( )} { (1), (2), (3),..., ( )}X i X X X X N

• We toss a die N times and get a set ofoutcomes

• Suppose we roll a die with N=6, we might get

CESdSP

{ ( )} { (1), (2), (3),..., ( )}X i X X X X N

{ ( )} {2,3,6,3,1,1}X i



Example of Expected Value(Discrete)

• But, empirically we have Empirical (MonteCarlo) estimate as Expected Value

6

1

{ } ( )Pr( ( ))

1 1 1 11 2 3 6

3 6 3 62.67

Xi

E x X i X X i

CESdSP

6

1

{ } ( )Pr( ( ))

1 1 1 11 2 3 6

3 6 3 62.67

Xi

E x X i X X i



Theoretical Expected Value

• But in theory, for a die

6

1

{ } ( )Pr( ( ))

1 1 1 1 1 11 2 3 4 5 6

6 6 6 6 6 63.5

Xi

E X X i X X i

1Pr( ( ))

6X X i

CESdSP

6

1

{ } ( )Pr( ( ))

1 1 1 1 1 11 2 3 4 5 6

6 6 6 6 6 63.5

Xi

E X X i X X i



Ensemble Average

i ensembles

1 1 2 2Ensemble Average of (1) (1)Pr[ (1)] (1)Pr[ (1)]

(1)Pr[ (1)]N N

x x x x x

x x

1 ensemble



ASP1-50

i ensembles

Ensemble Average

{ ( )}E x n



ASP1-51

{ ( )} ( ) ( ( )) ( )E x n x n p x n dx n

x

{ ( )}E x n

• I) Linearity



ASP1-52

{ ( ) ( )} { ( )} { ( )}E ax n by n aE x n bE y n

• II)

{ ( ) ( )} { ( )} { ( )}E x n y n E x n E y n



ASP1-53

{ ( ) ( )} { ( )} { ( )}E x n y n E x n E y n

• III)

{ ( )} ( ( )) ( ( )) ( )E y n g x n p x n dx n

x



ASP1-54

{ ( )} ( ( )) ( ( )) ( )E y n g x n p x n dx n

x

Autocorrelation

1 1( , ) { ( ) ( )}r n m E x n x mxx



ASP1-55

1 11 1 1 1 1 1( , ) ( ) ( ) ( ( ), ( )) ( ) ( )r n m x n x m p x n x m dx n x m

xx x x

1 1(1, 4) { (1) (4)}r E x xxx



ASP1-56

Autocorrelation

• n=m

2( , ) ( , ) { ( )}r n m r n n E x n xx xx



ASP1-57

2( , ) ( , ) { ( )}r n m r n n E x n xx xx

Autocorrelation Matrix

(0,0) (0,1) (0, 1)

(1,0) (1,1) (1, 1)

( 1,0) ( 1,1) ( 1, 1)

r r r N

r r r N

r N r N r N N

xx xx xx

xx xx xxxx

xx xx xx

R



ASP1-58

(0,0) (0,1) (0, 1)

(1,0) (1,1) (1, 1)

( 1,0) ( 1,1) ( 1, 1)

r r r N

r r r N

r N r N r N N

xx xx xx

xx xx xxxx

xx xx xx

R

Covariance

( , ) {[ ( ) ( )][ ( ) ( )]}c n m E x n n x m m xx



ASP1-59

( , ) {[ ( ) ( )][ ( ) ( )]}c n m E x n n x m m xx

Stationarity (I)

• I)

{ ( )} { ( )}E x n E x m n1



ASP1-60

n2

Stationarity (II)

• II)

( , ) { ( ) ( )}r n n m E x n x n m xx

1 1 1 1( , ) { ( ) ( )}r n n m E x n x n m xx



ASP1-61

1 1 1 1( , ) { ( ) ( )}r n n m E x n x n m xx

Expected Value of Error Energy

• Let’s take the expected value of error energy

2 2 2ˆ ˆ{ } {( ) 2( )( ) ( ) }

ˆ ˆ ˆ{( )( )} 2 {( )( )} {( )( )}

ˆ ˆ ˆ{ } 2 {( )( )} { }

ˆ ˆ ˆ2 {( )( )}

T T T T

T T T T T T

T T T T T T

T T T T

E e E

E E E

E E E

E

w x w x w x w x

w x x w x w w x w x x w

w xx w x w x w w xx w

w Rw x w x w w Rw



ASP1-62

2 2 2ˆ ˆ{ } {( ) 2( )( ) ( ) }

ˆ ˆ ˆ{( )( )} 2 {( )( )} {( )( )}

ˆ ˆ ˆ{ } 2 {( )( )} { }

ˆ ˆ ˆ2 {( )( )}

T T T T

T T T T T T

T T T T T T

T T T T

E e E

E E E

E E E

E

w x w x w x w x

w x x w x w w x w x x w

w xx w x w x w w xx w

w Rw x w x w w Rw

Vector-Matrix Differentiation

Î)ˆ

ˆ ˆ ÎI) 2ˆ

T

T T

w x xw

w xx w Rww

CESdSP

Î)ˆ

ˆ ˆ ÎI) 2ˆ

T

T T

w x xw

w xx w Rww



Partial diff. and set to zero

• Differentiation

• Result:

ˆ0 2 {( ) } 2ˆ

ˆ2 { } 2

ˆ2 2

TE

E d

w x x Rww

x Rw

r Rw

• Differentiation

• Result:

CESdSP

ˆ0 2 {( ) } 2ˆ

ˆ2 { } 2

ˆ2 2

TE

E d

w x x Rww

x Rw

r Rw

1ˆ w R rASP1-64

EECS0712 Adaptive Signal Processinghttp://embedsigproc.wordpress.com/eecs0712

Assoc. Prof. Dr. P.Yuvapoositanon

2-D Error surface

CESdSP

1ˆ w R r



Four Basic Classes of AdaptiveSignal Processing

• I) Identification• II) Inverse Modelling• III) Prediction• IV) Interference Cancelling



ASP1-66

• I) Identification• II) Inverse Modelling• III) Prediction• IV) Interference Cancelling

The Four Classes of AdaptiveFiltering



ASP1-67

System Identification



ASP2-68

Inverse Modelling



ASP2-69

Prediction



ASP2-70

Interference Canceller



ASP2-71

What are we looking for inAdaptive Systems?

• Rate of Convergence• Misadjustment• Tracking• Robustness• Computational Complexity• Numerical Properties

• Rate of Convergence• Misadjustment• Tracking• Robustness• Computational Complexity• Numerical Properties



ASP1-72

Introduction to adaptive signal processing

Education

Transcript of Introduction to adaptive signal processing