5th IEE Adaptive Signal Processing Professional Development Course

March 2006, Birmingham, UK

Bob StewartDept. of Electronic and Electrical Engineering

University of StrathclydeGlasgow G1 1XW

Scotland, UK

r.stewart@eee.strath.ac.uk

Adaptive IIR Filtering

Introduction to Adaptive IIR Filters .1

• In this section we will introduce adaptive IIR filters.

• Intuitive comment is made as to why adaptive IIR filters are not assimple and straightforward as FIR filters.

• The adaptive IIR LMS algorithms will be defined.

• Adaptive RLS algorithms will be presented.

• Applications ranging from speech coding to active noise control will bepresented.

• Classic decision feedback architectures (DFE) will be presented andshown to be forms of non-linear adaptive IIR filter.

• SystemView simulations of adaptive IIR algorithms will bedemonstrated.

Notes:For linear adaptive filters, two types can be identified: FIR and IIR.

Within these two classes, we can distinguish between RLS and LMS based algorithms.

Adaptive Algorithms

FIR IIR(non-recursive) (recursive)

Fein

tuch

’s LM

S

Equation ErrorOutput Error

Sta

ndar

dN

orm

alis

edFi

ltere

d-X

Filte

red-

Erro

rS

ignE

rror/D

ata

RPE

New

ton

Dec

isio

nFe

edba

ck

LMS

New

ton

Nor

mal

ised

Leas

tMea

n4t

h

QR

-IIR

RLS

PLR

New

ton

Blo

ckN

on-L

inea

r

Sta

ndar

dQ

RK

alm

an

LMSRLS

QR

-IIR

RLS

PLR

LMSRLS LMSRLS

Adaptive Filtering 2

• For a number of applications, adaptive IIR filters may have acompuatational and modelling advantage. Consider the inverse sys id:

• Using adaptive FIR filter, the inverse has many weights:

• Using adaptive IIR filter, the inverse may have only two weights:

s(k) 1 0.5

y(k)Σ

+

-e(k)Adaptive

Filter G(z)

d(k)

x(k)

Adaptive AlgorithmH z( ) 1 0.5z 1–+=

G z( ) H 1– z( ) 1 0.5z 1–– 0.25z 2– 0.125z 3–– 0.0625z 4– …+ + += =

G z( ) H 1– z( ) 11 0.5z 1–+-------------------------- = =

Notes:When using an FIR filter to find theinverse of the above system, around20-30 weights would be required toinvert the filter.

This is clearly a minimum phase systemas the zero of the transfer function(when ) is inside the unitcircle at . Hence the inverse ofthe system could have a single poleinside the unit circle if inverse modellingwas performed using an IIR filter.

Using an adaptive FIR filter, the inverse can be calcuated as the z-transform of the unknown system impulseresponse. About 20-30 weights should be sufficient to invert this system.

However, if the adaptive filter was in fact an IIR filter, then only two weights would be required to produced theinverse!

H z( ) 1 0.5z 1–+=

sample, k

h k( )

0 1 2 3 4

1

0.5

Real

Imag

0.5 1-0.5-1

0.5

1

-0.5

1

z-plane

1 0.5z 1–+ 0=z 0.5–=

1 0.5z 1–+ 11 0.5z 1–+

0.5z 1––

1 0.5z 1– 0.25z 2– 0.125z 3– …+–+–

0.5z 1–– 0.25z 2––0.25z 2–

0.25z 2– 0.125z 3–+

0.125z 3––

G z( ) H 1– z( ) 11 0.5z 1–+--------------------------- 1 0.5z 1– 0.25z 2– 0.125z 3– …+–+–= = =

sample, k

g k( )

0

1

0.5

-0.51 2

34

PolynomialDivision

....and so on

Adaptive IIR Filter - Output Error 3

• General Closed Loop Adaptive IIR Signal Processor:

“The aim is to adapt the digital filter such that the input signal x(k) isfiltered to produce y(k) which when subtracted from desired signal d(k),will minimise the power of the error signal e(k).” .

input y(k)

Σ

+

-

desired

e(k)

d(k)

x(k)signal

signal

error signal

Output signal

Σ

Adaptive Algorithm

+

+

Adaptive FIR Filter

Adaptive FIR Filter

Adaptive IIR Filter

z-1

A(z)

B(z)

Notes:One key aspect of the adaptive FIR filter was that the minimum mean squared error (MMSE) solution is unique,i.e. the performance surface had exactly one minimum which was the best or optimal solution in an MMSEsense.

For adaptive IIR filters the performance surface no longer has just one minimum, and there may exist more thanone solution (i.e. overdetermined in some sense) giving multiple “optimal” solutions).

Note that in the above figure the block denoted adaptive FIR filter in both feedforward and feedback sections isessentially an FIR filter block with adaptive weights:

As the filter is connected in a feedback loop, this forms the recursive filter weights.

Hence in the feedback section we explicitly show the delay that is required in the feedback loop (otherwise thereis a delayless loop!). The signal flow graph is presented in this was as later we will be considering another typeof adaptive IIR, namely the equation error IIR. Presenting in this style makes things somewhat easier.

We specifically refer to this architecture as the “output” error formulation. Later we will also view the “equationerror” IIR (which is in fact not an IIR at all!)

Adaptive FIR Filterx(k) y(k)

x(k)

y(k)

z-1 z-1 z-1

B z( )

Adaptive IIR Filter Signal Flow Graph 4

• Example with 4 feedforward, and 3 feedback weights:

a0 a1 a2

b3

a3

b1b2

y(k)Σ

+

-e(k)

d(k)

Output signal

x(k)

Adaptive Algorithm

Adaptive IIR Filterz-1 z-1 z-1

z-1 z-1 z-1

Y z( )X z( )------------ A z( )

B z( )------------

a0 k( ) a1 k( )z 1– a2 k( )z 2– a3 k( )z 3–+ + +1 b1 k( )z 1– b2 k( )z 2– b3 k( )z 3–+ + +

------------------------------------------------------------------------------------------------------= =

Notes:The general input output equation for an IIR filter with feedforward weights and feedback weights is:

In vector notation

,

where the weight and data vectors are respectively:

and

N M 1–

y k( ) anx k n–( )

n 0=

N 1–

∑ bmy k m–( )

m 1=

M 1–

∑+=

y k( ) aTxk bTyk 1–+ wTuk= =

wT aT bT,[ ] a0 a1 a2 … aN 1– b1 b2 … bM 1–, , , , , , , ,[ ]= =

ukT xk

T yk 1–T,[ ]=

x k( ) x k 1–( ) x k 2–( ) … x k N– 1+( ) y k 1–( ) y k 2–( )… y k M– 1+( ), , , , , , ,[ ]=

System Identification of Recursive System 5

• The optimal solution would appear to be

• Of course there are many “optimal” solutions obtainable simply bymultiplying numerator and denominator by pole zero pairs:

G z( ) A z( )B z( )------------ N z( )

D z( )------------= =

Σ

+

-Σ

Adaptive Algorithm

+

+

Adaptive IIR Filter

Unknown System

N z( )D z( )------------

WhiteNoise A z( )

B z( )

G z( )

z-1

G z( ) A z( )B z( )------------ N z( ) 1 qz 1–+( )

D z( ) 1 qz 1–+( )---------------------------------------= =

Notes:The point being made here is simple. Adaptive IIR filters are NOT as straightforward as adaptive FIR filters! Wesimply demonstrate that whereas the optimal (best) adaptive FIR solution is unique, this is not the case foradaptive IIR as demonstrated by showing that there are multiple solutions.

Similarly in the notes of Slide 2 we noted two different solutions, one based on feedforward weights only, andone based on feedback weights only.

For example, consider again the inverse system identification in Slide 2 where the inverse of was being calculated. If we used an IIR with say 20 feedforward weights and 20 feedback weights, then an exactsolution that would give an error of zero would be:

, i.e.

However another solution would be the result of the polynomial division in the notes below Slide 2:

, i.e.

with no significant coefficients on the denominator, i.e. all feedback weights are zero.

There are of course many other solutions formed if there are pairs of cancelling poles and zeroes, e.g.

is another “valid” solution.

In the slides that follow we will again note that there is not a unique solution, and that there are also otherproblems whereby

H z( ) 1 0.5z 1–+=

G z( ) 1 0z 1– 0z 2– … 0z 19–+ + + +1 0.5z 1– 0z 2– … 0z 19–+ + + +--------------------------------------------------------------------------------- 1

1 0.5z 1–+---------------------------= = G z( )H z( ) 1=

G z( ) 1 0.5z 1–– 0.25z 2– … 0.191 -5×10– z 19–+ +1 0z 1– 0z 2– … 0z 19–+ + + +

------------------------------------------------------------------------------------------------------------------= 2 19– 0.191 -5×10 ...= G z( )H z( ) 1≈

G z( ) 11 0.5z 1–+---------------------------

1 0.5z 1–+( )1 0.5z 1–+( )

-------------------------------- 1 0.5z 1–+( )1 z 1– 0.25z 2–+ +( )

---------------------------------------------------= =

Minimising the MSE 6

• Assuming signals and are ergodic and wide sensestationary (WSS), then from Slide 3:

and

and hence

x k( ) d k( )

MSE E e2 k( )[ ] 1M----- e2 k( )

k 0=

M 1–

∑= = , for large M

e2 k( ) d k( ) y k( )–( )2 d k( ) wTuk–( )2= =

d2 k( ) wT u k( )uT k( )[ ]w 2d k( )wTu k( )–+=

E e2 k( )[ ] E d2 k( )[ ] wTE u k( )u k( )T[ ]w 2wTE d k( )u k( )[ ]–+= E d2 k( )[ ] wTRuuw 2wTpdu–+=

Notes:Minimising the mean square error requires that the assumptions of and being ergodic and wide sensestationary. This is the same assumption as made for the adaptive FIR LMS filters.

We will note that unlike for the FIR filter, no one step linear Wiener-Hopf type solution exists. In fact the solutionfor this IIR filter, or ARMA (autoregressive moving average) filter is non-linear and virtually impossible to solveby analysis for other than trivial systems.

The presentation of the IIR filter is done using typical adaptive filter signal flow graphs and signal labelling.There are other forms of general recursive system identification known as Box-Jenkins and ARMA(autoregressive moving average). In many senses both Box-Jenkins and ARMA modelling are part of the moregeneral subset of adaptive IIR filtering, however there application is restricted to system identification and oftenmakes more detailed reference to any measurement noise that may be present on the signal. For adaptiveIIR filtering the problem may not be one of simply system identification. It could be noise cancellation, echocancellation, inverse system identification and so on. Hence we refrain from specifically using the restrictivenames of Box-Jenkins and ARMA in favour of the general name, “adaptive IIR filtering”.

Note that if the filter simplifies to an FIR filter (i.e. it has no poles), then the mean squared error (MSE) reducesto:

where is the autocorrelation matrix, and the cross correlation vector. Thisequation is quadratic in and can be solved relatively easily to find the minimium mean squared error.

In the equation on the above slide note that is in fact a function of past values of since all past outputswere created using .

x k( ) d k( )

d k( )

E e2 k( )[ ] E d2 k( )[ ] wTE x k( )xT k( )[ ]w 2wTE d k( )x k( )[ ]–+=

E d2 k( )[ ] wTRw 2wTp–+=

R E xkxkT[ ]= p E xkd k( )[ ]=

w

Ruu ww

The IIR “Wiener-Hopf”? 7

• Note that the square matrix of dimension is given by:

and the vector is:

• Recall for the adaptive FIR filter that the performance surface wasquadratic on and a “hyperparaboloid” with one minimum. Howeverfor this IIR performance surface note that and are functions of

.

Ruu N M 1–+

RuuE x k( )xT k( )[ ] E x k( )yT k 1–( )[ ]

E y k 1–( )xT k( )[ ] E y k 1–( )yT k 1–( )[ ]

Rxx Rxy

RTxy Ryy

= =

N M 1–+ pdu

pduE d k( )x k( )[ ]

E d k( )y k 1–( )[ ]

pdx

pdy

= =

wRyy Rxy

w

Notes:Note that is simply the correlation matrix normally used in the Wiener Hopf solution.

Recall that the performance surface is a plot of against filter weights.

For an FIR filter (no feedback), the classic Wiener-Hopf solution is realised by noting that the equation:

is quadratic in the vector w. Hence there is only one minimum value denoted MMSE (minimum mean squareerror) and which occurs at, wopt.

The MMSE is found from setting the (partial derivative) gradient vector, , to zero:

The solution to this equation is linear.

If the FIR filter has two weights the performance surface isa paraboloid in 3 dimensions::

If the filter has more than three weights then we cannot draw the performacesurface in three dimensions, however, mathematically there is only oneminimum point which occurs when the gradient vector (with respect to w) iszero. A quadratic performance surface with more than three dimensions isoften called a hyperparaboloid.

Rxx

E e2 k( )[ ]

E e2 k( )[ ] E d2 k( )[ ] wTRxxw 2wTp–+=

∇

∇w∂∂ E e2 k( )[ ] 2Rxxw 2p– 0= = =

wopt⇒ Rxx1– p=

MMSE

w0

w1

w0(opt)

w1(opt)

MSE, ζwT w0 w1,[ ]=

Multimodal Performance Surface 8

• Therefore unlike for an adaptive FIR filter, for the IIR filter theperformance surface equation:

is NOT quadratic and a straighforward gradient based solution doesnot exist.

• The performance surface is now multimodal (i.e. localminima) and in general the gradient function:

is non-linear and non-trivial to solve since is a function of (c.f.for adaptive FIR filter).

E e2 k( )[ ] E d2 k( )[ ] wTRuuw 2wTpdu–+=

E e2 k( )[ ]

∇ k( )w∂∂ E e2 k( )[ ]=

Ruu w

Notes:For an adaptive FIR filter, the minimum mean square error is found simply by locating the “bottom” of thehyperparaboloid. Mathematically this is done by finding the zero gradient vector using vector calculustechniques.

The adaptive IIR filter multimodal performance surface, basically means that there are now local minima aswell as the desired single global minimum:

Therefore although we can still look for a zero gradient point on the performance surface of the adaptive IIRfilter, it cannot be guaranteed that the point found is the global minimum, and it may in fact be a local minimum.Hence the term “stuck in local minima” is often used.

MMSE

MSE

wwopt

Local Minima

GlobalMinima

Gradient IIR Filtering Techniques 9

• Following the strategy for deriving the FIR LMS algorithm, an adaptivegradient based IIR LMS algorithm can be derived.

• Note however that the performance surface has local minima, andtherefore the solution achieved via this gradient descent technique maybe a local minimum (sub-optimal) rather than the global minimum.

• However the general IIR LMS solution has proved useful, and the localminima problem is less “severe” than first anticipated.

• Therefore the adaptive IIR LMS is NOT guaranteed to converge to theglobal minimum

Notes:Recall that the LMS is derived by estimating the mean squared error performance surface gradient,

by the instantaneous squared error surface gradient:

This simplification makes the problem mathematically tractable.

∇kE e2 k( )[ ]∂

wk∂--------------------------=

∇ˆ

ke2 k( )∂

wk∂-----------------=

Full Gradient IIR LMS 10

• From the FIR LMS derivation the weight vector update is:

and

• Therefore:

where & are used for notational convenience.

wk 1+ wk µ ∇ˆ

k–( )+=

∇ˆ

ke2 k( )∂w k( )∂

----------------- 2– e k( ) y k( )∂w k( )∂

---------------= =

y k( )∂wk∂

-------------- y k( )∂a0 k( )∂

----------------- y k( )∂a1 k( )∂

----------------- … y k( )∂aN 1– k( )∂

------------------------- y k( )∂b1 k( )∂

----------------- …, , , , , y k( )∂bM 1– k( )∂

-------------------------T

=

α0 k( ) α1 k( ) … αN 1– k( ) β1 k( ) … βM 1– k( ), , , , , ,[ ]=

αn k( ) βm k( )

Notes:

Previously for the FIR LMS the instantaneous gradient vector was simply . and

However for the IIR LMS, the gradient vector will more complex due to the fact that past outputs must bedifferentiated , all of which are functions of the weight vector.

The step size is given by the parameter .

2– µe k( )x k( ) y k( )∂w k( )∂

--------------- x k( )=

µ

Full Gradient IIR LMS....cont’d11

• It is (reasonably) straightforward to show that:

• The resulting algorithm is termed the full gradient IIR LMS, and isperhaps best represented via a signal flow graph......

αn k( ) y k( )∂an∂ k( )

----------------- x k n–( ) bj k( ) y k j–( )∂an∂ k j–( )

------------------------j 1=

M 1–

∑+≈=

βm k( ) y k( )∂bm∂ k( )

------------------ y k m–( ) bj k( ) y k j–( )∂bm∂ k j–( )

-------------------------j 1=

M 1–

∑+≈=

Notes:In derving the above equations we have implicitly made the assumption

to simplify the mathematics and achieve a definable adaptive filter.

We can write both and values in a recursive form:

leading to the signal flow graphs:

is z-domain representation of the current values of the feedbackweights.

Note that changes every update.

y k j–( )∂an∂ k j–( )

------------------------- y k j–( )∂an∂ k( )

---------------------=

α β

αn k( ) x k n–( ) bj k( )αn k j–( )

j 1=

M 1–

∑+=

βm k( ) y k m–( ) bj k( )βm k j–( )

j 1=

M 1–

∑+=

Σ+

+

Bk z( )

αn k( )x k n–( )Σ

++

Bk z( )

βm k( )y k m–( )

Bk z( ) b1z 1– b2z 2– … bM 1– z M– 1++ + +=

Bk z( )

Full Gradient IIR LMS SFG 12

Σ+

+y(k)

Σ+

-

d(k)

e(k)

LMSA LMSB

e(k)α0 k( ) αN 1– k( )

. . .

. . . β1 k( ) βM k( )

. . .

. . .

z-1

z-1

αN 1– k( )

z-1

z-1

β1 k( )

βM 1– k( )

β2 k( )

y(k-1)

y(k-2)

y(k-M+1)

y(k-M)

x(k)

x(k-1)

x(k-N+2)

x(k-N+1)

x(k)

Σ+

+

11 Bk z( )–------------------------

11 Bk z( )–------------------------

11 Bk z( )–------------------------

α0 k( )

α1 k( )

11 Bk z( )–------------------------

11 Bk z( )–------------------------

11 Bk z( )–------------------------

Bk z( )

Ak z( )

z-1

z-1

Notes:SFG - Signal Flow Graph

On each time iteration, the current feedback weights are used with the current and past values of inputsignal and with past outputs to produce the and in order to update the LMS equation.

It should be clear that:

Bk z( )x k( ) y k( ) αn k( ) βm k( )

Σ+

+

Bk z( )1

1 Bk z( )–------------------------

Simplified Gradient IIR LMS 13

• If it is assumed that the IIR filter weights are “slowly adapting” thennoting that is calculated using data sample and ....

....and similarly that is calculated using data sample and

the simplified gradient IIR LMS can be realised such that we assume:

• Hence can be “reasonably” approximately calculated using datasample and ......

.....and similarly can be calculated using data sample and

αn k( ) x k n–( ) Bk z( )

βm k( ) y k m–( )Bk z( )

Bk z( ) Bk 1– z( ) … Bk M– 1+ z( ) … Bk N– 1+ z( )≈ ≈ ≈ ≈ ≈

αn k( )x k n–( ) Bk n– z( )

βm k( ) y k m–( )Bk m– z( )

Notes:The slowly varying weights assumption can be somewhat forced by the algorithm designer by choosing a verysmall step size for the algorithm!

Note that although the weight update here uses simple LMS, more involved algorithms such as an RPE(recursive predictor error) could be used (see later notes). The RPE uses an estimate of the correlation matrixto accelerate the convergence. See also Shynk’s paper for more details.

Another form of IIR algorithm exists called the SHARF (simplified hyperstable recursive filter) which in a sensesmooths the error signal with an adaptive filter in order to try and stabilise the IIR LMS. See Widrow’s text fordetails.

See also Shynk’s paper for information on the filtered-error IIR algorithm.

Simplified IIR LMS SFG 14

Σ+

+y(k)

Σ+

-

d(k)

e(k)

LMSA LMSB

e(k)α0 k( ) αN 1– k( )

. . .

. . . β1 k( ) βM k( )

. . .

. . .

α0 k( )

αN 1– k( )

β2 k( )

z-1

y(k-1)x(k)

x(k)

z-1

z-1

z-1

α1 k( )

β0 k( )

βM 1– k( )

z-1

z-1

z-1

β1 k( )

11 Bk z( )–------------------------

Bk z( )

Ak z( )

11 Bk z( )–------------------------

z-1

Notes:The simplified gradient algorithm reduces the level of computation considerably.

For the full gradient technique the computation required at each new sample is of the order multiply-accumulates (MACs) per iteration whereas for the simplified gradient the complexity reduces to around

MACs per iteration.

NM M2+

2 N M+( ) 2M+

Feintuch’s IIR LMS 15

• Feintuch made a further simplification, where in Slide 11, it wasassumed all derivatives of past outputs with respect to current weightsare zero:

• The justification for Feintuch is simplification, and the “proof” is analgorithm that does in fact converge in many situations.

αn k( ) y k( )∂an∂ k( )

----------------- x k n–( ) bj k( ) y k j–( )∂an∂ k j–( )

------------------------

j 1=

M 1–

∑+≈=

x k n–( )=

0

βm k( ) y k( )∂bm∂ k( )

------------------ y k m–( ) bj k( ) y k j–( )∂an∂ k j–( )

------------------------

j 1=

M 1–

∑+≈=

y k m–( )=

0

Notes:When it first appeared in 1976 Feintuch’s algorithm caused some debate in the signal processing literature.Within the control literature there is a similar technique refered to as Landau’s method. A number of commentswere to the effect that this algorithm could not and should not work, whereas some others were very positive bynoting that not only did it converge in certain circumstances, but it was also more inherently stable than the fullgradient and simplified gradient due to a tendency to adapt poles away from the unit circle towards the origin ofthe z-plane.

This stabilising property is a result of the PLR (pseudo linear regression) assumption (i.e. all past outputderivatives with respect to current weights are zero) and is well known in both signal processing and controlliterature. Therefore not only is Feintuch simpler than the simplified gradient algorithm, it is also more likely tobe stable, with the adapted poles less likely to adapt out of the unit circle.

Feintuch’s algorithm is still widely used and anaylsed within the signal processing literature.

The computational complexity is around MACs per iteration.2 N M+( )

Feintuch’s Algorithms SFG 16

Σ+

+y(k)

Σ+

-

d(k)

e(k)

LMSA LMSB

e(k)α0 k( ) αN 1– k( )

. . .

. . . β1 k( ) βM k( )

. . .

. . .

α0 k( )

αN 1– k( )

β2 k( )

z-1

y(k-1)x(k)

z-1

z-1

z-1

α1 k( )

β0 k( )

βM 1– k( )

z-1

z-1

z-1

β1 k( )

Bk z( )

Ak z( )z-1

x(k)

Notes:Note that the computational complexity of this algorithm is now only .

In typical applications Feintuch will outperform the FIR LMS if the system being identified has high levels ofrecursion within it. However just using Feintuch to identify a non-recursive system, very little advantage will beseen.

For a typical system, up to 10 or even 20 poles might be used with Feintuch’s algorihm.

A filtered-error algorithm may also be used whereby the error signal is smoothed prior to input into theadaptive algorithm. The possible advantage here is that the IIR algorithm can be assisted in convergening underSPR (strictly positive real) conditions. For certain SPR conditions a PLR algorithm will converge to an arbitrarypoint on the performance surface. Using a filtered-error can often alleviate this problem. A simple filtered erroris given by:

the coefficients should represent a simple low pass filtering function.

O N M+( )

ef k( ) cje k j–( )

j 0=

P

∑=

cj

Feintuch’s IIR LMS 17

• Feintuch’s IIR LMS can be drawn in a more “familiar” form to moreclosely resemble the well known FIR LMS SFG:

• Note that if the number of recursive weights is zero, then the SFGsimplifies to the FIR LMS.The algorithm requires MACs per sample.

y(k)Σ

+

-e(k)

d(k)

x(k)Σ+

+

Adaptive FIR Filter

Adaptive FIR Filter

Adaptive IIR Filter

z-1

a k 1+( ) a k( ) 2µe k( )x k( )+= b k 1+( ) b k( ) 2µe k( )y k 1–( )+=

2 N M+( )

Notes:In vector notation

,

where the weight and data vectors are respectively:

and

and

y k( ) aTx k( ) bTy k( )+=

bT b1 b2 … bM 1–, , ,[ ]= aT a0 a1 a2 … aN 1–, , , ,[ ]=

x k( ) x k( ) x k 1–( ) … x k N– 1+( ), , ,[ ]= y k 1–( ) y k 1–( ) y k 2–( ) … y k M– 1+( ), , ,[ ]=

Performance of IIR LMS Algorithms 18

• Feintuch is the simplest to implement of the three algorithms in termsof MACs per iteration.

• The full gradient IIR LMS is rarely implemented due to its high level ofcomputation required. The performance advantages for this high levelof computation are also noted to be minimal.

• The simplified gradient IIR LMS is also rarely implemented with,again, minimal performance improvement seen over thecomputationally simpler Feintuch’s IIR LMS.

• Situations where the full gradient and simplified gradient IIR LMS’shave improved performance over Feintuch’ IIR LMS are usually inenvironments where the system being modelled has very high levels offeedback. However these scenarios are rare!

Notes:Feintiuch’s LMS is therefore simplest to implement, and likely to be the most stable, given the inherent stabilsingnature of PLR type systems.

Note that some form of “stability monitoring” may be performed when implementing an IIR LMS algorithm. Thisbasically means that at every iteration the poles of the filter are calculated and if any are outside the unit circlethe adaption is stopped and restarted. In general this form of stability monitoring is not convenient for real timeDSP implementation.

Alternatively some recent publications have suggested implementing the adaptive filter as a cascade of 2ndorder filter sections whereby the stability of each section is easy to calculate. If any sections become unstablethen this particular section is “zeroed” and the adaptiion restarted.

Stability and Convergence 19

• Feintuch can be shown to be more stable than the full gradient andsimplified gradient IIR LMS’s due to an inherent ability not to adaptpoles outside of the unit circle.

• Step-size selection for IIR LMS algorithms is less well defined than forFIR LMS.

One practical rule of thumb is to choose the feedforward step size asfor an FIR filter, and set the feedback step size to one tenth of thisvalue!

• Feintuch Algorithm System Identification

• Feintuch Algorithm Inverse System Identification

Notes:Observe the OHP demonstrations and note here the main comments from the simulations.

Newton/Recursive Predictor Error 20

• This algorithm uses an estimate of the correlation matrix, toaccelerate convergence.

• Computational overhead is high in updating the correlation matrix oneach iteration.

y(k)Σ

+

-e(k)

d(k)

x(k)Σ+

+

Adaptive FIR Filter

Adaptive FIR Filter

Adaptive IIR Filter

z-1

A(z)

B(z)

w k( ) w k 1–( ) µR 1– k( )z k( )e k( )+=

R k( )

Notes:The output of the filter is and z(k) is defined as

where x(k) and are the input and output data vectors:

w(k) is where a(k) and b(k) are the feedforward and feedback filter weights

The adaptive filter weights are updated as:

where is recursively updated as:

The a priori (before weights updated) error is: . is a forgetting factor typically between 0.9and 1Initial values: , and (hence ).

The basis of the RPE algorithms is similar to the basis of the Gauss-Newton familty of FIR LMS algorithmswhereby an estimate of the data correlation matrix is used to accelerate convergence

y k( ) zT k( )w k 1–( )= z k( ) xT k( ) yT k 1–( )T

=

y k 1–( )

x k( ) x k( ) x k 1–( ) … x k N– 1+( )T

= y k 1–( ) y k 1–( ) y k 2–( ) … y k M– 1+( )T

=

w k( ) aT k( ) bT k( )T

=

a k( ) a0 k( ) a1 k( ) … aN 1– k( )T

= b k( ) b1 k( ) b2 k( ) … bM 1– k( )T

=

w k( ) w k 1–( ) µR 1– k( )z k( )e k k 1–( )+=

R 1– k( )

R 1– k( ) 1λ--- R 1– k 1–( ) R 1– k 1–( )z k( )zT k( )R 1– k 1–( )

λ zT k( )R 1– k 1–( )z k( )+--------------------------------------------------------------------------------–=

e k( ) d k( ) y k( )–= λ

z 0( ) 0= w 0( ) 0= R 0( ) δI= R 1– 0( ) 1δ---I=

Adaptive QR-IIR Filter 21

• A QR-IIR filter can be implemented by essentially setting up a twochannel QR filter, and feeding back the output to be the input to one ofthe channels:

Σ

+-

d(k)

y(k)

QRx(k) e(k|k-1)

e(k|k)

÷γ(k)

(⋅)½-

x1(k)

x2(k)

input

desired

signal

signal error signal

Output signal

Notes:A two channel QR-RLS algorithm. The output of the filter is:

where

x1(k) and x2(k) are the first and second channel input signals respectively:

R(k) and p(k) are obtained as:

The a priori (before weights updated) and a posteriori (after weights updated) errors are respectively:

and The output γ(k) is obtained as:

γ(k) is also known as the conversion factor, as it converts the a priori estimation error e(k|k-1), obtained with theold filter weights, into the a posteriori estimation error e(k), obtained with the updated filter weights. Note that

, for all k.

The QR-IIR is a Pseudo-linear-regressive algorithm.

y k( ) zT k( ) R 1– k( )p k( )[ ]= z k( ) x1T k( ) x2

T k( )T

=

x1 k( ) x1 k( ) x1 k 1–( ) … x1 k N1– 1+( )T

= x2 k( ) x2 k( ) x2 k 1–( ) … x2 k N2– 1+( )T

=

R k( ) p k( )

OT *Q

λR k 1–( ) λp k 1–( )

zT k( ) d k( )=

e k k 1–( ) d k( ) zT k( ) R 1– k 1–( )p k 1–( )[ ]–= e k k( ) d k( ) y k( )–=

γ k( )e k k 1–( )

e k k( )--------------------------

1 2⁄–=

0 γ k( ) 1≤ ≤

QR IIR SFG - Given’s Rotations 22

• For RLS IIR filters the “Feintuch” assumption is more formally stated asthe PLR (pseudo-linear regression).

• The RLS-IIR filter, based on a QR decomposition can be representedas the classic QR triangular array signal flow graph (SFG):

Σ

d k( )

e k( )

-+

y k( )x k( )

Givens Generation

Givens Rotation

y k( ) d k( ) e k( )–=

3 feedforward and2 feedback weights.

Notes:The QR decomposition for adaptive FIR filtering was extensively presented earlier. Note the similarity of theabove SFG to that for an FIR filter. In some sense the IIR-QR can be thought of as a two channel QR where,the second channel is the output of the first fedback!

More details on the QR-IIR filter can be found in:

M.Harteneck, R.W. Stewart, I. Proudler, J. McWhirter. An effective QRD based approach to adaptive IIRfiltering. Proceedings of IEEE ICASSP, Atlanta, USA, May 1996.

M.Harteneck, R.W. Stewart. Acoustic echo cancellation using a pseudo-linear regression and QRdecomposition. Proceedings of IEEE ISCAS, Atlanta, USA, May 1996.

M.Harteneck, R.W. Stewart, I. Proudler, J. McWhirter. An algorithmically engineered fast multichannel adaptivefilter based on the QR-RLS. Institute of Mathmatics Conference, Mathematics in Signal Processing, Warwick,December 1997.

Development of the QR-IIR is jointly being undertaken by DRA Malvern and University of Strathclyde.

QR-IIR Algorithm Properties23

• To date the QR-IIR has proved to be an excellent algorithm which canoutperform IIR-LMS and RPE IIR LMS gradient based strategies.

• All “traditional” QR algorithm features are exploitable:

• direct residual extraction;

• fast algorithm implementation;

• multichannel implementation.

• Fast QR-IIR algorithms can also be dervied in the same style as for fastQR-FIR.

Notes:

Adaptive IIR Filter - Equation Error 24

• One alternative form of the adaptive IIR is the equation error, which infact is not actually a truely recursive system!

• This adaptive filter actually has no feedback weights (poles) and asimple LMS feedforward-type adaptive algorithm can be derived.

y(k)Σ

+

-e(k)

d(k)

x(k)Σ

Adaptive Algorithm

+

+

Adaptive FIR Filter

Adaptive FIR Filter

Adaptive IIR Filter

z-1

No outputfeedback

Notes:There exists a type of “pseudo” IIR filter usually called the equation error, whereby the adaptive filter is actuallytwo FIR filters and the desired signal has been used to approximate the signal . This can be seen bycomparing Slide 3 and Slide 24.

The resulting architecture is not actually recursive and consists of two FIR LMS filters. The equation errorsolution is only of use when the system is used in zero noise conditions, otherwise the solution is biased!

Note that instead of feeding back we feedback . The loose mathematical justification is that if thesystem were adapted with a truely zero error, then Hence starting of with approximated as

should allow adaption in conditions of zero noise.

The equation error adaptive IIR performance surface is hyperparaboloid and therefore has only one minimum.

Adaptive equation error systems actually form the basis of the widely used decision feedback equaliser (DFE).

d k( ) y k( )

y k( ) d k( )d k( ) y k( )= y k( )

d k( )

Equation Error LMS Algorithm 25

• Assuming signals and are ergodic and wide sensestationary (WSS), then from the LMS weight vector update is:

and

From:

• Therefore the equation error LMS update is:

where

x k( ) d k( )

w k 1+( ) w k( ) µ ∇ˆ

k–( )+= ∇ˆ

ke2 k( )∂w k( )∂

----------------- 2– e k( ) y k( )∂w k( )∂

---------------= =

y k( ) aT k( )x k( ) bT k( )d k 1–( )+ aT k( ) bT k( )x k( )

d k 1–( )= =

y k( )∂wk∂

-------------- y k( )∂a k( )∂

-------------- y k( )∂b k( )∂

--------------,T

xT k( ) dT k 1–( ),[ ] uT k( )= = =

w k 1+( ) w k( ) 2µe k( )u k( )+= wT k( ) aT k( ) bT k( )=

Notes:The equation error adaptive IIR performance surface is hyperparaboloid and therefore has only one minimum.

Equation Error Signal Flow Graph 26

• Unlike the output error adaptive IIR, the equation has only oneminimum.

(Note this is not in fact an IIR filter in the sense of having filter poles!)

y(k)Σ

+

-e(k)

d(k)

x(k)Σ+

+

Adaptive FIR Filter

Adaptive FIR Filter

Adaptive IIR Filter

z-1

a k 1+( ) a k( ) 2µe k( )x k( )+= b k 1+( ) b k( ) 2µe k( )d k 1–( )+=

Notes:Redrawing the SFG for an inverse system identification example indicates that there is not recursive filter part(i.e. poles) of the output error IIR.

.

In conditions of no observation noise the solution is unbiased.

Σ

+

-Σ+

+

Unknown System A z( )

B z( )z-1

Σ

Noise

Noise

Σ

Noise

Σ+

Σ+Unknown

System A z( )

B z( )z-1

-+

Adaptive Equalisation 27

• The feedforward/decision feedback (FF-DFE) equaliser is essentiallyan equation error adaptive IIR filter when in training mode.

Adaptive Filter

Σ+

−Mobile

ChannelPRBS

Training

PRBS Training

time

Implicit connectiontime

Σ+

z-1d k( )

y k( ) e k( )

M z( )

Adaptive Filter

+

Westwood, Los Angeles, USA Eastwood, Glasgow, UK

Notes:Note that this system when operating in training mode is essentially an equation error (EE) IIR filter. Althoughthe name IIR would suggest feedback, the EE-IIR is actually non-recursive as is fedback, rather than .

More discussion on equalisers can be found in the DSP Communications notes.

d k( ) y k( )

FF-DFE Decision Directed Mode (DDM) 28

• The feedforward/decision feedback equaliser is essentially an outputerror adaptive IIR filter when connected in decision feedback mode.

Adaptive Filter

Σ+

−Mobile

ChannelPRBS

Training

time

Σ+

d k( )

y k( )

e k( )

M z( )

Adaptive Filter

+

z-1

+1

−1

Slicer

Eastwood, Glasgow, UKWestwood, Los Angeles, USA

Notes:When the system switches to DDM, the adaptive filter then begins to resemble an output error IIR filter. Analysisis not straightforward given the non-linear slicer element in the feedback.

Adaptive LMS equalisers are widely used in data communications ranging from telephone channel modem(V32, V34, V90) to cable modems and ADSL / HDSL modems and equalisation for mobile channels. AdaptiveDFEs are widely used within these applications when the channel may have spectral zeroes, or noiseenhancement is a concern.

In the above examples we have only used 2 symbol data (0 and 1, represented as -1 and +1). In many realworld systems there will be many more symbols and hence the “slicer” represented above is somewhat morecomplicated.

For QAM based communications the data will be complex. In these applications the DFE and adaptive IIR LMS(equation and output error) will of course be complex.

Other Application Examples 29

• Adaptive IIR filters have very relevant application in any applicationwhere recursion is to be identified:

• Some forms of echo cancellation (ISDN)

• Acoustic systems

• Active noise control

• Speech compression - adaptive predictors.

• If however an adaptive IIR is being to use to attempt to model a non-recursive system using fewer weights than an adaptive FIR, then this isunlikely to produce useful results.

Notes:IIR filters should be used in applications where there is true recursion to be modelled. If there is not, then wewould recommend simply using an FIR filter with a “long enough” impulse response.

Active Noise Control 30

• Filtered-U IIR LMS based algorithms for real time noise cancellationcan be derived based on a slight variation of Feintuch’s IIR LMS:.

Errormicrophone

Reference microphone

He(f)

NOISE

Q(f)

T(f)

Loudspeaker

d(k)

+Filter

Zeroes a

y(k)x(k)

Heˆ

z( )

Σ+

Filter Poles b

f(k)

g(k)

bk 1+ bk 2– µekgk=ak 1+ ak 2– µekfk=

Heˆ

z( )

Notes:In the active noise control architecture the output of the filter passes through the acoustic environment beforeproducing the error signal by summing with the noise. Hence the adaptive algorithm must be rederived toproduce the filtered-U LMS. The filtered-U is the IIR equivalent of the filtered-X LMS for FIR filters.

Active noise control is based on the simple physical principle of destructive interference:

Applications include noise reduction inside aircraft, cars, factories, and so on.

Note that because the error signal in the above architecture is actually:

then the IIR LMS algorithm is modified slightly to realise the filtered-U LMS whereby the impulse response ofthe loudspeaker, room acoustics and microphone is necessarily taken into account. A derivation of the filtered-U can be found in the “draft” A-Z. For further details on filtered-U algorithms, please see:

L.J. Eriksson. "Development of the filtered-U algorithm for active noise control." Journal of the Acoustical Society ofAmerica, Vol. 89 (No.1), pp 27-265, 1991

R.W. Stewart, R. Duncan, and S. Weiss. "Multichannel adaptive active noise cancellation using the DSP56001." IEEEInternl Conf on Acoustics, Speech and Signal Proc., Vol I, pp. 51-54, April 1993.

ANCLoud-

speaker

NOISE

Quiet Zone:(destructive interference)

Anti-phasenoise

Periodicnoise

ek dk yk*he––

Two Channel ANC System 31

•

c11

c12

c11

c12

c21

c22

c21

c22

c11

c12

c21

c22

Secondary Loudspeakers

Error Microphones

y1(k)

y2(k) d2(k), NOISE

d1(k), NOISE

e2(k)

e1(k)

a12

LMSa12

b12

LMSb12

a11

LMSb11

b11

LMSb11

DAC

DAC

Amp

Amp

AD

CA

mp

AD

CA

mp

Figure 2: Example of a 2 channel adaptive IIR based ANC system implemented on the testbed

"Reference" Filter

ReferenceMicrophoneinput

"Reference" Filter

ReferenceMicrophoneinput

DSP56002

ADC

ADC

Notes:Strathclyde/ have developed a multichannel active noise control testbed . The system has been widely used,and the results below are shown for active noise control of car engine noise, and air ducting noise.

0 5 10 15 20 25 30 35 40 45 50

Time (ms)

Am

plitu

de 20001200

400-400

-1200-2000/2000

1200400

-400-1200-2000

0

0

0 100 200 300 400 500 600 700 800 900 1000

Frequency (Hz)

Mag

nitu

de (d

B)

0-8

-16-24-32

-40/0-8

-16-24-32-40

TIme Analysis

Power Spectra

Before ANC

After ANC

Before ANC

After ANC

0 5 10 15 20 25 30 35 40 45 50

Time (ms)

Mag

nitu

de (u

nits

)

25001500

500-500

-1500-2500/2500

1500500

-500-1500-2500

0

0

0 100 200 300 400 500 600 700 800 900 1000

Frequency (Hz)

Mag

nitu

de (d

B)

0-8

-16-24-32

-40/0-8

-16-24-32-40

TIme Analysis

Power Spectra

20 25 30

Before ANC

After ANC

Before ANC

After ANC

ANC Inside a Car ANC for Air Ducting

Adaptive Differential PCM (ADPCM) 32

• Adaptive differential pulse code modulation (ADPCM) achievescompression using an adaptive step size, and an adaptive IIR predictorfilter.

• The ITU G-721 and G726 (more recently) recommendation definesADPCM for 32 kbits/s and lower bit rate data transmission.

Adaptive IIR

Quantizer

Adaptive IIR

Σ∆(n)

xd(n)

x(n)Channel

xd(n)

Σ+

+Quantizer

Step size adapter

Step size adapter

Σ

+

+

Notes:ADPCM uses a 2 pole, four zero IIR filter for prediction purposes. Given that practically every PC has anADPCM compress facility this makes ADPCM one of the most implemented algorithms around. Hence to saythat adaptive IIR filters are rarely implemented is quite untrue given ADPCM is on every Windows PC and manydigital telephone systems such as DECT (Digital European Cordless Telephone).

Adaptive IIR Conclusions 33

• This presentation has introduced adaptive IIR LMS algorihms.

• We derived the most widely used adaptive IIR algorithm - Feintuch’sLMS.

• Convergence and stability have been demonstrated via real worldsimulation.

• Pseudo linear regression (PLR) strategies (RLS) were presented.

• The relation between adaptive equalisers and adaptive IIR filters washighlighted.

• A number of real world applications were highlighted.

Notes: