The Pennsylvania State University

The Graduate School

Department of Electrical Engineering

COMPARISON ON THE PERFORMANCE BETWEEN THE THREE STRUCTURES

OF IIR ADAPTIVE FILTER FOR SYSTEM IDENTIFICATION BASED ON

GENETIC ALGORITHMS (GA)

A Thesis in

Electrical Engineering

by

Xin Shao

2015 Xin Shao

Submitted in Partial Fulfillment

of the Requirements

for the Degree of

Master of Science

December 2015

The thesis of Xin Shao was reviewed and approved* by the following:

Kenneth W. Jenkins

Professor of Electrical Engineering

Thesis Advisor

John F. Doherty

Professor of Electrical Engineering

Kultegin Aydin

Professor of Electrical Engineering

Head of the Department of Electrical Engineering

*Signatures are on file in the Graduate School

iii

ABSTRACT

Genetic Algorithms (GA) are based on the principles of natural selection and natural

genetics that originate in biology. The Genetic Algorithm (GA) has been used for IIR

adaptive system identification to deal with its multimodal error surface. The Genetic

Algorithm (GA) can be very useful in the all three structures of IIR filters while the Gradient

Algorithm experiences many difficulties due to the recursive feedback. This thesis will focus

on the different performances of three structures of IIR adaptive filters based on the Genetic

Algorithm (GA) and Multi-Parents Genetic Algorithm (MPGA). Experimental results

demonstrate that, in general, the standard Genetic Algorithm (GA) direct form will have

lower Mean Square Error (MSE), while the cascade and parallel forms will have higher

convergence rates. The relative performance of three structures for Multi-Parents Genetic

Algorithm (MPGA) is similar to the 2-parent Genetic Algorithm, but the rate of convergence

is higher than the standard GA, which means the MPGA converges faster than the standard

GA. Furthermore the performances of the three structures for the IIR filter based on modified

Multi-Parents Genetic Algorithm (MPGA) are very similar. Simulation results demonstrate

that when compared with the GA, the MPGA operates similarly on the three different

structures, increases the rate of convergence rate and reduces the computational complexity.

Finally, the Genetic Algorithm and the Gradient Algorithm were combined on the

direct form to take advantage of each algorithm. When the rate of convergence decreases into

a steady level the Gradient Algorithm is then applied so that the MSE will decrease again to a

lower value, demonstrating that the combined algorithm obtains a more precise result and

improves the performance.

iv

TABLE OF CONTENTS

List of Figures .......................................................................................................................... v

List of Tables ........................................................................................................................... vi

Acknowledgements .................................................................................................................. vii

Chapter 1 Introduction ............................................................................................................. 1

1.1 Development of Adaptive Filtering ........................................................................... 1 1.2 Gradient Algorithm .................................................................................................... 1

1.2.1 Introduction of the LMS Algorithm ................................................................ 1 1.2.2 Limitation of the LMS Algorithm on IIR Adaptive Filtering ......................... 2

1.3 Genetic Algorithm ...................................................................................................... 3 1.4 Multi-Parents Genetic Algorithm ............................................................................... 4 1.5 Overall Thesis Structure ............................................................................................. 5

Chapter 2 FIR vs IIR Adaptive System Identifications ........................................................... 6

2.1 FIR Adaptive System Identification ........................................................................... 7 2.2 IIR Adaptive System Identification ........................................................................... 7

Chapter 3 Three Structures of IIR Filters Based on the Genetic Algorithm ............................ 10

3.1 Direct Form II Structure ............................................................................................. 10 3.2 Cascade Form Structure ............................................................................................. 11 3.3 Parallel Form Structure .............................................................................................. 12 3.4 Comparison of three structure .................................................................................... 12

3.4.1 Effect of Population Size ................................................................................. 13 3.4.2 Effect of Mutation Rate ................................................................................... 14 3.4.3 Effect of Input Signal ...................................................................................... 16 3.4.4 Effect of the Number of Parents ...................................................................... 17

Chapter 4 Combined Genetic Algorithm and Gradient Algorithm .......................................... 18

4.1 Combine Genetic Algorithm and Gradient Algorithm for the FIR filter ................... 18 4.2 Combine Genetic Algorithm and Gradient Algorithm on IIR filter ........................... 21

Chapter 5 Conclusion and Future Work .................................................................................. 23

v

LIST OF FIGURES

Figure 1- 1 GA circle ............................................................................................................... 4

Figure 2-1. Adaptive system identification configuration ....................................................... 6

Figure 2- 2. Simplified IIR Adaptive Filter Block Diagram[14] ............................................. 9

Figure 3-1. Direct form II IIR filter structure .......................................................................... 11

Figure 3-2. Cascade form IIR filter structure ........................................................................... 11

Figure 3-3. Parallel form IIR filter structure ............................................................................ 12

Figure 3-4. Effect of Population Size....................................................................................... 13

Figure 3-5. Effect of Mutation Rate ......................................................................................... 14

Figure 3-6. Effect of Mutation Rate ......................................................................................... 15

Figure 3-7. Effect of Input Signal Characteristics ................................................................... 16

Figure 3-8. Effect of the Number of Parents ............................................................................ 17

Figure 3-9. Effect of Modified Parents Amount in the MPGA................................................ 18

Figure 4-1. Gradient Algorithm Applied to the FIR filter ....................................................... 19

Figure 4-2. Genetic Algorithm Applied to the FIR filter ......................................................... 19

Figure 4-3. Combination of Genetic Algorithm and Gradient Algorithm Applied to

the FIR filter ..................................................................................................................... 20

Figure 4-4. Gradient Algorithm Applied to the IIR filter ........................................................ 21

Figure 4-5. Genetic Algorithm Applied to the IIR filter .......................................................... 21

Figure 4-6. Combination of Genetic Algorithm and Gradient Algorithm Applied to

the IIR filter ...................................................................................................................... 22

vi

ACKNOWLEDGEMENTS

I would like to thank my advisor, Professor W. K. Jenkins, for his careful and patient

guidance throughout all aspects of my thesis and graduate studies. He is more a mentor than a

teacher for me. What he taught me is beyond the knowledge itself. I would also like to thank

Professor John F. Doherty and my classmate Guoxin Sun for their advice, support and

assistance for my thesis.

I am also grateful to my parents for their support and love.

1

Chapter 1

Introduction

1.1 Development of Adaptive Filtering

Ever since Widrow and Hoff established the theory of adaptive filtering [1], adaptive

filters have been widely used in various fields such as inverse modeling, linear prediction and

noise and echo cancellation [2]. Adaptive filters are also applied in the system identification

area successfully. Finite impulse response (FIR) and infinite impulse response (IIR) adaptive

systems are two general classes of models used for unknown system identification.

An adaptive filter is a system which has a linear filter, of which the transfer function

is manipulated by a set of parameters, also known as tap weights. In each iteration, the system

calculates the cost based on a cost function, a criterion for optimization, and the cost serves as

a feedback to adjust those parameters (or transfer function) to minimize the cost on the next

iteration until an optimization is considered to have been reached. The mean square error

(MSE) of the signal is the most commonly used cost function, the most widely used

optimization algorithm that minimizes the MSE is known as the least mean square (LMS)

algorithm.

1.2 Gradient Algorithm

1.2.1 Introduction of the LMS Algorithm

The LMS algorithm is an application of the stochastic gradient descent method [2].

Because of its simplicity, ease of computation, requiring no off-line data estimation, the LMS

is used widely as an optimization algorithm for adaptive filtering. The LMS algorithm is also

2

very suitable for FIR adaptive filters when the error surface is unimodal and quadratic. The

LMS algorithm for FIR adaptive filters can be summarized as follows [2]:

y(𝑛) = �̂�H(𝑛)𝐮(𝑛), (1-1)

e(𝑛) = d(𝑛) − y(𝑛), (1-2)

�̂�(𝑛 + 1) = �̂�(𝑛) + μ𝐮(𝑛)e∗(𝑛), (1-3)

where n is the iteration number, u(n) is the input vector, y(n) and d(n) are the corresponding

output and desired responses, respectively, e(n) is the error signal, �̂�(𝑛) is an estimate of the

unknown tap-weight vector, w(n), of the adaptive system, and μ is the step size parameter.

The superscript H denotes Hermitian transposition and the asterisk denotes complex

conjugation.

1.2.2 Limitation of the LMS Algorithm on IIR Adaptive Filtering

Generally, the IIR filter is better than the FIR filter for system identification because

an FIR filter may not be able to accurately approximate the unknown system with a

reasonable length filter. Alternatively, an IIR filter has recursive characteristics and results in

less amount of memory requirements to design complex systems [3]. In many situations an

IIR system can represent a better model for real-world systems than an FIR system.

Although the LMS algorithm is a good choice for FIR adaptive filters, the application

on IIR adaptive filters exhibits some drawbacks. Different from FIR adaptive filters, the error

surface of an IIR adaptive filter is often multimodal with respect to the filter coefficients,

causing the LMS algorithm to become stuck at local minima rather than converge to the

global minimum [4]. Due to this reason, the application of IIR system identification is limited

until some other optimization algorithms are developed.

3

1.3 Genetic Algorithm

During the past few decades, many other optimization algorithms in the

computational intelligence area, such as the particle swarm optimization (PSO) algorithm, the

cat swarm optimization (CATSO) [6] and the genetic algorithm (GA) [7], etc., have been used

for overcoming the deficiencies encountered by the IIR LMS algorithm.

GAs are global optimization search techniques that are based on the principles of

natural selection originated in the late 1980s [7]. Thereafter, the GA has been widely used in a

broad variety of engineering applications such as pharmaceutical drug designs, antenna

designs, integrated circuit testing, and adaptive infinite impulse response (IIR) filtering [8].

The GA can be viewed as an evolutionary process, which is mainly a cycle composed

of three operators, namely, Selection, Crossover and Mutation. The cycle is summarized in

Figure 1-1. This process starts with a set of randomly generated N individuals (N is called

population size) with M parameters to be optimized. At the beginning of each generation, the

fitness function will evaluate the fitness of each individual. Then a portion, generally half, of

individuals (parents) with higher fitness are selected and the rest individuals will be removed

from the population. After that each two of the selected individuals form a pair to do

crossover and generate two new individuals (offspring). During this stage, two parents

generate two children and the size of the population remains the same (assume half of the

population are selected).

Then mutation occurs to all parameters of some of the individuals with a relatively

low mutation rate m for each individual. The parameter updated in this stage occurs with the

probability m and can be expressed as

p̃ik = pik + 𝑞,

where q is some small constant. After the mutation stage, the remaining individuals form the

new generation, whose fitness have been evaluated, and then the process continues until the

"best"

4

Old Generation New Generation

Selection Mutation

Crossover

Figure 1- 1 GA flow chart

individual is generated. The information exchange in the Crossover stage and the change of

information in the Mutation stage are able to get the search of the GA out of local minima

when the error surface is multimodal. In spite of that, the slow convergence rate of the GA

limits the use of its implementations somehow [3]. In order to improve the convergence rate,

a modified GA - the MPGA was proposed and will be introduced in the next paragraph.

1.4 Multi-Parents Genetic Algorithm

In recent years, many attempts have been proposed to improve GA because of its low

rate of convergence and high computational complexity, including the innovative multi-

parents genetic algorithm (MPGA), which adopts three or more parents, instead of two, in the

crossover stage of the GA. Since it generates more offspring in each generation, the MPGA is

believed to have positive effects on the rate of convergence. This will lead the way for

broader implementations of the GA in IIR adaptive system identifications in the future [3].

Since the increasing number of parents can increase the rate of convergence, in order

5

to achieve lower MSE while maintaining faster rate of convergence, Guoxin Sun proposed a

new modified multi-parent GA (MPGA)[3], which makes the number of parents decrease

automatically when the difference between the current MSE and the previous one is smaller

than a given threshold and the Permutation Crossover MPGA (PC MPGA) gradually becomes

the 2-parents GA. This modification (referred to as the variable parent (VP) MPGA algorithm)

reduces the computational complexity of the PCMPGA significantly since only the first

several generations use the PC MPGA.

1.5 Overall Thesis Structure

Firstly, the development of the adaptive filtering, gradient and genetic algorithms are

introduces in Chapter 1. Then in terms of the IIR filter and FIR filter based on the GA it will

be demonstrated that if the unknown system is an FIR model the adaptive will achieve very

good results which the IIR adaptive algorithm cannot. Hence, in the rest of thesis the IIR and

FIR Adaptive System Identifications will be briefly compared in chapter 2 and then turn to

the IIR filter. Three structures (direct form, cascade form, parallel form) of IIR filter based on

the GA and MPGA will be compared in chapter 3. In the last chapter (chapter 4) a new ideal

that combining Genetic Algorithm and Gradient Algorithm, which will result in a better

performance of Adaptive System Identifications, will be introduced.

6

Chapter 2

FIR vs IIR Adaptive System Identifications

Over the past several decades, adaptive filters have been applied in many areas,

including inverse modeling, linear prediction, noise and echo cancellation, and system

identification [2]. In applications dealing with system identifications, an adaptive filter is used

iteratively to provide an optimal model for an unknown system or plant, based on some error

function, i.e., mean squared error (MSE) of the signal between the output of the adaptive filter

and that of the plant. The optimal model is obtained when the error function is minimized.

The general adaptive filtering configuration for system identification is shown in Figure 2-1

below. The plant and the adaptive filter are driven by the same input signal. The error

function, attained by subtracting the adaptive filter output from the plant output and

calculated by Eq. 1-2, is used to update the adaptive filter parameters according to the

optimization algorithm adopted. As mentioned above, when the unknown system is nonlinear

or contains feedback, an FIR adaptive filter with a reasonable length may not provide an

accurate model of the system but an IIR adaptive filter can achieve a better result [3].

∑

Plantd+

y-

Adaptive

filter

e

u

System

output

System

input

Figure 2-1. Adaptive system identification configuration

7

2.1 FIR Adaptive System Identification

Finite Impulse Response (FIR) filters are digital filters with finite impulse responses.

They are also known as non-recursive digital filters as they do not have the feedback (a

recursive part of a filter), even though recursive algorithms can be used for FIR filter

realization [10]. The equation for FIR filters can be expressed as:

𝑦(𝑛) = ∑ 𝜔𝑘(𝑛)𝑢(𝑛 − 𝑘), 𝑛 = 0,1,2, … , ∞𝑀−1𝑘=1 , (2-1)

where parameters 𝜔𝑘(n), which are called tap weights, are allowed to change at every time

step and u(t) is the input signal. As mentioned above, various algorithms can be used to

update the tap weights and the LMS algorithm is the most widely used one. If the LMS

algorithm is used, then the equation, which updates tap weights, is the same as Eq. 1-3. Since

there is no feedback for FIR filters, it is simpler than the IIR. It does a good job if the

unknown system is FIR filter as well.

2.2 IIR Adaptive System Identification

Different from FIR filters, IIR filters are recursive filters that uses the output of the

filter as points in the numerator of the filter coefficients. An Infinite Impulse Response (IIR)

adaptive filter is a linear filter and its output depends not only on the weighted inputs but also

on the previous outputs, which can be expressed as:

𝑦(n) = ∑ 𝑎𝑘𝑗(𝑛)𝑥(𝑛 − 𝑘)𝑁

𝑘=0+ ∑ 𝑏𝑗(𝑛)𝑦(𝑛 − 𝑗)

𝑀

𝑗=1, (2-2)

where n is the iteration number; ak(𝑛) and b𝑗(𝑛) are the tap weights (coefficients of

numerator and denominator, respectively); and N and M are the orders of the numerator and

denominator of the IIR filter, respectively [3]. The vector w(n) has the numerator and

denominator filter coefficients concatenated and will be used as the previous filter values.

𝑤(n) = [𝑎0(𝑛)𝑎1(𝑛) ⋯ 𝑎𝑁(𝑛)𝑏1(𝑛) ⋯ 𝑏𝑀(𝑛)], (2-3)

8

The following equation is the error equation where e(n) is the error, d(n) is the desired signal

and y (n) is the output of the filter.

𝑒(n) = 𝑑(𝑛) − 𝑦(𝑛), (2-4)

The LMS algorithm uses stochastic gradients to update the filter coefficients and multiplies

them with the error to make new coefficients. The alpha and beta terms are the outcome of the

derivation of the gradient operator of the algorithm. The vector η(n), like w (n) from earlier,

is the concatenation of the alpha and beta gradient terms that leads to updated filter

coefficients.

𝛼𝑖(n) = x(n − i) + ∑ 𝑏𝑙(𝑛)𝛼𝑖(𝑛 − 𝑙)𝑀

𝑙=1, (2-5)

𝛽𝑖(n) = y(n − i) + ∑ 𝑏𝑙(𝑛)𝛽𝑖(𝑛 − 𝑙)𝑁

𝑙=1, (2-6)

𝜂(n) = [𝛼0(𝑛)𝛼1(𝑛) ⋯ 𝛼𝑁(𝑛)𝛽1(𝑛) ⋯ 𝛽𝑀(𝑛)], (2-7)

The final recursion is shown in equation (2-8). It can be seen that η(n), which is the vector

that contains the gradient terms, gets multiplied by the error and the μ value; then added to the

original filter weight. The constant μ controls the step size of the filter coefficient changes.

Equation (2-8) shows how the filter is ultimately updated.

𝑤(n + 1) = 𝑤(𝑛) + 2𝜇𝑒(𝑛)𝜂(n), (2-8)

The following figure shows a simplified IIR adaptive filter block diagram from Farhang[13]

and illustrates how the IIR adaptive filter works using the LMS algorithm.

9

Figure 2- 2. Simplified IIR Adaptive Filter Block Diagram [14]

As a recursive filter, the IIR filter can accomplish the filtering of a signal with a lower

order than FIR filter and it can also obtain the same performance. However, as the error

surface of an IIR adaptive filter tends to be multimodal, the LMS algorithm tends to become

stuck at local minima [4]. Furthermore, IIR adaptive filters are susceptible to quantization

effects and can more easily become unstable because of the recursive nature of the filter.

There are many solutions, which have been developed, to overcome the drawbacks of

the IIR adaptive systems. The issue of quantization effects can be reduced by applying a

different realization of the IIR structure, such as an adaptive IIR lattice. The drawback of the

instability can be solved by applying a stability check, such as converting the filter to an

equivalent cascade second order terms and bounding the denominator coefficients to the

stability triangle, or by using an IIR lattice structure [9]. To deal with the multimodal error

surface in this thesis, the Genetic Algorithm was implemented to update the tap weights rather

than the LMS algorithm.

10

Chapter 3

Three Structures of IIR Filters Based on the Genetic Algorithm

An IIR system can be expressed as equation (2-2), where the output of this kind of

system depends on the inputs as well as past outputs (with feedback). The unit impulse

response of an IIR system is infinite in length and it contains a feedback loop, which means

its structure is recursive. Let N = M in (2-2), then the transfer function H(z) is

𝐻(𝑧) =𝑌(𝑧)

𝑋(𝑧)=

𝑎0 + 𝑎1𝑧−1 + 𝑎2𝑧−2 + ⋯ + 𝑎𝑁𝑧−𝑁

1 + 𝑏1𝑧−1 + 𝑏2𝑧−2 + ⋯ + 𝑏𝑁𝑧−𝑁

There is not only one structure for IIR systems, but rather the same system function may have

different structures. The paragraph below will introduce and compare the three kinds of

structures called the direct form, cascade form and parallel form.

3.1 Direct Form II Structure

The direct form II structure is one of the most commonly used structures that is

shown in Figure 3-1. The direct form II structure can be represented by (3-2) and (3-3). The

delay variable u(n) at the middle top of Figure 3-1 satisfies (3-2), and the output y(n) in

Figure 3-1 satisfies (3-3). [11]

𝑢(𝑛) = 𝑥(𝑛) − 𝑏1𝑢(𝑛 − 1) − 𝑏2𝑢(𝑛 − 2) − ⋯ − 𝑏𝑁𝑢(𝑛 − 𝑁), (3-2)

𝑦(𝑛) = 𝑎0𝑢(𝑛) + 𝑎1𝑢(𝑛 − 1) + 𝑎2𝑢(𝑛 − 2) + ⋯ + 𝑎𝑁𝑢(𝑛 − 𝑁), (3-3)

Equation (3-2) and (3-3) are used to design an IIR filter. At the beginning u(n-1), u(n-2),…are

set to zero. At time n, a new input x(n) is acquired, and (3-2) is used to calculate u(n). Then

the filter output becomes

𝑦(𝑛) = 𝑎0𝑢(𝑛) + 0,

At time n+1, a newer input x(n+1) is acquired and delay variables are updated by (3-2)

𝑢(𝑛 + 1) = 𝑥(𝑛 + 1) − 𝑏1𝑢(𝑛) − 0,

(3-1)

11

The output is

𝑦(𝑛 + 1) = 𝑎0𝑢(𝑛 + 1) + 𝑎1𝑢(𝑛) + 0,

and so on, for time n+2, n+3,… For each specific time, a new input is acquired, then the delay

variables and the output are updated by (3-2) and (3-3).

Figure 3-1. Direct form II IIR filter structure

3.2 Cascade Form Structure

The transfer function in (3-1) can be factored as

𝐻(𝑧) = 𝐶𝐻1(𝑧)𝐻2(𝑧) ⋯ 𝐻𝑟(𝑧),

in terms of first- or second-order transfer functions. The cascade (or series) structure is shown

in Figure 2. An overall transfer function can be represented with cascaded transfer functions.

For each section, the direct form II structure or its transpose version can be used. The transfer

function H(z), in terms of cascaded second-order transfer functions, can be written as [11]

𝐻(𝑧) = ∏𝑎0𝑖 + 𝑎1𝑖𝑧−1 + 𝑎2𝑖𝑧−2

1 + 𝑏1𝑖𝑧−1 + 𝑏2𝑖𝑧−2

𝑁/2

𝑖=1

Figure 3-2. Cascade form IIR filter structure

12

3.3 Parallel Form Structure

The transfer function in (3-1) can also be represented as

𝐻(𝑧) = 𝐶 + 𝐻1(𝑧) + 𝐻2(𝑧) + ⋯ + 𝐻𝑟(𝑧),

that can be obtained via a partial fraction expansion (PFE) of H(z). The resulting parallel form

structure is shown in Figure 3. Each of the transfer functions H1(z), H2(z), . . . can be either

first-order or second-order functions.

Figure 3-3. Parallel form IIR filter structure

As with the cascade structure, the parallel form can be efficiently represented in terms

of second-order direct form II structure sections. H(z) can be expressed as [11]

𝐻(𝑧) = 𝐶 + ∑𝑎0𝑖 + 𝑎1𝑖𝑧−1 + 𝑎2𝑖𝑧−2

1 + 𝑏1𝑖𝑧−1 + 𝑏2𝑖𝑧−2

𝑁/2

𝑖=1

3.4 Comparison of three structure

Now the performances of these three structures will be compared to demonstrate the

differences between them.

13

3.4.1 Effect of Population Size

Figure 3-4. Effect of the Population Size

In this section the population size will be the variable and two experiments are

compared in the following paragraph. These experiments are based on 2parent GA-5th order

IIR adaptive filter, the magnitude of mutation is 0.08 and the probability of mutation is 0.2.

The population sizes are 60 individuals and 150 individuals.

From Figure 3-4 it can be seen that the more individuals it has the lower the MSE will

become, which means better results are obtained by increasing the number of individuals.

Under the same number of individuals the direct form has a faster rate of converge and the

lowest mean square error. Hence, the performance of the direct form is the best and parallel

form is the worst.

14

3.4.2 Effect of Mutation Rate

The mutation is set to modify the information (genes) of the individuals. It maintains

the genetic diversity from one generation to the next and is considered to be the key factor to

prevent the GA from becoming stuck in local minima. Mutation alters one or more gene

values of the individuals and occurs according to a user-defined probability [3].

Figure 3-5. Effect of the Mutation Rate

Figure 3-5 demonstrates that the mutation rate will affect the MSE directly. Mutation

plays an important role in Genetic Algorithm. When increasing the mutation it will increase

the possibility to change the weights during the algorithm and that will decrease the rate of

convergence. From the above we can clearly see the dotted lines, whose mutation is 0.4, have

a lower rate of convergence and a higher final MSE than the corresponding full line whose

mutation is 0.2. In term of filter’s structure, it can be seen that the direct form always has a

better performance than others. When the mutation is 0.4, the parallel form and cascade form

filters have the similar performance and when mutation is 0.2 the cascade form has a better

performance than parallel form.

15

Figure 3-6. Effect of the Mutation Rate

It can be seen from Figure 3-6 that when the mutation becomes large enough, the

MSE will increase after a little convergence. Hence, the mutation probability should not be

set too large; otherwise, the GA will become a random search and the system will become

unstable. It also cannot be too small, otherwise it would take a long time for the search to get

out of local minima.

It is also obviously to be seen that the cascade form and parallel form filters are easier

to be effected by the mutation than the direct form filter. Hence, the cascade form and parallel

form filter are more sensitive than the direct form on the mutation.

16

3.4.3 Effect of Input Signal

Figure 3-7. Effect of the Input Signal Characteristics

From figure 3-7 we can see the different performances for the different inputs. It is

easy to see out that when a color noise signal is used as the input the filter has a worse rate of

convergence and higher MSE. The input signal that with more noise will take more time to

converge.

In terms of structure of the filter, the type of input doesn’t change the relative

performance of the three filters, direct form has the best performance and parallel form has

the worst. It can be seen that the GA is not strongly affected by the structure of filters.

17

3.4.4 Effect of the Number of Parents

Figure 3-8. Effect of the Number of Parents

In figure 3-8 where the variable is the number of parents, it can be seen that when

increasing the number of the parents the rate of convergence increased, although, the MSE

becomes higher. In terms of their structure, the parallel form and cascade have the similar

performance but the direct form appears to have the best performance.

Now the performance of the 2-parent and the modified multi-parents will be

compared. For multi-parents, the modified MPGA changes the amount of parents

automatically. From figure 3-8 and figure 3-9 it can be clearly seen that the rate of

convergence of modified multi-parents is faster than the 2-parent one and multi-parents one.

But it has a higher MSE than the others. In term of their structure, parallel form and cascade

have the similar performance for 2-parent and direct form has a better performance. For the

modified multi-parents the direct form and the cascade have similar performance and the

parallel form has a worse performance.

18

Figure 3-9. Effect of the Modified Number of Parents in the MPGA

Chapter 4

Combined Genetic Algorithm and Gradient Algorithm

From early research we can see the advantages and disadvantages of each algorithm,

then an idea to combine them together came up. Following we will see if we can get better

performance when we first apply the Genetic Algorithm and then apply the Gradient

Algorithm. The first case will be to apply the combined algorithm approach to the FIR filter.

4.1 Combine Genetic Algorithm and Gradient Algorithm for the FIR filter

From figure 4-1 and figure 4-2 we can see Genetic Algorithm (GA) did a bad job on

FIR filter when the error surface converged the mean square error is still high. However, the

gradient algorithm did a good job on the FIR filter, its error converged to a lower mean square

19

error than the GA did. Now it will be shown how the FIR filter performs if we combine these

two algorithms together.

Figure 4-1. Gradient Algorithm Applied to the FIR filter

Figure 4-2. Genetic Algorithm Applied to the FIR filter

20

Figure 4-3. Combination of Genetic Algorithm and Gradient Algorithm Applied to the FIR

filter

From the figure 4-3 we can easily see the combination performed better than either of

those two algorithms separately. It is obvious that the combination improves the performance

of FIR filter. However, the LMS part in the combination did not work as well as it did

separately. Figure 4-1 indicated that the Mean Square Error (MSE) converged approximately

from 0 DB to -50 DB, but in the combination the LMS part only converge the MSE from -10

DB to -55 DB so the difference is 45 DB. And it is also to be seen that if we apply the LMS

algorithm at the moment that GA just converged, the total rate of convergence is higher than

both of them separately. However, the combination and the separate LMS approximately

converge to the similar MSE. Hence, it is not very obvious that the combination is able to

achieve a lower error but it clearly improves the rate of convergence of the FIR adaptive filter.

Now the same combination strategy will be applied to the IIR adaptive filter to

determine if this strategy will achieve a better result.

21

4.2 Combine Genetic Algorithm and Gradient Algorithm on IIR filter

Figure 4-4. Gradient Algorithm Applied to the IIR filter

Figure 4-5. Genetic Algorithm Applied to the IIR filter

22

The result is opposite to the FIR one, Genetic Algorithm performed better than

Gradient Algorithm. The MSE of Gradient Algorithm made the MSE converge to -1.5 DB

and the Genetic Algorithm made the MSE converge to -18 DB. The Genetic Algorithm (GA)

got a better result, which indicated the GA performed better on IIR filter than LMS algorithm.

Now the combination of these two algorithms will be demonstrated:

Figure 4-6. Combination of Genetic Algorithm and Gradient Algorithm Applied to the IIR

filter

From figure 4-6 it can be easily seen that the MSE converged to -34 DB and it seems

to be still converging, which is much better than both of these algorithm separately. From

these results can conclude that the combination strategy significantly improved the

performance of the IIR adaptive filter. It also can be seen that the LMS part of combination

converged lower than the LMS did separately. That means the convergence, which is

contributed by GA, allows the LMS algorithm to more easily converge to the global minimum.

Comparing the combination on FIR and IIR filters we can conclude that the improvement for

performance of combination on IIR is more obvious than it on FIR.

23

Chapter 5

Conclusion and Future Work

Based on the experimental results in the previous chapter, it can be concluded that,

for the performance of three different structures of IIR adaptive filters, the direct form has the

best performance, the cascade form is a bit worse than the direct form, and the parallel form

has the worst performance. It also can be seen that more individuals will result in a lower

mean square error, high mutation may make the result unstable, more parents tend to improve

the rate of convergence, and colored noise is more difficult to be handled than white noise.

Another conclusion is that no matter what conditions are changed the relative

performance of these three structures will not significantly change. This suggests that the

Genetic Algorithm can be applied to arbitrary filter structures and the performance will not be

strongly influenced by the type of filter structure. Future research will further investigate the

use of a final gradient stage to minimize the final converged mean square error.

In terms of FIR adaptive filters the Gradient Algorithm LMS generally works better

than the Genetic Algorithm. However, in terms of the IIR adaptive filter the Genetic

Algorithm appears to do a better job than the Gradient Algorithms, which means that the

Genetic Algorithm is suitable to be applied on the IIR adaptive filter and it will improve the

performance of the IIR adaptive filter.

In this thesis the research came up with a new strategy that combines the Genetic

Algorithm and Gradient Algorithm together. From the experimental results we can find the

combination really improves the performance of the adaptive filter and it improves the IIR

filter a lot. Since the IIR adaptive is very sensitive on Gradient Algorithm, we can use the

combination of Genetic Algorithm and Gradient Algorithm to improve the performance of the

adaptive filter. Future research will investigate the performance of three different structures

based on this combination strategy.

24

Bibliography

[1] B. Widrow and M. E. Hoff, "Adaptive Switching Circuits," 1960 Wescon Conv. Rec. pt.

4, pp. 96-104, August 1960.

[2] S. Haykin, "Adaptive Filter Theory," Englewood Cliffs, NJ: Prentice Hall, 2013.

[3] Guoxin Sun, "Performance of multi-parents genetic algorithms (MPGA) for IIR adaptive

system identification," M.S. Thesis, Department of Electrical Engineering, Penn State

University, State College, PA, 2014.

[4] B. Widrow and S. D. Stearns, "Adaptive Signal Processing," Englewood Cliffs, NJ:

Prentice Hall, 1985.

[5] D. J. Krusienski and W. K. Jenkins, "Design and Performance of Adaptive Systems

Based on Structured Stochastic Optimization Strategies," Circuits and Systems Magazine,

Vol. 5, No. 1, pp 8 –20, February 2005.

[6] J. So and W. K. Jenkins, "Comparison of CAT Swarm Optimization with Particle Swarm

Optimization for IIR System Identification," Proceedings of the Forty-Seventh Annual

Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, PP. 903-

910, November 4-7, 2013.

[7] David E. Goldberg, "Genetic Algorithms in Search, Optimization, and Machine

Learning," Addison-Wesley Longman Publishing Co., Inc. Boston, MA, USA, 1989.

[8] R. Nambiar, C.K.K. Tang and P. Mars, "Genetic and Learning Automata Algorithms for

Adaptive Digital Filters", Proc. IEEE Int. Conf. on ASSP, 1992, vol. IV, pp. 41-44.

[9] Shynk, J. J., "Adaptive IIR Filtering", IEEE ASSP Magazine, pp. 4-21, April 1989.

[10] Zoran Milivojević, "Digital Filter Design", MikroElektronika, 1st edition, 2009.

(http://www.mikroe.com/chapters/view/72/chapter-2-fir-filters/).

[11] Rulph Chassaing, "Digital Signal Processing and Applications with the C6713 and

C6416 DSK", Worcester Polytechnic Institute, Wiley Interscience, A JohnWiley & Sons,

Inc.

25

[12] Yao L., Sethares W. A., "Nonlinear Parameter Estimation via the Genetic Algorithm,"

IEEE Transactions on Signal Processing, vol.42, April 1994.

[13] Z. J. Cavaliero and W. K. Jenkins, " Comparative Analysis of the Use of IIR Adaptive

Filters vs. FIR Adaptive Filters to Implement Sequential Processing for Fetal

Electrocardiograms," [D] Penn State University, State College, PA

[14] Farhang-Boroujeny, B. Adaptive Filters: Theory and Applications, Sussex: John Wiley &

Sons, 1998. Print.