Study of a neural network-based system for stability ... 1... · Study of a neural network-based...
Transcript of Study of a neural network-based system for stability ... 1... · Study of a neural network-based...
Study of a neural network-based system
for stability augmentation of an airplane Author: Roger Isanta Navarro
Annex 1
Introduction to Neural Networks and
Adaptive Neuro-Fuzzy Inference Systems (ANFIS)
Supervisors: Oriol Lizandra Dalmases
Fatiha Nejjari Akhi-Elarab
Aeronautical Engineering
September 2013
i
Contents
1 Introduction to neural networks ......................................................................... 1
1.1 Main applications of Neural Networks ............................................................... 1
1.2 Advantages and disadvantages of Neural Networks .......................................... 2
1.3 Biological foundations of Neural Networks ........................................................ 3
1.4 The artificial neuron ........................................................................................... 4
1.5 Neural Network typologies................................................................................. 5
2 Required mathematical and numerical tools ........................................................ 7
2.1 Matrix inversion formula .................................................................................... 7
2.2 Least-Squares Method ....................................................................................... 7
2.3 Recursive Least-Squares Method ....................................................................... 8
2.4 Method of Steepest Descent .............................................................................. 9
3 ANFIS Networks ................................................................................................ 12
3.1 Fuzzy logic, fuzzy inference systems and the Sugeno fuzzy model ................... 12
3.2 Advantages of ANFIS over Multilayer Perceptrons (MLP) ................................ 13
3.3 Architecture ..................................................................................................... 14
3.4 ANFIS Hybrid Learning ...................................................................................... 16
3.4.1 Forward pass ............................................................................................ 17
3.4.2 Backward pass .......................................................................................... 17
ii
List of figures
Figure 1.1 Main neural elements .................................................................................... 3
Figure 1.2 Artificial neuron model .................................................................................. 4
Figure 1.3 Examples of step and sigmoidal activation functions .................................... 5
Figure 2.1 Generic convergence diagram of total output error with constant step ..... 10
Figure 2.2 Generic convergence diagram of total output error with variable step ...... 11
Figure 3.1 The Sugeno fuzzy model .............................................................................. 13
Figure 3.2 Diagram representation of a 3-inputs 3-rules ANFIS network ..................... 14
1
1 Introduction to neural networks
Artificial Neural Networks are nonlinear mapping systems whose structure is based in
the observed human and animal nervous systems; they are mathematical
approximations to the human brain function. Nevertheless, they are not directly
comparable to the brain, nor are their operation principles, for they do not base
themselves solely in biological networks operation, they only emulate a very simple
portion of human its functions and are not able to simulate the highly complex relational
neurobiological processes that occur within it.
Processing units are named neurons. Artificial Neural Networks (from now on Neural
Networks) comprise a large number of these processing units, each of which receives
weighted input from other units (or nodes) and generates a scalar output depending on
the available local information stored internally and the information arriving through the
incoming weighted connections.
A Neural Network is characterized by the following aspects:
- A set of processing units or neurons.
- An activation state for each unit, equivalent to the output of the unit.
- Connections between units, often defined by a weight that modifies the effect
of the input signal in the unit.
- A propagation rule, which specifies the effective entry of a unit as a function of
the external entries.
- An activation function based on the effective entry and the previous activation.
- An external input corresponding to a term known as bias for each unit.
- A method to gather information, corresponding to the learning rule.
- An environment in which the system is going to operate.
1.1 Main applications of Neural Networks
Applications of Neural Networks are diverse and so are the areas in which they are of
use. The main applications of Neural Networks are those in which the inferring
approximation and use of functions is required via observation or measure, these tasks
classify in following categories:
- Function approximation
- Data classification
- Data processing
- System control
2
System control is the most recent application of Neural Networks and is the field in
which most efforts are invested actually. It is also the field in which is developed this
study.
Specific areas of application of the Neural Networks are also varied, the most significant
follow:
- Automotive and transport: automatic pilot systems, failure detection by external
vibration detection, truck brakes diagnosis, fleet tracking systems.
- Banking and finances: checks and other documents reading, credit applications
and risks evaluation, real properties appraisal, loans assessment, price evolution
forecasting, fake identifications, signature interpretation and identification, use
of credit line analysis.
- Electronics: code sequence prediction, elements distribution in integrated
circuits, process control, failure analysis, artificial vision, voice recognition.
- Manufacturing: production and process control, product analysis and design,
failure diagnosis, automatic visual quality inspection.
- Medicine: EEG and ECG analysis, prosthesis design, optimization of transplant
times, recognition and prediction of infarctions via ECG, reduction of hospital
costs.
- Robotics: dynamic control of trajectory, controllers, optical systems.
- Security: adaptive security codes, cryptography, digital recognition of
fingerprints.
- Telecommunications: data compression, automation of information services,
real time translation of spoken language.
- Voice: voice recognition, voice comprehension, vowel classification,
transformation from text to voice.
1.2 Advantages and disadvantages of Neural Networks
Neural Networks present several advantages over other processing systems, the most
significant being:
- Neural Networks can synthesize algorithms through a learning process.
- To use neural technology it is not necessary to now the mathematical details. It
is only required familiarity with the job data.
- The solution of nonlinear problems is one of the strengths of the Neural
Networks.
- Neural Networks are robust. They might fail in some processing elements, but
the network keeps on working, as opposed as in traditional programming.
3
However, some disadvantages are also characteristic of Neural Networks:
- Neural Networks must be trained for each problem. Moreover, multiple tests
must be conducted to define the adequate architecture. Training might be long
and CPU time consuming.
- The training requirement involves large volumes of data.
Neural Networks present a complex issue for external observers who would want to
make modifications in it. To add new knowledge is necessary to change the interactions
of many units so that its unified effect might synthesize this new knowledge.
1.3 Biological foundations of Neural Networks
A biological neuron is a cell specialized in information processing. It divides, from a
simplified point of view, into the cell body (soma) and two different kinds of
ramifications: dendrites and the axon; the firsts drive pulse inputs to the soma and the
later transmits the output signals generated by the body of the cell.
Figure 1.1 Main neural elements
The signals that reach the neuron through the dendrites are weighted by a parameter
called weight, associated to the corresponding synapsis. These weights might excite the
neuron (positive weight synapsis) or inhibit it (negative weight synapsis).
The soma then integrates and combines the different weighted input signals, and emits
an output signal depending on the sum of the weighted entries: if the sum is higher or
equal to the activation threshold of the neuron, the output is generated and sent
through the axon.
4
The ability to adjust the signals, by modifying the weight values, results in a learning
mechanism which is the training methodology of the artificial neural networks.
1.4 The artificial neuron
As stated, an artificial neuron is a mathematical approximation to the biological neuron
function. Within the generic artificial neuron two processes occur: the first one is the
algebraic sum of the inputs of synapsis and the second is the evaluation of a nonlinear
function resulting in its output value.
Figure 1.2 Artificial neuron model
The algebraic sum that will be lately evaluated by the nonlinear function can be written
as a function of the synaptic weights , entry signals and a bias .
( ) ∑ ( ) ( )
(1.1)
where counts the total of incoming connections.
The polarization or threshold of the neuron is an external parameter, however it can be
considered as additional entry :
( ) ( ) ( ) ∑ ( ) ( )
( ) (1.2)
is called the activation potential.
The neuron output, results from the evaluation of the activation function , which takes
the activation potential as argument and may also take the previous output.
( ) ( ( ) ( )) (1.3)
5
If the previous output is not considered as argument, the resulting value of the whole
process of integration and evaluation within the neuron can be written as
( ) (∑ ( ) ( )
( )) (1.4)
Many activation functions are used as step, linear, hyperbolic tangent or sigmoidal
functions among others.
Figure 1.3 Examples of step and sigmoidal activation functions
1.5 Neural Network typologies
Artificial Neural Networks may be classified differentiating between neuron function,
single or multilayered networks, training methodologies, whether the flow of
information is recurrent or feed-forward, etc.
McCulloch and Pits, in 1943, introduced the simple artificial neuron. These are neurons
which accept values of zero or one as incoming signals and present an activation
threshold of one or two, thus allowing the implementation of a logic OR (in case of one
as activation threshold) or a logic AND (in case of the activation threshold being equal to
two). The integration of postsynaptic pulses is lineal in this kind of neurons.
In more realistic neuron designs the entry signals may be a real value and are subject to
varying weights.
Different neuron models have been designed to fulfill specific requirements such the use
of integration methods other than simple algebraic addition in which the entry signals
are defined as functions of time ( ) which describe changes in time on the
voltage ( ). Other neurons such as the ones used within the ANFIS network proposed
in this study have special functions such as the product of all their entries, a quotient or
the computation of a function linear or nonlinear to some defined or varying
parameters.
-0.5
0
0.5
1
1.5
y
u
-1.5
-1
-0.5
0
0.5
1
1.5
y
u
6
Neural networks may also be differentiated after their number of layers. Simplest layers
have only one or two layers, the second usually being a sum of the previous node
outputs, an example of such neural networks are Simple Perceptrons, whose goal is
usually the division of an n-dimensional space into two sub spaces according to a
criterion. A neural network differentiating between whether a color is dark or light may
be achieved through the implementation of a Simple Single Layered Perceptron. The
frontier of decision, which is equivalent to the reaching of the activation threshold in the
neuron, may be defined by one of many activation functions of which some have been
mentioned earlier.
To solve more complicated problems multilayered networks are usually required. There
is a great variety of such networks for they allow considerable specialization on the
problem at hand. The two most common multilayered neural networks being the
Multilayered Perceptron, an extension of the Simple Perceptron to a multilayer scheme,
and the Adaptive Neuro-Fuzzy Inference System (ANFIS) network, which is the one used
within this study.
The structure of the network and the subsequent flow of information are also significant
differences between networks, thus the so-called Feed-Forward networks are those in
which the information flows from a layer to a strictly superior layer for all layers and
nodes of the network. On the contrary, a recurrent network will be that in which one or
more nodes receive as input the output of another node in a subsequent layer.
Training methodologies are also a significant distinguishing characteristic and are
intrinsically related to the network structure. The linearity or nonlinearity to the global
network output of some of the its varying parameters will allow or prevent the use of
optimization methods like the Least Squares Method presented in subsections 2.2
and 2.3 and the nonlinearity or a recurrent structure will require the use of other
optimization procedures such as derivative-based methods as the Steepest Descent
presented in subsection 2.4 or derivative-free methods such as Genetic Algorithms.
7
2 Required mathematical and numerical tools
Within this section the main mathematical and numerical tools that will later be used to
develop the neural network training will be presented.
2.1 Matrix inversion formula
The matrix inversion formula will be of importance in the following Recursive Least-
Squares Method subsection.
This formula states that given two nonsingular square matrices and , then
( ) ( ) (2.1)
Proof of this statement can be found in [2].
2.2 Least-Squares Method
Least-Squares Method is a standard method to compute the set of parameters that will
best approximate the solution to an overdetermined system. To do so, the sum of the
squares of the errors is minimized.
Consider a linear system of equations consisting in equations and unknown
parameters
(2.2)
where is an matrix, is the unknown parameter vector and is the
solution vector. It is obvious that if , the matrix will be squared and the
exact solution to the system, provided that is nonsingular, can be easily calculated by
(2.3)
If , which is a very common situation in neural network training, as well as in
many other applications, an exact solution is not always possible, either because the
model is not appropriate enough to describe the system, or because of the existence of
noise or error contamination in the data. Equation (2.2) should be modified to account
for this error:
(2.4)
As defined previously, the main goal of the method is finding an approximate vector
which minimizes the sum of squared error.
8
∑( )
( ) ( ) (2.5)
where is the ith row of . The expression in the previous equation may be expanded,
derived and equated to as follows:
(2.6)
( )
(2.7)
At , ( )
(2.8)
If is non-singular, can be solved:
( ) (2.9)
2.3 Recursive Least-Squares Method
One important drawback of the previously disclosed Least-Squares Method resides in
the necessity of inverting the matrix: an operation requiring a high computational
cost if is not small enough. Moreover, the Least-Squares Method, as presented
previously, does not take advantage of the recently computed values, but given
additional equations to the system, requires the total recalculation of the method. The
following procedure presents a recursive method to account for an additional equation
in the system, that is, an extra training pair ( ), taking in consideration the previous
values.
From now on, the circumflex symbol denoting the approximate solution will be
omitted for simplicity.
Considering the unknown parameter vector at step
( ) (2.10)
the same vector at step can be written as:
([
]
[ ])
[ ]
[ ]
(2.11)
The following and are introduced
( ) (2.12)
9
([
]
[ ])
( ) (2.13)
Using the matrix inversion formula presented at the beginning of this section, the
computation of can be rewritten as an incremental formula:
(
)
(2.14)
Once known , an incremental expression for can easily be found.
( ) (2.15)
can be eliminated from this expression using the Equation (2.10)
(2.16)
( ) [(
) ] (2.17)
yielding the final incremental expression for :
( ) (2.18)
The sequential calculation of given , using Recursive Least-Squares Method can
be summarized in the following two steps:
{
( )
(2.19)
where the initial equals ( ) .
2.4 Method of Steepest Descent
Descent methods main goal is also the minimization of a function defined on an -
dimensional input space [ ] . The objective function might not have
linear form with respect to , as opposed as considered in the Least-Squares Method
(variations of Least-Squares Method exist for nonlinear models, however these will not
be considered since will not be used within this study). Also as opposed to the Least-
Squares Method, this local minimum is found iteratively due to the complexity and non-
linearity of .
Within one iteration, the next values vector, denoted by , is computed by a step
from of size in a direction so that ( ) ( )
10
(2.20)
( ) ( ) ( ) (2.21)
The computation of step is performed through two procedures: determination of
direction and determination of the step size . Many different methods exist whose
main difference lies in the computation of the first procedure, while the step size is
commonly determined by line minimization. However, some methods do not use line
minimization, which is the case of the two methods presented below.
The taken corresponds to the direction in which decreases more quickly, that is
( ).
( ) (2.22)
Equation (2.22) is commonly named simple steepest descent. In it, given a fixed the
magnitude of the step ( ) varies automatically because of the different gradients
of . Experimental results [2] show that the search is not efficient enough for
and will not converge for . A good value to choose is close to . Choosing
constant presents a considerable oscillation around the optimal solution since there is
no real control of the step size, as it depends directly on the value of . This effect is
shown in Figure 2.1.
Figure 2.1 Generic convergence diagram of total output error with constant step
Chang and Fallside introduced a heuristic method known as Backpropagation learning
rule with a momentum to update in which a descent direction is influenced by the
11
previous one, and the step size increases when the direction “looks good” according to
the relation between both directions [2].
If the gradient vector is normalized, Equation (2.23) is named normalized version of
steepest descent:
( )
‖ ( )‖ (2.23)
where is the actual Euclidean distance from to .
Again, may be chosen fixed or variable. If fixed, a small value ( ) will lead to an
inefficient search, while a high value ( ) will cause an efficient approach to the
objective value, but will then oscillate around it, loosing precision.
Figure 2.2 Generic convergence diagram of total output error with variable step
This automatically forces to consider a dynamically updated value of according to two
simple rules:
1. If the objective function undergoes consecutive reductions, increase by .
2. If the objective function undergoes consecutive combinations of one increase
one decrease, reduce by .
The values of and are typically set to and respectively, after
experimental observations.
12
3 ANFIS Networks
Adaptive Neuro-Fuzzy Inference Systems (ANFIS) are a class of adaptive neural networks
that are functionally equivalent to fuzzy inference systems (described in the following
subsection) and offer the combination of learning, adaptability and nonlinear, time-
variant problem solving characteristics of Artificial Neural Networks plus the important
concepts of approximate reasoning and treatment of information provided by the fuzzy
set theory.
ANFIS network control systems (or neuro-fuzzy systems) represent a hybrid platform for
solving actual complex problems that require the use of intelligent systems and are a
viable alternative to the conventional model-based control schemes. They allow dealing
effectively with the common issues of uncertainty and unknown variations in plant
parameters and structure, hence improving robustness of the control system.
3.1 Fuzzy logic, fuzzy inference systems and the Sugeno fuzzy model
Fuzzy logic [2] is a set of mathematical principles based on degrees of membership to
pre-established functions whose main goal is information modeling; it is a flexible tool
based on linguistic rules dictated by an expert. Fuzzy logic was developed to emulate
human logic and attain correct solutions in spite of the ambiguity of information. In
contrast with conventional logic where strict boundaries are set between the
membership of a variable to a set or another, fuzzy logic presents membership ranks
within the interval between the two sets, and offers a solution based on this dual or
higher membership.
As a simple example, the speed of a car may be classified as high or low for a given
circumstance, and the reaction of the driver when breaking will depend, among other
things, on whether he assigns his current speed to a set or another, or an intermediate
state between them. On one hand, conventional logic will establish strict boundaries
between the proposed sets; for example, driving at less than will be
considered slow and doing so at or more will be considered high speed. Clearly a
problem arises when driving at speeds around , since the response of the driver
subject to conventional logic will vary abruptly when crossing this boundary. On the
other hand, a driver subject to fuzzy logic reasoning will be able to assign partial
memberships to both functions, that is, considering partially high and partially
slow, and provide a much more accurate response. Moreover, if the parameters defining
the membership functions are variable, the fuzzy system will be able, provided a correct
training algorithm, to modify such functions to offer a better response.
13
The Sugeno fuzzy model aims to generate a systematic approach towards generating
fuzzy rules from a given input-output data set. Sugeno fuzzy rules are of the form:
( ) (3.1)
where and are fuzzy sets and ( ) is a function associated to the fact of
and pertaining to and respectively. ( ) will usually be a polynomial, in what
is then called the first-order Sugeno fuzzy model, in contrast with the zero-order Sugeno
fuzzy model if ( ) were constant.
Figure 3.1 shows the fuzzy reasoning procedure for a first-order Sugeno fuzzy model:
Figure 3.1 The Sugeno fuzzy model
3.2 Advantages of ANFIS over Multilayer Perceptrons (MLP)
In addition to the general advantages and disadvantages of the Neural Networks, ANFIS
networks present interesting advantages over Multilayer Perceptrons (MLP), which are
the most direct competitors in neural computing for the type of problem treated within
this study. These advantages result from the fact that ANFIS presents a much more
specific mathematical structure which enables it as a good universal adaptive
approximator. The most significant advantages of ANFIS in front of MLPs follow:
1. ANFIS presents a much better learning ability: for a similar network complexity,
a much smaller convergence error is achieved, and although the convergence is
slower the smallness of the error in ANFIS is able to compensate that fact.
2. MLP often present a sudden convergence preceded by a region of considerable
instability.
3. ANFIS can achieve highly nonlinear mapping, far superior to MPL and other
common linear methods of similar complexity.
14
4. ANFIS requires fewer adjustable parameters than those required in other Neural
Network structures and, specifically, backpropagation MPLs.
5. The ANFIS structure allows for parallel computation.
Finally, ANFIS presents two advantages exclusive to its method:
6. ANFIS networks present a well-structured knowledge representation.
7. ANFIS networks allow a better integration with other control design methods.
3.3 Architecture
ANFIS structure consists in layers of neurons, each of which having a very specific
behavior. From these, layers , and have a constant behavior, while layers 1 and 4
have varying parameters, the modification of which allows for the network training.
Figure 3.2 Diagram representation of a 3-inputs 3-rules ANFIS network
The definition of every node within the ANFIS structure follows. All nodes within a same
layer have identical behavior, subject to the dependence to varying parameters (layers
and ). Considering an -input and -rules ANFIS, the following nodal behaviors are
defined:
LAYER 1
Layer 1 consists of adaptive neurons in which the fuzzification is performed, that is:
the grade of membership to the defined membership functions of the input is evaluated.
15
( ) ( ) (3.2)
where is the input vector and ( )
denotes the neuron in of layer , associated to
input .
The membership function ( ) may vary; triangular and bell shaped functions are
commonly used. A generalized bell-shape function has been chosen for it is one of the
most common membership functions and is continuous, which allows better
differentiation when performing the backpropagation training:
( )
|
|
((
)
)
(3.3)
This function depends on three parameters:
- modifies the bell width.
- together with , modifies the bell slope at the point where .
- modifies the center position of the bell.
These parameters are commonly called premise parameters.
LAYER 2
Layer 2 nodes are fixed in which its output is the product of all their entries. The nodes
are commonly labeled and their respective outputs represent the firing strength of
the rules, or the rules inferences.
Depending on the problem at hand there will be a relationship between inputs, and the
whole set of possible combinations of each input rule will not be necessarily computed.
If however it is necessary to compute all of the possible combinations or a subset of
them the definition of ( )
will depend on those values of its corresponding
combination:
( ) ∏
(3.4)
LAYER 3
Layer 3 nodes, labeled are also fixes and their respective outputs represent the
normalized inferences, that is: the ratio of the corresponding rule’s firing strength to the
sum of all the rule’s firing strengths.
16
( )
∑
(3.5)
LAYER 4
Nodes in layer 4 are adaptive nodes in which the consequent evaluation inference is
calculated, its output is defined as:
( ) (∑
) (3.6)
where the set of parameters and is commonly called consequent parameters.
LAYER 5
Finally, a single node in layer sums all the outputs from layer to compute the
overall output of the network:
( ) ∑
∑
∑ (3.7)
3.4 ANFIS Hybrid Learning
ANFIS networks, as any other neural network, might be trained by backpropagation of
the resulting error and adjustment of the adaptive parameters according to this
propagation in order to minimize it. Nevertheless, ANFIS presents some linearity with
respect to some of its parameters, due to its structure, allowing for the application of
the much efficient Least Squares Method. The use of the combination of both Least-
Squares and Steepest Descent methods is referred to as Hybrid Learning.
As commented, the adaptive parameters of the ANFIS network divides into premise
parameters in layer 1, and consequent parameters in layer 4. The contribution of these
last set to the network output is linear:
∑ (∑
)
(3.8)
therefore, their computation using Least-Square Method or its recursive version is
advised, for the exact is obtained with no need of several iterations and requires less
computational effort and time on what a process named forward pass. On the contrary,
premise parameters will be computed using the Steepest Descent Method, by
17
backpropagating the error through the network, during the backward pass. Both
training steps constitute the Hybrid Learning methodology and are presented below.
3.4.1 Forward pass
In the forward pass the Recursive Least-Square Method is used to evaluate the
consequent parameters. Considering the notation in subsections 2.2 and 2.3 and a -
input network as the one shown in Figure 3.2, the vector will be defined as follows:
{ } (3.9)
and will be the desired output vector (not to be confused with the network results,
after Layer 5). Considering that the required number of training pairs to have a definite
system will be ( ), where it is recalled refers to the number of inputs and to
the number of rules, the vector would be as follows:
{ ( )} (3.10)
Meanwhile, matrix can be defined after Equations (3.6) and (3.7):
[
( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
]
(3.11)
where the superscript between parenthesis denotes the training pair number and has
been defined ( ) to simplify the notation.
Within the recursive part of the method, the vector corresponding to the training
pair is defined as any row of the matrix :
{ ( )
( ) ( )
( ) ( )
( ) } (3.12)
3.4.2 Backward pass
In the backward pass the error signal propagates backwards through the network until
the dependence of this error to each of the premise parameters is evaluated. Once
known this gradient, the parameters may be updated by Steepest Descent:
( ) ( )
‖ ‖ (3.13)
Chain rule is used to evaluate the partial derivatives:
∑
∑
(3.14)
18
where here stands for any premise parameter , or . The partial
derivatives are derived as:
[
( )
] (3.15)
[ (∑
)] (3.16)
{
(
∑
) ( )
(
∑
)
(3.17)
∏
(3.18)
The ⁄ derivative depends on the membership function used, and will be
different for each premise parameter , or . To simplify the notation, the
following function is introduced:
( )
(3.19)
It is also noticeable, that the absolute value within expression (3.3) may be rewritten in
order to ease differentiation:
( )
|
|
[(
)
]
( )
(3.20)
The following partial derivatives result. For premise parameter :
(
)
(3.21)
For :
( )
(3.22)
And for :
( )
(3.23)