Lecture 17: ANFIS Adaptive

In the Name of God

Lecture 17: ANFIS ‒ Adaptive Network-Based Fuzzy Inference System

Outline

▫ ANFIS Architecture▫ Hybrid Learning Algorithm▫ Learning Methods that Cross-Fertilize ANFIS and

RBFNRBFN▫ ANFIS as a universal approximator

What is ANFIS?

• There is a class of adaptive networks that are pfunctionally equivalent to fuzzy inference systems.

• The architecture of these networks is referred to ANFIS hi h t d f d ti t kas ANFIS, which stands for adaptive network-

based fuzzy inference system or semantically equivalently adaptive neuro-fuzzy inferenceequivalently, adaptive neuro-fuzzy inference system.

ANFIS Architecture

• Assume that the fuzzy inference system under y yconsideration has two inputs x and y and one output z.

• For a first-order Sugeno fuzzy model with two if-then rules:

Sugeno model and its corresponding equivalent ANFIS architectureequivalent ANFIS architecture

a) A two-input first-order Sugeno fuzzy model with two rulesb) Equivalent ANFIS architecture

Sugeno ANFIS, Layer 1g y

• Every node i in this layer is an adaptive node with a node function:function:

• In other words, O1,i is the membership grade of a fuzzy set Ai(or Bi).

Sugeno ANFIS, Layer 1g y

• Here the membership function can be any p yappropriate parameterized membership function, such as the generalized bell function:

• where {ai , bi , ci } is the parameter set.P t i thi l f d t• Parameters in this layer are referred to as premise parameters.

Sugeno ANFIS, Layer 2g y

• Every node in this layer is a fixed node labeled П, whose output is the product of all the incoming signals:

• Each node’s output represents the firing strength of a rule.• Any other T-norm operators that perform fuzzy AND (e.g.

min) can be used as the node function in this layer.

Sugeno ANFIS, Layer 3g y

• Every node in this layer is a fixed node labeled N. The i-th node calculates the ratio of the i-th rule's firing strength to the sum of all rules' firing strengths:

• For convenience, outputs of this layer are called normalized firing strengths.

Sugeno ANFIS, Layer 4g y• Every node i in this layer is an adaptive node with a

node functionnode function

• where wi is a normalized firing strength from layer 3 and {pi , qi , ri } is the parameter set of this node.

• Parameters in this layer are referred to as consequent parameters.parameters.

Sugeno ANFIS, Layer 5g y

• The single node in this layer is a fixed node labeled ∑, which computes the overall output as the summation of all incoming signals:

Structure of Sugeno ANFISg

• Note that the structure of this adaptive network pis not unique.

• For example, we can perform the weight normalization at the last layer.

Extension from Sugeno ANFIS to Tsukamoto ANFISTsukamoto ANFIS

• The output of each rule (fi) is induced jointly by a p ( i) j y yconsequent membership function and a firing strength.

a) A two-input Tsukamoto fuzzy model with two rulesb) Equivalent ANFIS architecture

ANFIS Architecture: two-input Sugeno with nine rulesSugeno with nine rules

ANFIS for Mamdani FIS

• For the Mamdani fuzzy inference system with y ymax-min composition, a corresponding ANFIS can be constructed if discrete approximationsare used to replace the integrals in the centroiddefuzzification scheme.

• The resulting ANFIS is much more complicated than either Sugeno ANFIS or Tsukamotothan either Sugeno ANFIS or Tsukamoto ANFIS.

Hybrid Learning Algorithmy g g

• When the values of the premise parametersp pare fixed, the overall output can be expressed as a linear combination of the consequent parameters.

• In Symbols, the output f can be written as

• which is linear in the consequent parameters p1, q1 r1 p2 q2 and r2q1, r1, p2, q2, and r2.

Hybrid Learning Algorithmy g g

• Goal: Calculating premise parameters and gconsequent parameters of ANFIS.

• Hybrid learning algorithm consist of two passes:F d P▫ Forward Pass

▫ Backward Pass• Forward Pass:Forward Pass:▫ Node outputs go forward until layer 4 and the

consequent parameters are identified by the least-th d (li l i ! h ?)squares method (linear learning! how?).

• Backward Pass:▫ The error signals propagate backward and the▫ The error signals propagate backward and the

premise parameters are updated by gradient descent.

Hybrid Learning Algorithmy g g

Forward Backward Forward pass

Backward pass

Premise parameters Fixed Gradients

descent

Consequent Parameters

Least-squares estimator FixedParameters

Signals

estimator

Node output Error signalsSignals Node output Error signals

Learning Methods that Cross-Fertilize ANFIS and RBFNFertilize ANFIS and RBFN

• Under certain minor conditions, an RBFN (radial , (basis function network) is functionally equivalent to a FIS.

• There are a variety of adaptive learning h i th t b d f b th d timechanisms that can be used for both adaptive

FIS and RBFN.

Hybrid Learning Algorithm for ANFIS and RBFNand RBFN

• An adaptive FIS usually consists of two distinct p ymodifiable parts: ▫ The antecedent part▫ The consequent part

• These two parts can be adapted by different optimi ation methods s ch as h brid learningoptimization methods, such as hybrid learning procedure combining GD (gradient descent) and LSE (least-squares estimator)LSE (least squares estimator).

• These learning schemes are equally applicable to RBFNs.

Learning Algorithms for RBFNg g

• Conversely, the analysis and learning algorithms for RBFNs are also applicable to adaptive FISRBFNs are also applicable to adaptive FIS.

• A typical scheme is to fix the receptive field (radial basis) functions first and then adjust the weights of th t t lthe output layer.

• There are several schemes proposed to determine the center positions (μi) of the receptive field f tifunctions:▫ Based on standard deviations of training data (Lowe)▫ By means of vector quantization or clustering

t h i (M d d D k )techniques (Moody and Darken)• Then, the width parameters σi are determined by

taking the average distance to the first several nearest neighbors of 'snearest neighbors of ui's.

Learning Algorithms for RBFN

• Once these nonlinear parameters (ui and σi) are

g g

p ( i i)fixed and the receptive fields are frozen, the linear parameters (i.e., the weights of the output

)layer) can be updated by either the least-squares method or the gradient method.

• Other RBFN analyses, such as generalizationproperties are all applicable to adaptive FISproperties, are all applicable to adaptive FIS.

ANFIS as a Universal Approximator

• When the number of rules is not restricted, a ,zero-order Sugeno model has unlimited approximation power for matching any nonlinear ffunction arbitrarily well on a compact set.

• This fact is intuitively reasonable.H t i th ti l f d• However, to give a mathematical proof, we need to apply the Stone-Weierstrass theorem (see the book for detailed analysis!)book for detailed analysis!).

Example

Rule 1: IF x is small (A1) AND y is small (B1) THEN f1=smalll i l ( ) i l ( ) f l

Example

Rule 2: IF x is large (A2) AND y is large (B2) THEN f2=large

A1:21

11

1)(

x

xA 212

1)(

y

yBB1: 1.01.01.01 yxf

21

221

y

A2:B2:

2214

1)( yB221)( xA 1010102 yxf2

2141

y22

291

)(

x

A 1010102 yxf

Given the trained fuzzy system above and input values of x=3 and y=4, find output of the Sugeno fuzzy system

ExampleExample

THEN

f1=0.1x+0.1y+0.1

f2=10x+10y+10f2=10x+10y+10

x y

ExampleExample

THEN

f1=0.1x+0.1y+0.10.5 0.5

f2=10x+10y+10f2=10x+10y+10

0.1 0.039

x=3 y=4

Example

A1x=3 0.5W =

0.25

Example

A2

3

x=3

0.25

N

W1=0.25+0.0039

W1=0.985

0.5

W0.0039

B1y=40.00390.1

N

W2=0.25+0.0039

W =0 0154B2y=4 0.039

W2=0.0154

Example

W1=0.985

Example

N w1f1=(0.985)x(0.1x3+0.1x4+0.1)=0.788

W2=0.0154

N w2f2=(0.0154)x(10x3+10x4+10)=1.232

Example0.7877

Example

0.7877+1.232 = 2.0197

O = 2.0197

1.232

What if “T-norm” is taken at layer 2 to perform AND operation

0.5

to perform AND operation

A1

A2

x=3

x=3

0.5

0.5N

W1=0.5+0.039

5

W 6min

3

0.50 039

W1=0.9276

B1y=4 0.0390.1

N

W2=0. 5+0.039

0.039

min

B2y=40.039 W2=0.0724

LAYER 3LAYER 2LAYER 1

What if “T-norm” is taken at layer 2 to perform AND operationW1=0.9276

to perform AND operation

N w1f1=(0.9276)x(0.1x3+0.1x4+0.1)=0.7421

W2=0.0724

N w2f2=(0.0724)x(10x3+10x4+10)=5.7920

LAYER 4LAYER 3

What if “T-norm” is taken at layer 2 to perform AND operation

0.7421

to perform AND operation

0.7421+5.7920 = 6.5341

O = 6.5341

5.7920

LAYER 5LAYER 4

Gradient Descent learning for ANFIS

fw i=1,2,3, …R # of rules

i i

i iii

ii w

fwfwF F is the calculated/estimated output value

(by ANFIS)

Error = e = (d – F)2 d = Actual/Real Output

( , , ....)e

x y

Gradient of ANFIS’s output: Making ANFIS’s output (O) closer to actual output (AO)

This can be done by updating values of the parameters (e.g., a, c,…) over n (iteration/step)( 1) ( ) ea n a n

a

η: learning ratea

Practical considerations

• In a conventional FIS, the number of rules is ,determined by an expert familiar with the target

• In ANFIS, it is chosen empirically▫ By plotting the data sets and examining them

visually (for less than three inputs)Or by trail and error▫ Or by trail and error

• The initial values of premise parameters are set in such a way that the centers of MFs arein such a way that the centers of MFs are equally spaced along the range of each input variable

Practical considerations

• These MFs satisfy the condition of ε-ycompleteness with ε = 0.5▫ Given a value x of one of the inputs, we can

l f d l l b l halways find a linguistic label A that μA ≥ ε• This helps the smoothnessε completeness can also be maintained during• ε-completeness can also be maintained during the optimization process

Extensions of ANFIS

• The MFs can be of any parameterized forms y pintroduced previously

• Any T-norm operator can be replaced the algebraic product

• A learning rule can decide the best T-norm ti h ld b doperations should be used

• The rules can be realized using OR operators instead of ANDinstead of AND

More on ANFIS

• ε-completeness ensures that for any given value p y gof an input variable, there is at least a MF with a grade greater or equal to ε

• This guarantees that the whole input space is covered properly if ε is greater than zero

l t b i t i d b th• ε-completeness can be maintained by the constrained gradient descent

More on ANFIS

• Moderate fuzzyness: within most regions of the y ginput space, there should be a dominant fuzzy if-then rule with a firing strength close to unity h f h f lthat accounts for the final output

• This preserve MFs from too much overlap and make the rule set more informativemake the rule set more informative

• This eliminate the cases where one MFs goes under the other oneunder the other one

More on ANFIS

• A simple way is to use a modified error measurep y

• where E is the original squared error, β is a weighting constant, and is the normalized fi i hfiring strength

• The second term is indeed Shannon’s information entropyinformation entropy

• This error measure is not based on data fitting alone and trained ANFIS has betteralone, and trained ANFIS has better generalizability

More on ANFIS

• The easy way of maintaining reasonably shaped y y g y pMFs is to parameterize the MF correctly

ExampleExample

ANFIS is used to model a two-dimensional sinc equation defined by

xyyxyxcz )sin()sin(),(sin

y

x and y are in the range [-10,10]

Number of membership functions for each input: 4

Number of rules: 16

Examplex y

Example

Initial membership pfunctions

Final (trained)(trained) membership functions after 100after 100 epochs

ExampleExample

Function approximation using ANFISFunction approximation using ANFIS

In this example, an ANFIS is used to follow a trajectory of the non linear function defined bytrajectory of the non-linear function defined by the equation

1cos(2 )x

First we choose an appropriate architecture for

2

1( )x

ye

First, we choose an appropriate architecture for the ANFIS. An ANFIS must have two inputs – x1and x2 – and one output – y.and x2 and one output y.

Thus, in our example, the ANFIS is defined by four rules, and has the structure shown below.ou u es, a d as t e st uctu e s o be o

An ANFIS model with four rulesAn ANFIS model with four rules

Layer 1 Layer 4Layer 2 Layer 3 Layer 5

N1 1x1

y yy y y

A1

x1 x2

1

y

N2A2 2 2

N3

x2 B1 3 3

N4B2 4 4

Function approximation using ANFIS

The ANFIS training data includes 101 training

Function approximation using ANFIS

The ANFIS training data includes 101 training samples. They are represented by a 101 3 matrix [x1 x2 yd], where x1 and x2 are input [ 1 2 yd], 1 2 pvectors, and yd is a desired output vector.

The first input vector, x1, starts at 0, increments by 0.1 and ends at 10.

The second input vector, x2, is created by taking sin from each element of vector x1, with elements of the desired output vector, yd, determined by the function equationdetermined by the function equation.

Learning in an ANFIS with two MFs assigned to each input (1 epoch)assigned to each input (1 epoch)

y

2

y

Training Data ANFIS Output

1

0

1

-3

-2

-1

1 3

-0.5

0 0.5

4 6

810

-10.5

02

x1x2

Learning in an ANFIS with two MFs assigned to each input (100 epochs)

y

assigned to each input (100 epochs)

1

2 Training Data ANFIS Output

-1

0

1

1-3

-2

24

68

10

-0.5

00.5

1

02

-1x1

x2

So, ...

We can achieve some improvement, but much

,

better results are obtained when we assign three membership functions to each input variable. In this case the ANFIS model will have nine rulesthis case, the ANFIS model will have nine rules, as shown in figure below.

An ANFIS model with nine rules x1 x2

11A1 N1

An ANFIS model with nine rules

1

2

1

x1 A2

1

N2

N

2

3

y

3

4A3

N3

N4

3

4

y

5

6B1 N6

N5 5

6

7

8B2

N7

N8

7

8

x2

9B3 N9 9

Learning in an ANFIS with three MFs assigned to each input (1 epoch)

y

assigned to each input (1 epoch)

2

y

Training DataANFIS Output

1

0

1

-3

-2

-1

1 3

-0.5

0 0.5

4 6

810

-10.5

02

x1x2

Learning in an ANFIS with two MFs assigned to each input (100 epochs)

y

assigned to each input (100 epochs)

1

2 Training Data ANFIS Output

-1

0

1

1-3

-2

24

6 8

10

-0.50

0.5

1

0 2

-1x1

x2

Initial and final membership functions of the ANFISthe ANFIS

08

1

08

1

0.2

0.4

0.6

0.8

0.2

0.4

0.6

0.8

0 1 2 3 4 5 6 7 8 9 100

x1 x2-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10

(a) Initial membership functions.

0.6

0.8

1

0.6

0.8

1

0 1 2 3 4 5 6 7 8 9 100

0.2

0.4

-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10

0.2

0.4

x1 x2(b) Membership functions after 100 epochs of training.

Key Features of Different Modeling ToolsTools

Technique Model free

Can resist outliers

Explains output

Suits small data sets

Can be adjusted for new data

Reasoning process is visible

Suits complex models

Include known facts

Least squares regression

Neural networks

Fuzzy Systems

ANFIS

YesNo

Partially

An outlying observation, or outlier, is one that appears to deviate

Adapted from Gray and MacDonell, 1997

y g ppmarkedly from other members of the sample in which it occurs.

Readingg

• J-S R Jang and C-T Sun Neuro-FuzzyJ S R Jang and C T Sun, Neuro Fuzzy and Soft Computing, Prentice Hall, 1997 (Chapter 12)(Chapter 12).