Hopfield Networks

Divneet Singh Kapoor

AP-ECE

What is Hopfield Network ??

Hopfield net is a form of recurrent artificial neural network inventedby John Hopfield.

Hopfield nets serve as content-addressable memory systemswith binary threshold units.

They are guaranteed to converge to a local minimum, butconvergence to one of the stored patterns is not guaranteed.

Hopfield Network

2

What are HN (informally)Hopfield Network

3

These are single layered recurrent networks

All the neurons in the network are fed back from all other neurons in the network

The states of neuron is either +1 or -1 instead of (1 and 0) in order to work correctly.

No. of the input nodes should always be equal to no. of output nodes

Following figure shows a Hopfield network with three nodes

Some Interesting Facts .

The recall pattern of Hopfield network is similar to our recall mechanism.

Both are based on content addressable memory

If some of the neurons of network are destroyed the performance is degraded but some network capabilities may be retained even with major network damage. Just like our brains

Hopfield Network

4

Did you know that we are

similar

Major Applications

Recalling or Reconstructing corrupted patterns

Large-scale computational intelligence systems

Handwriting Recognition Software

Practical applications of HNs are limited because number oftraining patterns can be at most about 14% the number of nodes inthe network.

If the network is overloaded -- trained with more than themaximum acceptable number of attractors -- then it won't convergeto clearly defined attractors.

Hopfield Network

5

Mathematical Modeling of HNs

Hopfield Network

6

Algorithm

Hopfield Network

7

Updation Types

Asynchronous one picks one neuron, calculates the weighted input

sum and updates immediately. This can be done in a xed order, or neurons can be

picked at random, which is called asynchronous random updating.

Synchronous the weighted input sums of all neurons are calculated

with- out updating the neurons. Then all neurons are set to their new value, according

to the value of their weighted input sum.

Hopfield Network

8

Hopfield Network

9

Mathematical Modeling of HNs

ii

n

ijj

jiji Tivwnet 1

TiWvnet

nnet

.

.

net

net

2

1

net

0

0

0

0

321

33231

21221

11212

nnn

n

n

n

www

www

www

www

w

ni

.

.

i

i

2

1

i

Consider signum function to be neurons activation function.

i.e.

vi = +1 if hi>0

vi = -1 if hi

Mathematical Modeling of HNs

Hopfield Network

11

)sgn(1 iikt

i

k

i Tivwv

Liapunov Energy function :-

vtviWvvtttE

2

1

Power Of Hopfield NetworksHopfield Network

12

We want to understand how to achieve this kind of performance form simple Hopfield networks

Learning HNs through simple example

Hopfield Network

13

Oa

Ob

Oc

W3,2

W1,2W2,1

W3,1 W1,3

W3,3

W2,2

W2,3

W1,1

There are various ways to train these kinds of networks like back propagation algorithm , recurrent learning algorithm, genetic algorithm but there is one very simple algorithm to train these simple networks called One shot method. I will be using this algorithm in order to train the network.

Learning HNs through simple example

Lets train this network for following patterns Pattern 1:- ie Oa(1)=-1,Ob(1)=-1,Oc(1)=1

Pattern 2:- ie Oa(2)=1,Ob(2)=-1,Oc(3)=-1

Pattern 3:- ie Oa(3)=-1,Ob(3)=1,Oc(3)=1

Hopfield Network

14

w1,1 = 0w1,2 = OA(1) OB(1) + OA(2) OB(2) + OA(3) OB(3) = (-1) (-1) + 1 (-1) + (-1) 1 = 1w1,3 = OA(1) OC(1) + OA(2) OC(2) + OA(3) OC(3) = (-1) 1 + 1 (-1) + (-1) 1 = -3w2,2 = 0w2,1 = OB(1) OA(1) + OB(2) OA(2) + OB(3) OA(3) = (-1) (-1) + (-1) 1 + 1 (-1) = -1w2,3 = OB(1) OC(1) + OB(2) OC(2) + OB(3) OC(3) = (-1) 1 + (-1) (-1) + 1 1 = 1w3,3 = 0w3,1 = OC(1) OA(1) + OC(2) OA(2) + OC(3) OA(3) = 1 (-1) + (-1) 1 + 1 (-1) = -3w3,2 = OC(1) OB(1) + OC(2) OB(2) + OC(3) OB(3) = 1 (-1) + (-1) (-1) + 1 1 = 1

Learning HNs through example

Moving onto little more complex problem described in Haykins Neural Network Book

They book used N=120 neuron and trained network with 120 pixel images where each pixel was represented by one neuron.

Following 8 patterns were used to train neural network.

Hopfield Network

15

Learning HNs through example

In order to recognizing power of HNs

For this they need corrupted image. They flipped the value of each pixel with p=0.25.

Using these corrupted images trained HN was run. And after certain number of iteration the output images converged to one of the learned pattern.

Next slides shows the results that they obtained

Hopfield Network

16

Learning HNs through example

Hopfield Network

17

Learning HNs through example

Hopfield Network

18

Flow Chart summarizing overall process

Network returns the decrypted pattern

Run the trained network with corrupted pattern

Update weight vectors of Network

Train HN using Standard patterns

Hopfield Network

19

Shortcomings of HNs

Training patterns can be at most about 14% the number of nodes in the network.

If more patterns are used then

the stored patterns become unstable;

spurious stable states appear (i.e., stable states which do not correspond with stored patterns).

Can sometimes misinterpret the corrupted pattern.

Hopfield Network

20

Thank you ..

Hopfield Network

21