Pattern Association

8/3/2019 Pattern Association

1/36

Klinkhachorn:CpE320

Pattern Association


2/36

Klinkhachorn:CpE320

Associative Memory Neural Nets

An associate network is a neural network

with essentially a single functional layer that

associates one set of vectors with another set

of vectors

The weights are determined in such a way

that the net can store a set of pattern

associations

Each association is an input-output vector

pair, s:t


3/36

Klinkhachorn:CpE320

Associative Memory (Cont.)

If each vector t is the same as the vector s

which it is associated, then the net is called an

autoassociative memory

If the ts are different from the ss, the net iscalled a heteroassociative memory

The net not only learns the specific patternpairs that were used for training, but also is

able to recall the desired response pattern

when given an input stimulus that is similar

but not identical, to the training input.

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall


4/36

Klinkhachorn:CpE320

Associated Pattern Pairs


5/36

Klinkhachorn:CpE320

Partial Data Presented

Who?

Astro

Rosie Look like?


6/36

Klinkhachorn:CpE320

Noisy Data Presented


7/36

Klinkhachorn:CpE320

Noisy Data Presented


8/36

Klinkhachorn:CpE320

Classification of ANN

Paradigms


9/36

Klinkhachorn:CpE320

Architecture of an Associative memory

Neural Net

Feedforward net: information flows from the

input units to the output units Example: Linear associator Network

Recurrent (iterative): there are connections among

the units that form closed loop

Example: Hopfield, BAM



10/36

Klinkhachorn:CpE320

Hebb Rule for Pattern Association

Simplest and most common method of

determining the weights for an associative

memory neural net

Can be used with patterns that are represented as

either binary or bipolar vectors


e = wixi

x1

xn

w1

wn

y = 1 if e 0= 0 if e < 0


11/36

Klinkhachorn:CpE320

Algorithm

Step 1: Initialize all weights (i=1,.,n; j = 1..,m)

wij = 0

Step 2: For each input training-target output vector pair

s:t, do Steps 3-5.Step 3: Set activations for input units to current

training input (i = 1,.,n):

xi=siStep 4: Set activations for output units to current

target output (j = 1,.,m):yj = tj

Step 5: Adjust the weights( I=1,n; j=1,.,m):

wij(new) = wij(old) +xiyi



12/36

Klinkhachorn:CpE320

Weights Calculation: Outer Product

Let s-t be the input vector-output vector pairs

The outer product of two vectors

s = (s1,..,si,.,sn)

and

t = (t1,...,tj,.,tm)Is the matrix product of the n x 1 matrix S = sT and

the 1 x m matrix T = t:



13/36

Klinkhachorn:CpE320

Weights Calculation: Outer Product (Cont)

To store a set of associations s(p) : t(p), p=1,.,P, where

s(p) = (s1(p),.,si(p),.,sn(p))and

t(p) = (t1(p),.,tj(p),.,tm(p))

The weight matrix W = {wij} is given by Pwij = si(p)tj(p) p =1

This is the sum of the outer product matrices required to

store each association separately. P

In general, W = sT(p)t(p) = sTt p =1



14/36

Klinkhachorn:CpE320

Perfect recall VS cross talk

Hebb rule:

If the input vectors are uncorrelated (orthogonal), the

Hebb rule will produce the correct weights, and the

response of the net when tested with one of the training

vectors will be perfectrecall

If the input vectors are not orthogonal, the response

will include a portion of each of their target values.

This is commonly calledcross talk



15/36

Klinkhachorn:CpE320

Hebb rule: an example

Suppose a Heteroassociative net is to be trained to

store the following mapping from input row vectors

s = (s1,s2,s3,s4) to output row vectors t = (t1,t2):


Pattern s1,s2,s3,s4 t1,t2

1st s (1, 0, 0, 0) (1, 0)

2nd s (1, 1, 0, 0) (1, 0)

3rd s (0, 0, 0, 1) (0, 1)4th s (0, 0, 1, 1) (0, 1)


16/36

Klinkhachorn:CpE320



x1

x2

x3

x4

x1

x1

w11w12

w21w22w31

w32

w41w42


17/36

Klinkhachorn:CpE320



Weights calculation: Outer Product

4W = sT(p)t(p) = sTt p =1

W=

1 1 0 0

0 1 0 0

0 0 0 1

0 0 1 1

.

1 0

1 0

0 1

0 1

=

2 0

1 0

0 1

0 2

=

w11

w12

w21

w22

w31

w32

w41

w42


18/36

Klinkhachorn:CpE320



Testing the net

for x = (1,0,0,0)

xW= (1,0,0,0)

2 01 0

0 1

0 2

= (2,0) > (1,0)


19/36

Klinkhachorn:CpE320



Testing the net - input similar to the training input

for x = (0,1,0,0) differs from the training vector (1,1,0,0)

only in the first component.

xW= (0,1,0,0)

2 01 0

0 1

0 2

= (1,0) > (1,0)


20/36

Klinkhachorn:CpE320



Testing the net - input not similar to the training input

for x = (0,1,1,0) differs from the training input patterns in

at least two components.

xW= (0,1,1,0)

2 01 0

0 1

0 2

= (1,1) > (1,1)

The two mistakes in the input pattern make it

impossible for the net to recognize it!


21/36

Klinkhachorn:CpE320

Iterative Autoassociative Net


22/36

Klinkhachorn:CpE320

Hopfield Nets (Recurrent)

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold


23/36

Klinkhachorn:CpE320

John Hopfield

Computational neurobiology /

biophysics

While the brain is totally unlike current digital computers, much of what it does can be described as computation.

Associative memory, logic and inference, recognizing an odor or a chess position, parsing the world into objects, and

generating appropriate sequences of locomotor muscle commands are all describable as computation. My current

research focuses on the theory of how the neural circuits of the brain produce such powerful and complex

computations. Olfaction is one of the oldest and simplest senses, and much of my recent work derives fromconsiderations of the olfactory system. One tends to think of olfaction as "identifying a known odor," but in highly

olfactory animals, the problems solved are much more complicated. Many animals use olfaction as a remote sense, to

understand the environment around them in terms of identifying both where and what objects are remotely located.

This involves at least coordinating when something is smelled with the wind direction, and untangling a weak

signal from an odor object from a background of other odors simultaneously present. The homing of

pigeons or the ability of a slug to find favorite foods are examples of such remote sensing.

http://www.molbio.princeton.edu/faculty/hopfield.html


24/36

Klinkhachorn:CpE320

Binary Hopfield Processing Unit

x1

x2

x3

xn

j

wj1

wj2

wj3

wjn

ej=xiwji i ij

yj = +1 if ej0= 0 if ej


25/36

Klinkhachorn:CpE320

Hopfield Architecture



26/36

Klinkhachorn:CpE320

Hopfield Net



27/36

Klinkhachorn:CpE320

Hopfield Net Algorithm


Step1: Compute the connection weights,Let

m be the total # of associated patterns, i.e.

X = (x1,x2,xp,..,xm)

and for each pattern p with n inputs

xp = (xp1,xp2,,xpi,..xpn)

Then, the connection weight from node i to node j, mwij = (2xpi-1)(2xpj-1) . If xp =(0,1), and wii = 0 p=1

or mwij = xpixpj . If xp =(-1,1), and wii = 0 p=1

or W = XTX . If xp =(-1,1), and wii = 0


28/36

Klinkhachorn:CpE320

Hopfield Net Algorithm (Cont)


Step2: Initialize the network with unknown input pattern,x = (x1, x2,.. , xn), by assigning output

yi(0) = xi, for i= 1 to n

where yi(0) is the output of node i at time 0

Step 3: Iterate (update outputs) until convergence

nyj(t+1) = F[ wijyij(t)], j = 1 to n

i=1

where F[e] = +1. if e0, or= -1(0). if e


29/36

Klinkhachorn:CpE320

Hopfield Net: Updating Procedure


Random UpdateAllows all units to have the same average update

Sequence UpdateRepeat the sequence until a stable state is attained

Most other neural network paradiagms have a layer ofprocessing units updated at the same time (or nearly thesame time).

Hopfield, however, suggested that random update of the

neuron has advantages both in implementation (each unitcan generate its next update time) and in function

(sequential updating can restrict the output states of the

network in cases in which different stable states are

equiprobable)


30/36

Klinkhachorn:CpE320

Hopfield Net: Updating Procedure



31/36

Klinkhachorn:CpE320

Hopfield Net: Analysis


Convergence**

Each state of the Hopfield network has an associated

"Energy" value,

Successive updating the network provides a convergence

procedure thus the energy of the overall network gets

smaller.

Consider that unit j is the next processing unit to be

updated,

E=

1

2 wjixjxii,j jj

Ej = 1

2wjixjxi =

i ,ij

1

2xj wjixi

i, i j


32/36

Klinkhachorn:CpE320



When unit j is updated,

If xj changes from 0 to 1, then

Ej = Einew Ejold = 1

2xj wjix i

where

xj = xj new x jold

xj = 1

and

wjixi 0i

Thus,

Ej 0


33/36

Klinkhachorn:CpE320



If xj changes from 1 to 0, then

The network guaranteed to converge, when E

taking on lower and lower values until it reaches

a steady state.

xj = 1

and

wjixi < 0i

Again,

Ej < 0


34/36

Klinkhachorn:CpE320

Hopfield Net: Recalling


T+10% Noise -->T T +20% Noise --> O


35/36

Klinkhachorn:CpE320

Hopfield Net: Application


A binary Hopfield net can be used to determine

whether an input vector is a known vector (i.e.,

one that was stored in the net) or an unknown

vector.

If the input vector is an unknown vector, the

activation vectors produced as the net iterates will

converge to an activation vector that is not one of

the stored patterns; such a pattern is called a

spurious stable state.


36/36

Klinkhachorn:CpE320

Hopfield Net: Storage Capacity


Hopfield found experimentally that the number of

binary patterns that can be stored and recalled in a net

with reasonable accuracy, is approximately

P 0.15 n, where n is the number of neurons in the net

A detailed theoretical analysis showed

P n/2 log2n

Pattern Association

Documents

Transcript of Pattern Association